this repo has no description
add robots.txt
+132
+131
assets/robots.txt
+131
assets/robots.txt
···
1
+
User-agent: AddSearchBot
2
+
User-agent: AI2Bot
3
+
User-agent: AI2Bot-DeepResearchEval
4
+
User-agent: Ai2Bot-Dolma
5
+
User-agent: aiHitBot
6
+
User-agent: amazon-kendra
7
+
User-agent: Amazonbot
8
+
User-agent: AmazonBuyForMe
9
+
User-agent: Andibot
10
+
User-agent: Anomura
11
+
User-agent: anthropic-ai
12
+
User-agent: Applebot
13
+
User-agent: Applebot-Extended
14
+
User-agent: atlassian-bot
15
+
User-agent: Awario
16
+
User-agent: bedrockbot
17
+
User-agent: bigsur.ai
18
+
User-agent: Bravebot
19
+
User-agent: Brightbot 1.0
20
+
User-agent: BuddyBot
21
+
User-agent: Bytespider
22
+
User-agent: CCBot
23
+
User-agent: Channel3Bot
24
+
User-agent: ChatGLM-Spider
25
+
User-agent: ChatGPT Agent
26
+
User-agent: ChatGPT-User
27
+
User-agent: Claude-SearchBot
28
+
User-agent: Claude-User
29
+
User-agent: Claude-Web
30
+
User-agent: ClaudeBot
31
+
User-agent: Cloudflare-AutoRAG
32
+
User-agent: CloudVertexBot
33
+
User-agent: cohere-ai
34
+
User-agent: cohere-training-data-crawler
35
+
User-agent: Cotoyogi
36
+
User-agent: Crawl4AI
37
+
User-agent: Crawlspace
38
+
User-agent: Datenbank Crawler
39
+
User-agent: DeepSeekBot
40
+
User-agent: Devin
41
+
User-agent: Diffbot
42
+
User-agent: DuckAssistBot
43
+
User-agent: Echobot Bot
44
+
User-agent: EchoboxBot
45
+
User-agent: FacebookBot
46
+
User-agent: facebookexternalhit
47
+
User-agent: Factset_spyderbot
48
+
User-agent: FirecrawlAgent
49
+
User-agent: FriendlyCrawler
50
+
User-agent: Gemini-Deep-Research
51
+
User-agent: Google-CloudVertexBot
52
+
User-agent: Google-Extended
53
+
User-agent: Google-Firebase
54
+
User-agent: Google-NotebookLM
55
+
User-agent: GoogleAgent-Mariner
56
+
User-agent: GoogleOther
57
+
User-agent: GoogleOther-Image
58
+
User-agent: GoogleOther-Video
59
+
User-agent: GPTBot
60
+
User-agent: iAskBot
61
+
User-agent: iaskspider
62
+
User-agent: iaskspider/2.0
63
+
User-agent: IbouBot
64
+
User-agent: ICC-Crawler
65
+
User-agent: ImagesiftBot
66
+
User-agent: imageSpider
67
+
User-agent: img2dataset
68
+
User-agent: ISSCyberRiskCrawler
69
+
User-agent: Kangaroo Bot
70
+
User-agent: KlaviyoAIBot
71
+
User-agent: KunatoCrawler
72
+
User-agent: laion-huggingface-processor
73
+
User-agent: LAIONDownloader
74
+
User-agent: LCC
75
+
User-agent: LinerBot
76
+
User-agent: Linguee Bot
77
+
User-agent: LinkupBot
78
+
User-agent: Manus-User
79
+
User-agent: meta-externalagent
80
+
User-agent: Meta-ExternalAgent
81
+
User-agent: meta-externalfetcher
82
+
User-agent: Meta-ExternalFetcher
83
+
User-agent: meta-webindexer
84
+
User-agent: MistralAI-User
85
+
User-agent: MistralAI-User/1.0
86
+
User-agent: MyCentralAIScraperBot
87
+
User-agent: netEstate Imprint Crawler
88
+
User-agent: NotebookLM
89
+
User-agent: NovaAct
90
+
User-agent: OAI-SearchBot
91
+
User-agent: omgili
92
+
User-agent: omgilibot
93
+
User-agent: OpenAI
94
+
User-agent: Operator
95
+
User-agent: PanguBot
96
+
User-agent: Panscient
97
+
User-agent: panscient.com
98
+
User-agent: Perplexity-User
99
+
User-agent: PerplexityBot
100
+
User-agent: PetalBot
101
+
User-agent: PhindBot
102
+
User-agent: Poggio-Citations
103
+
User-agent: Poseidon Research Crawler
104
+
User-agent: QualifiedBot
105
+
User-agent: QuillBot
106
+
User-agent: quillbot.com
107
+
User-agent: SBIntuitionsBot
108
+
User-agent: Scrapy
109
+
User-agent: SemrushBot-OCOB
110
+
User-agent: SemrushBot-SWA
111
+
User-agent: ShapBot
112
+
User-agent: Sidetrade indexer bot
113
+
User-agent: Spider
114
+
User-agent: TavilyBot
115
+
User-agent: TerraCotta
116
+
User-agent: Thinkbot
117
+
User-agent: TikTokSpider
118
+
User-agent: Timpibot
119
+
User-agent: TwinAgent
120
+
User-agent: VelenPublicWebCrawler
121
+
User-agent: WARDBot
122
+
User-agent: Webzio-Extended
123
+
User-agent: webzio-extended
124
+
User-agent: wpbot
125
+
User-agent: WRTNBot
126
+
User-agent: YaK
127
+
User-agent: YandexAdditional
128
+
User-agent: YandexAdditionalBot
129
+
User-agent: YouBot
130
+
User-agent: ZanistaBot
131
+
Disallow: /