The code and data behind xeiaso.net
update robots.txt
Signed-off-by: Xe Iaso <me@xeiaso.net>
+42
-2
+42
-2
lume/src/static/robots.txt
+42
-2
lume/src/static/robots.txt
···
1
-
User-Agent: *
1
+
# If your bot is in this list and you want to scrape my blog, please contact me to arrange for payment commensurate with your resource usage.
2
+
User-agent: AI2Bot
3
+
User-agent: Ai2Bot-Dolma
4
+
User-agent: Amazonbot
5
+
User-agent: anthropic-ai
6
+
User-agent: Applebot
7
+
User-agent: Applebot-Extended
8
+
User-agent: Bytespider
9
+
User-agent: CCBot
10
+
User-agent: ChatGPT-User
11
+
User-agent: Claude-Web
12
+
User-agent: ClaudeBot
13
+
User-agent: cohere-ai
14
+
User-agent: Diffbot
15
+
User-agent: DuckAssistBot
16
+
User-agent: FacebookBot
17
+
User-agent: FriendlyCrawler
18
+
User-agent: GPTBot
19
+
User-agent: iaskspider/2.0
20
+
User-agent: ICC-Crawler
21
+
User-agent: ImagesiftBot
22
+
User-agent: img2dataset
23
+
User-agent: ISSCyberRiskCrawler
24
+
User-agent: Kangaroo Bot
25
+
User-agent: Meta-ExternalAgent
26
+
User-agent: Meta-ExternalFetcher
27
+
User-agent: OAI-SearchBot
28
+
User-agent: omgili
29
+
User-agent: omgilibot
30
+
User-agent: PanguBot
31
+
User-agent: PerplexityBot
32
+
User-agent: PetalBot
33
+
User-agent: Scrapy
34
+
User-agent: Sidetrade indexer bot
35
+
User-agent: Timpibot
36
+
User-agent: VelenPublicWebCrawler
37
+
User-agent: Webzio-Extended
38
+
User-agent: YouBot
39
+
Disallow: /
40
+
41
+
User-agent: *
2
42
Sitemap: https://xeiaso.net/sitemap.xml
3
43
Disallow: /metrics
4
-
Disallow: /.within/health
44
+
Disallow: /.within/health