The code and data behind xeiaso.net
5
fork

Configure Feed

Select the types of activity you want to include in your feed.

update robots.txt

Signed-off-by: Xe Iaso <me@xeiaso.net>

+42 -2
+42 -2
lume/src/static/robots.txt
··· 1 - User-Agent: * 1 + # If your bot is in this list and you want to scrape my blog, please contact me to arrange for payment commensurate with your resource usage. 2 + User-agent: AI2Bot 3 + User-agent: Ai2Bot-Dolma 4 + User-agent: Amazonbot 5 + User-agent: anthropic-ai 6 + User-agent: Applebot 7 + User-agent: Applebot-Extended 8 + User-agent: Bytespider 9 + User-agent: CCBot 10 + User-agent: ChatGPT-User 11 + User-agent: Claude-Web 12 + User-agent: ClaudeBot 13 + User-agent: cohere-ai 14 + User-agent: Diffbot 15 + User-agent: DuckAssistBot 16 + User-agent: FacebookBot 17 + User-agent: FriendlyCrawler 18 + User-agent: GPTBot 19 + User-agent: iaskspider/2.0 20 + User-agent: ICC-Crawler 21 + User-agent: ImagesiftBot 22 + User-agent: img2dataset 23 + User-agent: ISSCyberRiskCrawler 24 + User-agent: Kangaroo Bot 25 + User-agent: Meta-ExternalAgent 26 + User-agent: Meta-ExternalFetcher 27 + User-agent: OAI-SearchBot 28 + User-agent: omgili 29 + User-agent: omgilibot 30 + User-agent: PanguBot 31 + User-agent: PerplexityBot 32 + User-agent: PetalBot 33 + User-agent: Scrapy 34 + User-agent: Sidetrade indexer bot 35 + User-agent: Timpibot 36 + User-agent: VelenPublicWebCrawler 37 + User-agent: Webzio-Extended 38 + User-agent: YouBot 39 + Disallow: / 40 + 41 + User-agent: * 2 42 Sitemap: https://xeiaso.net/sitemap.xml 3 43 Disallow: /metrics 4 - Disallow: /.within/health 44 + Disallow: /.within/health