ptr.pet / endpoint
data endpoint for entity 90008 (aka. a website)
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

allow claude-user so claude can fetch website in tool calls

dawn 0eeab661 8c75080a

+1 -1
+1 -1
eunomia/src/lib/robots.ts
··· 21 21 AgentType.SEOCrawler 22 22 ]); 23 23 lastFetched.set(Date.now()); 24 - return robotsTxt; 24 + return robotsTxt.replace("User-agent: Claude-User\nDisallow: /", ""); 25 25 } catch (error) { 26 26 console.error('failed to fetch robots.txt:', error); 27 27 return '';