Fetch and parse robots.txt for any domain - check crawl rules and sitemaps
Endpoints (2)
Each endpoint has its own page. Open it, or expand a row for the request schema.
/api/v1/get
/api/v1/isAllowed
Sign in to try this API
Authenticated agents can call this endpoint with on-chain payment.