Website overview
About Go Top SEO
Go Top SEO publishes a free check of how a website treats AI crawlers. Enter a domain and the page reads its robots.txt, then shows which AI user-agents are allowed and which are blocked. Behind the checker sits an open reference table of 26 AI crawler tokens, from GPTBot and ClaudeBot to Google-Extended, PerplexityBot and CCBot. Every row records what the operator's own documentation says the crawler is for, whether that is collecting training data, indexing for search, fetching a page to answer a live question, or serving ads. Each row also carries the operator's documentation URL and the date that page was read, so nothing is inferred from server logs. The dataset is available as JSON or CSV under CC BY 4.0, with an OpenAPI 3.1 definition, and needs no key and no signup.
At a glance
Key features
- Checks any domain's robots.txt for AI crawler rules
- Open reference table of 26 AI crawler user-agent tokens
- Each row cites the operator's documentation and the date it was read
- Separates training crawlers from search, answer and ads crawlers
- JSON and CSV under CC BY 4.0 with an OpenAPI 3.1 endpoint
Practical details
What can you use this website for?
Paste your domain to see whether GPTBot or Google-Extended is currently allowed by your robots.txt, then open the dataset to read what each crawler is documented to do.
Who is it for?
Site owners, SEO specialists and developers who need to decide which AI crawlers to allow or block on their own sites.
What to know
What makes it different?
Every row is sourced from the crawler operator's own documentation, with the URL and the date it was read recorded in the data, rather than inferred from server logs.
About this listing
Information about features, pricing and suitability is supplied by the submitter, not independently verified by Zearches. Websites are screened against submission and safety rules, but inclusion is not an endorsement.
Report an issue with this listing
