Type a website or a page address to see whether search engines and AI crawlers may fetch it, the robots.txt rule that decides for each one, and whether the site publishes llms.txt.
Free, no account needed.
How to use it
Every site can publish rules for crawlers at /robots.txt. The rules come in groups, each starting with one or more User-agent lines. A crawler follows the group that names it, and only falls back to the User-agent: * group when no group names it. That is why a site that blocks everything for * can still let GPTBot in, or the other way round.
Inside the group, the longest matching Allow or Disallow rule wins, and Allow wins a tie. A * in a rule matches anything and a $ at the end means the address must end there. If no rule matches, the crawler may fetch the page.
For the address you type, the checker reads robots.txt once and answers for every crawler on the list:
A missing robots.txt (an answer such as 404) means there are no rules, so every crawler may read the site. A robots.txt that cannot be fetched because the server fails (a 5xx answer or no answer) means crawlers must treat the whole site as blocked until it works again. Crawlers should follow at least five redirects to find the file.
robots.txt is a request, not a lock. Well-behaved crawlers follow it, and changes can take a day to reach them because they keep a copy.
Add a group for each one to robots.txt, for example User-agent: GPTBot followed by Disallow: /. Name every crawler you want to stop: a rule in the * group does not apply to a crawler that has its own group. Then run the check again to confirm the verdict changed.
No. Google says Google-Extended only controls whether content it crawls may be used for training its AI models, and that it does not affect inclusion or ranking in Google Search. Blocking Googlebot is what removes pages from search.
Because that crawler has its own group somewhere in the file. A crawler follows only the group that names it, so the rules under User-agent: * no longer apply to it. Add the Disallow rule to its own group as well.
A Markdown file at /llms.txt that gives AI assistants a short summary of a site and links to its most useful pages. It is optional and does not control access: robots.txt does that.
Not on their own. They are page tags that ask AI companies not to use the page or its images, and only some honour them. To keep a crawler out, block it in robots.txt.
Yes. The site AI access API returns the same verdicts per crawler as JSON, with the deciding rule, llms.txt and page tags, for any URL.
Get a key and make your first call in minutes.