Robots.txt Checker

Robots.txt is a text file at your site's root that tells search engine crawlers which pages they can and cannot access. It helps manage crawl budget and prevent indexing of private pages.

Robots.txt prevents crawling but not indexing. Google may still index a blocked URL if other pages link to it. To fully prevent indexing, use a noindex meta tag instead.

Robots.txt must be at your domain root: example.com/robots.txt. It won't be recognized in subdirectories. Each subdomain needs its own robots.txt file.