The short answer
A robots.txt file is a plain text file at the root of a website that tells search engine crawlers which pages or folders they may visit. It is a request, not a lock, and it does not by itself remove a page from search results.
What does robots.txt do?
Crawlers such as Googlebot and Bingbot read robots.txt before they visit your site. The file lists which areas they should avoid. Well-behaved crawlers follow it. Badly behaved ones may ignore it, so never rely on it to hide private information.
Where does it go?
In the top folder of your site, so it opens at your-domain.com/robots.txt. A file in a subfolder is not read. Each address (including each subdomain) needs its own file.
The basic rules
- User-agent: names the crawler the rules apply to. An asterisk means all crawlers.
- Disallow: a path the crawler should not visit.
- Allow: an exception inside a blocked path.
- Sitemap: the full address of your XML sitemap.
A simple example
This allows everything except the admin area, and points to a sitemap: **User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml**, each on its own line.
Common mistakes
- Leaving Disallow: / on a live site, which blocks everything.
- Blocking CSS and JavaScript files, which stops search engines seeing pages the way visitors do.
- Using robots.txt to keep a page out of results. Use a noindex tag instead, and leave the page crawlable so the tag can be read.
- Writing the file as robot.txt. The correct name has an s: robots.txt.
How to test it
Open your robots.txt in a browser to confirm it loads, then use the robots.txt report in Google Search Console to see how Google reads it.
Try the free tools
Frequently asked questions
Is it robot.txt or robots.txt?
The file must be named robots.txt, with an s. A file called robot.txt is ignored by crawlers.
Does robots.txt stop a page being indexed?
Not reliably. A blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag.
Does Google respect the Crawl-delay rule?
No. Google ignores Crawl-delay. Some other crawlers follow it.
How big can the file be?
Google reads up to 500 KiB of a robots.txt file and ignores anything beyond that, so keep it short.