Skip to content
ScriptGarden

Robots.txt Generator

Build a robots.txt file for your site. Start from a preset, block or allow the paths you choose, add your sitemap, and download the finished file.

Runs in your browserTopic: AI crawler controlFreeNo sign-upUpdated

Your robots.txt: upload it to the root of your site (example.com/robots.txt)

User-agent: *
Disallow:

What is a robots.txt file?

A robots.txt file is a plain text file at the root of a website that tells search engine crawlers which pages or folders they may visit. It is a request, not a lock: well-behaved crawlers follow it, but it does not hide pages from people or keep them out of search results by itself.

How it works

  1. 1

    Pick a preset that is close to your site, or start from scratch.

  2. 2

    Add the paths you want crawlers to avoid, and any exceptions.

  3. 3

    Add your sitemap address, and choose whether to block AI crawlers.

  4. 4

    Download robots.txt and upload it to the top folder of your website.

Mistakes to avoid

  • Leaving "Disallow: /" on a live site. That asks every crawler to stay away from the whole site. It is meant for staging sites only.
  • Blocking your CSS and JavaScript. Search engines need them to see your pages the way visitors do.
  • Forgetting the sitemap line. Adding it helps crawlers find your pages.
  • Not testing after changes. Open your robots.txt in a browser and check it in Google Search Console.

Frequently asked questions

Is it robot.txt or robots.txt?

The correct name is robots.txt, with an s. People often search for "robot.txt" or "robot txt", but a file named robot.txt is ignored by crawlers. This generator makes content for the correctly named robots.txt file.

What is a robots.txt file?

It is a plain text file at the root of your site (example.com/robots.txt) that tells crawlers which parts of the site they may visit. Well-behaved crawlers such as Googlebot and Bingbot follow it.

Where do I put the file?

In the top folder of your website, so that it opens at your-domain.com/robots.txt. A robots.txt file placed in a subfolder is ignored.

Does blocking a page remove it from Google?

Not reliably. Blocking a page stops crawlers reading it, but the address can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag and leave the page crawlable so the tag can be seen.

Is robots.txt a way to hide private pages?

No. The file is public and only polite crawlers obey it. Protect private content with a login or server settings.

Should I block AI crawlers?

That is your choice. Blocking asks AI companies not to use your pages for training or AI search results, and the bots that respect robots.txt will comply. It will not stop bots that ignore the file.

Does Google use the crawl delay?

No. Google ignores the Crawl-delay line. Some other crawlers respect it, so it is optional.

Related guides

Sources and further reading

More free tools