Robots.txt generator
Create a custom robots.txt file online — allow or block crawlers, add your sitemap, and copy the result. Free, instant, and private in your browser.
Adds Disallow: / for GPTBot, ClaudeBot, Google-Extended and CCBot. Google and Bing search are not affected.
What is a robots.txt file?
A robots.txt file sits at the root of your site (for example https://example.com/robots.txt) and tells crawlers which paths they may fetch. It is a public crawl-budget signal — not a security wall. Search engines read it before (or while) crawling so you can keep admin areas, staging folders, or thin parameter URLs out of the crawl queue.
How to use this robots.txt generator
1. Choose whether bots are allowed by default or blocked site-wide.
2. Optionally add your Sitemap URL so crawlers can discover URLs faster.
3. List folders to Disallow (one path per line), such as /admin or /tmp.
4. Set a crawl-delay if you need to slow polite bots (note: Googlebot ignores crawl-delay).
5. Copy the generated robots.txt and upload it to your site root.
Robots.txt directives cheat sheet
User-agent — which bot the rules apply to (* means all).
Disallow — paths that should not be crawled.
Allow — exceptions inside a disallowed tree (supported by major engines).
Sitemap — absolute URL to your XML sitemap.
Crawl-delay — seconds between requests for bots that honor it.
Example: block a private folder while keeping the rest open:
User-agent: * Disallow: /admin/ Disallow: /tmp/ Sitemap: https://example.com/sitemap.xml
How to block AI crawlers with robots.txt
Tick Block AI bots and the generator adds a rule for four AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) and CCBot (Common Crawl). Each one gets Disallow: /, which asks it not to use any page on your site.
They do different jobs. GPTBot and ClaudeBot collect content that may be used to train AI models. Google-Extended is not a crawler: it's a token that tells Google not to use your content for Gemini training and grounding. It doesn't change Google Search. CCBot builds Common Crawl's open web archive, a public dataset anyone can reuse.
The option does not block the crawlers behind AI search engines (OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot), so your site can still appear in their answers. It doesn't block the agents that fetch a page when a user asks for it either (ChatGPT-User, Claude-User, Perplexity-User). Keep in mind that robots.txt is a request, not a technical block: well-behaved bots follow it, others may not. It also only affects future crawling. It doesn't remove content that was already collected.
# Block AI bots User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: /
Common robots.txt mistakes
Blocking CSS/JS that Google needs to render pages, using robots.txt to “hide” confidential files (they can still be linked and discovered), forgetting the file must be at the domain root, or shipping a Disallow: / to production by accident. Check the robots.txt report and URL Inspection in Google Search Console after deploy.
Robots.txt vs meta robots
robots.txt controls crawling of URL paths. A meta robots noindex tag (or X-Robots-Tag header) controls indexing of a URL that was already fetched. To keep a page out of search results you usually need noindex — not only a Disallow line. Use both deliberately: Disallow for crawl waste; noindex for pages that must stay out of the index.
Frequently Asked Questions
What is a robots.txt file?
It is a text file in the root of your website that gives instructions to web robots (crawlers) about which pages to visit or ignore.
Where should I place the file?
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt).
Does robots.txt hide pages from users?
No, it only instructs bots. If you need to secure a page from humans, use authentication or password protection.
Will Disallow remove a page from Google?
Not reliably. Disallow can stop crawling, but URLs may still appear in results without snippets. Use noindex (and allow crawling of that URL) when you need de-indexing.
Is this robots.txt generator free?
Yes. It runs entirely in your browser — nothing about your site configuration is uploaded.
How do I block GPTBot and other AI crawlers with robots.txt?
Add a group for each bot with User-agent: and its name, then Disallow: /. For example, User-agent: GPTBot followed by Disallow: /. Tick "Block AI bots" in this generator to add GPTBot, ClaudeBot, Google-Extended and CCBot at once, then upload the file to your site root.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended doesn't affect whether your site appears in Google Search and isn't a ranking signal. It only controls whether Google can use your content to train Gemini models and for grounding in Gemini. Googlebot keeps crawling your site as usual.
Will blocking GPTBot stop ChatGPT from finding my site?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses a different bot, OAI-SearchBot, which this option doesn't block. If you also block OAI-SearchBot, your site won't be shown in ChatGPT search answers. When a user asks ChatGPT to open a page, it uses ChatGPT-User, and OpenAI says robots.txt may not apply.