How do I create a robots.txt file?
Add the paths you want to disallow, any paths you want to allow, and the full URL of your sitemap. Submit, then save the result as robots.txt in the root of the site, so it is available at https://example.com/robots.txt. This page writes the file. It does not upload it.
| Line | What it does |
|---|---|
| User-agent: * | The rule for crawlers that do not have their own group |
| Disallow: /admin | Asks crawlers not to fetch paths under /admin |
| Allow: / | Asks crawlers to fetch the rest of the site |
| Sitemap: https://example.com/sitemap.xml | Points at the sitemap |
What does Disallow mean in robots.txt?
Disallow names a path a crawler is asked not to fetch. Disallow: /admin covers /admin and anything under it. A path has to start with /. This builder drops a line that does not, including a full URL or a sentence.
Does robots.txt stop Google from indexing a page?
No. robots.txt is a crawl hint, not an indexing order. Google can still list a disallowed URL when another page links to it, often with no snippet. To keep a page out of search, use a noindex tag on a page Google is allowed to fetch. robots.txt also does not hide a URL from someone who already knows it.
Where does the robots.txt file go?
In the site root, next to the homepage, at /robots.txt. A file in a subfolder is not the file crawlers read for the whole host. After you publish it, open that URL in a browser and confirm the text matches what you built here.
User-agent: * Disallow: /admin Allow: / Sitemap: https://example.com/sitemap.xml
Which AI crawlers can this block?
The checkbox adds one group that asks these crawlers not to fetch the site. Google-Extended is Google’s AI-training crawler. It is not Googlebot, so Google Search can still crawl the site. A crawler can ignore the file. This list is not every AI crawler.
| User-agent | Who publishes it |
|---|---|
| GPTBot | OpenAI |
| Google-Extended | Google, for AI training |
| ClaudeBot | Anthropic |
| PerplexityBot | Perplexity |
| CCBot | Common Crawl |