How to create a robots.txt file and check it on a live site

To create a robots.txt file, publish a plain-text file at your site root (/robots.txt) with a User-agent line, the paths to Disallow (and optional Allow overrides), and a full Sitemap: URL. Then fetch that URL on production, confirm it returns HTTP 200, and make sure you did not block CSS, JavaScript, or your whole site with Disallow: /. robots.txt controls crawling, not indexing, so keep public marketing pages fetchable and use noindex when you need a URL out of search.

This page is a method guide for technical SEO: what the file does, a minimal template, mistakes that hide entire sites, and a short validation loop. It is not a substitute for the free robots.txt builder, which writes the file for you, or the broader SEO checker for websites, which audits on-page signals on a URL you paste. For the full developer crawl stack, see SEO for developers; for indexing versus crawling, see why your site is not showing on Google.

What robots.txt Does and Does Not Do

Crawlers request https://yourdomain.com/robots.txt before they crawl heavily. The file uses plain lines such as User-agent: *, Disallow: /admin, and Sitemap: https://yourdomain.com/sitemap.xml. Google documents robots.txt as a way to manage crawling, not as a guaranteed way to keep a URL out of results (Google robots.txt introduction).

That distinction matters in production:

• Crawl control: A disallowed path is a request not to fetch it. Google may still list the URL if it learns the address from links, often with a limited snippet.
• Not an indexing switch: Blocking a URL in robots.txt can prevent Google from reading a noindex tag on that URL. Teams sometimes block a page and wonder why it still appears.
• Not authentication: Anyone can read robots.txt. Do not treat Disallow as access control for secrets.

For SaaS and marketing sites, the goal is boring: let crawlers reach public pages and assets, keep admin and staging paths out of routine crawling, and point to one sitemap index.

A Minimal robots.txt You Can Ship Today

Start from a small template and edit paths to match your routes:

User-agent: *
Disallow: /admin
Disallow: /api/internal
Allow: /

Sitemap: https://example.com/sitemap.xml

Each line has a job:

• User-agent: * applies the group to crawlers that do not have a more specific rule.
• Disallow: names a path prefix crawlers are asked not to fetch. Use leading slashes; match how your app routes URLs.
• Allow: (optional) overrides a broader Disallow when you blocked a parent path but need a public subdirectory.
• Sitemap: must be a full URL to your XML sitemap or sitemap index, not a relative path.

If you prefer a form instead of typing lines, use Orchory's robots.txt builder: enter disallow paths, add the sitemap URL, copy the result, and commit it as public/robots.txt or the equivalent static file your stack serves at the domain root.

Numbered steps to create and publish

1. List paths that should not be crawled. Admin consoles, internal search, cart checkout flows, and preview hosts belong here. Public docs, blog posts, and asset directories usually do not.
2. Draft the file. Use the template above or the builder output. Keep one sitemap line that matches production.
3. Place it at the site root. The URL must resolve to /robots.txt on the live host, not under /public/robots.txt in the browser unless your server maps that way intentionally.
4. Deploy and fetch the live URL. Open https://yourdomain.com/robots.txt in a browser or with curl -sS -D - https://yourdomain.com/robots.txt. Expect HTTP 200 and text/plain.
5. Submit or refresh the sitemap in Search Console. The robots file and sitemap work together: robots sets crawl preferences; the sitemap lists URLs you want discovered efficiently.

Common robots.txt Mistakes

Most production incidents are copy-paste errors, not exotic syntax.

Disallow: / accidents

Disallow: / asks crawlers not to fetch anything on the host. Staging templates often ship this line; if it reaches production, organic visibility collapses even when pages look fine in the browser. Search Console may show URLs as blocked by robots.txt. The fix is to remove or narrow that line, deploy, and confirm the live file changed.

Blocking CSS, JavaScript, or images

Disallowing /assets/, /_next/static/, or similar paths can stop crawlers from rendering pages correctly. Google needs to fetch resources required for rendering (JavaScript SEO basics). Block private app areas, not the directories that hold public styles and scripts.

Staging and preview leaks

A staging host that allows full crawl while still linked from the public web wastes crawl budget and can expose draft URLs. Use HTTP authentication or network isolation on non-production hosts, and give staging its own robots rules. Do not rely on robots.txt alone to hide a publicly reachable URL.

Missing or wrong Sitemap line

A sitemap URL that 404s, points at HTTP while the site is HTTPS-only, or lists the wrong host confuses discovery. Generate the sitemap from the same source of truth as canonical URLs, then paste that live URL into robots.txt.

Using robots.txt as index removal

Teams sometimes disallow a landing page to "remove it from Google." That blocks fetch and can interfere with noindex. For public URLs you want out of the index, allow crawl and send noindex, or use Search Console removal tools after the page is fixed. The crawl-versus-index split is covered in more detail in the Google visibility checklist.

How to Check robots.txt on a Live Site

Validation is read-only and should run after every robots change.

1. Fetch the production URL. Confirm status 200, no accidental redirect chain to a login page, and body text that matches what you committed.
2. Read each Disallow line against your route map. Ask whether any line blocks marketing paths, sitemap files, or render-critical assets.
3. Open Search Console. Use the robots.txt report and URL inspection on a sample of important URLs to see whether Google reports a crawl block.
4. Spot-check a public page. Paste the URL into Orchory's free SEO checker for websites to catch related issues such as accidental noindex, missing titles, or broken canonicals on the same deploy.
5. Re-test after deploy. CDN and edge caches occasionally serve an old robots file. Purge or version the file if your platform caches static assets aggressively.

Orchory does not host a separate robots.txt fetch tool on the free tools page; the builder creates the file, and the SEO checker focuses on page-level signals. For a full crawl audit beyond robots.txt, send your site and the team can walk through what to fix first on a call.

robots.txt Versus noindex

Use the right control for the job:

Page-level robots meta tags are a different layer from the robots.txt file. Orchory's meta tag checker reads those tags from a URL; it does not replace a robots.txt review.

When robots.txt is clean and pages still underperform, the next step is usually sitemap hygiene, internal links, and on-page structure, not more disallow lines. Browse free SEO tools for generators and checkers, or request a walkthrough if you want help prioritizing fixes across the site.

FAQs