Free tools
Create a valid robots.txt file in seconds. Control search engine crawling, add your sitemap, preview the output, and download instantly — entirely in your browser.
Loading robots.txt generator…
A robots.txt file tells compliant crawlers which paths they may fetch. Used well, it protects admin and checkout URLs, reduces wasted crawl budget on thin parameter pages, and points bots to your XML sitemap. It does not securely hide private content — use authentication for that.
Allow and Disallow rules let you guide Googlebot, Bingbot and other agents. Start open for public content, then block carts, accounts, search facets and staging folders that create duplicate or low-value URLs.
Disallowing /admin, /wp-admin, or platform equivalents reduces accidental deep crawling of back-office surfaces. Remember: robots.txt is public. Do not put secrets or unpublished URLs you want to keep confidential into the file.
Blocking infinite filter combinations, add-to-cart query strings and internal search result pages helps crawlers spend more time on indexable product, collection and content URLs.
Use a clear User-agent group, explicit Allow/Disallow pairs when needed, and one or more Sitemap lines so crawlers can discover your canonical URL set faster.
Add absolute Sitemap URLs (for example example.com). Shopify, WordPress and most static generators expose a sitemap you can reference from robots.txt.
FAQ
Straight answers about crawl rules, indexing myths, Shopify, WordPress and sitemaps.
robots.txt is a plain-text file placed at the site root (example.com) that gives crawlers voluntary instructions about which paths they may request.
Compliant bots fetch /robots.txt before crawling. They match the User-agent group, then apply Allow and Disallow rules for URL paths. Sitemap lines advertise XML sitemaps. Non-compliant bots may ignore the file.
No. Disallow can stop crawling of a URL, but a page can still appear in search results if it is linked elsewhere. Use noindex (for example via meta robots or HTTP headers) when you need to keep a URL out of the index.
Not securely. The file is publicly readable, and blocked URLs can still be discovered through links. Use login walls, permissions and noindex for sensitive or private content.
Shopify merchants typically edit robots.txt through the theme’s robots.txt.liquid template (Online Store → Themes → Edit code) or related Shopify admin tooling. Generate rules here, then paste carefully and preview on a development theme first.
Place robots.txt at the site root, or manage it with your SEO plugin / host file manager. After publishing, open yoursite.com to confirm the live file matches what you generated.
Most public sites benefit from a simple robots.txt that allows public content and references a sitemap. Very small sites can still use a minimal Allow: / plus Sitemap line for clarity.
Yes, by targeting Googlebot (or using Disallow: / for User-agent: *). Blocking Google removes crawl access for that agent and can remove pages from Google Search over time. Use with caution on production sites.
Titles, descriptions, canonicals and robots meta tags.
Build JSON-LD for pages, products and organizations.
Create FAQPage JSON-LD for Google rich results.
Create campaign tracking URLs for GA4 and ads.
Technical and on-page SEO guidance for storefronts.
Technical SEO and growth support from Web6.
More free browser tools from Web6.
Talk through crawl strategy and SEO delivery.
Official references: Google robots.txt documentation and robotstxt.org.
