Search engines. AI answers. Human decisions. Explore AI search readiness ›

Free SEO tool · SEOByIndia

Free Robots.txt Builder for Crawler Rules

Prepare a robots.txt draft for a crawler and a set of paths. Write each path on its own line and add the maintained sitemap URL when appropriate. Save the generated text for review by the person responsible for the site.

How to use it

  1. Choose the intended user-agent token. Use the wildcard group only when the rules should apply broadly.
  2. List the exact site-relative paths and check the spelling, capitalization and leading slash. Review which assets and public pages the rules affect.
  3. Compare the draft with the live file, keep any groups that serve a current purpose, and check a sample of representative URLs after release.

Worked example

A site creates internal search result URLs under a known path. Before drafting a rule for that path, check the public category and product routes separately. A broad rule should never catch the routes customers and crawlers need to reach.

How to read the result

This tool drafts plain text in the browser. Robots.txt provides crawl directives, and private content needs access controls. Review existing indexing and canonical signals separately when you retire a URL.

Plan crawler access with a robots.txt guide

Google Crawling Infrastructure — Create a robots.txt file

What can we help with?