Calculator Converter
SEO

Robots.txt generator

Draft a robots.txt that allows or blocks crawling of paths you list.

Loading tool…

  1. 1 Start from allow-all or disallow-all.
  2. 2 Add paths to disallow, one per line, each starting with /.
  3. 3 Add your sitemap URL if you have one.
  4. 4 Download robots.txt and put it at the root of the site.

How it works

robots.txt is a request to crawlers, not a lock. This generator writes a file for User-agent: * with Allow and Disallow lines for the paths you add, plus an optional sitemap URL. Disallowing a page does not remove it from Google if other sites link to it. Do not disallow your CSS if you want pages rendered properly.

Frequently asked questions

Will this hide a page from Google?

A disallow asks Google not to crawl it. A noindex tag is the instruction about indexing. They are different.

Where does the file go?

At https://yourdomain/robots.txt, not inside a subfolder.

See all

Meta title and description length

Count characters in a title and meta description against common search limits.

UTM builder

Add utm_source, utm_medium, and utm_campaign to a URL.