Robots Txt Generator
Robots Txt Generator — runs in your browser. Inputs are not stored.
Generate robots.txt
Builds robots.txt text to deploy at the site root. You must upload it to the server yourself.
How to use
- Enter a User-agent. The default asterisk means all crawlers.
- Write Allow paths and Disallow paths, one path per line.
- Optionally add a Sitemap URL, and turn on Extra block: User-agent: AdsBot-Google → Disallow: / if you need it.
- Press Generate, review the robots.txt text below, then Copy.
- Upload the file to the site root yourself—this tool does not deploy it. Reset clears the fields.
Key concepts
The robots.txt generator assembles a text file that hints paths to crawlers. It is not a security barrier; protect private data with authentication. If you have Disallow lines and Allow is empty, crawlers are often read as allowed everywhere except the blocked paths. The generated file only takes effect when it is deployed at the root.
The AdsBot extra block is an example that stops an ads bot from seeing the whole site. Do not block everything if you need search visibility. The Sitemap line helps crawlers find your URL list. The tool only builds text; it does not upload to a server.
Example
User-agent asterisk, Disallow /admin/ and /private/, plus a sitemap URL, generates a file with those three rules. Turning on the AdsBot option appends a second block. Save the copied text as a public root file on your host. With Disallow only and Allow empty, crawls are read as allowed except blocked paths. Blocking an admin path without login still allows direct URL access—do not confuse this with security. After you put the file at the site root, check fetch in Search Console. When you retry the same input, reset first. On a shared computer, clear the fields when you finish. Closing the tab discards what you pasted; you do not need a server delete request. Button names match the tool above. If the result is empty, check required fields and hidden errors. This page is usage help, not a claim that input is stored elsewhere.
Related: Sitemap generator · Meta tag generator · hreflang generator
Frequently asked questions
Does the robots.txt generator upload to my server?
No. It only builds text. You must deploy it at the root yourself.
Is robots.txt a security feature?
No. It is only a crawler hint. Protect admin pages with login.
What is the AdsBot extra block?
An optional block that sets Disallow / for AdsBot-Google. It is an example for reducing ads-bot crawls.
Can I use Allow and Disallow together?
Yes. Write one path per line. More specific paths often take priority.
Is a Sitemap URL required?
Optional. It makes the list easier to find. An XML sitemap is a separate file.
Is the generated file submitted to search engines immediately?
No. Upload the file, then check it in Search Console.
Last reviewed: 2026-09-04