Compare Alternatives Best Tools Glossary Checklists Downloads Security Center Privacy Center
Free tool

Robots.txt Generator

Build a robots.txt with common rules, the right syntax, and your sitemap line.

Robots.txt Generator
Standard syntaxSitemap lineNo data leaves your device

Crawl rules

Disallowed paths

Sitemap

How it works

Pick which user-agents you want to address, the paths you want to disallow, and your sitemap URL. We assemble a syntactically correct robots.txt file you can copy and place at the root of your domain.

Methodology

Robots.txt is a polite-protocol file: well-behaved crawlers respect it, malicious ones ignore it. It is not a security control. Anything sensitive needs authentication, not a Disallow rule. Always include a Sitemap line so crawlers find your URL list quickly.

Frequently asked questions

Should I block AI training crawlers?

It's a policy choice. To block known AI crawlers, add User-agent rules for GPTBot, ClaudeBot, Google-Extended, and others, with a Disallow: /.

What if I need to block a page from indexing?

Robots.txt blocks crawling, not indexing. To prevent a page from appearing in search results, use a noindex meta tag in the page itself.

Other free tools

Tool

Password Strength Checker

Estimate the entropy and crack-time of any password — entirely in your browser.

 Tool

Password Generator

Create a strong, random password using your browser's secure RNG.

 Tool

Passphrase Generator

Generate a memorable random word-based passphrase for your master credential.

 Tool

Browser Privacy Snapshot

See exactly what your browser exposes to every website you visit.

 Tool

Security Headers Explainer

Paste your site's response headers — get a plain-English breakdown of each one.

 Tool

QR Code Generator

Generate a QR code from any text or URL — entirely in your browser.

 Tool

Word Counter

Count words, sentences, paragraphs, and reading time as you type.

 Tool

Character Counter

Count characters with and without spaces — useful for SEO meta tags and tweets.

 Tool

URL Encoder / Decoder

Convert text to and from percent-encoded URL form, in your browser.

 Tool

Base64 Encoder / Decoder

Encode text to Base64 or decode Base64 to text — locally.

 Tool

Meta Title SERP Preview

See how your <title> tag looks in a Google search result, with character feedback.

 Tool

Meta Description SERP Preview

Preview your meta description and check it fits in a search snippet.

 Tool

VPN vs Proxy Decision Helper

Answer four questions; get a clear recommendation.