Free tool

Free AI Crawler Rules Generator for robots.txt

The generator writes the robots.txt rules for each AI crawler, from training bots to AI search, and the machine-readable text and data mining opt-out that EU courts look for.

Build your AI crawler rules Generate Acceptable Use Policy
  • robots.txt rules
  • EU opt-out (TDMRep)
  • Content signals
  • Terms clause

Build your AI crawler rules.

Complete 3 steps and copy the results in step 4.

Tell us about your website

One per line, such as /admin/ or /cart/. Search engines skip them too.

Choose what AI crawlers are allowed to do

AI training

Crawlers that collect pages to train AI models.

Choose crawlers one by one
AI search and answers

Crawlers that index pages for AI assistants to cite and link in answers.

Choose crawlers one by one
Fetches when a person asks

Fetchers that open a page because a person asked an AI assistant about it.

Choose crawlers one by one

Choose where the rules apply

Blocked crawlers stay out of
Who gets to read your website

Each AI crawler, as your rules treat it. Blocked crawlers are marked.

Copy your AI crawler rules

Review the rules before you publish them. They follow your answers above.

Enter your website URL to see your robots.txt rules.

The generator writes rules from your answers, the crawler documentation each AI company publishes, the W3C TDMRep protocol and the Cloudflare Content Signals Policy. It isn't legal advice. robots.txt is a request that well-behaved crawlers follow, and courts still disagree on what counts as a valid opt-out.

Where to add your AI crawler rules.

Crawlers read the rules at fixed addresses on your website. Add each file once, then check it every few months.

  1. robots.txt. Upload the file to the root of your website, at /robots.txt. Add the AI groups to your current file if you have one, and keep a single User-agent: * group.

  2. Your website platform. On WordPress, Shopify, Wix or Squarespace, edit robots.txt in the SEO settings or with an SEO plugin, since the platform writes the file.

  3. tdmrep.json. Upload the file to /.well-known/tdmrep.json. On some hosts you create the .well-known folder yourself.

  4. Your terms. Add the clause to your Terms and Conditions, so the reservation is written in words as well as in the files.

  5. Your CDN or host. Turn on the bot settings of your host or CDN, such as Cloudflare, to block crawlers that ignore robots.txt.

  6. A reminder. Check the crawler list every few months: AI companies add and rename crawlers.

The rules

What the rules ask.

Each row names a law, a ruling or a standard, when it applies, and what it means for your opt-out.

Laws, rulings and standards for opting out of AI training, when each applies, and what it means for an opt-out
Law, ruling or standard When it applies What it means for you
EU Copyright Directive (2019/790), Article 4(3) Text and data mining of content published online in the EU, AI training included. The law allows anyone to mine lawfully accessible content unless the rightsholder reserves the right in an appropriate manner, such as machine-readable means for content online.
EU AI Act (2024/1689), Article 53, and the general-purpose AI Code of Practice Providers of general-purpose AI models placed on the EU market, since August 2, 2025. Providers need a policy to find and respect reservations of rights. Code signatories use crawlers that follow robots.txt and make reasonable efforts to follow other machine-readable protocols.
Kneschke v. LAION, Hamburg Higher Regional Court, December 10, 2025 A reservation written only in the website terms. An appeal is pending at the Federal Court of Justice. The court found that the natural-language reservation wasn't shown to be machine-readable when the images were copied. Use robots.txt or TDMRep, not only words.
DPG Media v. HowardsHome, Amsterdam District Court, October 30, 2024 A robots.txt file that names some AI crawlers. An appeal is pending. Blocking named bots didn't reserve rights against a crawler the file didn't name. A reservation for everyone, such as TDMRep, covers crawlers you don't know about.
Like Company v. Google, Court of Justice of the EU, case C-250/25 Pending. Referred by a Hungarian court in 2025. The court will say whether training an AI model copies content, and whether the text and data mining exception covers it.
Robots Exclusion Protocol (RFC 9309) Any crawler that chooses to follow it. Crawlers read /robots.txt and follow the group that names them, or the * group. It's a request, not access control.

Acceptable Use Policy Generator

Ban scraping in your Acceptable Use Policy.

The TermsFeed Acceptable Use Policy Generator has an optional rule that forbids scraping, crawling or extracting data from your website or service by automated means, except through the features you provide. Add the AI training clause as your own rule.

Generate Acceptable Use Policy

Frequently asked questions.

Answers to common questions about blocking AI crawlers.

How do I block AI crawlers from my website?

Add a group for each AI crawler to your robots.txt file with Disallow: /, and keep the User-agent: * group open for search engines.

  • Name each crawler, such as GPTBot, ClaudeBot or CCBot.
  • Add the EU text and data mining opt-out (TDMRep), which covers crawlers you don't name.
  • Use the bot settings of your host or CDN for crawlers that ignore robots.txt.

Will blocking AI crawlers hurt my search rankings?

No, blocking AI training crawlers doesn't affect search engines.

  • Google says Google-Extended doesn't affect Google Search or rankings.
  • Apple says pages that block Applebot-Extended still appear in its search results.
  • Blocking AI search crawlers, such as OAI-SearchBot or PerplexityBot, removes your pages from those AI answers.

Does robots.txt stop AI crawlers?

robots.txt is a request. Well-behaved crawlers follow it, and nothing forces the others to.

  • OpenAI, Meta, Perplexity and Amazon say their fetchers that open a page for a person don't always follow robots.txt.
  • Some crawlers hide their name.
  • To block crawlers for real, use the bot settings of your host or CDN, or firewall rules.

What are Google-Extended and Applebot-Extended?

They're robots.txt names that control AI training, not separate crawlers.

  • Googlebot and Applebot crawl your pages as usual.
  • Blocking Google-Extended keeps the pages out of Gemini training and grounding.
  • Blocking Applebot-Extended keeps them out of model training at Apple.

What is TDMRep?

TDMRep is a W3C community protocol for reserving text and data mining rights in machine-readable form.

  • It uses a file at /.well-known/tdmrep.json, an HTML meta tag or an HTTP header.
  • tdm-reservation: 1 means the rights are reserved.
  • It applies to every AI company, so it covers crawlers your robots.txt file doesn't name.

Is the AI Crawler Rules Generator free?

Yes, the AI Crawler Rules Generator is free.

  • The generator needs no account.
  • The crawler list comes from the documentation each AI company publishes.

Build your rules with the generator above.