Two different goals
A search crawler indexes pages so they can be ranked later. An AI crawler usually fetches pages to answer a specific question right now. That difference changes what you want from the interaction.
Decide per crawler
- Training crawlers: decide whether your content should train models.
- Answer crawlers: blocking these removes you from live answers.
- Commercial crawlers: these drive some AI search products; policy varies.
A single blanket block is rarely the right answer. Most teams want to be found in answers while keeping their content out of training sets, which means evaluating crawlers individually.
How robots.txt expresses this
robots.txt supports per-agent rules. You can allow one crawler and disallow another with separate User-agent blocks. Most teams should keep an explicit allow list rather than relying on default behaviour.
What llms.txt actually does
An llms.txt file is a proposed convention for offering a curated set of pages for language models. It is advisory: no major crawler currently requires it. Treat it as low-cost, possibly useful, and never as a substitute for correct robots.txt rules.
Check what you actually changed
After changing crawler policy, confirm the outcome rather than assuming it. Fetch your own pages from a clean environment and note what a model can now see.