robots.txt and AI crawlers: what to watch out for

AI providers run different crawlers for different purposes: indexing for classic search, collecting training data, and user-initiated fetches to answer a specific query.

A robots.txt configuration should deliberately decide which of these crawler categories get access. Always check current provider documentation before making changes, since crawler names and behavior can change.

Allowing a crawler alone doesn't guarantee a mention in generated answers — it's a necessary but not sufficient condition.