TechmadeazyBefore any content work, make sure AI tools can actually read your site. It takes five minutes to check, and a surprising number of stores fail it without knowing.
Step 1: open your robots.txt
Go to yourdomain.com/robots.txt in a browser. This file tells automated crawlers which parts of your site they may read. Look for lines that name AI crawlers followed by Disallow: /, which blocks the whole site for that crawler.
The crawlers worth knowing
| Crawler | Run by | What blocking it affects |
|---|---|---|
| OAI-SearchBot | OpenAI | Whether your pages can appear in ChatGPT's search results and citations |
| ChatGPT-User | OpenAI | Pages ChatGPT visits when a user asks it to look something up |
| GPTBot | OpenAI | Whether your content can be used to train OpenAI's models |
| PerplexityBot | Perplexity | Whether your pages can be surfaced in Perplexity answers |
| ClaudeBot | Anthropic | Whether your content can be collected for Anthropic's models |
| Google-Extended | Whether Google can use your content for its Gemini models. It does not change how you appear in normal Google Search |
Each company documents its crawlers, and the list changes, so check their current documentation before making permanent decisions.
Step 2: check what sits in front of your site
Robots.txt is not the only gatekeeper. Some CDNs, firewalls and bot-protection apps now block AI crawlers by default or with a single toggle. If your robots.txt looks fine but your pages never show up as citations, check your hosting, CDN and security app settings for an AI bot or crawler blocking option.
Step 3: decide deliberately
There are sensible reasons to block training crawlers while allowing search and retrieval crawlers. A brand that wants to be recommended in AI answers should, at minimum, let the crawlers that power live answers read its product and category pages. Whatever you choose, make it a decision, not an accident left by an old plugin setting.
Step 4: make the important facts readable
Even when crawlers are allowed in, facts that only appear inside images, or only load after heavy scripts run, may be missed. Prices, sizes, materials, ingredients and policies should appear as plain text on the page.
Our free self-test includes a quick link to your robots.txt, and every AI Visibility Audit covers crawler access in full.
