AI access control service: visible in answers, in training on your terms.

We wrote this guide for publishers, ecommerce and corporate site teams who want their content to appear as a source in artificial intelligence (AI) answers but can’t clearly see which bot comes to their site and why, or whether the rules they’ve written are being followed. Every rule we describe is based on the IETF robots.txt standard; on the bot documentation of Google, OpenAI, Anthropic, Perplexity and Common Crawl; on the llmstxt.org proposal; and on Cloudflare’s documentation. We verified bot names against the source as of September 2026; because providers update their lists often, we check them again during setup. Our own field observations are marked separately under the heading “Expert note”.

Our AI access control work runs in three layers: building a crawler inventory from your logs and verifying identities, writing rules for each bot from robots.txt to the CDN, and setting up llms.txt and monitoring access. This page goes deeper into the technical access layer of GEO consulting.

What does an AI access control service solve?

AI access control makes sure the bots that carry you into AI answers, including those from the systems now discussed as superintelligence (ASI), can read your content, while bots collecting data for model training work according to your decision. AI bot traffic to your site isn’t a single “AI bot”; providers split their bots by purpose and give each its own name in robots.txt. OpenAI manages OAI-SearchBot for ChatGPT search and GPTBot for model training separately, and writes that these settings are independent: you can allow OAI-SearchBot and block GPTBot.¹ Anthropic’s split between ClaudeBot, Claude-SearchBot and Claude-User follows the same logic.²

This split opens up room for decisions. A publisher can want to be cited in answers while restricting training use; an ecommerce site opens product pages to AI search but closes cart and account pages to all bots; a corporate site wants to control crawl load. In none of these cases is the answer “open everything” or “close everything”; rules are written bot by bot, purpose by purpose.

Why do AI crawlers such as GPTBot, ClaudeBot and PerplexityBot visit?

AI crawlers come for three purposes: building a search index, fetching a page in real time for a user’s question, and collecting data for model training. These groups treat robots.txt differently; the table summarizes the product names we see most often in logs, using the providers’ own definitions.

The most important distinction is fetchers that come on a user’s behalf. Google splits its requests into three as well: common crawlers that always obey robots.txt, special-case crawlers whose behavior depends on their function, and user-triggered fetchers that ignore robots.txt because a user asked for the page.⁷ OpenAI also states separately that ChatGPT-User doesn’t crawl automatically and doesn’t determine appearance in Search.¹ That’s why we measure traffic that comes on a user’s behalf on a separate line. There’s one more bot for advertisers: OAI-AdsBot visits only landing pages submitted as ads in ChatGPT, to check policy compliance, and the data it collects isn’t used for training;¹ details are on our ChatGPT Ads consulting page.

How do you build a bot inventory from server logs?

A bot inventory is built by reading each request’s user-agent, IP address, requested path and status code together in server and CDN logs. Under RFC 9309, a crawler identifies itself in robots.txt with a product token, and that name is expected to appear in the request’s User-Agent header too;⁸ what we search the logs for is this name, not the full string. Because the Chrome version number in these strings changes, Google recommends using a wildcard instead of the version when searching.⁴ OpenAI writes that it may add a “robots.txt” marker to the user-agent on robots.txt requests;¹ this marker makes it easier to separate robots.txt reads in logs that don’t record the path.

After grouping, we look at four questions for each bot: which paths does it request, how often does it come, which status code does it get, and does it send requests to paths we’ve closed? Whether search bots reach product pages, and the load training bots put on the server, are read from this table. OpenAI writes that on sites that allow both bots, it may use the result of a single crawl for both purposes;¹ so request counts don’t show each purpose’s share exactly.

How do you verify that a bot really is that bot?

A bot’s identity is verified not by its user-agent string but by the requesting IP address being on the list the provider publishes, or resolving to the right domain. Google writes that the HTTP user-agent string can be spoofed;⁴ Common Crawl also warns about fake crawlers that identify themselves as CCBot.⁶ The verification order:

1. Reverse DNS: a reverse DNS lookup is run for the IP in the log; for Google crawlers, the name must end in googlebot.com, google.com or googleusercontent.com.⁷ For Common Crawl, the reverse DNS name of IPv4 addresses ends in crawl.commoncrawl.org.⁶
2. Forward DNS: a forward lookup is run for the returned name, and you check that the result matches the IP in the log.⁷
3. IP lists: at high volume, providers’ JSON lists are used: Google’s common-crawlers.json and fetcher lists,⁷ OpenAI’s searchbot.json, gptbot.json and chatgpt-user.json files,¹ Anthropic’s bots.json list,² Perplexity’s two lists⁵ and Common Crawl’s ccbot.json file.⁶
4. Decision: if the name and the IP match, the request is verified; if the name claims a known bot but the IP isn’t on the list, the request is reported separately as an impostor.

How do you block or allow AI bots in robots.txt?

In robots.txt, each bot follows the rules of the group that matches its product token; if it has no group of its own, it follows the asterisk (*) group, and if there’s none, it isn’t bound by any rule.⁸ Name matching is case-insensitive; groups matching the same name are merged into one.⁸ Within a group, the longest, meaning most specific, path match wins; if an equivalent allow and disallow conflict, allow wins.⁸ The file sits at the service’s root path as /robots.txt;⁸ every subdomain needs its own file, and Anthropic also says blocking must be done separately for each subdomain.²

The order of decisions follows from this. First, we keep search bots open; OpenAI writes that sites blocking OAI-SearchBot aren’t shown in ChatGPT search answers and can appear only as navigational links.¹ Next, we write the brand’s preference for training bots and Google-Extended, and finally, the paths no bot needs to crawl, such as cart, account and on-site search, separately in each group.

What does Google-Extended do, and what doesn’t it do?

Google-Extended is a robots.txt control token that determines whether content Google crawls can be used to train Gemini models and for grounding (supplying content from the Search index to the model at answer time); it isn’t a separate crawler.⁴ It has no user-agent string of its own, and crawling is done with Google’s existing strings;⁴ searching the logs for a request named “Google-Extended” turns up nothing. The preference affects Gemini apps and the Gemini API in Vertex AI.⁴

Google-Extended doesn’t affect Search: Google writes that this token doesn’t affect a site’s inclusion in Search and isn’t a ranking signal.⁴ AI Overviews and AI Mode, however, are part of Search; here Google says the control is the robots.txt rules written for Googlebot, and that nosnippet, data-nosnippet, max-snippet or noindex should be used to limit the information shown.³ For a page to be a supporting link in an AI Overview, it must also be indexed and eligible to be shown with a snippet.³

When do you need meta robots and X-Robots-Tag?

Meta robots and X-Robots-Tag are needed for page- and file-level indexing and display decisions that robots.txt can’t make. The meta tag sits inside the HTML page; X-Robots-Tag carries the same rules in the HTTP response header, and Google says to use this header for files such as PDFs, images or video.⁹ A rule with no bot name applies to all crawlers; when rules conflict, the more restrictive one applies.⁹ These tags are read when the URL is crawled: rules on a page blocked from crawling by robots.txt can’t be found and are ignored.⁹ Blocking a page in robots.txt and also adding noindex means the noindex is never seen.

Google documents meta robots and X-Robots-Tag in detail; OpenAI, Anthropic and Perplexity describe control through robots.txt.¹²⁵ That’s why we write the training preference in robots.txt.

How do CDN rules step in where robots.txt isn’t enough?

CDN rules step in to stop requests that don’t obey robots.txt, that come on a user’s behalf and ignore the rule, or whose identity can’t be verified, before they reach the server. RFC 9309 writes that robots.txt rules aren’t access authorization, don’t replace security measures, and that the paths listed in the file become public.⁸ Writing a hidden path into robots.txt doesn’t hide it; it makes it findable.

On Cloudflare, this job is done by AI Crawl Control: it lists AI crawlers, request counts and robots.txt violations bot by bot, and you choose allow or block for each.¹⁰ Blocking is written as a custom WAF rule and path-based exceptions can be added; on paid plans the bot can be sent a 403 or a 402 (payment required) response, while on the free plan detection is limited to crawlers that identify themselves by user-agent.¹⁰ Cloudflare splits bots into three behavior classes: search, agent and training; the agent class covers chat fetchers and browser agents working on a user’s behalf.¹¹ Cloudflare also announced a default for 15 September 2026 under which, on new domains, the training and agent classes are blocked on pages that show ads, while search bots stay open.¹¹ In other words, an ad-supported publisher page can also be closed to the real-time fetch an assistant makes for a user. Google also recommends allowing crawling at the CDN and hosting infrastructure as well as in robots.txt;³ that’s why we keep CDN settings in the same table as robots.txt decisions.

What is llms.txt, and what is it for?

llms.txt is a Markdown file placed in the root directory or a subpath to give language models and AI agents a short summary of a site and links to its important pages.¹² It’s a standardization proposal Jeremy Howard published in September 2024; today it’s live under the heading “v2”, last updated in August 2026.¹² The file is written in this order:¹²

• H1 title: the site’s name; the only required section.
• Summary blockquote: a short description needed to understand the rest of the file.
• Description sections: detail in paragraphs and lists without headings.
• H2 file lists: one Markdown link per line, with an optional note.
• “Optional” section: secondary links that can be skipped when context is tight.

The proposal asks that pages containing information also be published as a clean version with the .md extension at the same URL.¹² robots.txt says which access is accepted; llms.txt is read when an agent needs it while helping a user. A sitemap doesn’t do this job, because it doesn’t list LLM-friendly versions and is often too large to fit in the context.¹²

We keep the file limited to the pages an agent needs to answer correctly, such as company information, product or service groups, pricing and delivery policies, and support pages. Our free AI visibility analysis audits your crawler access and generates an llms.txt draft; during setup, we finalize this draft together with your robots.txt decisions.

How does an AI access control service work in the first 30 days?

The job of the first 30 days isn’t to change the rules right away; it’s first to measure who comes, then get the decision for each bot in writing, and roll out the rules after testing them.

1. Week 1 · Inventory: server and CDN logs are collected, requests are grouped and verified with IP and DNS; the current robots.txt, meta robots and CDN rules are mapped out.
2. Week 2 · Decision table: an allow, restrict or block decision is written for each bot together with you; the training preference and load limits are settled.
3. Week 3 · Setup: robots.txt is written bot by bot, meta robots or X-Robots-Tag is added where needed, CDN rules are set up, and llms.txt goes live.
4. Week 4 · First reading: the logs are read again; requests that don’t follow the rules, search bots blocked by mistake and robots.txt status codes are reported.

The effect of changes isn’t immediate: RFC 9309 says a cached copy of robots.txt can be used for up to 24 hours;⁸ OpenAI writes that changes take about 24 hours to take effect,¹ and Perplexity up to 24 hours.⁵ That’s why we do the first reading a few days after setup.

Copied template, CDN toggle or log-based service?

There are three common ways to manage AI bots; the difference is what the decision rests on:

How to choose an AI access control partner

1. Verification method: bots are verified by IP list and DNS, not by user-agent name; spoofed requests are reported separately.
2. Search and training kept apart: it is shown rule by rule that search bots stay open while training bots are closed.
3. robots.txt status code: after every change, the firewall is tested to return 200 for robots.txt, the sitemap and llms.txt.
4. Testing the live file: rules are read from the live URL, including lines added by the CDN and plugins, not from the copy on the server.
5. The decision is yours: allow, limit or block is decided with you for every bot, in a written table.
6. Guarantee language: no one should promise that llms.txt or a bot setting will bring visibility in AI answers; access is a precondition for visibility, not a guarantee of it.

AI access control pricing: what drives the cost?

The fee for this service depends on scope; instead of package prices, we share estimated ranges. The deciding factors:

We work out the scope together in the first meeting and then send a written proposal; a one-off setup and ongoing monitoring can be discussed as separate items.

AI access control pricing: estimated ranges

Estimated range, excluding VAT; rows marked “/ month” are monthly, the others are one-off. Scope is confirmed in a written proposal after the first meeting. As of October 2026.

When talking to a partner, it helps to get concrete answers to these questions: how do you verify bots, how do you separate search and training bots, how do you keep CDN rules consistent with robots.txt, and what do you measure after a change? If you’d like to read which bots your site is open to today together, pick a meeting day that suits you below.