Merchant playbooks

How to allow good AI agents through Cloudflare without opening the door to bad bots

Listen · 9 min

Narrated by an AI voice. Code examples are on the page.

Short answer

Allow AI agents by identity, not by name. In Cloudflare, set the Agent behavior (and any Personal Agents you want) to Allow in AI Crawler and Agent access policies, and add a WAF custom rule with the Skip action for verified bots if Super Bot Fight Mode or other rules are in the way. Keep your bot protection on for everything else, and never allow traffic just because its user-agent string names a well-known agent.

Key takeaways

  • Check Security Analytics and AI Crawl Control before changing anything.
  • Allow agents by Cloudflare's verification, never by user-agent string alone.
  • Bot Fight Mode cannot be skipped; Super Bot Fight Mode can.

Why does Cloudflare block AI agents you want?

Usually because a security setting is doing its job too broadly. Cloudflare's bot tools are built to stop automated traffic, and an AI agent is automated traffic, even when a real customer sent it.

OpenAI puts it plainly in its guide for site owners: "ChatGPT’s cloud browser can already use your site. You don’t need to add it to an allowlist. However, your security settings may unintentionally block these visits."

Several Cloudflare layers can stop an agent:

  • AI Crawler and Agent access policies. You pick an action for Search, Agent and Training traffic. If Agent is set to Block, agents acting for people are blocked across the zone.
  • Per-crawler blocks in AI Crawl Control. When you block a crawler there, Cloudflare creates or updates a WAF custom rule to enforce it.
  • Bot Fight Mode (free plan) or Super Bot Fight Mode (Pro, Business and Enterprise). Both can challenge or block automated traffic.
  • Your own custom rules, rate limits and security level.

The aim is not to switch these off. It is to make one narrow exception for agents Cloudflare has verified, and leave the rest of your protection in place. If you are still deciding which kinds of AI traffic you want at all, start with should you block AI agents?.

How do you find out what Cloudflare is blocking?

Look before you change anything. Two places show it.

  • Security Analytics. Cloudflare's Bot Fight Mode docs point to Security Analytics as the place to see blocked AI bot traffic. Filter to the blocked or challenged requests and note which product acted. That tells you which layer needs an exception.
  • AI Crawl Control. Available on all plans, it lists the AI crawlers requesting your content and whether you allow or block each one. On the free plan it identifies crawlers by their user-agent strings, and Cloudflare says upgrading enables more thorough detection.

Write down what you find: which agent, which product blocked it, and on which pages. A block on your product pages matters more than a block on your login page.

How do you allow ChatGPT agent through Cloudflare?

OpenAI's Agent Access guide gives two Cloudflare options, and says Cloudflare verifies ChatGPT's request signatures automatically, so you do not need to check them yourself.

Option 1: AI Crawler and Agent access policies. OpenAI says Cloudflare shows its cloud browser as dots, OpenAI under Personal Agents.

  1. In the Cloudflare dashboard, select your domain and go to Security, then Settings.
  2. Filter by Bot traffic and open AI Crawler and Agent access policies.
  3. Under Personal Agents, set dots, OpenAI to Allow (do not block).
  4. Save.

Cloudflare's own docs add two useful details. Personal Agents inherit your Agent setting by default, and "Cloudflare recommends Allow (do not block) for Personal Agents." For new domains, Search, Agent and Training all default to Allow (do not block).

Option 2: a WAF custom rule. OpenAI lists the identifiers Cloudflare uses: bot tag chatgpt-agent and detection ID 129220581. Create a custom rule under Security, WAF that matches Bot Detection ID equals 129220581 (OpenAI says to search for "ChatGPT Operator"), and set it to allow or skip. This is useful when another rule, not the access policy, is the one blocking.

Behind both options sits Web Bot Auth. OpenAI says the guidance "applies to signed cloud browser requests, not web crawlers", and its signed requests carry a Signature-Agent header of "https://chatgpt.com". That signature is what lets Cloudflare tell the real ChatGPT agent from a copy. Other agents work the same way: Google says it is "experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity" for Google-Agent.

How do you let verified agents past Super Bot Fight Mode and your own rules?

Use a WAF custom rule with the Skip action that matches verified bots, placed before the rules that block them.

Cloudflare's docs say: "To configure exceptions to Super Bot Fight Mode, create a custom rule with the Skip action." The Skip action can bypass "rate limiting rules, Managed Rules, Super Bot Fight Mode rules, and several other security products." The skip options include all remaining custom rules, all rate limiting rules, all Super Bot Fight Mode rules, all managed rules, User Agent Blocking, Browser Integrity Check and Security Level.

Cloudflare's rules language has a field for this. cf.client.bot "Indicates whether the request originated from a known good bot or crawler." Since 1 July 2026, Cloudflare counts signed agents as verified bots, as our explainer on the change covers. An illustrative rule expression:

(cf.client.bot)

Cloudflare's field reference does not spell out whether every newly verified agent matches this field, so check in Security Analytics that the rule catches the agents you want.

Set the action to Skip and tick only the products that were blocking agents in your analytics. Skip what you must, not everything. For example, you might skip Super Bot Fight Mode but keep rate limiting, so a verified bot that sends too many requests is still slowed down.

On Enterprise plans with Bot Management, cf.bot_management.signed_agent marks a request from "a known agent that self-identifies with Web Bot Auth", now classified as a verified bot or agent labeled Intermediary. That lets you write a rule for signed agents alone instead of all verified bots.

Note. Bot Fight Mode on the free plan is different. Cloudflare says: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." It recommends Super Bot Fight Mode for exceptions. If Bot Fight Mode is challenging agents you want, your options are to turn it off or upgrade.

How do you keep bad bots out at the same time?

Make every exception depend on Cloudflare's verification, and keep your other controls running for everyone else.

  • Never allow by user-agent string. A rule like "user agent contains ChatGPT" lets in anyone who types that name. Our guide to fake AI crawlers shows how common that is. Match on cf.client.bot, detection IDs or the access policies instead.
  • Keep Super Bot Fight Mode on for unverified traffic. The skip rule only covers verified bots. Definitely and likely automated traffic still meets your normal settings.
  • Keep rate limits. A verified operator can still send more requests than your site can handle.
  • Decide Training separately. Allowing agents does not mean allowing training crawlers such as GPTBot. The Training behavior has its own setting, including a Disallow option that adds a robots.txt instruction. Cloudflare notes robots.txt directives are voluntary.
  • Remember who is behind an Intermediary agent. Cloudflare labels agent services that many end users operate as Intermediary. Verification tells you which company runs the browser, not who typed the instructions. Keep your normal fraud and order checks at checkout.
  • Review regularly. Cloudflare's agent categories changed in July 2026 and may change again. Check AI Crawl Control and your analytics after any change, and re-run your own test purchase.

How Sill helps

Sill shows which AI agents and bots visit your site, the pages they read, and what they ask for and buy. It verifies signed agent requests (Web Bot Auth) on Cloudflare and WordPress installs, for agents that sign. When an agent places an order, your rules approve it, decline it or hold it for a person to review, and every decision is signed and recorded. To see what agents can find on your site today, run the free readiness scan, or work through our agent-ready checklist.

Questions people ask

Do I need to allowlist ChatGPT agent on Cloudflare?

Usually not. OpenAI says its cloud browser can already use your site and you don't need to add it to an allowlist, but security settings may block it by accident. If that happens, OpenAI's guide gives two Cloudflare fixes: allow its Personal Agent entry, or add a WAF rule for its detection ID.

Can I allow AI agents on Cloudflare's free plan?

Partly. AI Crawl Control is available on all plans and lets you allow or block each AI crawler, but Bot Fight Mode, the free plan's bot tool, cannot be bypassed with WAF custom rules. If it gets in the way of agents you want, your choices are to turn it off or move to a plan with Super Bot Fight Mode, which supports skip rules.

Is it safe to allow all Cloudflare verified bots?

Verified means Cloudflare has confirmed who the bot is and that it meets its behavior rules. It does not mean every request is one you want. Keep rate limits, choose your Training setting separately, and keep order checks for agents that buy.

Why does an AI agent still get blocked after I allowed it?

Another layer is probably acting first, such as Bot Fight Mode, a custom rule, a rate limit or a security level setting. Look up the blocked request in Security Analytics to see which product acted, then add an exception for that product.

Sources

  1. OpenAI: Agent Access (allow verified ChatGPT browser visits) · checked 17 Nov 2026
  2. Cloudflare docs: AI Crawler and Agent access policies · checked 17 Nov 2026
  3. Cloudflare docs: Verified bots · checked 17 Nov 2026
  4. Cloudflare docs: Bot Fight Mode · checked 17 Nov 2026
  5. Cloudflare docs: Super Bot Fight Mode · checked 17 Nov 2026
  6. Cloudflare docs: Skip action (custom rules) · checked 17 Nov 2026
  7. Cloudflare docs: Available skip options · checked 17 Nov 2026
  8. Cloudflare docs: AI Crawl Control · checked 17 Nov 2026
  9. Cloudflare docs: Manage AI crawlers (AI Crawl Control) · checked 17 Nov 2026
  10. Cloudflare docs: cf.client.bot field · checked 17 Nov 2026
  11. Cloudflare docs: cf.bot_management.signed_agent field · checked 17 Nov 2026
  12. Cloudflare blog: The age of agents: cryptographically recognizing agent traffic · checked 17 Nov 2026
  13. Google: Google's user-triggered fetchers · checked 17 Nov 2026

Written by The Sill Signals Team. Last reviewed 17 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free