Merchant playbooks

Is your website ready for AI agents? A 2026 checklist

Listen · 13 min

Narrated by an AI voice. Code examples are on the page.

Short answer

A website is ready for AI agents when agents can reach it, read it, use it and be trusted on it. That means robots.txt and firewall settings that let the agents you want in, clear pages with product data and an llms.txt map, machine-readable entry points such as an MCP server, agent card or UCP profile if you sell, and a way to verify agents and set rules for what they may buy.

Key takeaways

  • Check access first: robots.txt, CDN bot settings and firewall rules.
  • Make pages readable: clear text, product data, sitemap and llms.txt.
  • Verify agents by signature or IP, and set rules for orders.

What does it mean for a website to be ready for AI agents?

It means an AI agent can do on your site what a customer would ask it to do: find you, understand what you offer, and, if you sell online, help someone buy. And you can tell which agents are real and decide what they may do.

AI traffic comes in three broad kinds, and each needs something different from you:

  • Training crawlers collect pages to train AI models, such as OpenAI's GPTBot.
  • Search and answer bots fetch pages so an AI can answer a question or cite you, such as OAI-SearchBot and ChatGPT-User.
  • Agents act for a person in real time, often in a browser: reading, clicking and filling in forms. ChatGPT agent and Google-Agent are examples.

The checklist below runs in the order agents meet your site: can they get in, can they read it, can they use it, and can you trust them. Each item links to a deeper guide. Here is the short version:

Area Check Why it matters
Access robots.txt names the AI bots you allow and block Search and training bots read it first
Access CDN and firewall let verified agents through Bot protection can block real customers' agents
Readable Key facts are in page text, not only images or scripts Agents read what they can parse
Readable Product structured data and a sitemap Prices, stock and pages are easy to find
Readable llms.txt at your site root A Markdown map of your key pages
Usable MCP server, agent card or API catalog Lets agents call tools instead of guessing
Usable UCP profile, if you want agents to check out A standard entry point for agentic checkout
Trusted Verify agents by signature or IP, not by name User-agent strings can be faked
Trusted Rules for what agents may order You decide what is approved, declined or held

Can AI agents reach your website?

Check this first, because nothing else matters if agents are turned away at the door. Two things control access: your robots.txt file and your CDN or firewall.

1. Review robots.txt for AI bots. robots.txt tells crawlers what they may fetch. It is a request, not a lock. The standard says: "These rules are not a form of access authorization." Well-behaved crawlers follow it.

Make a separate choice for each kind of bot. OpenAI says its robots.txt settings are independent: "Each setting is independent of the others", so you can block GPTBot from training and still allow OAI-SearchBot for ChatGPT search. OpenAI also says "it can take ~24 hours from a site’s robots.txt update for our systems to adjust." Our robots.txt guide for AI agents has the bot names and example rules, and should you block AI agents? helps you decide.

Know the limit. Bots that fetch a page because a person asked often do not follow robots.txt. Google says of its user-triggered fetchers: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." OpenAI says robots.txt rules may not apply to ChatGPT-User for the same reason.

2. Consider content signals. Cloudflare's Content Signals Policy, published in September 2025, adds a line to robots.txt that states how your content may be used after it is fetched, with three signals: search, ai-input and ai-train. Its example:

Content-Signal: search=yes, ai-train=no

Cloudflare is clear that "content signals express preferences; they are not technical countermeasures against scraping." See content signals for more.

3. Check your CDN and firewall. Bot protection can block agents you want. OpenAI says its cloud browser "can already use your site", but "your security settings may unintentionally block these visits." On Cloudflare, AI Crawler and Agent access policies let you choose Allow, Block or Block on pages with ads for Search, Agent and Training traffic, and for new domains all three default to Allow. If you use Cloudflare, our step-by-step guide on allowing good AI agents through Cloudflare covers the settings and skip rules.

Can AI agents read and understand your pages?

Agents do best with pages that state facts plainly in text. If a person has to guess, an agent will too.

4. Put key facts in page text. Prices, sizes, stock, delivery times, returns and contact details should be readable text on the page. Information that only appears inside images, or only after a script runs, is harder for some bots to see. Keep your policies on their own pages with clear headings.

5. Add product structured data. Google says that with Product structured data "your product information can appear in richer ways in Google Search results", such as price, availability, review ratings and shipping. The same data gives agents clean facts about each product. Most store platforms and SEO plugins can add it; check that the price and stock it shows match the page.

6. Keep a current sitemap. A sitemap lists the pages you want found. Remove dead pages and make sure new products appear.

7. Publish an llms.txt file. llms.txt is a proposal from September 2024. In its own words: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content." It lives at your site root, and the only required part is an H1 with your site or project name. A good one adds a one-line summary and links to your key pages and policies. It is a map, not a ranking signal, and it does not replace robots.txt. Our llms.txt guide for online stores shows what to put in it, and the llms.txt glossary entry has the short version.

Can AI agents use your website, not just read it?

Reading is enough for many questions. To let agents search, check stock or start a purchase reliably, you can publish machine-readable entry points. None is required; pick the ones that fit your business.

8. An MCP server. "MCP (Model Context Protocol) is an open-source standard for connecting AI applications to external systems." For a store, that means tools such as product search or order lookup that an assistant can call. Our guide does your store need an MCP server? walks through when it is worth it. See also MCP.

9. An agent card. An agent card comes from the A2A protocol. The A2A specification describes it as "A JSON metadata document published by an A2A Server", listing what the agent can do and how to reach it, at /.well-known/agent-card.json.

10. An API catalog. RFC 9727, published in June 2025, defines /.well-known/api-catalog so automated clients can find a publisher's APIs and their descriptions. It is mostly useful if you already offer an API.

11. A UCP profile, if you want agents to check out. UCP, the Universal Commerce Protocol, describes itself as "The common language for platforms, agents, and businesses." Its specification says: "Businesses publish their profile at /.well-known/ucp as the discovery entry point." Platforms read it to learn what you support. Selling inside a specific AI app, such as Google AI Mode, also needs that platform's own setup and approval; see how to sell in Google AI Mode and Gemini. For the wider picture, read agentic commerce protocols explained and ACP vs UCP.

If you run Shopify or WooCommerce, start with the platform guides: AI agents and Shopify and AI agents and WooCommerce.

Can you tell which AI agents are real and control what they do?

Being open to agents only works if you can tell real ones from fakes and set limits on what they may do. This is the part most sites skip.

12. Verify agents by proof, not by name. Anyone can put "GPTBot" or "ChatGPT" in a user-agent string. Stronger checks are the IP lists and reverse DNS that operators publish, and cryptographic signatures. With Web Bot Auth, an agent signs each request. OpenAI says its signed cloud browser requests carry a Signature-Agent header of "https://chatgpt.com", and Google says it is "experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity." Cloudflare verifies bots by Web Bot Auth or IP validation. Our AI agent identity guide covers every method, and how to tell if a visitor is really ChatGPT is the hands-on version.

13. Remember who is behind an agent. A verified agent tells you which company runs it. Cloudflare labels agent services that many end users operate as Intermediary, because the end user driving each request is a separate question. That is the idea behind know your agent (KYA).

14. Set rules for what agents may order. Decide in advance which orders an agent may place without a person checking: a spending limit, product types, new or returning customers. Our guide to order rules for AI agents has examples.

15. Guard against prompt injection. Text on a web page can try to steer an agent that reads it, for example instructions hidden in a review. This is called prompt injection. Read prompt injection and shopping agents to see what to watch for.

16. Watch your agent traffic. Check which bots visit, which pages they read and whether they follow your rules. Review again whenever you change robots.txt, your CDN settings or your checkout.

How do you check where your website stands today?

Start with what is public. Open your own /robots.txt and /llms.txt in a browser, look for the /.well-known/ files above, and test a product page with a structured data testing tool. Then look at your CDN's bot analytics to see what is being blocked.

You do not need to finish every item. A small business site can stop after the first seven. A store that wants agents to buy should work through all sixteen, starting with access and verification.

How Sill helps

Sill's free readiness scan needs no account. It checks what AI agents see on your domain today, including your agent card, MCP server, llms.txt, AI access in robots.txt, structured data, API catalog, sitemap and content signals, plus UCP for stores, and shows the exact URL it checked for each row. If you install Sill, it publishes the files agents read (llms.txt, an agent card, an MCP server and a catalog) and keeps them current, shows which AI agents visit and what they read, ask for and buy, and verifies signed agent requests on Cloudflare and WordPress installs for agents that sign. When an agent places an order, your rules approve it, decline it or hold it for a person to review, and every decision is signed and recorded.

Questions people ask

Do I need an MCP server to be ready for AI agents?

No. Many agents read ordinary web pages. An MCP server helps when you want AI assistants to search products, check stock or look up orders through defined tools instead of reading pages.

Does llms.txt make AI agents visit my site?

No. llms.txt is a proposal for a Markdown map of your site that AI tools can read. It does not invite or rank anything by itself, and it does not replace robots.txt.

How long does a robots.txt change take to work for AI bots?

It depends on the bot. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. Bots fetched on a user's request may not follow robots.txt at all.

Is there a free way to check if my site is agent-ready?

Yes. Sill's free readiness scan at sill.so/scan needs no account. It checks the public files agents look for, such as llms.txt, an agent card, an MCP server and AI rules in robots.txt, and shows the URL it checked for each one.

Can AI agents buy from any website?

Some agents use a browser to fill in checkout forms like a person. Others buy through protocols such as UCP or a platform's own programme, which need setup and sometimes the platform's approval.

Sources

  1. RFC 9309: Robots Exclusion Protocol · checked 17 Nov 2026
  2. OpenAI: Overview of OpenAI crawlers · checked 17 Nov 2026
  3. OpenAI: Agent Access (allow verified ChatGPT browser visits) · checked 17 Nov 2026
  4. Google: Google's user-triggered fetchers · checked 17 Nov 2026
  5. Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy · checked 17 Nov 2026
  6. Cloudflare docs: AI Crawler and Agent access policies · checked 17 Nov 2026
  7. Cloudflare docs: Verified bots · checked 17 Nov 2026
  8. llms.txt proposal · checked 17 Nov 2026
  9. Google Search Central: Product structured data · checked 17 Nov 2026
  10. Model Context Protocol: What is MCP? · checked 17 Nov 2026
  11. A2A Protocol specification · checked 17 Nov 2026
  12. RFC 9727: api-catalog well-known URI · checked 17 Nov 2026
  13. UCP: Universal Commerce Protocol · checked 17 Nov 2026
  14. UCP specification overview · checked 17 Nov 2026

Written by The Sill Signals Team. Last reviewed 17 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free