Machines already made up about half of web traffic before AI arrived in force. Most of it is search engines, monitoring tools and attackers, yet a new kind of machine visitor has joined them, and it reads a firm’s website for a different purpose.

Cloudflare, which says it manages traffic for about 20% of the web, found that bots other than AI were responsible for half of requests for web pages at the start of 2025. It measured AI bots at 4.2% of those requests over the year, close to the 4.5% that came from Google’s search crawler alone, and found that crawling done by assistants on a person’s behalf grew more than fifteenfold. The share is small and the growth is fast. In a September 2026 letter, Cloudflare’s founders wrote that automated traffic passed human traffic in May 2026, more than a year ahead of their forecast, and credited “the rise of agents and AI crawlers.”

Three Kinds of AI Visitor Matter to a Firm

Training crawlers, search crawlers and assistants acting for a person each visit for a different purpose.

Most automated traffic is of no use to a firm, and a good deal of it is hostile. Three kinds of AI visitor are different. OpenAI’s guidance for website owners gives each its own name, a pattern other AI companies follow.

  • Training crawlers, such as OpenAI’s GPTBot, collect pages to improve AI models. Cloudflare found in 2025 that training “now drives nearly 80% of AI bot activity.”
  • Search crawlers, such as OAI-SearchBot, gather pages so that an AI service can cite them when it answers a question.
  • Assistants acting for a person, such as ChatGPT-User, fetch a page because someone asked an assistant to look something up or complete a task.

The three differ in their value to a firm. A training crawler takes content and may never send a visitor in return. A search crawler determines whether the firm appears in an AI-generated answer. An assistant acting for a person may be a prospective customer’s request arriving by another route.

The three also overlap in practice. Some crawlers serve training and search at once. Google states that websites allowing its AI training control “are eligible to appear in responses and receive clicks” in its Gemini app, so refusing training can also remove a firm from some answers.

The exchange is uneven. In the same 2025 study, Cloudflare measured how many pages each service crawled for every visitor it referred to a website. In July 2025 the ratios ran from about 5 for the largest search engine to about 1,100 and 38,000 for two AI services, and the higher figures had been falling.

Machines Read a Different Page Than People Do

Many AI crawlers read only the page a server first sends, before any script runs.

A person sees a page after the browser has run its scripts. Many AI crawlers do not run them. A December 2024 study by the web host Vercel found that “none of the major AI crawlers currently render JavaScript,” meaning they did not run a page’s scripts, with Google’s and Apple’s as the exceptions. Text that appears only after scripts run may never reach such a crawler. Agents that operate a browser are a further exception, since they can see the finished page.

That favors sites that send complete pages from the server, as Astro does by default and as a well-built WordPress site does. A script-driven map shows the problem. A regional insurance agency that loads its office locations that way shows them to every person and not to several of the most used assistants.

The website is also only one of an assistant’s sources. Bain found that 89% of prompts naming no brand were answered from third-party sources, so the same facts need to be right in a firm’s business listings and directory entries.

Two Decisions Belong to the Firm

A firm decides what machines may read and, as agents begin to act, what they may do.

What machines may read. A file named robots.txt, found at a site’s address followed by /robots.txt, tells crawlers which parts of a site they may visit. It is a request that well-behaved crawlers honor, and OpenAI notes that its rules “may not apply” when a person asks an assistant to fetch a page.

Hosting and security services add controls of their own, and those change. Cloudflare began asking the owners of newly added domains about AI crawlers in July 2025. In July 2026 it replaced that single switch with separate settings for search, agent and training traffic, with new defaults from September 15. A firm on Cloudflare therefore holds a setting for each of the three, whether or not anyone chose it. Google added a control in 2026 as well: a setting in Search Console, its reporting tool for site owners, includes a site in Google’s AI features or removes it.

For a firm that sells services, the case for admitting search crawlers and assistants is strong, because they are the route into AI answers. Training is the open decision, and it turns on how much of the firm’s value sits in what it publishes.

What machines may do. Agents are beginning to act as well as read, filling in forms, booking appointments and placing orders. WordPress has added a standard way for a site to list the actions an AI assistant may take. In October 2026 the project published a connector that exposes those actions to assistants such as Claude and ChatGPT, and it still describes that connector as experimental.

A workable rule separates low-risk actions from the rest. Reading a page, checking availability and sending an inquiry are low risk. Anything that changes a record, spends money or cancels a booking is better left to a person’s confirmation. What to switch on and what to wait on covers the detail.

A Machine-Ready Site Is Mostly a Well-Built One

A site that sends complete pages from the server is mostly ready for machines.

Google’s 2026 guide says no new machine-readable files or special markup are needed to appear in its AI features. The properties that matter are older.

  • Pages arrive complete from the server, with the important text present before any script runs.
  • Facts a buyer needs, such as services, locations and hours, appear as plain text and not only inside images or maps.
  • Structured data, the labels that tell software what a page contains, matches what the page visibly says.
  • The robots.txt file and the host’s AI settings are read at least once a year.

Two of these take minutes to check. The robots.txt file opens in any browser. The browser’s View Page Source command shows the page as a crawler that runs no scripts receives it, so a fact missing there is missing for that crawler.

The AI share of visits is likely to keep rising, and AI-generated answers already change how often people click through. Firms that decide what machines may read, and check the settings where that decision is enforced, will know which AI services can use their pages.