What you Need to Know About Bot and Agentic AI Traffic on your Website

As frontier AI models become more capable, cheaper, and better at using tools, people are finding that they can delegate increasingly vague tasks to them. OpenClaw was a big craze earlier this year (which you might have noticed if you were in the market for a Mac Mini). Many frontier chat apps already had built-in web-search tools, and we’ve seen them get further embedded within users’ environments, from desktop companions that have a view of everything the user is doing, to ones living directly in the browser.

These agents are not only better integrated but also able to interact with apps and the web in the same way humans do. Tools like Claude Cowork and Gemini in Chrome can use computer vision to “see” website content, and take over your pointer and keyboard to navigate and interact with UI elements. This is a big step up from bots that previously scraped with raw HTTP requests or agents that launched headless browser instances to navigate the web, and more difficult to distinguish from genuine user traffic (since they are literally controlling the mouse from the user’s everyday browser window). Many signals for detecting bot traffic fall apart at this stage (and one might even begin to question what defines “bot activity” anymore), but let’s take a step back and think about some broader questions - do you even need to know whether a visitor on your platform is really a human, a bot, or an agent? What benefits do you get from being able to distinguish between them? Do existing bot detection solutions suffice for this? (spoiler: no).

Traditional bots on the web, and ways for dealing with them

The internet is no stranger to machines visiting and interacting with sites. Automated bots have been a part of it since the beginning, performing tasks such as crawling pages to index them for search results, archiving, scraping structured content in lieu of missing APIs, and more.

What are some examples of “good bots” you would want to visit your site? The best-known examples are crawler bots such as Googlebot and Bingbot, which scrape much of the web to update Google’s and Bing’s massive search indexes. Another type of bot is a link unfurler, which fetches a page to build a preview, often the article title and a thumbnail image, that platforms such as Slack and WhatsApp can display. Other useful bots include performance checkers like Lighthouse and uptime monitors.

And then there are malicious bots - ones that do credential stuffing (checking leaked password information in large batches against login endpoints), scalping on shopping and event ticketing sites, unsolicited vulnerability scanning, and DDoSing. Much of the internet’s infrastructure is built around preventing these bots from affecting the security or availability of your website.

Features people frequently use to control what pages on their site are indexed or crawled are the Noindex, Nofollow, and Disallow directives. Techniques for detecting these bots include CAPTCHA, rate limiting, IP reputation, reverse DNS verification, device fingerprinting, and so on. Some commonly used platforms that handle this for you include Cloudflare, Akamai, DataDome, and HUMAN Security. However, these often fall short of detecting agent traffic, for reasons discussed below.

AI agents: not all are created equal

It’s important to make a distinction between the types of AI-lab crawlers that populate web traffic. They are frequently divided into crawler bots (bots that scrape the web to collect content for future model training), search bots (bots that index pages to surface in search results), and user bots (bots that fetch content specifically when acting on a user’s request). Only user bots make requests that are directly driven by an AI model accessing a website to answer a user’s query, and thus can be considered agentic AI traffic. For example, below is a breakdown of Anthropic’s bots, taken from their website:

Perplexity similarly has a search bot and a user bot, described here, but none that crawls training content. OpenAI documents its bots here, describing a similar split: GPTBot for training content, OAI-SearchBot for search results, and ChatGPT-User for requests representing user queries.

Do I want AI traffic on my website?

It wouldn’t be unreasonable to want to immediately block all AI bot traffic to deal with the sheer volume these new tools generate. The bots that scrape content for training feed the next generation of foundation models, and LLMs have repeatedly been shown to reproduce source text verbatim when prompted. If you’re concerned about protecting any creative or intellectual property, it’s important to keep those bots away from relevant pages.

On the other hand, it’s worth considering the entire ecosystem of the internet, and AI agents now make up a medium that millions of people are using as their primary way to interface with the web and surface information they need. Just as allowing crawlers such as Googlebot or preview generator bots is useful for a website, being visible to the right search bots and user bots can be hugely beneficial to the goals of a site operator, whether it’s getting hits in AI queries for products from your e-commerce site or making sure information about your offerings is accurate and well-sourced. A recent post by Peter Walker at OpenRouter showed that the category with the highest token usage on the platform was “workflow execution”, representing arbitrary tasks and workflows that include agentic browsing and decision making, coming in well above code generation and content writing. A survey by Pew Research showed that 42% of U.S. adults who use chatbots do so to search for information.

AI agents have the flexibility of human intent and the speed of machines, and many common processes on the internet start to look inadequate when we consider them through this new lens, especially e-commerce, which often has cumbersome flows designed to be navigated by humans. Internet standards bodies are quickly moving to accommodate agents and the ways they operate on the internet. x402 is an HTTP-native way to use stablecoins for payments at an API-request level, letting you gate web content easily and letting agents pay autonomously for access as they navigate (perhaps within a budget). UCP is a high-level commerce standard that supports product discovery, carts, and checkout flows from within agents, letting AI searches quickly surface directly purchasable products.

All of these allow agents accessing websites to take desirable actions such as paying for content or letting users purchase products from where they are. AEO (Answer Engine Optimization) is an emerging practice, similar to SEO, that aims to make your resources rank well in relevant queries. These standards are already being implemented in applications, and supporting them early will let your content be not just discoverable but authoritative!

How much of the traffic on the internet is actually made up of agentic bots?

A non-trivial amount, and it’s growing every day. At Vouched, we measure AI traffic with over 80 unique signals to detect agents on our clients’ websites. Back in June, we reported that across our portfolio, we had seen agentic traffic more than double from 3.06% to 7.38% within 30 days.

We also noted that agentic traffic behaved very differently from “regular” bot traffic, taking multi-step actions and spawning many more detections per active session than a non-agentic bot would. This demands more fine-grained control over who is able to access what content, but it also creates an opportunity to index more information and actions and make them ready to present to users.

Okay, how do I tell which ones are coming to my platform?

Before we go over the tools, let’s take a brief look under the hood. When a web server receives a request, it arrives with multiple pieces of information attached, called the request headers. These include crucial details about which content types the client accepts, cache directives, any authentication credentials such as session cookies or tokens, and more.

Among these is a header called the User-Agent (different usage of “agent” here!), which identifies information like the client application, operating system, and vendor making the web request.

This is what a User-Agent from a user with Chrome on macOS might look like:

And this is what one from Claude-User might look like:

Some AI bots like ChatGPT put this information in a slightly different header named Signature-Agent (or in addition to User-Agent), which uses a feature named HTTP Message Signatures to add authenticity to the request, allowing the receiver to verify it really is coming from the claimed source (e.g., OpenAI) and not someone else pretending to be them.

Server operators can then decide what to do with the request based on their policy and the above information. If they don’t wish to receive certain types of requests at all, they can say so in a file named robots.txt, placed at a top-level route on their website (e.g., my-pendant-store.com/robots.txt). There, you can define policies that let you do things such as blocking all bots with specific user agents, blocking only training bots but allowing user-facing bots, and more.

Great! Sadly, things get murky here. The User-Agent header is easy to spoof. Any actor who wants to scrape information without being blocked can fake it to resemble a regular, human-operated browser instance. Similarly, the Signature-Agent header only provides authenticity if it is included, and can simply be left out if the operator wants to hide that the traffic is from a bot.

And many AI agents, indeed, choose to spoof it. In August of last year, Cloudflare published a report showing that Perplexity, when confronted with content that it was blocked from accessing, attempted again while hiding that it was a bot. In fact, for the category of bots accessing websites to follow up on user queries, OpenAI, Google, and Perplexity all explicitly state that they will not honor robots.txt for such queries because the fetch is “initiated by a user”. Anthropic is the only major operator to state it will respect robots.txt for all three types of bots.

A request comes from a source IP address, and in some cases entities like Anthropic will publish the range of IPs their bots operate from, allowing those bots to be identified. But IP ranges change frequently and are easy to rotate to evade detection.

Additionally, there’s a whole class of agents, such as Claude Cowork, Perplexity Comet, and Gemini in Chrome, that now operate from within the user’s tools, taking direct control of their desktop or browser. Because of this, they don’t show up as distinct user agents to site operators at all, appearing instead as normal users. If you’re looking to spot these, it becomes a lot harder, as you may have to rely on behavioral signals instead of request headers and metadata.

What are services that I can use to detect agentic AI site visits?

AI agent detection is a newer field than traditional bot detection. Cloudflare offers Verified Bot detection, which identifies known bots and lets you set policies for handling them on its platform. This rests on two things: standards like Web Bot Auth, which require operators to sign their request headers, proving who they are (or verification through IP or reverse DNS), and additional checks by Cloudflare ensuring that bots from operators behave well, keeping request rates reasonable and honoring robots.txt. Together, they aim to ensure that all bot operators are transparent about their crawling and intent, and that they respect the preferences site owners express.

Vouched offers Agent Checkpoint Detect, a collection of bot and agent detection methods that work at different layers of the stack. For example, on the client side (the user’s browser), you can include a simple marketing pixel or install the JavaScript Beacon, both of which identify bots from page load and browser signals for analytics and marketing teams. There’s also a Node.js middleware package that inspects signals from request headers (like the User-Agent mentioned above), request frequency and timing, navigation patterns, and more. Gateway, which is deployed at the edge, can access lower-level network signals such as the TLS fingerprint. Together, these tools combine information signals from different slices of a session lifetime, and you’re free to mix and match them to fit your requirements for confidence, latency, and privacy. Detect also gives you more fine-grained decision making, as the tools report confidence scores and metadata (such as the verification method and risk level).

We also have a dashboard showing analytics on what activity is hitting your site, and which agents and bots are visiting. To get started, you can visit this link and select Checkpoint Lite to see your analytics. You can then use our Enforce platform to create policies for which bots can access and interact with certain parts of your applications.

What does this mean for me?

If you operate a website that receives any traffic, you probably have basic protections in place to prevent malicious bots from overwhelming it with DDoS attacks or doing credential stuffing. At the same time, most Content Management Systems include features for tailoring content to bots, such as Open Graph tags for link previews or search engine optimization, which is critical for reaching your target audience.

We’ve seen that agentic traffic is on the rise, and many tech and e-commerce players are developing tools and protocols to facilitate not just agentic browsing, but shopping and payments as well. So it matters just as much that your site is ready for these agents, and that you are able to enforce policies that represent your intent. At Vouched, we hope to see more development in this space, and are excited to offer tools for detecting and governing agent traffic today (namely Detect and Enforce), powered by open-source standards such as KYA-OS that bring identity and delegation to AI agents.

References

If you operate a website that receives any traffic, you probably have basic protections in place to prevent malicious bots from overwhelming it with DDoS attacks or doing credential stuffing. At the same time, most Content Management Systems include features for tailoring content to bots, such as Open Graph tags for link previews or search engine optimization, which is critical for reaching your target audience.

We’ve seen that agentic traffic is on the rise, and many tech and e-commerce players are developing tools and protocols to facilitate not just agentic browsing, but shopping and payments as well. So it matters just as much that your site is ready for these agents, and that you are able to enforce policies that represent your intent. At Vouched, we hope to see more development in this space, and are excited to offer tools for detecting and governing agent traffic today (namely Detect and Enforce), powered by open-source standards such as KYA-OS that bring identity and delegation to AI agents.

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

• https://kyc-chain.com/importance-audit-trails-enterprise-compliance/

• developers.openai.com/api/docs/bots

• https://docs.perplexity.ai/docs/resources/perplexity-crawlers

• https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting

• https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/