If you can see the agents, you can sell to them

Most teams discover they have an agent problem the same way: a marketing dashboard shows traffic spiking, conversion cratering, and nobody can explain it. The traffic is real. The shoppers are not.

Imperva's 2025 Bad Bot Report found that automated requests crossed 51% of total web traffic in 2024, the first time non-human traffic has been the majority in a decade. Cloudflare's 2025 figures on AI-specific crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, and friends) show year-over-year growth of 200–400% across the board, with GPTBot alone now responsible for more crawl requests on top-1000 retail sites than Googlebot.

The default response is to ignore them, or to block them. Both are mistakes. Agents are buyers. They convert at materially different rates than humans (38% higher on AI-referral traffic per Adobe Analytics), they shop different categories, and they fail in different places. None of that is visible in Google Analytics.

If you want to make informed decisions about agents (what to expose, what to price, what to rewrite) you have to measure them first.

What "tracking an agent" actually means

There are four signals worth separating, because the right action for each is different.

1. Crawlers. GPTBot, ClaudeBot, PerplexityBot, Bytespider, Applebot-Extended, Google-Extended. These don't transact. They index your content for retrieval at inference time. Block them and you disappear from the corresponding model's answers. The decision is strategic, not technical.

2. Real-time retrievers. When a user asks ChatGPT about your product, the model may issue a live fetch through OpenAI's ChatGPT-User agent (distinct from GPTBot). Same for Perplexity's Perplexity-User. These hit your site during a shopping session. They're directly upstream of conversion.

3. Browser agents. Operator, Claude for Chrome, the upcoming wave of agent SDKs that drive a real browser on a user's behalf. These look almost identical to human traffic: a Chrome user-agent string, a real session, mouse events, the works. The tells are subtle: viewport sizes that cluster around fixed defaults, no scroll dwell, page-to-page intervals measured in hundreds of milliseconds.

4. API/MCP traffic. Agents calling structured endpoints rather than crawling HTML. If you have an MCP server, an ACP endpoint, or a public catalog API, the traffic shape is wholly different: high cardinality, low session length, no funnel.

Conflating these is why most "bot detection" dashboards are useless for commerce. Blocking a Bytespider crawler and blocking a ChatGPT shopper running a real-time retrieval are very different decisions with very different revenue consequences.

What to actually log

For each request, the minimum useful record is:

• User agent string: for declared agents, this is the cheap, honest signal. GPTBot/1.2, ClaudeBot, PerplexityBot/1.0, etc. Anthropic, OpenAI, and Perplexity all publish verifiable IP ranges so you can reject spoofers.
• Referrer: chat.openai.com, perplexity.ai, claude.ai, gemini.google.com referrers are the clearest signal that a human is in the loop and the agent is acting on their behalf.
• Path and method: agents disproportionately hit product detail pages, schema.org JSON-LD, sitemaps, robots.txt, and structured-data endpoints. The shape of which paths get hit is itself a behavioral fingerprint.
• Timing: time-on-page, page-to-page latency, dwell. Humans pause. Agents don't.
• Render path: did they execute JavaScript? Most crawlers don't. Most browser agents do. This single bit splits two very different populations.
• Outcome: did the session end in an add-to-cart, a checkout, an abandoned funnel, or a parse-and-leave?

You can collect most of this from your existing server logs and edge platform. The work isn't collection. It's segmentation: pulling agent traffic out of the human aggregate so it stops contaminating your conversion math.

The decisions this unlocks

Once you can see agents as a distinct cohort, the questions you can answer get a lot more interesting.

Which models are sending you the most buyers? AI referral traffic is now ~20% of referrals to major retailers per Similarweb, but the breakdown by model is wildly uneven by category. Apparel, electronics, home goods, and travel each have a different dominant agent today. Knowing yours tells you where to focus content optimization.

Where are agents failing? Agents abandon at structured-data ambiguities: missing availability, inconsistent price fields, broken JSON-LD, missing variants. A 2025 Princeton/MIT study on agent commerce benchmarks found agents fail roughly 60% of completion tasks on real e-commerce sites, with parsing errors and ambiguous policies as the top two causes. If you can see which pages your agent traffic bounces off, you have a punch list.

What should you price differently? Agent shoppers are notably less price-sensitive on commoditized SKUs and notably more sensitive on premium items where they can comparison-shop instantly. That cuts against most marketers' instincts. Without measurement, you'll guess wrong.

Which crawlers are paying you back? Letting GPTBot crawl your full catalog only makes sense if the resulting ChatGPT-User retrievals (and downstream referrals) outweigh the bandwidth and competitive cost. The ratio is measurable. Most teams haven't measured it.

What content should be agent-first? If a structured policy page (returns, sizing, materials) gets disproportionate agent fetches and correlates with downstream conversion, that's a high-leverage page to rewrite for clarity and machine-readability. If a glossy editorial page never gets parsed, it's not earning its keep in the agent funnel.

The instrumentation gap

The honest version: doing this well is harder than it sounds. Standard analytics (GA4, Mixpanel, Amplitude) deliberately filter known bots out of their reports. That's the right default for human-funnel analysis. It's the wrong default for the question we actually want to answer.

You need a parallel pipeline:

1. Capture raw edge logs before bot filtering.
2. Classify each request into the four categories above using user agent, referrer, IP verification, and behavioral fingerprint.
3. Join agent sessions back to outcomes (orders, carts, errors) through a stable session or order identifier.
4. Report on the agent cohort separately, with its own conversion, AOV, and funnel breakdowns.

Most off-the-shelf tools stop at step 1 or 2. The reporting and outcome-joining is where the work lives, and where the insight is.

What we're building

Agent observability is one of the four pillars we think the agent-ready internet needs. We're building the layer that does the classification, the outcome-joining, and the reporting, so commerce teams can stop guessing about agent behavior and start making decisions with it.

If you're sitting on a pile of edge logs and want to know what's actually shopping your site, come talk to us.