What Building AI Agents Inside a Production Startup Taught Me About the Future of the Web
In March, Cloudflare’s CEO told an audience at SXSW that bot traffic would surpass human traffic online by 2027. On June 3, he posted that it had already happened: automated systems now generate 57.5% of HTTP requests to web content, against 42.5% from humans, per Cloudflare Radar, the first machine majority the company has recorded. The driver is agentic AI: a single agent performing a shopping task can visit a thousand times more sites than a person would. Those numbers describe the web I am actually shipping into: we build measurement and attribution infrastructure for the agent economy, working directly with the merchants and agencies trying to figure out how to make their businesses legible to LLMs and agents. From that vantage point, almost every assumption the internet was architected around is quietly breaking.
The assumptions that are breaking
Rate limits, sessions, identity, and analytics were all built around a single assumption: a human at a browser, leaving a consistent footprint as they moved through a site. That assumption no longer holds, and it doesn’t fail in one direction. It fails in many.
When we tested how every major AI system actually interacts with websites, we found at least six distinct architectures in production: raw HTTP fetches with no JavaScript or analytics, headless browsers running from datacenter IPs, browser extensions like Claude in Chrome running inside the user’s own session with their cookies and auth state, screenshot-based agents like ChatGPT Agent Mode operating remote Chrome on Azure VMs, OS-level computer use that controls applications beyond the browser, and emerging protocols like WebMCP that bypass the page interface altogether. Some are visible in server logs but not analytics. Some are visible in analytics but indistinguishable from humans. Some are invisible everywhere. The same buyer might use three of them in a single purchase, producing three different footprints across one journey. Identity is similarly fragmented: an agent in a browser extension carries the user’s real cookies and auth, while an agent on a remote VM carries none of it.
What actually works in production
Before OpenLens, I was on the Fundamental AI Research team at AWS AI Labs, and before that a research technician at Caltech. From that vantage point, the thing that strikes me most about the current public conversation on agent reliability is that it treats agents as one category. They are not. Agents that touch customers and agents that don’t are completely different problems with completely different playbooks. Conflating them leads to bad decisions on both.
Internal agents, the ones your engineers and operators use to ship faster, draft work, parse logs, summarize meetings, are pure productivity. The output gets reviewed by a human who owns the result. The most counterintuitive thing I’d tell engineering leaders right now is to point AI at your hardest, most critical work, not your easiest. Most people underestimate what current models can already do, and silo them off on tasks they don’t actually need to verify. That is exactly backwards. The bottleneck is no longer model capability. It is human discipline about verifying output before it ships. Teams that delegate trivial tasks and verify nothing produce slop. Teams that take swings at hard problems and rigorously check the output produce leverage.
Customer-facing agents are a different category entirely. The human ownership chain breaks the moment the agent is the interface. That is its own engineering discipline with its own playbook, and most of the public conversation muddles the two. The leaders worth listening to are precise about which problem they are actually solving.
Building on rented land
Building on top of a frontier model is a bit like building on top of someone else’s rented land while they renovate the house. Models update every few weeks. Pricing moves. Sometimes specific behavior on a specific prompt changes without warning.
Two examples from our own stack. We use Haiku to parse LLM responses for brand mentions. The prompt is written to describe what we want clearly, not to exploit quirks of a specific model version. As Haiku improves, the parser improves with it. We do not have to touch the code.
Last year, on an earlier version of our stack, we hard coded the API name for Claude 3.5 Sonnet in roughly ten different places. When that model was deprecated mid-deployment, the failures cascaded through a pipeline that touched users. We rebuilt the abstraction layer afterwards.
The lesson is to write to capability, not to versions. Code that depends on what a specific model does today breaks when that model changes. Code that describes what you actually want gets better as the models do.
What I think this all points to
The next generation of products, consumer and B2B, will be judged on how gracefully they handle agent traffic, not just human traffic. Not in some abstract “AI-ready” sense but concretely: can agents discover your product, authenticate with bounded scope, perform the action the user actually delegated, leave a clean audit trail, and do all of that without degrading the experience for the humans who still show up. Most companies I talk to have not started thinking about this, but the ones that have are going to compound an advantage that is very hard to catch up on later.
The post What Building AI Agents Inside a Production Startup Taught Me About the Future of the Web appeared first on SD Times.
Tech Developers
No comments