The Internet is Adapting for Machine Utilization

Cloud infrastructure has historically been designed for human users who navigate the internet through predictable actions like searching, clicking, scrolling, and streaming. In contrast, AI agents function differently, capable of generating substantial activity by creating multiple sub-agents that can quickly query various databases, sift through documents, and call APIs in mere seconds, only to disappear just as rapidly.

In response to this shift, Amazon is overhauling a core component of its cloud infrastructure. AWS recently launched its next-generation OpenSearch Serverless, a fully managed search and vector database — effectively a scalable system for information storage and retrieval — specifically engineered for workloads driven by agents. AWS asserts that this new platform can instantly scale up when agents initiate tasks and reduce to zero during periods of inactivity.

This release highlights a growing recognition in the tech industry: infrastructure that was originally developed for a human-centric internet is increasingly ineffective in an era where agent-driven interactions are on the rise.

Although AI agents currently account for a relatively small portion of internet traffic, machine-generated activity is already substantial and expected to grow. Cloudflare notes that bots made up 31% of all HTTP traffic over the past six months, with AI crawlers, search engines, and assistants making up about a quarter of all bot requests during that period.

“Non-human traffic is expected to outpace human traffic by the first half of 2027,” stated Li Yi Ohlsen, senior product manager at Cloudflare, during an interview with TechCrunch.

At Google’s I/O developer conference last week, the company revealed that users would soon be able to delegate tasks to AI systems, such as conducting product research, booking travel, browsing the web, and interacting with various applications. Yet, advancements aren’t limited to consumer AI agents; businesses are increasingly implementing agents for internal processes and client interactions, resulting in new layers of machine-generated traffic in the background.

Consequently, cloud providers and infrastructure companies are faced with the challenge of adapting systems originally designed for human use to accommodate a landscape where agents continuously and autonomously retrieve data, utilize tools, and generate machine-to-machine traffic.

This is where AWS’s new OpenSearch Serverless becomes critically important.

“The timing is evident. Agents are moving from experimental phases to production, generating traffic patterns that existing infrastructures are not prepared to handle,” Tia White, general manager for Amazon OpenSearch Service, informed TechCrunch. “Their activity levels can surge unexpectedly and then become idle suddenly, requiring a search solution capable of adapting without incurring costs for unused compute resources.”

One major technical leap in this new version is the decoupling of compute from storage, allowing compute resources to scale up in mere seconds to accommodate spikes in agent traffic and scale down to zero during idle times, ensuring that customers incur no costs during these periods.

“In our previous Serverless version, at least one operational instance was required due to the coupling of storage and compute resources,” White explained. “You couldn’t scale compute automatically at the necessary pace, leading to reserved idle compute resources whether they were being used or not.”

It’s similar to always paying for a parking space even when it’s not in use. With AWS’s enhanced Serverless version, it’s more akin to paying for a metered parking spot.

Upon launch, OpenSearch Serverless will seamlessly integrate with AI development platforms such as Vercel and Kiro, allowing developers to deploy production-ready search and vector backends for agents without the need to manage underlying infrastructure.

This shift is becoming increasingly common in the cloud industry. Databricks and Snowflake are rebranding themselves as AI memory and retrieval systems tailored for enterprise data. Microsoft has rolled out enhancements to Azure aimed at managing surges in AI agent activity and facilitating memory sharing among agents. Similarly, Cloudflare introduced a new infrastructure last month designed to provide agents with consistent environments and immediate scalability.

As more organizations adopt AI agents, the demand to redesign infrastructure around machine-generated workloads will likely increase, making the deployment of agents more cost-effective and simpler at larger scales.