Eighty-three percent of large organizations believe their infrastructure needs improvement before deploying AI agents in production, according to a Google Cloud study based on responses from 1,402 technology leaders. The challenge isn’t just about buying more GPUs: integrating with legacy systems, inference costs, security, data management, and energy consumption are among the main obstacles.
The essentials of infrastructure for agentic AI in 30 seconds
- Only 17% fully trust their current infrastructure to run critical agents.
- Inference now accounts for 47% of AI budget, compared to 28% for training.
- 81% identify operational complexity as a hidden cost.
- Security, governance, and MLOps concern 79% of leaders.
- 84% plan to strongly invest in infrastructure and operations over the next two years.
The report reflects the shift from AI focused on generating responses to systems capable of planning, maintaining context, consulting tools, and executing actions. A request can trigger database searches, API calls, inventory checks, communications with other agents, and system modifications.
This ongoing activity requires a different infrastructure than what’s needed for testing a chatbot or training a model temporarily. Agents need memory, fast data access, observability, unique identities, permissions, and the ability to sustain processes over extended periods.
The research was completed in January 2026 by Google Cloud and GBK Collective. Participants included IT leaders from companies in 12 countries with over 1,000 employees, except in Asia-Pacific and Latin America, where the minimum was 500. All respondents used or planned to use generative AI within the next 12 months. The results describe large organizations already interested in the technology, not the entire business landscape.
Only one in six companies considers itself prepared
The preparedness chart on page 6 of the report divides organizations into five groups. 12% need fundamental changes, 29% must update core systems, and 27% believe minor integration and adjustments are required.
Another 16% could deploy pilot agents with little effort but still lack confidence in using them on core services. Only 17% feel ready for production agents involved in critical processes.
| Current Infrastructure Readiness | Percentage |
|---|---|
| Needs fundamental changes | 12% |
| Requires significant updates to core systems | 29% |
| Minor integration and adjustments needed | 27% |
| Can run pilots easily | 16% |
| Ready for critical production agents | 17% |
The sum exceeds 100% due to rounding, but Google Cloud sums up that 83% of organizations still need some improvements before full readiness.
The main technical gap isn’t accelerators. 43% report difficulty connecting agents with legacy APIs and data sources. 36% lack high-performance vector databases, and 35% find existing security measures insufficient for multi-system access.
| Main deficiencies in deployment | Percentage |
|---|---|
| Legacy API and data integration | 43% |
| Insufficient vector databases | 36% |
| Limited security for multi-system access | 35% |
These percentages are responses to multiple options—organizations can face all three issues, which is common given that agents often need to interact with varied applications built at different times and across environments.
The challenge lies in giving context without granting unrestricted access. An agent preparing an order might need to check sales, inventory, suppliers, and logistics, but shouldn’t have permissions to modify all systems equally.
Inference exceeds training in budget allocation
Ongoing use of models is shifting costs from training to inference. Inference now accounts for 47% of the AI budget, versus 28% for training, according to the report.
| AI Budget Distribution | Percentage |
|---|---|
| Inference | 47% |
| Training | 28% |
| Model optimization | 16% |
| Experimentation | 9% |
This is significant because agents produce multiple responses and operate continuously. They can hold lengthy conversations, use tools, review results, and re-query models—all steps that consume compute, memory, storage, and network resources.
Google Cloud describes this as an “inference tax”—costs that arise when scaling workloads on legacy platforms. The term is commercial, but data clarifies that 62% cite infrastructure and usage costs, including storage, data transfers, and underutilized accelerators.
The most cited expense, however, is operational complexity. 81% recognize the engineering work needed to integrate models, databases, tools, access controls, and monitoring platforms as a hidden cost.
| Hidden costs when scaling AI | Percentage |
|---|---|
| Operational complexity and engineering | 81% |
| Infrastructure and direct consumption | 62% |
| Talent and training | 57% |
Additionally, 96% believe economic efficiency is key when choosing infrastructure. 32% see it as extremely important, 49% as very important, and 15% as important.
This concern favors managed services, as providers handle provisioning, scaling, and maintenance. The report notes that advanced AI adopters are 50% more likely to use fully or primarily managed services.
Such reliance can also increase vendor dependence. 78% now buy generative AI solutions directly from their main cloud provider—up from 48% in 2025. Google interprets this as a desire for integrated, centralized governance, but it also raises issues about portability, exit costs, and platform switching.
Security, hybrid cloud, and energy are next on investment priority
Agents extend the attack surface by acting on behalf of individuals or business processes. 79% cite security, governance, and MLOps as major concerns at inference scale.
| Top challenges to scaling inference | Percentage |
|---|---|
| Security, governance, and MLOps | 79% |
| System alignment with business | 64% |
| Model performance and efficiency | 64% |
| Cost management | 46% |
| Security as a specific concern | 39% |
Within infrastructure security, AI supply chain security ranks first. It concerns models from third parties, open weights, libraries, training data, and components that may introduce vulnerabilities or unintended behaviors.
| Security concerns related to AI | Percentage |
|---|---|
| AI supply chain and MLOps | 48% |
| Data in multi-tenant environments | 41% |
| Unauthorized access or model theft | 39% |
The study also points to risks like indirect instruction injection and tool manipulation. An attacker could introduce malicious commands via pages, emails, or documents that an agent later processes.
The recommended response involves separating identities, minimum permissions, activity logs, and human approval for sensitive operations. The aim isn’t just to know what response the model produced, but to reconstruct what data it accessed, which tools it used, and what changes it made.
Regarding deployment, 52% now use a hybrid multicloud architecture—up from 41% in the previous year. 90% see value in running AI at the edge, with 72% rating it as very or extremely important.
| Architecture and deployment indicators | Percentage |
|---|---|
| Uses a hybrid multicloud strategy | 52% |
| Values AI at the edge | 90% |
| Considers edge very or extremely important | 72% |
| Prioritizes data residency controls | 48% |
| Considers an integrated platform a must-have | 69% |
Edge processing can reduce latency, ensure service continuity during connectivity outages, and limit queries sent to the cloud. It doesn’t fully replace data centers but allows task distribution based on specific needs.
Data residency also influences distribution. Nearly half (48%) prioritize infrastructures with controls over data location. Since agents may access personal, industrial, or financial data, jurisdiction impacts databases, records, and services involved in each operation.
Energy consumption is another concern. 91% consider energy use when selecting inference hardware, and 61% see it as a major or significant factor.
| Energy impact on hardware choices | Percentage |
|---|---|
| Energy use influences selection | 91% |
| Major or significant factor | 61% |
It’s not just about electricity costs. High-density equipment may require new power lines, liquid cooling, racks, and data center modifications. In some regions, the available grid capacity directly limits the installation of new accelerators.
Therefore, investments are mainly focused on the technological foundation. 84% plan to allocate significant resources to infrastructure and operations over the next 12-24 months, compared to 67% for data and 55% for security.
| Planned investments in next 12-24 months | Percentage |
|---|---|
| Core infrastructure and operations | 84% |
| Data | 67% |
| Security | 55% |
Results show that deploying agents isn’t just about more powerful models. Companies need to modernize interfaces, consolidate data, monitor costs, and define permissions before handing over real actions to autonomous systems.
The report, produced by Google Cloud, emphasizes integrated platforms, managed services, and its own infrastructure. Despite the commercial angle, the data reveal a clear tension: many organizations have advanced in AI faster than they’ve prepared the systems to support it.
Frequently Asked Questions
What is an agentic AI system?
An application capable of planning tasks, maintaining context, consulting tools, and executing actions in multiple steps. It can operate with some autonomy, though sensitive operations should be overseen by humans.
Why isn’t adding more GPUs enough?
Most issues relate to data integration, legacy APIs, security, permissions, and operational complexity. Accelerators improve computation but don’t resolve these core limitations alone.
What percentage of companies are fully ready?
Only 17% are fully confident in their infrastructure to run critical agents in production.
Where will organizations invest most?
84% plan to significantly invest in core infrastructure and operations over the next year or two. Data and security come after, at 67% and 55% respectively.
Source: Google Report State of infrastructure in the agentic AI era 2026
