The rise of generative artificial intelligence (GenAI) is transforming the tech industry at an unprecedented pace, while also pushing data centers to their limits. Sky-high energy costs, scalability issues, and technical constraints are some of the challenges already shaping the present—and influencing the future—of cloud computing.
When ChatGPT was launched in November 2022, few people imagined the global impact it would have just two and a half years later. The widespread adoption of applications based on generative language models, combined with the explosive growth of users—hundreds of millions worldwide—has compelled giants like Meta, Google, and Microsoft to significantly ramp up their infrastructure investments.
By 2025, these three companies alone are expected to spend over $200 billion on data centers, while OpenAI’s Stargate project plans to invest $500 billion over four years to deploy new facilities. These nearly government-scale figures are necessary to support a technology that continues to grow in complexity and demand.
The business model of generative AI relies heavily on a key factor: the ability to train and deploy large language models (LLMs) and serve them in real-time to millions of users. This need has fundamentally changed the rules of design, operation, and economics in data center management.
In the AI industry, everything is measured in tokens—the smallest output unit of a generative model. Each query to a model, every phrase produced, and each image generated consumes computing power and energy, which, multiplied by billions of interactions, results in astronomical operating costs.
Although providers are rapidly expanding their capacity, OpenAI continues to restrict usage of its models, both via API and the free ChatGPT version. The reason is that available computational resources aren’t sufficient to meet demand without jeopardizing system stability.
This presents inference operators—the ones responsible for delivering results to end users—with a classic dilemma: subsidize usage to promote adoption or shift costs to customers from the start, thereby slowing growth. In either case, profit margins are shrinking, and the business model becomes more uncertain.
One major issue is energy consumption. According to SemiAnalysis, by 2030, AI data centers could account for 4.5% of global electricity generation.
The numbers are alarming:
– A next-generation Nvidia GPU could draw up to 1,800 kW, four times more than the A100.
– Current AI racks, like those with GB200 chips, already surpass 100 kW per rack, over five times the traditional cloud standard.
– Nvidia’s Rubin Ultra roadmap targets racks over 500,000 W, approaching the scale of small power plants.
This forces a reassessment of data center design from the ground up. Some are already built near power generation sources to reduce transmission losses, and liquid cooling is quickly replacing air cooling systems. For example, Meta is developing Hyperion, a cluster capable of scaling up to 5 GW of power.
This issue extends beyond the tech industry, as local power grids begin to feel the strain of this colossal demand. Governments worldwide are exploring ways to balance energy access between data centers and the rest of society.
While model training often takes center stage, the real bottleneck is inference—providing fast, reliable responses to users. Generative models are highly dependent on memory, and traditional GPUs are not optimized for these workloads, resulting in high latencies. Generating an image with GPT-4, for instance, can sometimes take over a minute.
To address this, data centers need inference-optimized accelerators and more efficient architectures. Without these, user experience suffers, and the perceived value of these tools diminishes.
Training and deploying AI models at scale is unlike any other technological challenge. Today, some clusters already feature over 100,000 interconnected GPUs, with leading providers working on systems exceeding 300,000 GPUs across multiple campuses.
The scale creates unprecedented issues in orchestration and management: maintaining low latency, ensuring reliability, and maximizing hardware efficiency are as complex as building the physical infrastructure itself. Software plays a critical role here, requiring advanced scheduling, load balancing systems, and ultra-fast interconnections to unify these clusters into cohesive systems.
Reimagining the entire tech stack is essential. This includes:
– Data center level: optimizing power delivery, liquid cooling, and new physical designs.
– Compute platform level: developing accelerators optimized for inference, not just training.
– Software level: creating compilers, runtimes, and orchestrators designed for massive AI workloads.
– Model level: designing lighter, more efficient architectures that maintain accuracy without excessive power consumption.
As D-Matrix emphasizes in its analysis, the path forward involves co-developing hardware and software from first principles. Simply adding more GPUs is no longer enough; all components in the chain must be redesigned to operate synergistically.
Generative AI promises to revolutionize entire industries—from education to biomedicine—but its uncontrolled expansion risks becoming unsustainable from both an energy and economic perspective.
The next decade will be decisive. If the industry manages to innovate in efficiency—through new architectures, optical interconnects, and smarter orchestration strategies—data centers can support the surging demand. Otherwise, AI growth may hit a wall of costs, energy constraints, and reliability issues.
As Aseem Bathla, CEO of D-Matrix, warns:
“The key isn’t building endless numbers of data centers; it’s building them better—with infrastructures truly optimized for generative AI.”