Tantra Analyst

The revelation in 2024 that Arm was planning to develop its own silicon sent chills down the spines of its licensees. However, its AGI CPU announcement allayed those fears and showed a pathway for the company’s ambition to become an AI silicon player without trampling its licensees.

AGI CPU is carving out a sizable $100 billion slice of the gigantic $1 trillion AI infrastructure silicon market, going head-to-head against its traditional rivals Intel and AMD. But more importantly, it’s accomplishing that without directly competing with behemoths such as Nvidia, hyperscalers, or Arm’s ecosystem partners. With Meta as the lead customer and collaborator, and support from more than 50 players, Arm positions itself as a formidable AI player.  

While the ecosystem benefits from AGI CPU in the short term, Arm’s future silicon ambitions will decide the former’s fate. 

AI infrastructure market dominated by GPUs, with room for CPUs

In the AI data center silicon market, Nvidia and its GPUs get all the limelight. However, the landscape is much more complex and nuanced, with many architectures, topologies, and players. At a very high level, it can be divided into GPU and AI accelerators, CPUs, and interconnects. For our discussion, only the first two are relevant.

By all accounts, GPU and AI accelerators account for the largest share of the market. Currently, the GPU market is dominated by Nvidia, with AMD playing second fiddle. The AI accelerators market is a mix of hyperscalers, including Google, AWS, Microsoft, and Meta, developing their own silicon, largely for their own use, and merchant silicon players such as Groq (owned by Nvidia), Cerebras, Tenstorrent, and others.

Before the AI boom, CPUs with x86 architecture were the staple for data centers, dominated by Intel. Even today, Intel holds a large share of the CPU market, followed by AMD. Arm architecture is gaining traction. Many hyperscalers have their own Arm-based CPUs, which they again primarily utilize for their own needs. AWS’s Graviton and Microsoft’s Cobalt are some good examples. Nvidia’s Vera CPU is also based on the Arm architecture. There are also some merchant players such as Ampere and Fujitsu.

But yesterday’s cloud data centers are rapidly transitioning to AI data centers. Initially, even now, the focus is on training, where GPUs and AI accelerators run the bulk of the workload. But the shift toward inference and the growth of agentic AI are making CPUs important again.

During the announcement keynote, Arm CEO Rene Haas presented an informative slide highlighting the role of CPUs in modern agentic data centers.

The agentic data center is where a user interacts with specialized AI agents. (Source: Tantra Analyst)

In cloud data centers, CPUs managed all the queries. In AI data centers, queries are answered by tokens generated by GPUs. But when we move to agentic AI, the system doesn’t just answer queries; agents perform complex, multi-step tasks based on those queries. Orchestration of all those agents and their tasks is performed primarily by CPUs.

Agentic AI is growing rapidly (e.g., massive popularity of OpenClaw). Arm projects that close to 120 million CPUs will be needed for a 1 GW AI data center. Given power constraints in data centers, more efficient CPUs will be in demand. Again, Arm projects the market at around $100 billion by 2030, a sizable portion of the overall AI data center silicon market that Arm aims to serve.

AGI CPU: a smart balancing act between customers, competitors, and licensees

Arm should be commended for carving out a sizable market without making most of its licensees utterly unhappy or competing with the giants of the AI infrastructure world. First, it is not competing with GPU giants Nvidia and AMD, or hyperscalers building their own AI accelerators. This is a stark contrast to other merchant silicon vendors trying to compete with GPUs using Arm-based AI accelerators.

Second, it is not directly competing with hyperscalers that make their own CPUs. From the outside, AGI CPUs might seem like direct competition to Google, AWS, and Microsoft’s own CPUs—but not so, when looking more closely. One of the main reasons these hyperscalers started developing their own CPUs is the inability of the x86 architecture to scale power efficiency and its slow evolution.

Additionally, these hyperscalers aren’t selling their CPUs to others. So, AGI CPU won’t directly compete with them in the marketplace. On the contrary, if AGI CPU performs as well as Arm claims, they might even consider using it. And just as Meta closely collaborated on the AGI CPU, they might consider collaborating with Arm to create CPUs optimized for their workloads.

Most of these assertions are apparent from the slew of endorsements Arm has received for this launch, to the point that Nvidia CEO and senior executives from AWS and Google spoke during the keynote. The only exception is Ampere, against which AGI CPU will directly compete.

Arm is butting heads directly with Intel and AMD, who, as mentioned before, provide the majority of merchant CPUs to data centers today. Despite its recent strides, x86 is still not as power efficient as Arm. Arm shared impressive power and performance comparison charts during the event. Looking at the history of these architectures, the results look plausible. Additionally, Intel’s current fab and other challenges will only make this easy for Arm.

What AGI CPU means for Arm licensees, now and in the future

AGI CPU has, for the time being, calmed the nerves of Arm licensees. To my written questions regarding the effect of AGI CPU on licensees, Arm replied, “Neutrality and ecosystem openness remain foundational for Arm. We continue to provide broad and equal access to architecture, IP, and CSS, and the addition of silicon is meant to expand optionality, not reduce it. This model gives partners and customers flexibility to choose what works best for them, whether that is building custom silicon using Arm IP and CSS, or deploying Arm-designed silicon. It expands access to Arm technology rather than restricting it and reinforces our role as a neutral platform provider.”

But a lot depends on how wide Arm plans to spread its silicon net. AGI CPU has a clear long-term roadmap. But Arm was very cryptic about what comes beyond that, saying only that there’s a lot more to come, with a possible addressable market of more than $1 trillion by 2030.

In the short term, for most licensees, AGI CPU is likely to be very positive. It gives full legitimacy to the architecture in the AI data center space and accelerates the maturity of software, tools, and infrastructure. Arm’s EVP of Cloud AI business, Mohamed Awad, stated that Arm will contribute many foundational platform elements, software, validation, tooling, and other components to the Open Compute Project, which will benefit all licensees.

But $1 trillion is a staggering target. That might indicate that Arm’s silicon ambitions are going far beyond this intelligently carved-out market segment and extend to AI accelerators, edge AI, and even smartphones and compute devices. That will hit at the heart of almost every licensee’s market. And Arm’s entry there will fundamentally change the dynamics—even the possibility of licensees adopting alternatives more aggressively, such as RISC-V.

Closing thoughts

The AI and AI infrastructure markets are still in their infancy. The projected opportunity is massive, and there’s room for enough players. Hence, currently, everyone is treating everyone else as frenemies. There are many unknowns about how this will evolve, such as the quantum of training vs. inference, the growth of agentic AI, artificial general intelligence (AGI), edge AI, new architectures and topologies, and more.

It seems today’s demarcation between GPUs, CPUs, accelerators, etc., will blur, and AI will run across the stack. What is certain is that power will be at a premium. All these processors will ultimately have to compete for the fixed rack power envelope. And processors that deliver the best performance per watt for specific AI workloads will win. The future is interesting and exciting, for sure.

If you want to read more articles like this and get an up-to-date analysis of the latest tech industry news, sign up for our monthly newsletter at TantraAnalyst.com/Newsletter, or listen to our Tantra’s Mantra podcast.