Elon Musk’s xAI Reportedly Working on a Supercomputer

Tech mogul Elon Musk told investors that his AI startup xAI plans to build a supercomputer to power the next generation of its AI chatbot Grok, The Information reports. Dubbed the Gigafactory of Compute, the supercomputer is expected to be finalized by fall 2025 and might be built in partnership with Oracle.

In his presentation for xAI’s investors, Elon Musk revealed that the new supercomputer will require 100,000 Nvidia H100 GPUs, making it four times larger than the largest existing GPU clusters. As a reference, training the Grok 2 model required about 20,000 Nvidia H100 GPUs. Nvidia’s H100 GPUs are in high demand in the AI data center chip market. They are so hard to obtain that even OpenAI’s CEO reportedly wants to start making his own.

It’s worth mentioning that these aren’t Nvidia’s latest generation GPUs. The company will soon ship H200 compute GPUs for AI and HPC app development. Furthermore, its Blackwell-based B100 and B200 GPUs are expected to ship later this year. Earlier this year, reports claimed that Nvidia is forming a new unit to enter the $30 billion custom chip market.

It is unclear why Musk intends to use older generation GPUs for xAI’s supercomputer, but the substantial cost and 2025 project deadline might play a role. According to The Information’s citations from Musk’s presentation to investors, he promised to hold himself “personally responsible for delivering the supercomputer on time.”

The Gigafactory of Compute supercomputer could be a game changer for the development of xAI’s large language model. After all, Musk funded the AI company to challenge Microsoft-backed OpenAI and Google’s DeepMind. He recently said he expected xAI to catch up with these AI leaders by the end of 2024.

Interestingly, Musk and the startup’s CEO, Sam Altman, were among OpenAI’s co-founders. Musk supported OpenAI with just under $45 million, promising to up his investment to $1 billion. The partnership never progressed due to Musk’s resignation from the board in 2018. Musk and Altman have been in an ongoing feud ever since, culminating with Musk filing a lawsuit against OpenAI this year.

Musk aims to create AI that surpasses human cognitive abilities. At VivaTech 2024 in Paris, he predicted AI could outperform humans in all tasks by the end of 2025, potentially linked to xAI’s supercomputer release. If Musk’s scenario plays out, human’s future role will be to “give AI meaning,” he suggested.