Supercharging AI

Nvidia has utterly dominated the market for AI chips by repurposing semiconductors originally designed for video games. Now GROQ, a tiny eight-year-old startup, is taking on one of the world's most valuable companies with a purpose-built chip designed for AI from scratch.

- Richard Nieva

Jonathan Ross' first inkling that something was wrong came back in February while he was speaking to a host of Norwegian Parliament members and tech execs in Oslo. Ross, the 42-year-old CEO of AI chip startup Groq, was in the middle of a demo he hoped would revitalize the languishing company: an AI chatbot that could answer questions almost instantaneously, faster than a human can read. But it was lagging slightly. It unnerved Ross, who was pitching a Groqpowered European data center that would showcase the specialized chips responsible for those superfast answers. "I just kept checking the numbers," he recalls. "People didn't know why I was so distracted."

The culprit was an influx of new users. A day before Ross's Oslo meeting, a viral tweet from a tech founder raving about "a lightning-fast AI answer engine" sent tons of new traffic to the online demo, buckling the company's servers. It was a problem, but a good one to have.

When he cofounded Groq eight years ago, Ross' idea was to design AI chips explicitly for what's known in the industry as "inference": the part of artificial intelligence that mimics human reasoning by applying what it has learned to new situations. It's what enables your smartphone to identify your dog as a corgi in a photo it's never seen before, or an image generator to imagine Pope Francis in a Balenciaga coat. It's quite different than Al's other computational suck: training the massive models to begin with.

But until OpenAI released ChatGPT in late 2022, touching off a global AI frenzy, the demand for superfast inference was limited, and the company was limping along. "Groq nearly died many times," Ross says from inside the startup's semiconductor lab in San Jose, California, recalling one low point in 2019 where the startup was a month away from running out of money. "We started Groq maybe a little bit early."