AMD acquires AI chip startup Taalas to boost inference performance by etching models into silicon
The Hot Take: That sounds alot like an ASIC...
In AMDâs latest bid to upset Nvidia's dominance in AI hardware, the House of Zen has acquired AI chip company Taalas, which bakes model weights directly into silicon in a process that promises to boost inference performance by an order of magnitude or more. The deal, announced at market close on Thursday, appears to be framed in much the same context as Nvidiaâs $20 billion licensing deal with Groq last December: make high-performance âpremiumâ inference services prized for AI agents, like code assistants, faster and cheaper to run. AMD didnât disclose the terms of the deal, but from what we understand, this is an actual acquisition rather than an acquihire. Founded in 2023 and based in Toronto, Taalasâ approach to inference is radically different from conventional GPUs or the dataflow architectures that underpin Groq LPUs or Cerebras' waferscale accelerators. A model-specific integrated circuit The startupâs chips donât rely on HBM to store the model weights but rather etch them directly into the silicon. In a sense, Taalasâ chips are really model-specific integrated circuits or MSICs. Perhaps more importantly, Taalasâ tech isnât just conceptual. In February, the startup revealed its first test chip fabbed on TSMCâs 6nm process tech, which it called the HC1. Initial benchmarks saw the chip serve Metaâs Llama 3.1 8B at a blistering 16,960 tokens a second â when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. While Llama 3.1 is ancient by todayâs standards, having made its debut all the way back in mid 2024, the reticle-sized chip was really intended to prove the concept. Taalas has been incredibly secretive about how its chips actually work, but we know its processors are comprised of two main regions: the mask-ROM recall fabric where model weights are etched, and the SRAM recall fabric where KV caches and fine-tuning adapters are stored. For its second-gen HC2 chip due out this summer, Taalas aims to boost parameter count to 20 billion parameters. That might not sound like much, but just like with GPUs for larger models, weights are simply distributed across multiple accelerators using pipeline parallelism. At 20 billion parameters per chip, youâd need just 50 accelerators to support a trillion-parameter model, and AMD just so happens to have a rack-scale compute platform and in-house system design team that can comfortably accommodate that. Thatâs quite a bit more space and power efficient than Nvidiaâs recently unveiled LPX systems, which would need a few dozen GPUs and at least 2,000 Groq LPUs to serve the same model. From what we understand, AMD intends to pair its Instinct-based Helios racks with chips based on Taalasâ tech, which implies a disaggregated architecture where compute-heavy prompt processing is done on GPUs while token generation is offloaded to Taalas-based accelerators. Itâs also possible that AMD could adopt a sort of tick-tock cadence in which customers initially deploy and validate models on Instinct accelerators and, once theyâre satisfied with them, transition to Taalas accelerators. We can only speculate at this point, but hereâs what AMDâs SVP of AI, Vamsi Boppana, had to say about it in a canned statement: âAMD is building a full-stack AI platform that gives customers the flexibility to deploy the right compute solutions for every AI workload." You better really love that model While the tech is blazing fast, if you hadnât already figured it out, it comes with a pretty substantial downside. Once the chips are deployed youâre stuck with that model. Any change bigger than something like a LoRA adapter is going to require a re-spin of the chips, which is not only expensive but time-consuming. Nearly four years into the AI boom, new models are rolling out on a nearly monthly basis. In order to benefit from Taalasâ tech, AMDâs customers are going to have to be really sure about their choice of models, which will be easier for some than others. However, if the startup is to be believed, the situation isnât quite as bad as it sounds. While new models will require a re-spin, it doesnât require starting over from scratch. Instead, just two layers of metal need to be changed, which is a lot cheaper and less time-consuming. With that said, we strongly suspect this tech will largely be deployed by AI model devs, their infrastructure providers, and a handful of inference providers. In an interview with our sibling site The Next Platform in February, the company suggested that etching a model's weights into silicon is 100x less expensive than training a frontier model. AMD is certainly in a position to negotiate those deals. OpenAI, Anthropic, and Meta are all major Instinct customers. Given the close working relationship between the model houses and the chip designer, it wouldn't be surprising to see a GPT or Claude deployed on a combination of Taalas and instinct accelerators. The tech also has implications for model development. One of the ways developers have cut down on hallucinations is by trading time for accuracy. The technique, called test-time scaling, is quite simple in practice, and involves allowing a model to âthinkâ for longer before responding. One drawback of test-time scaling is that it consumes substantially more tokens, which makes it expensive, and means users have to wait longer for the chatbot, code assistant, or agent to respond. If AMDâs Taalas buy can drive down the cost per token and boost output speeds by 10x or 20x, model devs may opt to extend the reasoning time even further. In any case, we may not have to wait long to see just how Taalas fits into AMDâs broader vision. Subject to regulatory approval, the deal is expected to close in the fourth quarter. ÂŽ
Read the full article