As OpenAI, Google, and other tech giants chase ever-larger models, with each claiming a new benchmark score record every month, enterprises face a quieter but far more practical challenge: inference.
As OpenAI, Google, and other tech giants chase ever-larger models, with each claiming a new benchmark score record every month, enterprises face a quieter but far more practical challenge: inference.
The process of running trained AI models to analyze new data and generate answers might sound simple in theory, but at scale, it’s where most companies stumble. GPUs, originally engineered for graphics rendering, excel at raw computation yet falter under the weight of millions of real-time queries. This leads to soaring costs, latency issues, and massive energy demands.
IBM, which has often positioned itself as the architect of enterprise computing, is stepping into that gap. Instead of chasing larger models, the company is positioning itself as the AI enabler, the connective layer that turns intelligence into execution. Its latest ecosystem bet focuses on the invisible but critical foundation of modern AI: inference infrastructure.
Through new partnerships with Anthropic and Groq, the California-based startup known for its Language Processing Units (LPUs), IBM aims to reimagine how enterprise AI operates in production environments.
“Data is everywhere, multiple clouds, edge, on-premises, and enterprise AI must be able to work across hybrid environments. We have a layered model strategy, balancing IBM-built innovation with strategic partnerships to accelerate outcomes,” Rob Thomas, SVP and chief commercial officer at IBM, told me. “We take advantage of various models, small language models like Granite, large language models from our partners like Mistral and Meta, and frontier models through our partnership with Anthropic, and use the best model for each use case.”
With Groq’s inference hardware now integrated into IBM’s watsonx Orchestrate, the company claims enterprises can run agentic AI systems up to five times faster and more cost-efficiently than traditional GPU-based setups.
“AI is still stuck in the ‘dial‑up’ era – models can give accurate answers but to give high-quality research-grade answers can mean waiting up to 10 minutes while an LLM or agent goes off and thinks,” said Jonathan Ross, CEO and founder of Groq. “Faster processing also drives up usage and thus compute costs, so speed must be coupled with cost efficiency.”
Traditional GPUs, Ross explained, excel at parallel, batch-oriented workloads such as model training. But when it comes to low-latency, multi-step reasoning, the kind of dynamic execution required for agentic AI, GPUs falter. LPUs use a software-controlled, assembly-line architecture, moving data in a deterministic flow to eliminate bottlenecks common in GPUs and deliver real-time AI performance.
“Agentic AI improves LLM outputs by decomposing a task into a series of explicit steps and executing each step sequentially. This “think‑in‑steps” approach yields better results, though it also multiplies the compute required, driving up both latency and cost,” he told me. “GPUs are the right tool for training, or the creation of AI models. LPUs are the right tool for inference, or the running of AI models.”
