Megatrend · Artificial Intelligence

A model is "trained" once, but "run" a billion times — and this is the chip built for those billion runs

The AI chip the world knows (NVIDIA's GPU) is built to be great at "training" a model. But training happens only a few times. Running the model to actually answer real questions (inference) happens every second, millions and billions of times. Once the volume of running dwarfs training, the cost per answer (cost per token) and the speed (tokens per second) become the new battleground — and that opens the door to a new breed of chip that does "only the running," faster and cheaper than a general GPU. This lesson introduces Groq, Cerebras, d-Matrix, and the reason most of the real players are still private companies.

Category Artificial Intelligence Level Specific topic Maturity Scaling Read time ~13 min

A giant factory carefully casts a single chip statue, just one. That statue is then copied into countless tiny shadows that flow out into the hands of people all over the world, conveying training a model once and then running it again on a massive scale. — **Built once, used without end.** An AI model is trained just once, but it gets "run" to answer real questions billions of times — and those billions of runs are this chip group's arena.

01What it is — a chip made to "run," not to "train"

Picture building a dictionary. Writing and editing the dictionary takes years and a huge team — but you do it only once. After that, what happens every day is people looking up words, millions of times. The world of AI is exactly the same. "Training" a model is writing the dictionary; "running" it (inference) is looking up a word — and this node is about the chips designed specifically for "looking up words," not for writing.

On the megatrend map, this node is a leaf under AI Compute & Accelerator Silicon, inside the larger trend Artificial Intelligence. Its definition is straightforward: "an architecture designed for low-latency, high-throughput model execution." In plain language — a chip that makes AI answer fast and answer well at the lowest possible cost per answer.

Why does this job need its own chip? Because "running" and "training" are two different jobs. Training has to compute backward and adjust trillions of weights — it needs flexibility and enormous raw power. Running is a one-way forward computation, repeating the same thing, predictable — and a job where you "know in advance what you'll do" can be handled by a purpose-built chip far faster and more power-efficiently than a general-purpose one. That's why new breeds of chip like Groq's LPU or Cerebras's wafer-scale machines came to be.

Key terms

Training · Inference · Token · Latency

Training = teaching a model with massive data; happens once / rarely · Inference (running) = taking the trained model and answering real questions; happens all the time · Token = the small unit of text (roughly a word or word-piece) the model produces one piece at a time; one paragraph of an answer can be hundreds of tokens · Latency = the wait before you get an answer — the lower it is, the more "instant" it feels

02Why it matters — when "running" gets bigger than "training"

What changes everything is that the share of compute is flipping. In 2023, training a model ate about two-thirds of compute and running took just one-third. But once AI left the lab and reached real use by millions, everything inverted — by 2025 running rose to about half, and by 2026 it's expected to reach two-thirds of all AI compute. Many shops see running eating 70–80% by 2028–2030.

The weight of compute is shifting from "train" to "run"

Share of all AI compute that is inference (running) — 2028 is an estimate

Source: Deloitte TMT Predictions 2026 (inference ~2/3 of compute in 2026), Futurum/Silicon Analysts — 2028 is the midpoint of the estimate range

Once running becomes the big chunk, the market for chips that do this work balloons too. Different shops size the inference-chip market differently depending on what they count, but the direction agrees: Deloitte sees chips designed for inference alone breaking $50 billion in 2026, while the broad inference-chip market runs around $105 billion in 2025, with many shops seeing it grow to $300–570 billion in the early 2030s at roughly 25–28% a year.

~2 of every 3 units of AI compute in 2026 will be "running" (inference) — a flip from just 1 in 3 back in 2023. That's why chips built specifically for running have become the new field everyone wants into.

More important than the size is that its economics are completely different from training. With training, you pay a lot once to get a good model. But with running, you pay again every time someone asks. So if you can cut the cost per answer even a little, multiplied by a billion times it becomes enormous money — and that's the entire reason this chip group exists: not to be good at everything, but to do one job fastest and cheapest per token.

03How it works — train once, then run without end

The heart of this node is the asymmetry between training and running. Let's trace the life of a single model to see why "running" becomes the big cost that decides everything.

The asymmetry that changes everything. Train once, but run without end — the cost over a model's lifetime concentrates in "running," which makes speed and price per token the deciding factor.

So how do inference chips make it faster? The key is memory. When you run a model, the multiplier isn't the bottleneck — it's moving the model's weights out of memory and into the chip that's slow. A typical GPU keeps the weights in HBM memory that sits "off-chip," sending data at roughly 8 TB/s, while Groq chose to keep the weights right on the chip in SRAM, which runs at 80 TB/s and up — more than ten times faster. The result: the chip doesn't have to wait for data, so it fires out tokens far more rapidly.

Key terms

SRAM vs HBM · Deterministic execution

HBM = high-bandwidth memory placed beside the chip; holds a lot but is slower to reach (see HBM & AI Memory) · SRAM = memory embedded right on the chip; very fast but holds little — some inference chips trade capacity for speed · Deterministic execution = scheduling every operation in advance, clock cycle by clock cycle, at compile time, so there's no random "stutter" — Groq uses this to squeeze out maximum speed and certainty

04How it differs from GPU and ASIC — and what it connects to

This node has siblings in the same family you need to tell apart, because they can all "run" a model — but their philosophies differ:

Versus GPU & Merchant Accelerators: a GPU is a general-purpose chip that can both train and run. Its strength is flexibility and the software (CUDA) the world's developers know. But that flexibility carries overhead you don't need when running — a specialized inference chip cuts that overhead away in exchange for better speed/price per token on pure running
Versus Custom Silicon / ASIC: the line is very thin — a specialized inference chip is itself a kind of ASIC. The difference is "who builds it and for what." Node 10010200 centers on ASICs the tech giants build themselves to use in their own cloud (like Google's TPU), while this node centers on inference chips that independent chip companies build to sell or serve to everyone (Groq, Cerebras, d-Matrix)
Relies on HBM & AI Memory and Foundry & Contract Fabrication: whether they pick SRAM or HBM, these chips still have to be built with the most advanced process and advanced packaging, just like every other AI chip

Looking beyond the chip family, this node is the "engine" that ignites other trends:

Powers Agentic AI directly: an AI agent that works in steps calls the model many times per task — each call is a new run. The more agents spread, the more inference demand multiplies. These are the customers hungriest for speed and the lowest cost
A partner to Semiconductors (Logic): an inference chip is a logic chip designed specifically for AI work — view this node through the "AI-demand lens," while Logic looks through the lens of the whole industry's chip-manufacturing technology. Two views of the same thing
Draws power from Energy Transition & Power Demand and relies on Cloud & Digital Infrastructure: running that happens every second draws power continuously (unlike training, which finishes in rounds), so efficiency per watt is the main selling point of this chip group. And it "enables" both Robotics and Biotech, which need AI to answer fast

Perspective An easy way to remember it: GPU = the "Swiss Army knife" (can train and run) · the tech giants' ASIC = "a knife sharpened for its own kitchen" · the independent inference chip = "a knife sharpened for slicing only, then sold to every kitchen." They all cut, but the more specialized it is, the faster and cheaper it cuts that one job — in exchange for not being able to do anything else.

05Where it stands now

This field is hot right now, with two layers of wonder. The first layer is speed numbers that tear up the chart. Specialized chips run an LLM literally orders of magnitude faster than a single GPU — Groq does about 300–750 tokens per second versus a typical GPU's roughly 100, while Cerebras on a wafer-scale machine does over 2,500 tokens per second per user on a large Llama model. And d-Matrix claims its platform runs faster, costs less, and uses several times less power than a GPU system.

Specialized chips fire out tokens many times faster than a single GPU

Approximate LLM running speed (tokens per second per user) — depends on the model and the setup

Source: Groq (300–750 tok/s), Cerebras (~2,522 tok/s on Llama 4 Maverick), SemiAnalysis — figures vary by model/setup

The second layer is that most of the real players are still private companies — Groq, SambaNova, and d-Matrix aren't yet listed on the stock market, so retail investors can't buy them directly. d-Matrix just raised $275 million in a Series C at the end of 2025 at a $2 billion valuation, while Cerebras, the wafer-scale pioneer, is among the first in the group to step into the stock market, with 2025 revenue of about $510 million, up 76%, backed by a compute-purchase contract worth over $20 billion from OpenAI.

On the other side, incumbents like NVIDIA and AMD aren't just standing by. Both have shipped chips/versions tuned specifically for inference — NVIDIA unveiled the Rubin CPX, designed for large long-context runs (expected late 2026), while AMD pushes its Instinct family (MI350/MI400) as the cost-per-compute option and landed OpenAI as a strategic partner. So the inference field isn't just "startups vs. giants" — it's a competition along the axes of speed–cost–flexibility that everyone has joined.

A single square chip fires out a stream of tiny text grains as a strong, rapid jet, in contrast to the chip beside it that releases them one drop at a time, conveying a specialized inference chip producing tokens many times faster than a typical GPU. — **It's measured by speed per answer.** A chip designed specifically for running fires out tokens many times faster than a single GPU — and in a job repeated a billion times, that speed is money.

Key players in this field

Cerebras SystemsCBRS · US

United States · wafer-scale pioneer

Builds an AI chip the size of an entire wafer (the Wafer-Scale Engine), keeping the model's weights on the whole chip, so it runs an LLM at ~2,500 tokens/sec. 2025 revenue ~$510 million, up 76%, backed by a compute contract worth over $20 billion from OpenAI — and among the first in the group to go public.

core · speed leader

NVIDIANVDA · US

United States · incumbent that came down to play

Known as the king of training chips, NVIDIA already holds most of the inference work and ships specially tuned chips like the Rubin CPX for large long-context runs (expected late 2026) — the reason the inference field isn't the incumbent's "weak spot" that many assume.

core · incumbent

AMDAMD · US

United States · cost challenger

Pushes its Instinct family (MI350/MI400) as the option focused on total cost per compute for running work, and landed OpenAI as a strategic partner — the price pressure that makes the inference field compete on cost, not just speed.

core · cost competitor

Groqprivate

United States · private

The maker of the LPU (Language Processing Unit), which keeps the model's weights in on-chip SRAM (bandwidth of 80 TB/s and up) and schedules every operation in advance, clock cycle by clock cycle (deterministic), to squeeze out maximum speed and certainty — runs an LLM at about 300–750 tokens/sec. Still a private company.

core · low-latency leader

SambaNova Systemsprivate

United States · private

Designs a dataflow architecture tuned to run large models efficiently, sold as a full system to enterprises and governments — one of the specialized inference challengers still a private company.

core · full-system

d-Matrixprivate

United States · private

Uses in-memory compute (computing inside memory) on its Corsair chip to break the "memory wall" — claims speed and energy efficiency several times better than a GPU system. Just raised $275 million in a Series C (end of 2025) at a $2 billion valuation. Still private.

core · in-memory compute

06The future — the era of models that "think a lot"

The biggest tailwind for this node is reasoning models. The new generation of models doesn't answer right away — it "thinks to itself" in steps before answering, and that internal thinking is the production of a huge number of tokens the user never sees. The result: a single question can eat 5–10 times the compute in the running stage of a previous-generation model (and heavy jobs can hit 70–100 times in energy terms). So inference demand grows not just with the number of users, but with the surging "tokens per question" too.

What looks contradictory but actually reinforces it is that the price per token is dropping very fast. The cost of running a GPT-3.5-class model fell more than 280-fold from late 2022 to late 2024 — but the cheaper it gets, the more people use it and the more the models think, so total usage grows faster than the price falls. This is the "AI cost paradox" that keeps AI companies' compute bills ballooning even as the unit price drops — and the reason everyone is chasing the chip that runs cheapest.

Price per token plunges, but usage surges even faster

Approximate index (base = 100 in 2023) — price per token for a same-capability model vs. the volume of tokens used across the whole system

Source: Epoch AI & arXiv (inference price falling ~200x/year on average after early 2024), Deloitte — a composite index to convey the direction

The second direction is that running will spread out. Not every job has to run in a giant data center — some inference is starting to shift closer to the user (see Edge & On-device AI Silicon) for low latency and privacy, opening room for especially power-efficient architectures to play where the big GPU can't reach.

A brain-shaped lump of thought whose interior is filled with long, tangled, winding paths before finally arriving at a single short one-sentence answer at the end, conveying a model that thinks in very long steps before answering. — **The more it thinks, the more tokens it eats.** A reasoning model produces a huge amount of "internal thinking" tokens before answering — so running demand grows with intelligence, not just with the number of users.

07Challenges & risks

The first risk is that the incumbents have come down into the running field too. Many people think inference is NVIDIA's "weak spot," but the truth is NVIDIA already holds most of the inference work and ships specially tuned chips (like the Rubin CPX) to defend its turf, while AMD chases with an attractive total cost — so an inference startup has to be fast and cheap enough to be worth "leaving the familiar software ecosystem," a wall that's harder to cross than the speed numbers on a slide.

The second risk is that most of the standout players are still private companies. Groq, SambaNova, and d-Matrix aren't on the stock market, so ordinary investors find it hard to reach them directly. And as young companies still burning cash to build a customer base, their road to profit is still to be proven — some may be acquired, some may not survive. This is a field where "great technology" doesn't guarantee "the business survives."

The third risk is dependence on a handful of customers, and the question of cycles. The big demand for inference chips comes from a few AI companies and cloud giants that pour in capital all at once (like the giant contract between Cerebras and OpenAI). If returns from AI come slower than expected or investment slows, demand that looks bottomless could shrink fast. And because the customers are concentrated, losing a single big deal can shake the whole company.

Bottom line for investors Inference-Optimized Silicon is a bet that "when AI is used for real all over the world, running will be many times bigger than training, and will reward whoever runs fastest and cheapest per token." Three keys: (1) models that "think a lot" push running demand to grow with intelligence, not just user count = a structural tailwind · (2) the real advantage is a "cost per token + speed" good enough to be worth moving out of the GPU ecosystem · (3) many of the real players are still private, and the established incumbents hold inference too — value will go to those who can prove they're "truly faster and cheaper on real work," not just on paper.

In short: this node is the chip born for the job AI does most often — "running" a model to answer real questions billions of times. As running keeps getting bigger and models keep thinking more, whoever can run fastest and cheapest per token will hold one of the key keys to the entire AI economy.