Core conclusion

In the training era, the bet was that "the strongest model would eventually recoup its costs in inference workloads" — costs couldn't be quantified, so everyone chased extreme performance. In the inference era, users pay per million tokens, and the cost of producing those tokens can be worked out precisely, so the goal of a chip shifts from "faster" to "better value". The technical watershed is this: at the decode stage, every token produced requires reading the entire set of model weights from storage into the compute units, and that is how the bottleneck moved from compute to memory bandwidth.

Underlying logicTraining's math doesn't close; inference's does

Training itself carries no direct commercial return, so when you try to work out the cost of training, the exercise is unquantifiable — the hope is "I bet on the most powerful training cluster, so that I own the most powerful model, and later I recoup every dollar I put in through inference workloads." Inference is the opposite: how much a user is willing to pay per million tokens, and what it costs me to infer 1 million tokens (the money spent buying GPUs plus the electricity burned while running inference) — that is an account you can calculate clearly. Inference systems therefore pursue price-performance, not peak performance.

"In the era of training, there was no way for anyone to earn any commercial return from training... So when people start calculating the cost of training, this thing becomes unquantifiable." [04:56]

"Inference is about pursuing price-performance rather than peak performance — it brings cost into the picture far more." [05:42]

Technical dividePrefill is compute-bound, Decode is bandwidth-bound

Inference has two phases. In the prefill phase every token is already known, so there is ample parallelism: you read the model from HBM into the compute units once and can process an entire input in a single pass — it is compute-intensive, which lands right on the GPU's sweet spot. The decode phase is strictly autoregressive: you must know the output for the current token before you can compute the next one, and the cost of producing each token is reading the whole model's weights all over again.

"The cost of producing each token is really reading your entire model — a 1.6T model — from your storage medium into the compute units." [06:51]

The GPU's answerBatching: the GPU's clever move, and also the ceiling on the experience

A GPU cannot read the entire model for one user's single token, so it does batching: it gathers inference jobs from tens of thousands of users and computes everyone's nth word together. For any one user, part of the wait is really spent waiting on the other 9999 users in the same batch. That is also why, in agent and reasoning scenarios, how fast inference feels is fairly tightly constrained.

"You are not only waiting for the GPU to finish inferring your one token; you are also waiting for the GPU to finish computing this one token for the other 9999 users in your batch." [09:44]

Definition"Inference chip" is a very loose term — the endgame is a "system", not "a single chip"

Inference is complicated enough on its own: it splits into prefill and decode, and decode further splits into Attention and FFN. Both guests' judgment about the future is that an ideal inference system will contain many different kinds of chips, each with a more targeted new architecture for the part it serves, rather than one general-purpose chip doing every job.

"The inference system we hope for may well contain many different kinds of chips, so that the actual process of decoding tokens can be made cheap enough and fast enough." [10:22]

Core conclusion

The trade-off among the three media is at heart a trade-off between "capacity and distance": DRAM is cheap but far away, HBM is fast but expensive and in short supply, and SRAM is the fastest yet takes 6 transistors to store 1 bit, leaving it roughly two orders of magnitude below HBM in capacity. The real first principle is this: HBM has high bandwidth because it moves storage into the same package; the next step up is to put storage directly inside the compute chip, so the wires go from "package traces" to "lithographed wires" — a difference of orders of magnitude. That is why the whole industry is converging on SRAM, and SRAM is simply the most mature and reliable carrier available right now.

IntuitionDRAM is cheap but far, HBM is fast but expensive, SRAM is fastest but can't hold enough

DRAM is the memory stick in your computer: it holds a lot, but it sits far away, fetching data back and forth takes time, and it needs constant refreshing. HBM is essentially 3D-stacked DRAM that has been moved next to the compute chip, with many channels opened up to deliver high bandwidth; its capacity is decent and it is fast, at the price of being expensive — and global supply is extremely tight and severely short. SRAM is different: it can sit directly inside the compute chip, making it the fastest option, but it takes 6 transistors to store 1 bit — to hold the same amount of data it needs tens, even over a hundred times, the chip area of DRAM.

"DRAM's direction is cheap but slow; HBM is fast but expensive and impossible to get hold of; SRAM is the fastest but it can't fit it." [15:06]

First principlesSwap "package wires" for "lithographed wires"

The core of bandwidth is the density of your wires and their frequency. HBM manages high bandwidth because it puts storage and compute in the same package, shortening the wires from the motherboard to inside the package. The next step up is to put storage directly inside the compute chip: the wire goes from a "package trace" to a "lithographed wire inside the compute chip," and these two kinds of wire differ fundamentally in density, cost, and energy. So the SRAM route is not a bet on any particular medium — it is a bet that "locality can be raised one more level," which is exactly where Bill Dally's line in Chapter 5 lands technically.

"When you take this wire from a package trace to a lithographed wire inside the compute chip, the two kinds of wire differ fundamentally in wire density, in wire cost, and in energy consumption." [12:10]

Industry convergenceNot just Groq and Cerebras: the whole industry is moving toward SRAM

American AI inference-chip startups, including d-Matrix and MatX, are all heading toward SRAM; among the big players, each generation of TPU and Trainium comes with ever-larger SRAM. Each company's particular way of using SRAM differs — Google leans toward using SRAM to hold the KV cache — but the essence is the same: make good use of a high-bandwidth medium so that memory access across the whole system is faster and cheaper.

"Compared with the other media that provide storage today, SRAM — from a bandwidth standpoint, whether you mean the absolute quantity or the price-performance you get per dollar — is extremely high. So what we really see is everyone converging in this direction." [16:24]

Core conclusion

Where the two companies diverge is not the medium but "how to solve communication scheduling for a large cluster." Groq moves scheduling up front to compile time, using static arrangement and determinism to eliminate runtime overhead, at the cost of being unable to absorb the runtime dynamism of MoE — some chips are simply left idle. Cerebras replaces chip-to-chip communication with the physical proximity of a whole wafer, at the cost of yield — not a one-time "tooling" expense, but "of every 10 you make, 9 are unusable." Both bills ultimately land on token cost.

Shared starting pointFirst sacrifice capacity, then swap the bottleneck from "capacity" to "communication"

A single chip's SRAM has roughly two orders of magnitude less capacity than HBM and cannot hold the whole model. The most direct fix is to make the system very large — hundreds, even thousands of chips — purely so that all of the model's weights can sit in 6-transistor SRAM. Everything does fit, but fitting alone produces no tokens; you also have to be able to read it back out effectively and compute on it. And once a system gets big enough, the efficiency bottleneck lands squarely on how you do communication.

"It all fits — as long as my system is built big enough, it fits. The problem is that once it is just sitting there in storage, it doesn't actually produce tokens." [17:55]

Groq routePut scheduling into compile time and push determinism to the extreme

Groq's judgment is that scheduling at runtime tends to be very expensive. So it wants to have worked out, at compile time, all of the computation and communication the entire system will ever do — which clock cycle performs which computation, which clock cycle starts communicating; once the layout is set, there is no scheduling problem left at runtime. What comes with it is determinism: whether a read from the storage medium takes 100 nanoseconds or 3 nanoseconds does jitter, and with SRAM plus a purpose-built chip design you can push both determinism and static clock synchronization to the extreme. This line of thinking fits the founder's background perfectly — Jonathan Ross came out of the compiler side of Google's TPU, and all of Groq is "hardware built around the compiler."

"Groq's whole approach really happens around the compiler... take deterministic compilation, then build the hardware around that compilation, so the hardware was redesigned starting from static compilation and static scheduling." [23:27]

Groq's weak spotMoE is decided at runtime, and static compilation can't predict it

An MoE network has a great many experts, and at inference each token activates only a small subset of them — pick 8 out of 128 experts, say — and which 8 get picked is decided at runtime, unpredictable at compile time. Meanwhile a single chip's SRAM cannot hold the whole network either, so experts are necessarily spread across different chips, which means tokens must be routed at runtime to the right chip — exactly the part static scheduling handles worst. The two conservative responses each carry a cost: either send the tokens everywhere and let some chips idle, or slice the model a different way — but working it through, without splitting along the expert dimension, today's models are very hard to partition cleanly and efficiently.

"Say you pick 8 out of 128 experts: there is no way to predict that at compile time, because different tokens pick different experts." [20:11]

Why it's still acceptableIdling only hurts price-performance, not the ability to be fast

Groq made the technical choice of fully static compile-time scheduling before it ever ran into dynamism as large as MoE's — when it was founded in 2016, Transformer did not exist yet. The good news is that the math still works: SRAM holds a two-to-three order of magnitude advantage over HBM in bandwidth price-performance, so after giving up one order of magnitude it is still ahead; and chips idling do not stop it from being "fast." There is a commercial layer on top: once you have built something faster than everyone else's that is also actually cheaper, you do not go telling people your costs are lower — because some customers will pay a premium for "fast."

"It starts with a three-order-of-magnitude advantage; I give up one order of magnitude and I can still beat it." [22:41]

"When you have built something faster and cheaper than everyone else's, you don't tell people your costs are actually lower." [23:04]

Cerebras routeStuff an entire cabinet onto a single wafer, trading physical distance for communication overhead

Cerebras's thinking is more direct: cluster communication has a cost because several hundred chips always have physical distance between them — so take everything in a cabinet and put it onto one big wafer. The physical distance shrinks, and the bandwidth of the wires you can run gets better too. But the cost lands on yield: by conventional design the single-chip size limit is around 800 square millimeters, a wafer can be diced into thirty to fifty chips, and at 50% yield some twenty good ones are left; when the product is the entire wafer, though, the moment on-die defects pass a certain proportion, the whole wafer is scrapped. Cerebras's real engineering barrier is "how to accept yield" — partial good design, accepting that 30% of the sites are bad while keeping the product usable, plus software that uses the logic of several hundred thousand cores to route around bad blocks, and making this completely different system plug into the rack.

"Where it breaks is that once your whole wafer fails to meet its yield requirement, the entire wafer is effectively scrapped." [27:11]

"For every 10 you make, perhaps 9 of them are unusable, and what that really hits is your cost in actual production." [28:02]

Cost conversionOne takes a haircut for determinism, the other for yield

What everyone ultimately cares about is the cost of tokens, and datacenter costs come in two parts: the one-time cost of buying the system, and the electricity and energy consumed while it runs. To get a feel for the orders of magnitude: if there were no yield problem, a single chip shaped like Groq's could cost one tenth of NVIDIA's with ten times its bandwidth; but Cerebras, because its manufacturing-side costs are so high, is probably "ten times NVIDIA's bandwidth at three times the cost of HBM." So Cerebras takes a haircut for yield cost and Groq for determinism, and both haircuts ultimately land on token cost. This also explains why Cerebras is more expensive than Groq — and why "fast" itself can command a premium: if others deliver 100 token/s and it can do 750 or even 2000 token/s, users will pay a large premium for that.

"Cerebras has to take a haircut for its complicated yield costs, and in the end that haircut lands on the cost of tokens." [30:10]

NVIDIA's choiceWhy spend $20 billion to acqui-hire Groq

The reasoning is straightforward: the large bandwidth demands that inference creates are something the GPU architecture simply cannot handle well. For NVIDIA, CUDA is both a moat and, in a sense, a burden on innovation; and Jensen Huang clearly recognized that in inference, compute is not necessarily the only thing that matters — the bandwidth capability still has to be added. So why not do it in-house? Mark has an NVIDIA Research perspective: they were doing architecture-level innovation very early on and published plenty of papers on things like CGRA, but at a mature company, pushing a disruptive research idea onto the product teams is extremely difficult — it is easier to simply acqui-hire a new technology team with no ties to the existing one.

"Jensen recognized very clearly that in future inference, compute will not necessarily be the only thing — this technical capability of adding bandwidth is needed." [33:01]

"Why not just acqui-hire a new technology team that has no connection to the previous teams." [33:50]

Context calibrationWhat happened to this deal afterward

According to public reporting (information from outside this episode): on December 24, 2025, NVIDIA acquired Groq's core chip technology and founding team for roughly $20 billion, through a non-exclusive IP license plus talent absorption, with founder Jonathan Ross and others joining NVIDIA; Groq's cloud business was spun off and raised funding independently to keep operating. At GTC 2026, NVIDIA released the Groq 3 LPU as a coprocessor dedicated to decode within the Vera Rubin platform — the Rubin GPU handles the compute-intensive prefill, while the LPU produces tokens with low latency. This is exactly the "ideal division of labor" the two guests describe in this episode: which part the GPU handles and which part Groq handles.

"The way Groq now works together with NVIDIA really is close to the ideal we had envisioned — for example, which part of the tasks the GPU handles, and which part Groq handles." [11:07]

Software stack"In the inference era, CUDA will definitely be bypassed"

This prediction is one hundred percent going to happen: look at the inference chips from the big players — TPU, Amazon's Trainium — and not a single one says anything about being CUDA-compatible. The reason TPU software is called hard to use is rooted not in "many hard problems" but in "one hard problem" — as long as you write out all of Transformer's operators on that chip, Transformer runs, and the business logic works. Anthropic can buy TPUs in large volumes because many of its engineers came from Google and know how to use JAX; swap in an engineer who has never built a compiler, and it still feels hard — at the software layer, TPU is still a black box. The real shift is this: in the training era, people were willing to pay for "how usable the software is"; in the inference era, the scales will tip toward absolute token cost and price-performance (DeepSeek runs on GPU chips, yet is willing to write very low-level PTX assembly for deep optimization), and with AI coding capability improving, operator and runtime development has been dramatically accelerated.

"From the inference-chip angle, I think bypassing CUDA is one hundred percent going to happen." [34:13]

"As long as it is not very many hard problems but one hard problem, then once I solve that problem, the matter is settled. I only need to write out all of Transformer's operators on this new chip, and then my Transformer can run." [34:55]

Core conclusion

To evaluate a cloud inference chip, you must look at three numbers at the system level: per-token cost, per-user tokens/s, and power consumption (utilization is only an intermediate variable that affects cost). And the value of "fast" is the most commonly misunderstood — it is not about seeing the same answer sooner, but about letting the model do more internal thinking within the same time budget and therefore become smarter. On the technical route, they chose system-level heterogeneity: networking over a thousand SRAM chips to hold the entire model, and leaving Attention and Prefill to the GPU.

ApproachNetwork a thousand-plus chips so the links between them are never the bottleneck

The goal is not to carry the chip layer's two-to-three-orders-of-magnitude bandwidth price-performance advantage over to the system layer intact, but to have the system layer retain roughly 10x price-performance plus a one-to-two-orders-of-magnitude speed gain. The arithmetic is plain: how much SRAM fits on a single chip is public knowledge — on the order of a few hundred MB; today's models are around 1T parameters, and even after FP8 quantization that is still about 1TB — divide one by the other, and you need roughly a thousand chips just to fit the whole model in capacity. Their core work is to connect those thousand-plus chips with some topology so that the connections between chips do not become the inference bottleneck, while making sure every chip's SRAM bandwidth gets fully used (there will inevitably be losses). In terms of data flow, the hidden state trickles bit by bit through the chip network, from one side to the other, and produces the final result.

"What we are doing is figuring out how to connect these thousand-plus chips with some topology, so that the connections between each of the chips in the middle do not become the bottleneck when I do inference." [37:56]

System-level heterogeneityCut the model in two and build only the half we are best at

Their chip and system will not do everything for the whole model, because they believe "the most suitable hardware should be used for the corresponding part of the application." The whole model gets split into an attention (Attention) side and a non-attention FFN side, and they build the latter half — quite similar to Groq but not entirely the same. There is also an honest statement of Amdahl's law here: because it is a heterogeneous system, they solve 90% of the problem and leave 10% to the GPU, so even if that 90% is solved with 1000x efficiency, the whole system can improve by at most 10x.

"We solve 90% of the problem, but we leave that 10% for the GPU to solve, so at most our system can improve by 10x." [39:10]

Evaluation frameworkIn the cloud, only three numbers matter: cost, speed, and power consumption

In the inference era, the first three metrics are the important ones — the cost per token produced, inference speed, and the power consumption per token; the fourth (utilization) affects the first, but in the end what everyone cares about is certainly cost. And because future systems are heterogeneous, a single chip cannot, on the strength of compute, bandwidth, and capacity alone, shape anyone's overall picture of the system — judging whether a chip is good must be done at the system level. On the edge, the logic is entirely different: for a chip in a box or a phone, the cost per token is not the most important thing; what matters is "for at most $100, this $100 buys inference at a certain speed" — that is a product you can sell.

"Going forward, whether a chip is good must be measured at the system level." [41:45]

"For this kind of edge AI chip, what is being considered is an entirely different thing." [42:10]

China-US differencesJensen talks token per watt, while customers in China care more about token per dollar

Cloud power usage is on the smaller end of today's total cost of ownership, but there is a fairly big difference between the US and China: America is short on electricity and energy, so it cares more about how many tokens you get per watt; customers in China are less sensitive to power consumption and care more about how many tokens you get per dollar. This difference becomes a key variable later on, when explaining OpenAI's storage choice.

"This actually shapes how much people care — for instance, whether it is Jensen talking token per watt every day, or everyone caring about token per dollar." [42:53]

Consistency note: the episode gives two not-fully-consistent figures for "electricity's share of total cost" (one spot says roughly 1% and 2%, another says about one third in China and about two thirds in the US). Only the qualitative conclusion is kept here — America's hard constraint is electricity supply, not just the price of power, which is why it is more sensitive to token per watt.

CounterintuitiveThe faster the speed, the smarter the AI

Since agents exploded at scale, a huge share of the tokens produced by language models is actually never read by humans: chains of thought are generated by the model for itself, and the tokens an agent produces are read by other agents. Humans can only read tokens so fast — at 100 token/s we already cannot keep up — which is why many systems today are designed backwards from 100 token/s (or even 50 token/s); but for agent systems, that speed has, to a degree, no ceiling — the faster the better. To the feeling that "inference speed is already good enough," the guests' response is: it will certainly feel good enough, but it will be realized somewhere else — users cannot accept a question being thought about for two days, and even an hour is hard to accept; that hour is a hard constraint. So pushing tokens faster does not mean faster interaction; it means that within the same interaction time, the model can use 10x more tokens for internal self-reflection, or run lots of experiments agent-style, and thereby become more intelligent. The horizontal axis of that chart in Jensen Huang's keynote literally reads smarter AI; Bill Dally, for his part, prefers to call this dimension interactivity.

"At the same one-minute interaction speed, your model can use 10x more tokens for more internal self-reflection, raising its level of intelligence." [46:23]

"We want more intelligence within a fixed time limit, and that translates into fast." [48:28]

Forward-looking methodFacing model iteration, grab the "invariants"

Chips are a long-cycle business (traditionally a year and a half to two years), so you must look two or three years ahead and find the essential invariants that will shape chip design. The judgments to make come in two layers: first, whether the chip supports a given class of models (the heterogeneity choice), and second, if it does, how to do it on your own chip. The structural fact they observe is this: on the attention-mechanism side of Transformer, there is still a great deal of algorithmic innovation — every new lab that emerges publishes some new ideas; but the FFN on the other side, since becoming MoE, has turned into a relative invariant. So they prioritize serving the part that has already converged. Separately, the Diffusion models used in image and video generation compute differently from Transformer and are extremely compute-dense, fitting a different paradigm; but if in the future they become autoregressive, or the two are combined, then it naturally fits their design.

"That latter FFN section, since it became MoE, has actually become a relative invariant." [50:49]

"We hope to find the chip-to-system solution that best matches today's bandwidth needs, and I don't think this future is tied to any specific model." [51:29]

Core conclusion

The barrier to chip startups lies not in any single technical point but in "walking the entire chain yourself" — from requirements analysis to delivery and networking, with yield, supply chain, and capital in between. And China's advantage is not just supply chain and infrastructure: an often-overlooked variable is the "open-source ecosystem" — America's most mainstream models are not open, making it hard for Groq and Cerebras to find a reference point they could run through on their own commercially; China is exactly the opposite, with the strongest models being more open at the architecture level.

The vanishing computeWhy spec-sheet compute never runs at full: bottlenecked by bandwidth and by fragmentation

Two reasons. First, many workloads have a compute-to-memory-access ratio that does not land on NVIDIA GPU's sweet spot — by the basic structure of the Roofline model, once you are stuck on bandwidth, the compute simply cannot be used. Second, inference is made up of a huge number of small, fragmented operators (a few hundred tokens must come out per second, and each token corresponds to a very fine-grained compute task), so compared with training's uniform large matrix multiplications, it incurs more scheduling losses. Responsibility here sits on three levels: the users, the chip design, and the software layer. The two relevant metrics are MFU (model FLOPs utilization, which essentially only counts the matrix-multiplication portion) and MBU (memory-bandwidth utilization) — bandwidth saturated at 100% does not mean the chip is 100% utilized, because there are at least two units involved: memory bandwidth and compute. What is called "the chip waiting for data" is, at bottom, the interconnect being stuck.

"If you are bandwidth-bound, then obviously your compute cannot be used." [52:47]

"Your MFU — we basically only count the matrix-multiplication part. So once we find the matrix multiplications spinning idle, we end up with a fairly low MFU." [54:01]

The full processThe life of a chip: from requirements to delivery

Mark walked through the entire process: requirements analysis (what this chip is positioned as, what problems it must solve, which are technical problems and which are customer problems such as security) → high-level architecture design (what actually ships in the future may be a server or even a cluster, so requirements must be decomposed layer by layer across system, server, and single chip) → microarchitecture definition (how the compute units are designed, matrix units versus vector units, which instruction set to use, where storage sits) → a very detailed feature list → RTL coding (the actual coding may take only two or three months; most of the time goes into the upstream requirements and architecture) → verification (large-scale test cases, or formal verification and mathematical proofs) → backend (place and route, how the wires are routed, the aspect ratio of storage cells, plus design rule checks based on information from the Fab) → tape-out (the photomasks are made by the wafer fab; this applies to the first tape-out) → package testing (an NVIDIA chip looks nearly half the size of a phone, yet the portion that is actually lithographed is a small fraction of it; most of the packaging exists to solve "how to bring such fine lithographic lines out onto an actual board"; testing, meanwhile, requires a systematic method that tells you which chips in a batch are good and which are bad) → delivery (PCB design, the control chips and switch chips on it, network ports and interfaces, and then power, cooling, and networking for the whole cluster). He stresses that this is not a solo fight — many of the later stages involve back-and-forth with suppliers.

"People say that for a large chip, the cost of tape-out is extremely high, so everyone hopes that at tape-out they can guarantee the chip will have no functional problems." [56:51]

▸Three optional entry points for a startup: build servers and clusters, build chips, or build an integrated chip-plus-software solution

▸The real time distribution: upstream requirements and architecture take the bulk, while RTL coding itself takes only two or three months

▸Where the barrier lies: countless technical challenges, tape-out costs in the millions of dollars, yield control, supply chain

▸As the guests relay it: many NVIDIA scientists do not leave to start companies because "the industry chain you have to move is too big" — but the payoff odds are big enough

China's advantagesSupply chain, infrastructure, and the overlooked variable of the "open-source ecosystem"

Supply chain will be a very critical issue: China's entire chip market may stay in a state where demand exceeds supply for the next two to three years, so the key to success or failure may not be how well the chip is designed, but very possibly whether you can manufacture enough chips. Architectural innovation, counterintuitively, delivers a reverse dividend here — because the design involves a fairly large innovation, it relaxes the requirement for the most advanced process node, opening up the chance to make excellent use of mature nodes that others cannot exploit and even consider obsolete, and to build, on mature process technology, a solution that beats advanced-process systems on tokens per second and tokens per dollar. Mark's return to China was not entirely about starting a company: there were already good job opportunities at home when he graduated; China's supply chain is complete and full of suppliers; its ability to build inference infrastructure (where data centers go, water and power) is a very strong advantage; and it is itself a large market that can turn entirely on its own. There is an even more overlooked point: Groq's and Cerebras's commercialization has been difficult, partly because America's most mainstream models are not open source — you build a new chip and cannot get people who do not understand your chip to adapt to it, and it is just as hard to adapt yourself to a model that is not open; and the open-source models of the time (Qwen, Llama) were not America's most mainstream ones. China is the reverse: the strongest models (DeepSeek above all, plus Zhipu) are more open at the architecture level, and the best model APIs that can be reliably obtained in China are themselves these open-source models.

"The hard part for Groq and Cerebras back then was that America's most mainstream models were not open source." [01:02:25]

Competitive landscapeHow chip startups compete with model vendors

If the models are open source, a model vendor building its own chip has no fundamental advantage over us building chips — because building this system requires thinking through the other parts of the system, and that is not what model vendors do. Model vendors' real advantages are two: the more advanced models have not been open-sourced; and the hardware and software teams can coordinate (for example, Google's TPU can factor in how the chip should be designed even as the model keeps moving forward). The guests divide model vendors into two kinds: the traditional internet giants, which have their own training needs and a wildly varied model portfolio (everything from dialogue to image generation), so they prefer chips that are not too narrowly targeted, and what they actually build is mostly a replacement for existing general-purpose AI chips, skewing toward training or generality; and the pure model vendors (DeepSeek, Zhipu, Kimi), for whom "designing a good model" and "designing a good chip" require entirely different capabilities — integrating two completely different capabilities into a single company is not necessarily a good thing, and this itself is something that wants to be heterogeneous.

"Companies whose culture suits algorithm design should do the models, and companies whose culture suits chip design should do the chips." [01:05:08]

Core conclusion

Bill Dally's two guiding principles form the underlying explanation for every technical route in this episode: first, "everything revolves around locality" — temporal and spatial locality are smoothed away by the algorithms in AI's decode process, so locality must be deliberately manufactured in the architecture; that is the real reason for the SRAM route. Second, "to escape a local minimum, you must first be clear about what you will sacrifice" — micro-innovations can preserve the larger picture, but order-of-magnitude gains demand clarity about what can be given up. Amazon's two philosophies (reasoning backward from the system to the hardware, and customer obsession) offer the engineering version of the same thing: start with the end in mind, and design for "the two years out when the chip actually arrives," not for "today."

MentorshipHow Bill Dally finds "problems worth doing"

Mark went to Stanford in 2017 for his PhD, with Bill Dally as his advisor. Bill was deeply involved in many of the technical directions and decisions as NVIDIA made the transition from early GPU graphics card chips to AI chips; his stream-processing research from before 2000 was later commercialized by NVIDIA into today's GPU. In his senior year, Mark was pulled in by Song Han to work as a research assistant, on model pruning and quantization and on making models run more efficiently on hardware; Song Han served as his weekly mentor, but there were also weekly meetings with Bill. Bill's first impression on him was: extremely smart, and someone who looked at problems at a very fundamental level and never picked low-hanging fruit—as a senior professor, he had no pressure to publish papers in the short term, so he picked problems that were very hard to solve but would have an enormous impact on both academia and industry once solved.

"He won't try to solve some low hanging fruit... the things he sees that are more fundamental are some of the technical points he is actually interested in." [01:06:29]

The betFilling in the "strong logic" half of future AI

AI development has two branches: the "perception" branch, leaning toward deep learning and neural networks, and the "strong logic" branch (building knowledge graphs and establishing very rigorous logic, such as using AI to prove mathematical theorems). Bill's judgment in 2017 was that future AI would be "strong logic plus strong perception", but on the hardware side almost nobody cared about the strong-logic half — people were mostly staying at the software level. So they designed a brand-new chip architecture from scratch, a dedicated accelerator completely different from CPUs, GPUs, and everything that came after, to accelerate this. Where that judgment lands today: how to better combine algorithms for solving logic problems (SAT problems) with the Transformer, letting the Transformer break real, complex math and physics problems down into these strong-logic problems — that is what many AI for Science companies are trying to do, and at the algorithm level things have not converged yet; but if it turns out this route really works and the bottleneck of scientific exploration really comes down to "solving a very complex logic problem faster", then the demand for chips and hardware is certain.

"The way Bill sees it, future AI is strong logic plus strong perception, but at the time, from that angle, basically nobody cared about the strong-logic part, because almost no one had ever built dedicated hardware for it before." [01:08:14]

First principles"Everything revolves around locality"

The line that shocked Mark most from Bill: computer architecture is like real estate — everything is about location. Temporal locality is "I used a piece of data, so over the coming stretch of time it will very likely be used again"; spatial locality is "I used a piece of data, so I will very likely soon use some of the data around it". Cache design in a CPU is itself an application of these two kinds of locality. The key insight: in AI, and especially in decode, temporal and spatial locality are "flattened out" at the algorithm level — so you have to go the other way and build more locality at the architecture level, in actual hardware, to compensate. That is the real reason they want to use SRAM to hold model weights: SRAM provides the best locality.

"For the whole chip design, everything unfolds around locality." [01:10:43]

"Computer architecture is like real estate — it's all about location, location, location." (a paraphrase of Bill Dally's original words, quoted in English on the show) [01:10:51]

"We need to build more locality in actual hardware at the architecture level to make up for this." [01:11:42]

Decision principleEscaping the local minimum: first get clear on what to sacrifice

Bill's logic: find the hardest thing, find the point with the biggest impact on the whole system, then try to solve it. There is a key distinction here — when you are making only small improvements, you can preserve the whole without disruptive changes; but if you want an order-of-magnitude improvement, you absolutely must think through, very clearly, what can be sacrificed and what is truly critical. Only once you have thought that through can you make the more critical choices at the system level, escape the local minimum, and find a bigger, freer global solution. This principle is borne out repeatedly in this episode: Groq sacrificed per-chip capacity for determinism, Cerebras sacrificed yield for physical distance, OpenAI sacrificed per dollar for per watt — all instances of "getting clear on what to sacrifice". The cost is a high probability of failure.

"When you are only making small improvements, you can actually preserve the whole picture... but when you are going for something hard, an order-of-magnitude improvement, you have to think very clearly about what you can afford to sacrifice and what is truly critical to you." [01:12:01]

View on failureFailure doesn't negate whether the problem itself is right

Mark says failures like this are extremely common, but it has nothing to do with success or failure — it is the logic of how you work, of what kind of things are worth doing. Failure only means you may not have found the right solution, or that a solution may not even exist; it does not mean the problem you set out to solve was itself the wrong problem.

"It doesn't mean the problem you set out to solve wasn't the right problem — it just may be that you didn't find the right solution, or that a solution may not exist at all." [01:12:55]

Startup adviceBill gave exactly one hard piece of advice — plus one contrast

The hard advice was just one line: never use code written by PhD students — a painful lesson from Bill's own previous startup. More interesting is the contrast: at heart Bill is closer to a scientist; in his eyes "the essence of research is gaining the most knowledge at the smallest cost", which is why back then he would advise students not to tape out for tape-out's sake — because before tape-out you already have 99% of the knowledge, and taping out brings nothing beyond engineering implementation. But a startup is precisely about solving the remaining 1% of engineering, and it may consume 99% of your energy: its contribution to "whether this thing can be built" is not that large, yet you must pour enormous time into practical design problems like cooling and power delivery.

"The advice Bill gave me was: never use code written by PhD students, because it was a painful lesson from Bill's previous startup." [01:13:32]

"Before doing any engineering, you can already know from a design standpoint where the technical bottleneck of this thing is and whether it can be done." [01:14:37]

Engineering philosophyAmazon: customer obsession and "begin with the end in mind"

Ziyang works on the software stack for the Trainium chip at Amazon's Annapurna Labs. Software sits at the very top of the system and users come in through software, so Amazon has two philosophies: first, work backwards from the system to the hardware; second, customer obsession — get clear on what the user ultimately wants, then use the software stack to work backwards through every layer in front of it. Put another way, it's beginning with the end in mind: be very clear about what the workload you want to accelerate looks like, which class of models it is, whether it's training or inference or both — this directly affects every layer of the concrete design as you work backwards from the high-level system all the way down to the chip. One key parameter is "the ratio of compute bandwidth to memory-access bandwidth" — it determines how many computations you need to do after reading a piece of data to keep both compute and memory access busy, and it directly decides which class of models or applications works best on this chip. The actual practice: first pin down what the final workload is, how many nodes the system will have, how many nodes it takes to physically fit the model and whether that can be accepted, then work backwards to where this ratio should be set. Once the hardware is fixed, it in turn determines how you optimize for different software. The most essential point: what you need to know is not necessarily what users are using "today", but what users will want "when the chip is done" — which is two or three years out.

"First you have to know what the user wants, and that may not be what users are using today — it may be what users will want at the point when the chip is actually built." [01:17:12]

"Because a chip, by its nature, is a long-cycle endeavor." [01:17:22]

Key takeaway

Jalapeño did not go the SRAM route; instead it stacked HBM4 bandwidth to 15.4 TB/s per package. But this departure is not a contradiction — it is a stronger conclusion: the choice of memory route is not a contest of technical superiority, but a question of "whether your first constraint is money or power". Building heterogeneity inside a single chip and using "dark silicon" to switch off unused units to save power was a solution the two guests were seeing for the first time. The endgame will not be a single route — different market conditions will converge on different points, and that is exactly the two layers of meaning behind "heterogeneity".

CharacterizationA general-purpose inference chip, not a dedicated accelerator

OpenAI's chip was jointly developed with Broadcom and released on June 24, 2026 — exactly the day this episode was recorded, so the show did a supplementary recording before publishing. In character it is a general-purpose AI inference chip, fairly close to a GPU: its generality shows in that it can run all kinds of models — the demo even ran a game. It wraps up in a single chip the parts — prefill and decode, Attention and FFN — that were previously thought to require system-level heterogeneity; the design also adds support for small-dimension matrix multiplications, so performance stays quite good even at small batch sizes. Public information even says its next generation will support training. Another small feature is its use of AI: for kernels written in its kernel language, much of the optimization was to a large extent handed to Codex; and the chip design itself saw big gains in both performance optimization and time to market (project kickoff to tape-out) compared with previous chip projects — reportedly taping out in 9 months.

"Its generality shows in that it can run all kinds of models — it even demoed running some games." [01:18:22]

"They say it took 9 months from kickoff to tape-out, and that speed is quite fast." [01:20:08]

Why the most expensive routeBecause its first constraint is power, not money

What US data centers lack right now is not capital or space but power supply. So OpenAI badly needs more tokens out of the same power capacity — producing more intelligence also means more revenue — hence it is extremely sensitive to power efficiency and comparatively less sensitive to capital expenditure or the cost of building chips. This explains a seemingly counterintuitive choice: the most essential advantage of the SRAM route is "several orders of magnitude more bandwidth per dollar than HBM", which translates directly into "more tokens per unit of price or per unit of chip capacity"; but what OpenAI cares about more is not per dollar, it is "how many tokens per watt or per megawatt" — and on that metric this chip is above NVIDIA's Rubin architecture. So "expensive" is not necessarily a disadvantage — it depends on which item you treat as the constraint.

"What OpenAI sees is that it is extremely, extremely sensitive to power efficiency... the same power capacity producing more tokens, and producing more intelligence also means more revenue." [01:18:33]

"For now, what OpenAI cares about more is another metric — the power angle." [01:21:54]

SRAM's discomfort zoneKV cache: capacity demand is dynamic, and changing fast

Long contexts plus cache hits push the KV cache's capacity demand way up, and it is dynamic: it keeps growing with the number of users and the length of contexts. That puts a demand on capacity for the money, and this falls precisely in SRAM's less comfortable range — which is why both guests' own designs, and Groq's combined-with-GPU design, all put the KV cache for the Attention part on a separate chip rather than on the big-SRAM chip. Another variable is that KV cache demand itself is changing fast: the first chart after DeepSeek's new model release showed per-token KV cache falling generation by generation. SRAM's biggest drawback is therefore pinned down as "capacity for the money" — 6 transistors to store 1 bit, and generally, to reach the bandwidth, it has to sit on the same die; stacking it actually lowers the bandwidth you get out. But demand splits into two categories: a "fixed quantity" like model weights can be handled by clustering many chips together, still within acceptable SRAM territory; while a "highly variable and possibly ever-growing quantity" like the KV cache — the capacity-limited drawback makes it unsuited to that scenario.

"If the data you need to store varies a lot and may keep growing in the future — say, the KV cache we just mentioned — then we consider that its capacity-limited drawback makes it unsuited to this scenario." [01:26:29]

The windowWill SRAM's window of advantage close within two or three years?

A reasonable worry: each HBM generation improves bandwidth faster than SRAM improves density, while the chip cycle is only a year and a half to two years (Jalapeño even 9 months). The guests' judgment: HBM bandwidth growth is actually not as rosy as people think — even "where the bandwidth growth after this generation comes from" is not very certain. First-principles thinking comes back to "wires" — HBM's connections are still some kind of wire within the package, not wires drawn by lithography, so its density has a natural bottleneck. A repeatedly mentioned direction is 3D DRAM, turning "wires along the edge" into "wires across the plane". The conclusion: on the point of "absolute bandwidth and value for money", HBM and HBF still struggle to reach SRAM's level; and all these directions are merely different points in the same trade-off space.

"Thinking from first principles: your connections still go through some kind of wire... but it is not a wire drawn by lithography, so its density does face some natural bottlenecks." [01:24:57]

Dark siliconFolding heterogeneity into a single chip, saving power by switching units off

The guests offered a small revision to the judgment that "the industry is converging on SRAM", but the direction holds: in large system-level heterogeneity, to improve token value for money, many companies will still converge on SRAM. What is genuinely interesting about Jalapeño is that it moves heterogeneity from the system level into the interior of a single chip — many of the parts inside the chip are heterogeneous to one another; at the same time it brings in the concept of Dark Silicon: use this one chip to do different things, and when doing some of them, switch off some of its parts or lower their power to improve energy efficiency. The cost is paying more for the chip, so from a per dollar standpoint it is not a great deal. By contrast: Groq, Cerebras, the two guests' own designs, as well as PD disaggregation and training-inference separation, are all "heterogeneity happening at the system level" — multiple chips each doing its own job; NVIDIA's GPU can also be understood as intra-chip heterogeneity, but it emphasizes generality more, and does not, the way OpenAI does, put different demands together and then switch things off in various ways to improve energy efficiency. The guest said this was the first time he had seen such a solution.

"What it really did was build that heterogeneity into a single chip." [01:27:06]

"This is the first time I've seen a solution like this." [01:29:21]

EndgameWill it still be NVIDIA's world?

The guest's answer was candid: this is hard to answer, but "we certainly don't want it to be NVIDIA's world" — if it were, it would mean that the many companies exploring new architectures would not survive, including their own. Their reasoning: NVIDIA, with the GPU at its core, faces certain limits in iteration speed and choice of approaches when exploring other directions; that is why many new approaches in fact do not originate from NVIDIA — and that is exactly why it is spending $20 billion to acqui-hire Groq. Their hope is to make the whole AI chip space a bit more colorful, with more approaches — which also matches their understanding of heterogeneity.

"We certainly don't want it to be NVIDIA's world — with so many companies doing (new) exploration... and that sentence would mean those companies won't survive, including us." [01:29:51]

"We hope the whole AI chip space becomes a bit more colorful, with a few more approaches — that also matches our understanding of heterogeneity as a whole." [01:30:26]

Key takeaway

The body of this report already presents everything in its correct form; the table below is for verification and extended background only. The items marked "Context calibration" come from public reporting and fill in the companies' follow-up moves mentioned in the episode.

Term calibrationTranscription mis-hearing → correct form

Context calibrationA few timeline items to fill in from outside this episode

To be clear: the items above are for completing the context; the core judgments and quotes in this episode all come from the original audio and do not depend on this external information. The recording took place on June 24, 2026, and the Jalapeño part was a supplementary recording made before the episode was published.

▸2025-12-24 | NVIDIA acquired Groq's core chip technology and founding team for about $20 billion, via non-exclusive IP licensing plus talent absorption; Groq's cloud business was spun off to operate independently.

▸2026-01 | Cerebras signed a multi-year, multi-billion-dollar compute purchase agreement with OpenAI.

▸2026-05-14 | Cerebras IPO'd on Nasdaq (ticker CBRS), closing its first day with a market cap above $67 billion.

▸2026-06-24 | OpenAI and Broadcom released the first in-house inference chip Jalapeño; 9 months from kickoff to tape-out.

▸2026 Hot Chips | Jalapeño's architecture details revealed: 216 GB HBM4, 15.4 TB/s bandwidth, 700 W, TSMC 3nm-class process, 64 core slices each paired with an HBM stack (NUMA-style).

Reading notesTwo versions and how to use them

This report comes in two versions: a PC reading version (this file, JavaScript-rendered, just open it locally) and a WeChat paste version (pure inline styles; select all and copy in a browser, then paste into the WeChat official account editor and the formatting holds). All quotes in the body are tagged with timestamps from the original audio, making them easy to trace and verify. The cover prompt file can be pasted directly into an image model to generate a white-background line-art cover image for the WeChat official account.