AI Chip and Memory: From Transistors to HBM
Type: Overview — synthesizes industry reports + multiple papers Companion: AI Data Center: Building the AI Factory Sources: “AI 생태계와 AI 반도체 시장 전망” (2026-03-25), “AI 데이터 센터 동향 및 전망” (2026-03-06), Kim Young-Beom (KIS)
The central shift: processor-centric → memory-centric
For four decades, computing was about making the CPU faster. Moore’s Law packed more transistors onto each chip, and software rode that wave. The deep-learning era inverted the bottleneck. Modern LLM workloads don’t choke on arithmetic — they choke on moving the weights and the KV-cache fast enough into the arithmetic units. So the locus of innovation moved from logic to memory.
Concretely, an Nvidia B200 GPU’s bill of materials is about 50% HBM by dollar value (HBM3e 6,400 BOM). Half the cost of the world’s most coveted chip is memory. The phrase “AI infrastructure is no longer processor-centric, it is memory-centric” is not a slogan — it is what the cost sheet says.
This page surveys (1) the chip taxonomy — GPU vs. ASIC, (2) the memory hierarchy — SRAM/HBM/DRAM/NAND — that AI workloads now traverse, (3) HBM and the packaging stack that holds it together, and (4) the market structure that follows.
1. Chip taxonomy: CPU, GPU, ASIC
A modern AI accelerator is one of three things, plus a quantum outlier:
| Class | Role | Strengths | Weaknesses | Vendors |
|---|---|---|---|---|
| CPU | General-purpose control | Versatile, mature software | Low parallelism for matrix math | Intel, AMD, ARM, RISC-V |
| GPU | Massively parallel SIMD | Thousands of cores, mature CUDA stack, general-purpose enough to run anything | Power-hungry, memory-bottlenecked | Nvidia (dominant), AMD |
| ASIC (NPU/TPU) | Fixed-function neural accel | Best perf/watt, smallest die for the same throughput | Locked to specific ops, NRE cost, slow to evolve | Google (TPU), AWS (Trainium/Inferentia), Meta MTIA, OpenAI / Broadcom, Tesla Dojo |
| QPU | Quantum sampling/search | Polynomial speedup on specific problems (factoring, search) | Cryogenic, error-corrected, narrow problem fit | IBM, Google, IonQ, Quantinuum |
The 2024–2026 trend is ASIC share rising at GPU’s expense at hyperscaler scale, because Nvidia’s 80%+ gross margin on B200 (BOM ≈40,000) gives every cloud incentive to design its own silicon. Gemini 3.0’s strong showing on Google TPU has further normalized “ASIC-first” thinking. GPUs remain dominant for the long tail of customers who cannot amortize a custom design.
A second axis is the move toward in-memory compute. Conventional Von Neumann architectures shuttle data between memory and ALU; that traffic dominates AI workloads. Two responses:
- PIM (Processing-in-Memory): put simple arithmetic units inside the memory die so data is reduced where it lives.
- CXL (Compute eXpress Link): a new bus that lets CPU/GPU treat external memory as cache-coherent, expanding effective HBM via pooled DRAM.
2. The memory hierarchy AI workloads cross
A single training step on a 70B-parameter model reads every weight from HBM, runs it through registers/SRAM, writes activations back to HBM, and periodically checkpoints to SSD. The hierarchy is steep — five orders of magnitude in price per GB, four in bandwidth.
| Tier | Where | Typical size (per GPU) | Bandwidth | Cost/GB | Role in AI |
|---|---|---|---|---|---|
| Registers | Inside the tensor core | KBs | — (single-cycle) | — | Operand staging |
| SRAM (L1/L2) | On the GPU die | ≈100–300 MB | TB/s on-die | ≈**$10,000** | FlashAttention works here; eliminates KV-cache round-trips |
| HBM | 3D-stacked, on the GPU package via interposer | 80–288 GB (B200: 192 GB, B300/Rubin: 288 GB–1 TB) | 3.4–15+ TB/s | ≈**$16.6** | Model weights + KV cache for active inference |
| LPDDR5X (CPU DRAM) | On the system board | 480 GB (Grace), 128–2048 GB on x86 hosts | ≈0.5 TB/s | ≈**$4–5** | Host memory, paging, CXL pool |
| NVMe SSD (NAND) | PCIe drives in the rack | 30 TB+/tray, EB-scale per AIDC | ≈14 GB/s | ≈**$0.15** | Checkpoints, swap, hot training data |
| Object storage (HDD) | Data lake / archive | EB+ | ≈0.2 GB/s | ≈**$0.02** | Raw corpora, completed models, cold logs |
Roughly: SRAM > HBM > LPDDR5X DRAM > NAND, with each tier ≈10–100× cheaper and slower than the one above. Sizing a 1GW AI data center built from GB200 NVL72 racks (≈ 497,952 GPUs):
- SRAM: ≈149 TB on GPUs + ≈25 TB on Grace CPUs = ≈174 TB total
- HBM: ≈96 PB (B200 192 GB) to ≈143 PB (B300 288 GB)
- CPU DRAM (LPDDR5X): ≈120 PB
- Local NVMe (NAND): ≈3.7 EB across compute trays
- External all-flash + HDD lake: EB-scale
Two engineering consequences follow. First, FlashAttention (Dao 2022) reorganizes attention to keep Q/K/V tiles in SRAM, slashing HBM round-trips and making long contexts tractable. Second, the design target shifts from FLOPS to bytes moved per FLOP — the so-called arithmetic intensity. Anything that increases reuse (KV caching, MoE, MLA) wins disproportionately.
3. HBM: how DRAM grew a vertical axis
HBM (High Bandwidth Memory) is the answer to the GPU’s memory-wall problem. Eight to twelve DRAM dies are stacked, connected vertically by TSVs (Through-Silicon Vias), and placed millimeters from the GPU on a silicon interposer. The result is 1024-bit-wide, 3–15 TB/s buses — far beyond what board-level DDR can deliver.
Stacking technologies
The hard problem is bonding hundreds of microscopic copper bumps between thinned dies without warping the stack or trapping voids. Industry calls these packaging technologies even though they happen at memory fabs, not at OSATs.
| Method | Analogy | Mechanism | Notes |
|---|---|---|---|
| TSV | Elevator | Copper-filled via through silicon, 5–10 µm pitch | Foundation: enables vertical signaling at all |
| TC-NCF (Thermo-Compression with Non-Conductive Film) | Iron-pressing a sandwich | Film between dies, heat + pressure | First-gen 8-Hi method; warpage and bump-pitch limits |
| MR-MUF (Mass Reflow + Molded Underfill) | Oven-bake then pour epoxy | Stack first, reflow all bumps at once, then mold-flow liquid epoxy into gaps | SK hynix’s lock: better thermal path, higher yield, simpler |
| Hybrid Bonding | Vacuum-fused copper-to-copper | Bumps and solder eliminated entirely — copper pads bond directly with dielectric | Samsung’s bet for HBM4/4e: enables 16-Hi+, lower thermal resistance, fundamentally smaller pitch |
The competitive read: SK hynix’s MR-MUF advantage carried HBM3/3e (and Nvidia’s H100/B200 ramps). Samsung’s pivot to hybrid bonding is the play for HBM4/4e (24–36 GB per die, 12-Hi+ stacks, 288 GB–1 TB per cube). Whoever solves hybrid bonding at scale captures the post-2026 HBM market.
Why HBM is so expensive — wafer-yield math
On a 12-inch wafer (≈4,000 for HBM3e because of TSV equipment, copper fill, and grinding steps):
| Standard DDR5 | HBM3e | |
|---|---|---|
| Die area | 60–65 mm² | 80–85 mm² (TSV holes + thermal margin add ≈40% area) |
| Gross dies/wafer | ≈1,000 | ≈750 |
| Wafer yield | 90% | 60% |
| Net good dies | ≈900 → $2.8/die | ≈450 → $8.9/die |
Then HBM3e stacks 8 good dies into one 8-Hi cube. One die failing kills the cube, so stack yield further compounds. Per cube the math runs ≈8.9 core dies) + 33.9 (packaging) ≈ **400 per 24 GB cube (3,200 worth of memory — half the GPU’s BOM.
A 1 GW AI data center (≈ 500k GPUs) therefore needs ≈4 million HBM cubes ≈ 89,000 wafers. For context, SK hynix’s projected 2026 4Q capacity is about 205,000 wafer-starts/month and Samsung’s about 210,000 — of which only a fraction can be HBM, since HBM cannibalizes general DRAM capacity at ≈2× the wafer cost per equivalent good-die count.
4. Advanced packaging: CoWoS-S → CoWoS-L → CoPoS
Stacking memory is half the battle; the other half is putting that stack near the GPU at thousand-pin densities. Conventional PCBs can’t do this. The industry runs on TSMC’s CoWoS family (Chip-on-Wafer-on-Substrate):
- CoWoS-S (Silicon interposer): A single large silicon die acts as the interposer between GPU and surrounding HBM stacks. H100 sits on a CoWoS-S package about 3.3× the standard EUV reticle (≈ 858 mm²). Reticle size is the ceiling — beyond it, you cannot expose the interposer in one shot.
- CoWoS-L (Local Silicon Interconnect): Instead of one giant silicon die, small silicon “bridges” (LSI tiles) only connect adjacent chiplets, with a cheaper organic substrate underneath. Reticle limit is broken; Blackwell B100/B200/GB200 are the first generation built on CoWoS-L. Bigger packages, more HBM cubes, but the LSI bridges become a precision-alignment problem.
- CoPoS (Chip-on-Panel-on-Substrate): Move from round 12-inch wafers to rectangular panel-level packaging (similar to display substrates, ≈600 × 600 mm). Far more package area per process step; intended for Rubin-and-beyond GPUs that exceed even CoWoS-L’s effective reach. Pilot lines from 2026, volume from 2028+.
Why this matters at the system level: CoWoS supply is the single largest bottleneck in AI compute. TSMC’s CoWoS-L capacity has been the rate-limiter for every Nvidia ramp from H100 onward. Cloud GPU price floors and HBM allocation politics ultimately reflect “how many CoWoS-L slots TSMC can finish per quarter.”
5. Market structure and supply chain
The AI-chip value chain has the classical semiconductor split, plus a few new roles:
- IDM (Integrated Device Manufacturer): Designs and fabricates — Intel, Samsung, SK hynix.
- Fabless: Designs, outsources fab — Nvidia, AMD, MediaTek, plus the rising hyperscaler ASIC designs (Google TPU, AWS Trainium, Meta MTIA).
- Chipless (IP): Licenses design blocks — ARM, Synopsys, Cadence. RISC-V is the open-source contender.
- Foundry: TSMC (≈ 60% global), Samsung Foundry, GlobalFoundries. TSMC is the single point of failure for the AI economy: it makes the GPU die, owns CoWoS, and is the only supplier qualified for cutting-edge HBM logic dies.
- OSAT: ASE, Amkor, JCET — back-end assembly and test, though HBM and CoWoS have pulled most value back into the foundry.
- Memory (“memory IDMs”): SK hynix, Samsung, Micron — the three vendors that matter for HBM.
Forecasts (industry consensus, late-2025 baseline):
- AI chip market: ≈49% CAGR from 2025 to ≈**$1T by 2029** (HBM included).
- HBM: ≈52B (2026); HBM4 ramps from 2H 2026.
- AI infrastructure capex (data centers + chips combined): crosses $1T in 2026.
- Memory super-cycle: DRAM revenue ≈445B (2029); ASP downside historically curbed because HBM cannibalizes general DRAM capacity at the foundries.
The single most important geopolitical fact: Nvidia → TSMC → SK hynix/Samsung is a three-link Asian supply chain on which the entire frontier of AI training depends.
6. Roadmap (2024–2030)
| Year | Nvidia GPU | HBM | Memory/GPU | Bandwidth | Packaging |
|---|---|---|---|---|---|
| 2022 | H100 (Hopper, 4N) | HBM3 8-Hi | 80 GB | 3.4 TB/s | CoWoS-S |
| 2024 | H200 / GH200 | HBM3e 8-Hi | 141 GB | 4.8 TB/s | CoWoS-S |
| 2025 | B100 / B200 / GB200 (Blackwell, 4NP) | HBM3e 8-Hi | 192 GB | 8.0 TB/s | CoWoS-L |
| 2025 H2 | B300 Ultra / GB300 | HBM3e 12-Hi | 288 GB | ≈8.2 TB/s | CoWoS-L |
| 2026 H2 | R100 (Rubin) | HBM4 8-Hi | 288 GB | ≈15 TB/s | CoWoS-L → CoPoS pilot |
| 2027 H2 | R300 (Rubin Ultra) | HBM4e 12-Hi | up to 1 TB | — | CoPoS |
| 2028+ | Feynman | HBM4e+ | — | — | CoPoS / Hybrid bonding mainstream |
Two themes ride underneath: (1) memory per GPU roughly doubles per generation while bandwidth grows even faster; (2) hybrid-bonded HBM4e and panel-level CoPoS are the bets on which the post-2027 industry will be won.
Where this connects to the book
- Chapter 2 introduces the semiconductor value chain (IDM/Fabless/Foundry/OSAT) and packaging primitives (TSV, MR-MUF, hybrid bonding). This page extends that with the CoWoS-S/L/CoPoS evolution and the wafer-yield economics.
- Chapter 8 uses these chips at data-center scale — 1 GW = 500k GPUs = 89k wafers — and locates them in the wider AI stack.
- AI Data Center: Building the AI Factory is the companion overview on what happens once you have the chips: power, cooling, network, economics.
See also
Foundations (chip + memory history):
- shannon-1938-symbolic-analysis-of-relay-and — Boolean circuits as the logical basis of every transistor
- neumann-1945-first-draft-of-a-report — The von Neumann bottleneck whose modern echo is the memory wall
The connectionist demand pull:
- vaswani-2017-attention-is-all-you-need — Attention’s Q/K/V matrix multiplications are what HBM bandwidth feeds
- kaplan-2020-scaling-laws-for-neural-language — Scaling laws are the demand curve for HBM and CoWoS-L
- brown-2020-language-models-are-few-shot-learners — GPT-3 at 175B parameters made HBM mandatory
- hu-2021-lora-low-rank-adaptation-of-large — LoRA reduces inference-time memory pressure
- han-2023-parameter-efficient-fine-tuning-for-large-models — PEFT trades compute for less HBM occupancy