AI COMPUTE
AI compute encompasses specialized silicon, high-bandwidth memory, and interconnect fabrics powering deep learning training and inference economics at scale.
Part of: AIAI compute refers to the specialized hardware architecture, semiconductor packaging, and high-speed networking fabric engineered to execute the massively parallel matrix multiplications and tensor algebra underlying deep learning training runs and real-time model inference.
The Silicon Spectrum: Architectures Compared
Modern neural network workloads are processed across diverse specialized silicon architectures optimized for distinct computational profiles:
| Architecture | Representative Silicon | Execution Paradigm | Memory Architecture | Primary Strength | Architectural Bottleneck |
|---|---|---|---|---|---|
| GPU (Graphics Processing Unit) | NVIDIA H100/B200, AMD MI300X | Massively parallel SIMD with dedicated Tensor Cores | Multi-stack HBM3e/HBM4 via CoWoS packaging | Universal framework support (CUDA), high peak FLOPs | High thermal dissipation, CoWoS packaging limits |
| TPU (Tensor Processing Unit) | Google TPU v5p / TPU v6 | 2D Systolic Arrays for 128x128 matrix multiplication | Stacked HBM with custom inter-die optical links | Extreme cluster scalability, deterministic execution | Locked to custom compiler ecosystems (XLA/JAX) |
| LPU / Custom ASIC | Groq LPUs, Tenstorrent | Deterministic Tensor Streaming Core | Ultra-fast on-chip SRAM (no off-chip HBM) | Ultra-low latency single-user token decoding | Extremely low memory capacity per chip (requires hundreds of chips) |
| CPU (Central Processing Unit) | Intel Xeon, AMD EPYC | Superscalar out-of-order execution | High-capacity system DDR5 DRAM | Dynamic control flow, preprocessing, OS management | Low matrix arithmetic throughput per watt |
The Roofline Model: Compute-Bound vs. Memory-Bound Regimes
Evaluating hardware efficiency requires the Roofline Model, which plots operational intensity (arithmetic operations per byte of DRAM traffic) against achievable performance:
Attainable Performance [FLOP/s] = min(Peak Compute Performance, Peak Memory Bandwidth × Operational Intensity)
In large language models, compute behaves differently across execution phases:
- Pre-fill Phase (Context Processing): When a model processes a multi-thousand-token prompt, it calculates attention in parallel across all tokens. This phase exhibits high operational intensity and is strictly compute-bound, saturating tensor cores.
- Decode Phase (Token-by-Token Generation): During autoregressive generation, the model must read all parameters (e.g., 70 billion weights) from High Bandwidth Memory to output a single token. This phase has very low operational intensity and is memory-bandwidth bound, leaving execution units idle while waiting on DRAM bus transfers.
This memory wall explains why silicon vendors prioritize HBM bandwidth over raw theoretical compute density for inference-optimized systems.
Interconnect Topologies and Cluster Scaling
Individual accelerators cannot fit frontier models within their localized memory. Distributing a multi-hundred-billion parameter model requires partitioning weights and tensor activations across thousands of chips, turning networking into a critical performance governor:
- Intra-Node Interconnect (Scale-Up): Protocols such as NVIDIA NVLink provide up to 1.8 TB/s bidirectional bandwidth per GPU, enabling multi-GPU nodes to treat memory as a unified coherent cache.
- Inter-Node Fabric (Scale-Out): Nodes connect across clusters of 10,000 to 100,000 accelerators using Remote Direct Memory Access (RDMA) over InfiniBand or RoCE (RDMA over Converged Ethernet). Without non-blocking leaf-spine network topologies, gradient synchronization stalls cluster training efficiency.
The Geopolitical and Capital Allocation Constraints
AI compute is constrained by unprecedented supply-chain centralization and macroeconomic friction. Advanced sub-3nm nodes rely on extreme ultraviolet (EUV) lithography monopolies (ASML) and foundries concentrated in Taiwan (TSMC). Advanced packaging techniques, such as Chip-on-Wafer-on-Substrate (CoWoS), constitute persistent physical bottlenecks.
Simultaneously, the enormous capital cost of hyperscale clusters alters enterprise capital allocation. Cloud providers and frontier AI labs commit tens of billions of dollars to multi-gigawatt datacenter builds, where access to grid power and energy economics replaces silicon availability as the ultimate binding constraint on computational growth.
Part of the knowledge graph at The Best Blog Ever — reference definitions for ideas that matter.
