AI Inference Is Now a Memory Problem, Not a Compute Problem

The AI infrastructure conversation has centered on GPU count and raw FLOPs for years. That framing is increasingly out of date. Inference now accounts for roughly two-thirds of all AI compute in 2026, up from about a third in 2023, and the bottleneck that actually constrains inference at scale has shifted to memory bandwidth, not compute, because inference workloads are memory-bound in a way training workloads generally aren't.

Where hyperscaler money is actually going

Memory is projected to absorb roughly 30% of hyperscaler AI data-center spending in 2026, a fourfold jump from 2023, specifically because of this shift toward inference-heavy, memory-bandwidth-bound workloads. The four major US hyperscalers committed a combined $630 to $690 billion in 2026 capex, with the majority now aimed at inference infrastructure rather than the training supercomputers that dominated capex conversations two years ago. Inference now represents roughly 80% of AI infrastructure budgets industry-wide.

Why this matters if you're not buying GPUs yourself

Even if your team never touches a data center directly, this shift explains a lot of what you're actually experiencing as an AI SaaS buyer or self-hoster: latency and cost at scale increasingly track memory bandwidth availability, not headline GPU specs. If you're evaluating infrastructure vendors, or doing the math on self-hosting your own model, the relevant constraint to ask about is memory bandwidth and HBM supply, not just GPU count, because that's where the actual bottleneck for inference workloads has moved, and it's the harder constraint to simply buy your way out of with more chips.

This also explains part of why inference pricing hasn't fallen as uniformly as raw compute cost trends would suggest. A GPU-count-based mental model of infrastructure cost is increasingly the wrong model for reasoning about where your inference bill is actually going.