The von Neumann Bottleneck Is the AGI Bottleneck — Why All Modern Chips Are Still Stuck in 1945
I published a deep dive into APU vs GPU vs NPU architectures earlier today. It explains how AMD packs CPU cores and GPU CUs onto the same die, how Apple’s unified memory eliminates the PCIe copy, and how NPU systolic arrays achieve 10× better energy efficiency than GPU tensor cores.
But there’s a deeper pattern that post hints at without naming: every single one of these architectures is a von Neumann machine. And that matters — not just for benchmark scores, but for whether we can actually build AGI on silicon.
John von Neumann’s 1945 EDVAC paper described an architecture where the program and data share a single memory, accessed through a single bus. Eighty years later, that bus is still there. We’ve widened it, shortened it, and wrapped it in coherence protocols — but we haven’t removed it.
What the von Neumann bottleneck actually is
The von Neumann bottleneck has three dimensions that compound:
| Dimension | What it means | Why it hurts |
|---|---|---|
| Bandwidth | Bytes per second across the bus | ALUs sit idle waiting for data |
| Latency | Nanoseconds per round-trip | Every cache miss stalls the pipeline |
| Energy | Picojoules per bit moved | Moving data costs more than computing on it |
The energy dimension is the one most people miss. A 64-bit FP multiply costs about 3.7 pJ on modern silicon. Moving the same 64 bits from DRAM costs about 1,000-2,000 pJ — roughly 300-500× more energy to move the data than to compute on it. This is why anything memory-bound burns power while doing very little actual math.
Every chip in the APU post is von Neumann
The Ryzen APU: closer, but still separated
The APU’s Infinity Fabric eliminates the PCIe copy between CPU and GPU. That’s a real improvement — pointer-passing instead of DMA transfer. But both CPU cores and GPU CUs still sit on one side of the memory controller, and DRAM sits on the other. The fabric reduces latency (no PCIe round-trip) but doesn’t increase bandwidth (still ~51 GB/s shared DDR4). The compute units still fetch data from a physically distant memory pool through a shared bus.
The Apple M4: the cleanest von Neumann you can buy
Apple’s unified memory is the most elegant implementation of the von Neumann model in consumer silicon. Single pool of LPDDR5x, every IP block (CPU, GPU, NPU, Media Engine) sees the same address space, and the System Level Cache acts as a coherence sponge. But the fundamental architecture hasn’t changed: compute is on one side of the memory bus, data is on the other. The SLC helps by caching hot lines, but the SLC itself is a cache — it mitigates the bottleneck, it doesn’t remove it.
The NPU: gets closest to breaking free
The NPU’s systolic array is the least von Neumann thing in a modern SoC. There’s no instruction fetch, no program counter, no branch prediction. The computation IS the dataflow — weights are stationary in the array cells, activations pulse through, and partial sums ripple out. This eliminates the instruction-fetch half of the bottleneck entirely.
But the NPU still has to load weights from somewhere. For a 7B-parameter LLM quantized to INT4, that’s ~3.5 GB of weights that have to be loaded into the NPU’s scratchpad SRAM before inference can start. The SRAM is fast (~1 TB/s per bank), but it’s small (~10-50 MB). The weights ultimately come from DRAM, across the system bus, through the von Neumann bottleneck.
The AGI bandwidth math
Here’s where it gets concrete. Let’s ask what a brain-scale neural simulation actually requires:
Now compare this to the best hardware we actually have:
| Hardware | Memory bandwidth | Memory capacity | Power | % of brain bandwidth |
|---|---|---|---|---|
| Human brain | ~50 TB/s (estimate) | ~200 TB equiv | 20W | 100% |
| NVIDIA H200 | 4.8 TB/s | 141 GB | 700W | 9.6% |
| AMD MI300X | 5.3 TB/s | 192 GB | 750W | 10.6% |
| Apple M4 Max | 546 GB/s | 128 GB | ~50W | 1.1% |
| Ryzen 7 7730U | 51 GB/s | 32 GB | 28W | 0.1% |
A single H200 hits ~10% of the brain’s estimated bandwidth — at 35× the power draw. And that’s bandwidth, not intelligence. That’s just moving the data around. The bandwidth gap means you physically cannot stream brain-scale synaptic updates through a von Neumann bus fast enough for real-time operation, no matter how fast your ALUs are.
This is the von Neumann bottleneck at AGI scale. Not “can we build enough transistors.” Not “can we write the software.” Can we get data from memory to compute and back fast enough, using little enough energy, to match what biology does with 20 watts of glucose?
The memory wall: why inference is already bottlenecked
You don’t need AGI to see the von Neumann bottleneck in action. It’s already the dominant constraint for LLM inference today.
During auto-regressive token generation with batch size 1 (a single user chatting with an LLM), the GPU does almost no compute. It spends 95%+ of its time waiting for weights to arrive from VRAM. A 70B model has 140 GB of weights in FP16. Every single token requires streaming all 140 GB through the memory bus to perform matrix-vector multiplies where the arithmetic-to-data ratio is abysmally low.
An H100 rated at 989 TFLOPS (FP16) achieves perhaps 5% utilization during token generation. The other 95% of the silicon is dark — powered on, consuming 700W, producing zero math, just waiting for the von Neumann bus to deliver the next chunk of weights.
This is the roofline model in practice: performance is bounded not by how fast the chip can compute, but by how fast data can cross the memory interface. You hit the memory-bandwidth roof long before you approach the compute roof.
What would a non-von-Neumann chip look like?
Compute-in-Memory (CIM)
Instead of moving data from DRAM to ALU, CIM embeds multiply-accumulate circuits inside the memory array itself. A memristor crossbar array performs analog matrix-vector multiplication in O(1) time by exploiting Ohm’s Law and Kirchhoff’s Current Law: the current at each column is the dot product of input voltages and memristor conductances. No data movement, no bus, no fetch-decode cycle.
Samsung’s HBM-PIM and Mythic’s M1076 analog matrix processor are shipping products that do this. The energy efficiency is staggering — Mythic claims Tensor Core-class matrix math at 1/10th the power. But the precision is limited (analog noise), the capacity is small (a few GB per chip), and the programming model is radically different from anything software engineers know.
Neuromorphic: computing like biology
Intel’s Loihi 2 and IBM’s NorthPole abandon the von Neumann model entirely. Instead of a central clock, instruction fetch, and shared memory bus, they use spiking neural networks where computation is event-driven, asynchronous, and localized. Each “neuron” has its own small memory (synaptic weights) and only activates when input spikes exceed a threshold.
IBM’s NorthPole packs 22 billion transistors into a single chip, with 256 cores each containing local memory. There is no off-chip DRAM — all weights are stored on-die, distributed across cores. The energy per synaptic operation is measured in picojoules, not nanojoules. It’s the closest silicon has come to the brain’s power envelope.
The catch: these chips don’t run PyTorch. They don’t run transformers natively. The software stack is research-grade, the model formats are proprietary, and the ecosystem is tiny compared to CUDA.
Wafer-scale: if you can’t narrow the bus, remove the distance
Cerebras takes the blunt-force approach: build a chip the size of an entire silicon wafer (46,225 mm² — 56× larger than an H200 die), put 900,000 cores on it, and keep all model weights in on-chip SRAM. The WSE-3 stores 44 GB on-die with 21 PB/s of aggregate memory bandwidth. That’s ~4,000× the effective bandwidth of an H200 for on-chip data.
The limitation is obvious: 44 GB isn’t enough for a 70B-parameter model. You still need a cluster. And wafer-scale chips have yield problems — a single defect can kill a $2M wafer.
Photonic: compute at the speed of light
Lightmatter’s Envise and Lightelligence’s PACE use Mach-Zehnder interferometers to perform matrix multiplication with photons instead of electrons. Light doesn’t generate resistive heat, doesn’t suffer from RC delay, and can be wavelength-multiplexed (multiple computations on different colors of light in the same waveguide). The energy per MAC is measured in femtojoules — 1,000× less than electronic MACs.
Photonic computing remains early-stage. Laser sources, modulators, and photodetectors add bulk and cost. Precision is limited by shot noise. But the physics is sound, and the energy math is compelling.
So: von Neumann kills AGI?
Not exactly. The von Neumann bottleneck makes brute-force AGI — scaling up today’s transformer architectures on today’s GPU clusters until they match brain-scale — thermodynamically infeasible. The energy gap is too large, the bandwidth gap is too large, and neither is closing fast enough.
But three things might change the calculus:
Path 1: Smarter algorithms, not bigger chips
The brain doesn’t compute softmax attention across 128K tokens every forward pass. It uses recurrent circuits, sparse activation, and local connectivity. State-space models (SSMs) like Mamba and RWKV achieve linear-time sequence processing — O(n) instead of O(n²) — by maintaining a compressed hidden state that represents the entire history without attending to every token.
If AGI-level cognition can be achieved with architectures that are inherently less memory-bandwidth-hungry than dense transformers, the von Neumann bottleneck matters less. You still need bandwidth, but you need far less of it.
Path 2: Mixture-of-Experts and sparse compute
The brain keeps ~99% of its neurons silent at any moment. A transformer loads every parameter for every token. Mixture-of-Experts (MoE) architectures route each token to a small subset of “expert” layers — a 1.8T-parameter MoE model might only activate 40B parameters per token, achieving brain-like sparsity. This dramatically reduces the memory bandwidth demand: you only load the active expert weights, not the entire model.
Path 3: The end of the von Neumann era
Transistor scaling hits fundamental physical limits around 1nm — at that scale, quantum tunneling makes gates leaky, and Dennard scaling (the observation that smaller transistors use proportionally less power) has been dead since 2006. The von Neumann architecture was designed for an era when transistors were discrete vacuum tubes and memory was mercury delay lines. It’s a miracle it scaled this far.
The next architecture won’t look like a CPU with attached DRAM. It will look like a memory chip that can also compute — or a photonic chip that computes with light, or a neuromorphic chip that spikes like neurons. In all cases, the separation between compute and memory disappears.
What this means for the APU discussion
The APU post celebrated the elimination of the PCIe bus — and rightly so. Zero-copy buffer sharing through Infinity Fabric is a genuine architectural improvement over discrete GPUs. But it’s an improvement within the von Neumann paradigm, not a departure from it.
| Architecture | Bus removed? | Bottleneck status |
|---|---|---|
| Discrete GPU | No — PCIe 4.0 ×16 (32 GB/s) | Worst: two memory pools, explicit copies, PCIe tax |
| APU (Ryzen 7730U) | PCIe copy eliminated | Better: unified pool, pointer-passing, still bus-limited at 51 GB/s |
| Apple M4 | All external buses removed | Best von Neumann: unified LPDDR5x + SLC, still compute-memory separated |
| NPU (XDNA 2) | Instruction bus eliminated | Partially post-von-Neumann: dataflow compute, but DRAM still feeds SRAM |
| Loihi 2 / NorthPole | All buses eliminated | Post-von-Neumann: compute IS memory, event-driven, no fetch |
The NPU’s systolic array is the most interesting case. It’s halfway out of the von Neumann model — it eliminated the instruction side of the bottleneck but not the data side. A fully post-von-Neumann architecture eliminates both.
The post ends where it should: with the recognition that the chip in your laptop, however advanced, is still organized around John von Neumann’s 1945 architecture. The AGI bottleneck is not “can we put enough transistors on a chip.” It’s “can we reorganize those transistors so the distance between memory and compute collapses to zero.” Biology figured this out hundreds of millions of years ago — neurons that are both storage and processor, connected by synapses that are both wire and weight. Silicon is still catching up.
The APU vs GPU vs NPU post: /apu-vs-gpu-vs-npu-architecture. Brain bandwidth estimates from the Whole Brain Emulation roadmap and AI Impacts. GPU bandwidth numbers from vendor specifications. Energy-per-op comparisons from Bill Dally (NVIDIA) and Horowitz (Stanford) VLSI energy tables.