Memory Bandwidth AI Speed: The Hidden Factor

The Need for Speed: Decoding Token Generation Constraints

When you buy a computer for gaming, you look at the graphics card’s “GHz” or “VRAM.” When you buy a computer for video editing, you look at the “CPU Cores.” But when you buy a computer for OpenClaw and local AI, there is only one metric that truly defines your experience: Memory Bandwidth.

In 2026, memory bandwidth (measured in Gigabytes per second, or GB/s) is the real “speed limit” of intelligence. You can have the fastest processor in the world, but if your memory bandwidth is narrow, your AI will feel like it’s thinking through a straw.

For additional context on the core components discussed above, consider reviewing how computing bandwidth is calculated.

Learning Objectives

By the end of this guide, you will understand:

1. Why Memory Bandwidth (GB/s) is the primary bottleneck for local AI speed.
2. The Difference between the Compute-Bound “Prefill” phase and the Memory-Bound “Decode” phase.
3. How to Calculate the theoretical maximum tokens per second (tok/s) for any hardware.
4. Why Unified Memory architectures (Apple M-Series, AMD Strix Halo) excel at AI inference.

The Bottleneck: Decode Phase vs. Prefill Phase

AI inference happens in two stages, and they use your hardware in very different ways.

1. The Prefill Phase: This is when the AI reads your prompt. It’s a massive burst of calculation. This is “Compute-Bound,” meaning it uses your GPU or NPU’s raw muscle.
2. The Decode Phase: This is when the AI writes the response, one word (token) at a time. To generate each word, the AI must “read” every single parameter of the model from your RAM. This is “Memory-Bound.”

Because the “Decode” phase is what you actually see and wait for, your “Tokens per Second” (tok/s) speed is almost entirely determined by how fast your RAM can feed data to the processor.

Fire Hose vs. Garden Hose: A Visualization

Imagine your AI model is a giant fire (the task). Your processor is the fire truck (the compute), but the Memory Bandwidth is the hose connecting the truck to the water supply (the data).

• The Snapdragon X Elite (135 GB/s): A high-quality garden hose. Great for small models (8B), but it will struggle to put out a “70B model” fire quickly.
• The AMD Strix Halo (256 GB/s): A commercial-grade hose. Now you’re talking. It can handle medium-to-large models with a respectable flow.
• The Apple M5 Max (614 GB/s): A literal fire hose. It delivers data so fast that the processor can barely keep up. This is why a high-end Mac can “talk” as fast as you can read.

The Math of Speed: How GB/s Translates to tok/s

You can actually calculate your theoretical maximum speed using a simple formula:

(Memory Bandwidth in GB/s) / (Model Size in GB) = Maximum Tokens per Second

Let’s look at a quantized 32B model (which weighs about 20GB):

• On a Standard PC (50 GB/s): 50 / 20 = 2.5 tok/s (Glacial).
• On a Snapdragon Laptop (135 GB/s): 135 / 20 = 6.7 tok/s (Slow).
• On an AMD Strix Halo (256 GB/s): 256 / 20 = 12.8 tok/s (Good).
• On an Apple M5 Max (614 GB/s): 614 / 20 = 30.7 tok/s (Human Reading Speed).

This is why Unified Memory is so critical. By soldering the memory directly next to the chip and using wide data paths, these systems deliver bandwidth that traditional “sticks of RAM” simply cannot match.

Comparing the 2026 Titans

Quantization’s Secret Service: Reducing the Load

You can “cheat” the bandwidth limit using Quantization. By compressing a model from 16-bit to 4-bit, you make the model 4x smaller.

Since the model is smaller, the processor has to read less data from the RAM to generate a token. This effectively quadruples your token speed without changing your hardware. This is why we focus so heavily on Quantization Guides for local AI.

Frequently Asked Questions

Is bandwidth more important than RAM capacity?

Capacity determines if the model runs. Bandwidth determines how fast it runs. If you have 128GB of RAM but only 50 GB/s bandwidth, a 70B model will load, but it will be painfully slow.

Why is laptop bandwidth so much lower than desktop GPUs?

Desktop GPUs use specialized memory (GDDR7) with massive 384-bit buses that consume 400 watts. Laptops have to balance speed with battery life, which is why Unified Memory is such a breakthrough–it offers high speed at low power.

Does CPU speed matter at all?

Yes, but only for the “Prefill” phase. Once the AI starts typing, your CPU is mostly just waiting for the RAM to send it the next set of data.

Can I increase my bandwidth?

Usually no. Bandwidth is a physical property of the memory bus on your motherboard or chip. This is why you must choose the right chip from the start.

What is ‘LPDDR5X-8533’?

It is the latest standard of high-speed laptop memory. The “8533” refers to the speed in Mega-transfers per second. The higher the number, the higher the bandwidth.

Is dual-channel RAM enough for AI?

For small 8B models, yes. For 32B or 70B models, you really want a Quad-Channel or massive “Unified” bus to avoid a bottleneck.

Does SSD speed affect AI speed?

Only for the initial “Load” time. Once the model is in your RAM, the SSD doesn’t do anything.

Why is the M5 Max so much faster than the M5 Pro?

The “Max” series typically doubles the memory bus width (e.g., from 256-bit to 512-bit), which instantly doubles the bandwidth and the AI speed.

What is ‘Arithmetic Intensity’?

It’s a fancy way of saying “How much work the CPU does per byte of data.” In AI inference, the arithmetic intensity is low, which is why bandwidth is the bottleneck.

Conclusion

Memory bandwidth is the invisible hand that guides your AI experience. If your OpenClaw agent feels “Laggy” or “Slow to respond,” it’s likely not a lack of CPU power–it’s a lack of bandwidth.

When choosing your next AI workstation, look past the marketing hype and focus on the GB/s. It’s the difference between an AI that works for you and an AI that makes you wait.

Ready to see how bandwidth affects specific models? Read our Deep Dive into 70B Model Performance.