For decades, computing performance was easy to reduce to a few headline specifications. CPUs were sold on clock speed and core count, GPUs on shader throughput and later teraFLOPS , memory on frequency, and storage on sequential transfer rates. Those numbers still have value, but modern processors have become so extraordinarily capable of performing arithmetic that another problem increasingly dictates how much of that theoretical performance can actually be used: getting data where it needs to go, when it needs to be there .

A modern CPU core can execute multiple instructions every clock cycle. A high-end GPU contains thousands of arithmetic units operating in parallel. AI accelerators push matrix multiplication throughput into the petaFLOP range. Yet none of these units can calculate anything useful without operands, instructions, textures, geometry, model weights, intermediate results, or some other form of data.

This creates a fundamental imbalance. In effect, arithmetic throughput has scaled enormously throughout the years, but moving data is comparatively much more expensive in terms of latency, bandwidth, energy, and physical chip resources. As a result of this, many modern workloads spend a surprising amount of effort simply trying to keep execution units fed .

That does not mean that every workload is "memory-bound." Far from it, in fact. Performance can still be constrained by compute throughput, instruction dependencies, branch prediction, synchronization, software overhead, storage, networking, or numerous other factors. What has changed is that understanding memory behavior has become indispensable to understanding modern computing performance.

And "memory performance" itself goes far beyond frequency or capacity. Latency, bandwidth, locality, cache capacity, memory-level parallelism, access patterns, prefetching, capacity, interconnect topology, and even where the data physically resides all interact with one another.

To understand why, we have to start with perhaps the most important distinction of all: how the data is being accessed.

Imagine reading a 1 GB file from beginning to end. The location of the next piece of data is perfectly predictable. Once the system realizes what you are doing, it can start fetching future data before it is explicitly requested .

This is a sequential access pattern, and modern computers are extremely good at handling it.

CPUs contain hardware prefetchers specifically designed to recognize predictable memory streams. DRAM controllers can keep multiple transactions in flight and reorder requests. SSD controllers distribute accesses among many NAND dies and channels. GPUs can combine memory requests from neighboring threads into larger transactions .

The hardware can essentially build a pipeline. Unfortunately, random access is a much harder hurdle to conquer.

Consider traversing a linked list where each entry contains the address of the next one. The processor may have no idea where the next access will go until the current one completes. That destroys much of the opportunity for prefetching and prevents the machine from efficiently overlapping accesses. This is why two workloads can move the same amount of data yet still run at dramatically different speeds .

The distinction becomes even more important when latency dependencies are involved. If a CPU requests four unrelated cache lines , it may be able to keep several cache misses in flight simultaneously and overlap much of their latency. If request B depends on the result of request A, however, B cannot even begin until A finishes.

This is commonly illustrated through pointer chasing . Each load reveals the address of the following load, forcing the processor to pay much more of the actual memory latency instead of hiding it through speculation or parallelism.

The same principle appears everywhere in computing. A database performing small unpredictable lookups presents a very different challenge from copying a large file. A GPU sampling coherent neighboring texels behaves differently from thousands of threads chasing unrelated addresses. An SSD reading hundreds of kilobytes sequentially sees a fundamentally different workload from retrieving scattered 4 KB blocks.

This also explains why we cannot simply "turn random access into sequential access” for improved performance. Still, software developers absolutely try to do so. Databases reorganize data. Game engines batch work. GPU algorithms reorder requests. Compilers change data layouts. Structures-of-arrays can sometimes replace arrays-of-structures . Sorting may transform scattered accesses into relatively coherent ones.

But doing so is not always possible. In fact, if the next address depends on the result of the current operation, then the dependency is real. If a ray can bounce toward an unpredictable part of a 3D scene, then the GPU cannot know every future access beforehand. If a database user asks for a specific record, then reading a terabyte of adjacent records first would obviously defeat the purpose.

The objective therefore becomes less about eliminating random accesses and more about making expensive accesses happen as rarely, predictably, and concurrently as possible.

Fortunately, real software rarely accesses completely arbitrary data all the time. In fact, programs usually tend to exhibit both spatial locality and temporal locality .

Spatial locality means that if a program accesses one piece of data, it is relatively likely to need nearby data soon afterward. Temporal locality means that recently accessed data has a reasonable chance of being accessed again.

A loop processing an array has excellent spatial locality. Frequently referenced game world data may have strong temporal locality. Instructions inside a hot loop — a section of computer code that runs very often — exhibit both types of locality. These properties are what make caches viable .

When a CPU needs a byte from DRAM, it generally does not transfer only that byte. Data is moved through the cache hierarchy in fixed-size blocks called cache lines. On mainstream x86 CPUs, these are typically 64 bytes. AMD's own optimization documentation , for example, describes software placement and access considerations around 64-byte cache lines.

If the program subsequently accesses neighboring data, it may already be sitting inside the cache. This introduces another critical concept: the working set .

A program may technically allocate tens of gigabytes of memory while repeatedly operating on only a small portion of it during a particular phase. If that active working set fits inside a fast cache, performance can be excellent. Increase the working set slightly beyond the available cache capacity, and suddenly many more accesses spill into a slower level of the hierarchy. This can create surprisingly abrupt performance changes.

It is also why cache capacity improvements sometimes produce enormous gains and sometimes almost none. Increasing a cache from 32 MB to 96 MB is extremely valuable if it allows a 60 MB frequently reused working set to remain on-chip. It offers considerably less benefit if the workload repeatedly streams through several gigabytes of data that it never touches again.

Caches therefore work best when software gives them something worth remembering .

This raises an obvious question. If cache is so useful, why not simply build an enormous ultra-fast cache and forget about slower memory? Because the properties we want from memory conflict with one another .

The closest CPU caches are made from SRAM and optimized for extremely low access latency and very high bandwidth. But SRAM consumes a considerable amount of silicon area. Larger structures also require more wires, longer physical paths, more lookup logic, and frequently more energy to access.

As capacity increases, maintaining the speed of a tiny local cache becomes progressively harder, and this is the main reason why the hierarchy exists.

An L1 cache is small enough to sit extremely close to an execution core and serve data with very low latency. L2 provides more capacity at somewhat higher latency. The last-level (usually L3 on modern CPUs) cache provides still more capacity and may be shared between cores, but accessing it is generally slower again. Miss every level and the request eventually has to travel to the much slower DRAM.

Cache design also goes far deeper than capacity. Associativity affects where data blocks may be placed and can reduce conflict misses, but more lookup flexibility adds complexity. Cache banking can increase parallel access capability. Multiple ports improve throughput but cost area and power. Private caches give individual cores fast local access, while shared caches can use total capacity more efficiently and simplify certain forms of data sharing.

Intel has discussed this exact design tension in its architectural research , describing cache organization as a balancing act between size, associativity, latency, bandwidth, private versus shared organization, scalability, and energy efficiency.

Then there is coherence . When several CPU cores cache copies of the same memory, the system must ensure that they do not operate indefinitely on contradictory versions of the data. Maintaining this coherent view generates additional communication and bookkeeping, particularly as core counts rise. More cache therefore provides no universal free lunch .

AMD's 3D V-Cache technology is an excellent modern example of engineers finding another way through the trade-off. Instead of dramatically expanding the CPU core die horizontally, AMD stacks an additional cache die vertically. Its second-generation implementation adds a 64 MB L3 cache die using direct copper-to-copper bonding and through-silicon vias .

The popularity of Ryzen X3D processors in gaming provides a practical demonstration of what happens when additional cache aligns with a workload's working set. Games constantly manipulate world state, draw call data, physics structures, animation, visibility information, and numerous other data structures . Keeping more of that frequently reused information close to the CPU can reduce expensive trips to DRAM.

But even here, the gains vary considerably between games. Some already fit their critical data into conventional/smaller caches. Others are constrained elsewhere. Some benefit enormously.

That variability is exactly what memory hierarchy theory predicts .

Eventually, data that is not available in cache must come from main/system memory, or DRAM.

DRAM offers far greater capacity than SRAM at a fraction of the cost per bit, but it is considerably slower. Engineers therefore compensate with bandwidth and parallelism.

Modern DRAM is divided internally into banks and bank groups. Within those banks are rows of memory cells. Accessing data typically involves activating a row into a row buffer before reading or writing the desired columns.

This leads to three simplified scenarios: if the required row is already active, the controller gets a row-buffer hit . If no relevant row is open, one has to be activated. If the bank currently has the wrong row open, the existing row may first need to be closed before the new one can be activated.

The memory controller therefore does much more than blindly forward CPU requests. It tracks outstanding transactions and attempts to schedule them efficiently across channels , ranks , bank groups and banks while respecting an extensive collection of memory timing restrictions.

This is one reason memory bandwidth and memory latency should not be treated as interchangeable measurements.

In fact, latency describes how long a particular operation takes. Bandwidth describes how much data can be transferred over time. A system can have enormous aggregate bandwidth and still be relatively poor at servicing a chain of dependent random memory accesses.

Conversely, a workload capable of keeping hundreds of independent transactions in flight may tolerate substantial latency while still approaching impressive overall throughput.

DDR5 memory illustrates how strongly modern memory design depends on parallelism. Compared with DDR4, DDR5 increases the number of bank groups, doubles the default burst length from eight to sixteen, and divides a DIMM into two independent subchannels. Micron specifically notes that increasing bank groups improves the probability of using less restrictive bank-group timings and allows more memory pages to remain open simultaneously.

It also explains why DRAM tuning sometimes gives frustratingly inconsistent results. Increasing transfer rate raises theoretical bandwidth. Tightening timings can reduce various delays. Adding ranks may affect available parallelism. The memory controller, CPU architecture, and application behavior then determine how much of any improvement reaches actual software.

A compression benchmark, game, database, and large file copy can therefore react very differently to the same memory upgrade .

GPUs require enormous bandwidth because thousands of execution lanes may request data concurrently. Consumer GPUs therefore use GDDR memory connected through comparatively wide interfaces and operating at very high transfer rates.

AI and HPC accelerators push this still further with High Bandwidth Memory , or HBM .

Instead of placing memory packages around the processor and driving them through conventional board-level interfaces, HBM stacks DRAM dies and uses extremely wide interfaces located very close to the processing device through advanced packaging.

The result is extraordinary bandwidth and good energy efficiency per transferred bit, although at substantially greater packaging complexity and cost.

AMD's Instinct MI355X provides a useful indication of the scale involved today. A single accelerator carries 288 GB of HBM3E and exposes up to 8 TB/s of theoretical memory bandwidth . Eight-GPU systems therefore contain 2.3 TB of HBM.

Those figures would be absurdly excessive for a desktop CPU. For modern AI, however, feeding the compute hardware has become one of the central architectural problems .

Modern generative artificial intelligence — mainly in the form of Large Language Models (LLMs) — provides perhaps the clearest example of why raw arithmetic throughput can tell only part of the story.

Large neural networks involve huge numbers of matrix operations, which is precisely what modern GPUs and dedicated AI accelerators are built to execute. But the processor must still retrieve model weights , activations , and intermediate data before those arithmetic units can do useful work.

Training is especially memory intensive because the system has to maintain far more than a model's weights. Depending on the training technique and optimizer , it may also need gradients , optimizer states, activations retained for backpropagation , and temporary workspace.

Inference removes some of those requirements but creates its own memory challenges.

Take a large language model with seven billion parameters. At 16-bit precision, the weights alone require roughly 14 GB before accounting for the KV cache and runtime overhead.

Quantization helps because reducing each parameter from 16 bits to 8 or 4 bits cuts the amount of memory required to store and move the model. This is why low-precision formats improve more than storage capacity. Fewer bytes traveling from memory can directly increase performance when memory traffic is the limiting factor.

The larger lesson extends well beyond AI. Recomputation and data movement are often interchangeable costs. Sometimes storing a result is cheaper than calculating it again. Sometimes recomputing a value is cheaper than retrieving it from distant memory.

Modern architecture increasingly tries to determine which side of that trade-off wins .

Games are particularly interesting because there is no single "gaming workload".

One moment the CPU may be traversing visibility/culling structures and preparing draw calls . Another thread may be updating physics or AI. The GPU simultaneously processes geometry, samples textures, reads material data, writes render targets , evaluates shaders, and accesses acceleration structures . In an open-world title, assets may also be streamed from storage into system memory and VRAM in the background. Some of those operations are highly coherent, but others are not. This helps explain several hardware trends.

On the CPU side, AMD's X3D processors use large last-level caches to capture more of the working sets used by games. The benefit can be substantial when critical game data would otherwise repeatedly travel to DRAM.

On GPUs, large last-level caches similarly reduce external VRAM traffic, boosting effective memory bandwidth and dramatically lowering access latency for critical data. NVIDIA, AMD, and Intel have all changed cache structures substantially across GPU generations because avoiding off-chip transactions can improve effective memory performance without requiring an equally large increase in physical memory bandwidth.

Real-time ray tracing makes data behavior more difficult again. In fact, rays traverse Bounding Volume Hierarchies , or BVHs, which are tree-type data structures that determine which pieces of scene geometry they may intersect. Unlike conventional rasterized workloads, rays can diverge and travel toward completely different regions of the scene.

NVIDIA describes this as both execution and data divergence : neighboring GPU threads may follow different code paths while also accessing memory addresses that are difficult to coalesce or cache. Its Shader Execution Reordering technology was designed partly to regroup ray-tracing work in ways that improve execution and data locality.

This is an important example because it shows that faster memory is not always the only solution to a memory problem, as sometimes the better option is reorganizing the work so the existing memory subsystem can be used more efficiently.

Games can also become constrained by capacity rather than bandwidth or latency. If required textures, geometry, and render resources exceed VRAM capacity, then the system may have to move resources across the PCI-Express bus from system memory or evict and reload them. A GPU with enormous theoretical bandwidth cannot compensate for data that is not resident in its local memory in the first place.

Memory performance therefore has several dimensions even within one frame: how much data fits locally, how quickly it can be accessed, how effectively accesses coalesce, how well caches capture reuse, and how much parallel traffic the architecture can sustain .

Storage provides perhaps the easiest demonstration of how misleading a single bandwidth figure can become.

Modern PCIe NVMe SSDs advertise spectacular sequential throughput, but sequential transfers represent an extremely favorable workload. Large contiguous requests allow the controller to distribute work efficiently across NAND channels, dies and planes while keeping many operations in flight. Real applications, however, frequently ask for something very different.

Operating systems, games, databases and applications may request thousands of relatively small pieces of data scattered around the drive. This is why SSD specifications distinguish sequential throughput from random IOPS , queue depth and latency.

Solidigm's own workload documentation defines sequential accesses as adjacent blocks, random accesses as blocks spread throughout the media, and queue depth as the number of outstanding I/O requests. Crucially, increasing queue depth can improve throughput by exposing more parallel work, but this comes at the expense of latency.

This is conceptually very similar to DRAM. In fact, a NAND flash memory-based SSD contains many flash dies operating in parallel behind a controller. NAND itself is organized into pages and larger erase blocks, and writes cannot be treated like overwriting a byte in DRAM. The controller performs address translation, wear leveling , garbage collection , and other background operations to make NAND look like an ordinary block device.

These processes can create additional internal traffic known as write amplification , where the SSD physically writes more data than the host requested. Garbage collection can also produce latency spikes during sustained workloads.

Consumer SSDs frequently add another layer to the hierarchy through fast write caching. In fact, a portion of NAND may temporarily operate in an SLC -like mode, accepting writes rapidly before data is later folded into denser TLC or QLC storage. Some drives also include DRAM for mapping data, while DRAM-less designs can use technologies such as Host Memory Buffer to keep portions of their mapping structures in system memory.

Once again, the storage device has effectively built its own memory hierarchy. And once the fast cache is exhausted, sustained write performance can look very different from the short benchmark burst shown on a product page .

At this point, it should be clear why memory benchmarks require context.

A large file copy wants sustained sequential bandwidth.

A latency-sensitive database may care far more about small random accesses and tail latency.

A scientific simulation operating on dense arrays can make excellent use of streaming bandwidth and vectorization.

A compiler may spend considerable time chasing complex data structures with relatively poor spatial and/or temporal locality.

A CPU-limited game can benefit enormously from a larger last-level cache if its hot working set fits inside it, while another game may barely react.

AI training needs tremendous compute throughput, capacity, and bandwidth simultaneously. Low-concurrency LLM decode can lean much more heavily on memory bandwidth. Long-context inference adds the KV cache and turns available memory capacity into another central resource.

Even the word bandwidth needs qualification. There is theoretical interface bandwidth, sustained application bandwidth, cache bandwidth, DRAM bandwidth, storage bandwidth, and interconnect bandwidth. A workload may saturate one while barely using another.

What ultimately determines performance is how the application's data access pattern interacts with the entire hierarchy .

This is why some of the most interesting improvements in modern processors are not simply faster arithmetic units.

Caches keep reused data close to execution resources, prefetchers predict future accesses, memory controllers reorder requests to expose DRAM parallelism, GPU thread scheduling attempts to improve coherence, tiling divides large problems into working sets that fit inside smaller memories, compression reduces the number of bytes that must be transferred, quantization does much the same for AI models, HBM moves large amounts of memory physically closer to accelerators through extremely wide interfaces, 3D stacking adds capacity without forcing every bit of memory onto the same planar die, chiplets allow designers to combine specialized compute, cache and I/O structures more flexibly, and increasingly sophisticated interconnects attempt to prevent communication between CPUs, GPUs and accelerators from becoming the next wall.

The common theme is simple: use expensive data movement as efficiently as possible.

Raw compute performance will continue rising. Memory bandwidth will rise too, caches will grow, HBM will become faster, SSDs will push further into double-digit gigabytes per second, and advanced packaging will bring previously separate components closer together.

However, physical distance, energy, capacity and latency ensure that no single memory technology can provide enormous capacity, enormous bandwidth, negligible latency and negligible cost simultaneously.

The memory hierarchy therefore is not going anywhere, and if anything, it is becoming deeper and more sophisticated.

Modern performance increasingly depends on putting the right data in the right level of that hierarchy before the processor asks for it, and on organizing software so that the same data can be reused rather than moved again .

A CPU core waiting hundreds of clock cycles for a dependent memory access is not useful compute. A GPU with thousands of idle arithmetic units waiting on VRAM is not useful compute. An AI accelerator capable of petaFLOPS but starved of model data is not useful compute. Neither is a 14 GB/s SSD if the application cannot efficiently issue and process the requests required to use it.

The fastest operation, after all, is often not fetching data more quickly. It is avoiding the fetch altogether.

Follow Wccftech on Google to get more of our news coverage in your feeds.