Buckle up as the ongoing memory- and HBM-related bottleneck is about to get materially decongested, thanks to SanDisk's initiative to develop High-Bandwidth Flash (HBF) in concert with SK hynix, which should eventually lead to a bifurcation in tasks that are currently performed solely by the HBM.

Before discussing the HBF standard, let's first go over the memory wall issue that often comes up with HBM.

HBM and the memory wall issue

An AI model with 100 billion parameters and operating at fp16 compute level - so 2 bytes per parameter - requires 200GB (100x10⁹x2) of HBM just for storing model weights.

For the benefit of those who might not be aware, weights tell a model how much importance it should accord to any given concept or a string of words. Basically, they are the sum total of a model's stored knowledge. As a general rule, the larger the model, the more weights it needs to store.

But that's not all. Consider a scenario: you are writing a story, but hampered by terrible short-term memory. Whenever you write a new word, you are compelled to read whatever you've written so far just to remember what has already been inked. Obviously, as the text length increases, so does this laborious process.

Key-Value or KV cache is similar to taking notes on a separate sheet so that you remain abreast of what has been written so far. This speeds up the entire process by orders of magnitude. Even so, as context increases so does KV cache, which is generally stored within the HBM given the speed and frequency with which a model needs to access it.

Now comes the hard part. HBM4 stacks are soldered alongside the GPU and come equipped with limited memory bandwidth. For instance, a generic 12-Hi HBM4 stack holds 36GB of data, while its 16-Hi counterpart can push it to 48GB.

So, HBM not only stores model weights but also KV cache. And, to increase HBM, you generally have to increase the GPU count, which can get very expensive very quickly.

Now add HBF to this equation

As we detailed earlier today, SanDisk is now developing an HBF standard with SK hynix, one that offers 512GB of storage along with a bandwidth of 0.4TB/s to 3TB/s .

Basically, just as HBM stacks DRAM, HBF stacks NAND die on top of each other, with Through Silicon Vias (TSVs) connecting the dies together and a controller logic die bonded to this NAND array.

The problem with using NAND cells, however, is their turtle-like speeds. An SRAM offers read speeds of just around a nanosecond vs. ~100 nanoseconds for DRAM and a whopping ~100 microseconds for NAND. This means that NAND offers 1,000x slower read speeds than a DRAM-based HBM .

HBF, however, leverages the power of parallelism to increase bandwidth by orders of magnitude. Basically, the logic die schedules thousands of parallel reads of NAND cells simultaneously. So, while each individual read of a given NAND cell is around 1,000x slower, thousands of parallel reads can deliver a cumulative bandwidth of 0.3TB/s to 3TB/s vs. just around 6.4GB/s for a JEDEC-spec HBM4.

Even so, an HBF can't counter NAND's atrocious write speeds and the attendant fragile write endurance. This means that KV cache will still need to be stored on the HBM as it requires frequent writes. Model weights, however, can shift to HBF.

Now, as a final point, do note that HBF arrays are not expected to be SSD-level cheap, especially as they still require advanced packaging, which itself is a costly process. Even so, they can be quite a bit cheaper than HBM, which should unlock substantial economies of scale. Also, freeing up HBM for KV cache means model context can become longer without compromising on the underlying output accuracy, which adds another substantial benefit to this emerging paradigm.

Follow Wccftech on Google to get more of our news coverage in your feeds.