SanDisk held an Investor Day presentation earlier this week. Ordinarily, that would be considered par for the course. However, then some of the presentation slides - laden with supposed inaccuracies and key omissions - started doing the rounds on the social media, leading to a backlash from the tech-savvy community.
An AI model basically consists of a series of transformer blocks that are made up of two key elements:
- The Attention Layer looks at how words in a sentence connect with each other. For instance, if you give the model a prompt that says Paris is the capital of France, the attention layer will link the word 'capital' to 'France' and identify 'Paris' as the answer. Even so, this layer does not know what is meant by France or Paris, but would save these notes in the form of KV cache. As context increases, so does KV cache.
- The Feed-Forward Network (FFN) holds the sum knowledge of a model in the form of weights, and it is this layer that uses deep mathematical formulas to unearth the meaning behind each word. For instance, it would look at France and activate its country encyclopedia, and so on.
Currently, both the model weights and KV cache are stored in HBM. However, HBM stacks are soldered alongside the GPU and come equipped with limited memory-related capacity. What's more, to increase HBM, you generally have to increase the GPU count, which can get very expensive very quickly.
High-Bandwidth Flash (HBF), however, represents one possible solution vector, and SanDisk is currently developing an HBF standard with SK hynix, one that would offer 512GB of storage along with a bandwidth of 0.4TB/s to 3TB/s . The company is aiming for a commercial rollout by 2028/2029.
Basically, just as HBM stacks DRAM, HBF stacks NAND die on top of each other, with Through Silicon Vias (TSVs) connecting the dies together and a controller logic die bonded to this NAND array.
The problem with using NAND cells, however, is their turtle-like speeds. An SRAM offers read speeds of just around a nanosecond vs. ~100 nanoseconds for DRAM and a whopping ~100 microseconds for NAND. This means that NAND offers 1,000x slower read speeds than a DRAM-based HBM .
HBF, however, leverages the power of parallelism to increase bandwidth by orders of magnitude. Basically, the logic die schedules thousands of parallel reads of NAND cells simultaneously. So, while each individual read of a given NAND cell is around 1,000x slower, thousands of parallel reads can deliver a cumulative bandwidth of 0.3TB/s to 3TB/s.
Even so, an HBF can't counter NAND's atrocious write speeds and the attendant fragile write endurance.
This brings us to the core of today's topic. SanDisk dropped the ball on accuracy and best practices in its presentation this week. For instance, given the fact that HBF will only roll out by 2028/2029, it should have compared its specs with those of 16Hi HBM4E, which will likely be the dominant HBM solution at that time. As such, this HBM variant has a bandwidth of 32 TB/s, around 3x higher than the 12.8 TB/s that SanDisk seems to have pencilled in.
SanDisk also used bf16 as its chosen quantization when most models these days either use fp8 or fp4.
What's more egregious, SanDisk made no mention of the write endurance that can be expected of HBF, which is a serious oversight, one that could potentially render the tech unsuitable for KV cache.
While we are sure SanDisk can put up a reasonable defense on all of these oversights and prevarications, they do paint a picture that is a fair bit desperate.
Follow Wccftech on Google to get more of our news coverage in your feeds.