correct me if I'm wrong but does this mean, for a maximally used server with the same per user speed, its now 5x cheaper for pro and 7.6x cheaper for flash in terms of serving costs?
I could be mistaken, but as I'm understanding it, that is slightly wrong.
You would not get any per server savings. However, if each server is producing tokens faster, you might need fewer servers for the same amount of users.
for service level agreement locked at 120(?)TPS it clearly shows its now running at 761% more throughput. meaning a single server now can output 6.61x more tokens. meaning at the same cost of serving as the previous method they generate 6.61x more tokens (probably there is a slightly higher load on the GPU but I dont think that will change electricity usage that much)
however at a more realistic serving speed of 80 (which they also point at and the openrouter speed currently is) its 51% more throughput. that is still a 33% price reduction if we assume the price is dictated by allotted server space + electricity.
Adding on this: because most claimed “speed up” out there are fake: multi-token generation acceptance are heavily context dependent, people often claim these in math context, which tends to have higher acceptance rate. What DeepSeek measure is more interesting: they kept the tps unchanged in their production system and measured the throughput increase. It is not useful for single-user use case (in this sub), but it is more fair arguably since it is real traffic and real money saved.
WithoutReason1729 [ ] [score hidden] stickied comment
Your post is getting popular and we just featured it on our Discord!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.
FullOf_Bad_Ideas
They did it again. Their API is now the fastest DeepSeek provider on OpenRouter.
Recoil42
Nyghtbynger
So... deepseek flash, but flasher ?
Edit : damn, imagine being a breakthrough model and your creators calls you "checkpoint"
Kryohi
That's the real backstory of Skynet
AlwaysLateToThaParty
I'll show you what a 'checkpoint' is buddy.
Flamboyant_Nine
Are their API models already using this architecture?
Ok_Zookeepergame8714
Probably, the thinking goes by much quicker than before.
oxygen_addiction
Flash got way faster this week, so most likely.
ProfessionalSpend589
It also started token generating in what I assume is Chinese when I prompt it in English. And I have to prompt it again to type in English.
Just now I made the connection that this may have been the change that triggered it.
corruptbytes
this is my world cup
woadwarrior
Amazing work! But needs 38TB of disk space to train a drafter for something as tiny as Qwen3-4B, damn!
Seideun
Compared with the size of Deepseek v4 pro it's not a big deal at all :)
woadwarrior
Imagine how big the target cache was for training DeepSeek v4 Pro’s drafter.
quantier
I need a NVFP4 version 😍
Every-Walrus
correct me if I'm wrong but does this mean, for a maximally used server with the same per user speed, its now 5x cheaper for pro and 7.6x cheaper for flash in terms of serving costs?
SufficientPie
Isn't speculative decoding faster but more expensive?
Every-Walrus
their paper shows an increase of at least 50% throughput for the same single user TPS which is, at the end of the day, all we care about.
squngy
I could be mistaken, but as I'm understanding it, that is slightly wrong.
You would not get any per server savings.
However, if each server is producing tokens faster, you might need fewer servers for the same amount of users.
Every-Walrus
for service level agreement locked at 120(?)TPS it clearly shows its now running at 761% more throughput. meaning a single server now can output 6.61x more tokens. meaning at the same cost of serving as the previous method they generate 6.61x more tokens (probably there is a slightly higher load on the GPU but I dont think that will change electricity usage that much)
however at a more realistic serving speed of 80 (which they also point at and the openrouter speed currently is) its 51% more throughput. that is still a 33% price reduction if we assume the price is dictated by allotted server space + electricity.
the-username-is-here
Come on, vLLM devs, everyone's waiting! :)
Getting local Flash from 40 to 50-60 tps would be HUGE.
UltraFOV
I get 97tk at IQ6. So now It gets faster?
horeaper
Searching in this subreddit for DSpark to see if anything else adopted this architecture, and all I get is DGX Spark 😑
zyxciss
Would be fun if it comes for Qwen models too
Tomr750
how does this compare to using ds4 on Macs with 128gb ram?
squngy
If I understand it right, this basically adds another MTP to them.
Dany0
This is a clear improvement over DFlash. But damn is it expensive to train the drafters...
[deleted]
[deleted]
StupidScaredSquirrel
They do in figure 7 of the paper
liuliu
Adding on this: because most claimed “speed up” out there are fake: multi-token generation acceptance are heavily context dependent, people often claim these in math context, which tends to have higher acceptance rate. What DeepSeek measure is more interesting: they kept the tps unchanged in their production system and measured the throughput increase. It is not useful for single-user use case (in this sub), but it is more fair arguably since it is real traffic and real money saved.
Mountain-Dragonfly46
In the abstract (first paragraph) of the paper linked above.
(OP could have quoted it in his message)
Go read it, Deepseek papers are always interesting.