Feature Request: DFLASH support (from 40 tok/sec to 400 tok/sec)

[KotPrezesOfficial]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

It allows to generate tokens so much faster, without any loss in quality.
More here: https://github.com/z-lab/dflash

Motivation

Because it allows faster generation

Possible Implementation

No response

[love01211]

+1

[am17an]

I'm experimentally working on this. Likely needs to wait till #18039 is merged

[enginefuture]

niubi

[rhythmyf]

+1

[giveUphope]

Nice job!!!

[snapo]

I'm experimentally working on this. Likely needs to wait till #18039 is merged

could you keep in mind maybe that we have dflash + ddtree together... both together would provide a 150% boost compared to 100% for dflash alone

[heyunxu]

+1

[zedbytes]

I'm experimentally working on this. Likely needs to wait till #18039 is merged

could you keep in mind maybe that we have dflash + ddtree together... both together would provide a 150% boost compared to 100% for dflash alone

dflash + ddtree + rotorquant ?

[snapo]

I'm experimentally working on this. Likely needs to wait till #18039 is merged

could you keep in mind maybe that we have dflash + ddtree together... both together would provide a 150% boost compared to 100% for dflash alone

dflash + ddtree + rotorquant ?

rotorquant afaik only reduces kv cache consumption

dflash uses a small diffusion model to speed up token generation through speculative decoding
ddtree on top of dflash creates a tree (from what i did read) to estimate the path of the diffusion prediction to use even less iterations in the diffusion process...

but i am just a noob, no clue if i got it right, as i understand the GPU things not enough, i can code but GPU kernels and ML kernels are somehow still a little over my head.

[ITankForCAD]

We should really start to look into speculative prefill solutions now that context size and context ingestion have become more important with agentic workflows.

[gyzerok]

We should really start to look into speculative prefill solutions now that context size and context ingestion have become more important with agentic workflows.

What are you waiting for? Start looking into it

[ruixiang63]

For anyone interested in DFlash, please take a look at this PR: #22105
@am17an Welcome to collaborate on this, there’s still much we can improve.

[fudingyu]

+1