backend fallback draft #50
Conversation
Signed-off-by: alec-flowers <[email protected]>
| - To enable KV Cache Routing we need to have tokens in frontend in order to calculate hashes to match against the indexers hashes created from KV events emitted from the backend. | ||
|
|
||
| REQ 3: PostProcessing SHOULD move to the backend engine | ||
| - Post-processing is currently done on the engine side, however the intention was to co-located it with the backend where there are many CPU's sitting idle alongside the GPU's. |
There was a problem hiding this comment.
Is this meant to be "currently done on the frontend side"?
| """Combine chat template + tokenization in one call.""" | ||
| pass | ||
|
|
||
| class VllmTokenizer(TokenizerProtocol): |
There was a problem hiding this comment.
Would it conceptually allow the sglang tokenizer as well?
There was a problem hiding this comment.
Yes, I was just providing this as a first example.
|
|
||
| **Benefits**: Offloads CPU work to backend servers, scales with engine instances | ||
|
|
||
| #### Mode 2: vLLM Native Processing (`--vllm-processing`) |
There was a problem hiding this comment.
Will this conceptually work with SGL and vLLM paths as well?
|
|
||
| **Benefits**: Offloads CPU work to backend servers, scales with engine instances | ||
|
|
||
| #### Mode 2: vLLM Native Processing (`--vllm-processing`) |
There was a problem hiding this comment.
Can we think of this in terms of a "hello world backend" example, CLI args, and all - and see if it generalizes to vllm, sglang, trtllm, etc. that each implement the "interface" we defined out of the box for users of our officially supported backends?
For example, maybe any backend would accept args like --dyn-backend-preprocessor and --dyn-backend-postprocessor (subject to change, just following --dyn- prefix convention for dynamo specific args rather than framework args we passthrough), etc. and for vllm backend that happens to map to using vllm's tokenizer?
There was a problem hiding this comment.
Yea I wasn't sure the best UX so just tried to write something down. Open to what you are suggestion as well.
| @@ -0,0 +1,222 @@ | |||
| # Native Framework Processing Fallback | |||
|
|
|||
| **Status**: Draft | |||
There was a problem hiding this comment.
Suggestion : Let's version it. example V.Alpha
|
|
||
| REQ 1: User SHOULD be able to select between dynamo processing and the backend enginge processing. | ||
|
|
||
| REQ 2: Preprocessing MUST happen on the frontend. |
| """Combine chat template + tokenization in one call.""" | ||
| pass | ||
|
|
||
| class VllmTokenizer(TokenizerProtocol): |
There was a problem hiding this comment.
Where this implementation lives, you will need to import vLLM and all its tokenization utilities, correct?
|
|
||
| ```rust | ||
| // Rust passes tokenizer path to Python factory | ||
| let tokenizer_path = card.get_tokenizer_path()?; |
There was a problem hiding this comment.
By passing only the path here, we assume the factory can figure out the type of tokenizer from that path?
| // Rust passes tokenizer path to Python factory | ||
| let tokenizer_path = card.get_tokenizer_path()?; | ||
| let adapter = Python::with_gil(|py| { | ||
| let py_tokenizer = factory.call1(py, (tokenizer_path,))?; |
There was a problem hiding this comment.
Should there be a protocol for the factory too?
No description provided.