Poor TPOT when decoding

[gty111]

Now we have cuda graph support. But according to nsys, there are large bubbles (20-30ms) between two decode batch when the batch size is large. We should find the root cause and optimize it.

[gty111]

The root cause is rank0's self.model_runner.prepare_input(schedule_seqs) in schedule_forward is very time-consuming (~5-500ms, proportional to the number of tokens in each batch). We need to optimize the bottleneck.

[gty111]

We find the overhead of zmq (method send_schedule_seqs and recv_schedule_seqs) is great.

[gty111]

Method send_schedule_seqs is serial for all ranks. The overhead is accumulated when the number of ranks is increasing.

[gty111]

We should cache metadata of sequence for each worker and suggest the following changes:

• The frontend process not only send new requests to the rank 0 worker but also other workers.
• Rank 0 worker send the ipc_package not only to the frontend process but also other workers. So each worker can maintain local cached seq pool.
• Rank 0 worker schedule a batch of seqs to run and send only the seq_id to other workers.

We need to maintain following metadata for each seq:

• new token id
• new page table
• new computed_token_num and to_computed_token_num

We change the architecture to #158. Post schedule remove token_ids if possible and send to_compute_tokens only.