What would you like to be added: Add a new mode "hybrid" in concurrency detector plugin that saturates as soon as either request or token based concurrency saturates.
Why is this needed: Request based concurrency detector is not enough as prefill heavy workloads tends to consume more GPU compute and saturate. In this case token based detector works but it falls short when OSL heavy workload arrives as it suffer from estimation problem. In such cases "request" based concurrency fits well since it puts an upper limit on number of requests that can be dispatched to endpoints.
What would you like to be added: Add a new mode "hybrid" in concurrency detector plugin that saturates as soon as either request or token based concurrency saturates.
Why is this needed: Request based concurrency detector is not enough as prefill heavy workloads tends to consume more GPU compute and saturate. In this case token based detector works but it falls short when OSL heavy workload arrives as it suffer from estimation problem. In such cases "request" based concurrency fits well since it puts an upper limit on number of requests that can be dispatched to endpoints.