[CLI] [Bug] App registration log streaming from unrelated Task Executor

[jaywonchung]

I deployed an app that uses one encoder task and one llm task (examples/mllm/app.py with encoder_fission=True) and after it has been deployed, another app that uses one llm task (examples/mllm/app.py with encoder_fission=True). The two llm tasks are not equivalent with each other and thus the second app registration spawns a second instance of vLLM. However, for some reason, the current stdout log output of the first llm (part of the first app) appeared in the output of cornserve register.

cc. @yizhuoliang

[majunze2001]

I believe this is because the CLI log streamer is using prefix-based matches of UnitTask names instead of exact matches for pod names. The new LLMUnitTask has exactly the same UnitTask names regardless of the value of encoder_fission, and the CLI cannot distinguish different task executor pods from their names. The easy way is to use different UnitTask names that depends on encoder_fission.

[jaywonchung]

Deployed unit task names are not supposed to be unique. Design has to be revised.

[yizhuoliang]

To decide whether to launch new task-manager/executor, the gateway checks if the required unit task is equivalent to existing ones. This checks all fields of the unit task, including encoder_fission.

So I propose to add a new "kind number" in the executor pod name.
te-vllm-[model_id]-[kind_number]-[gpu_rank]
In this way, the kind_number is decided by the gateway, and if two pods have different kind_number, these pods are distinct executors.

Then, in Jae-won's screenshot case, the gateway can early return the prefixes like te-vllm-qwen2-vl-7b-instruct-1- and te-vllm-qwen2-vl-7b-instruct-2-, so the LogStreamer can distinguish them as well. How about this.

[majunze2001]

Sidenote: the trailing numbers are the global ranks of GPUs. This is intended as it illustrates the association between a task executor and the sidecar it's using.

[yizhuoliang]

I see, then things are simpler. The gateway won't worry about the trailing number at all, the gateway can increment a kind_number whenever it launches a distinct new executor.

[jaywonchung]

I think relying on naming is brittle. Cornserve users are expected to write descriptors themselves too, and then they need to really adhere to the naming convention well or otherwise the system will break. This is relying on unstructured strings for structured system control signals, which makes the system vulnerable to human programming errors.

The core issue here is how to propagate the mapping between a newly spawned pod and an app. And this relationship is likely going to be part of what the upcoming CRD-based unit task management project will express natively; when a new unit task instantiation is deployed, it will create a new custom resource. All components (Resource Manager, Task Manager, etc.) will operate based on the CRD instance. And part of that, when the Task Manager creates a new Task Executor pod, it will fill in the pod_names field of the unit task instantiation custom resource, and everyone in the cluster can query that, including the CLI.

As a temporary fix, I thought of a concept using K8s labels below, but I don't think we have to do it -- we can fix this issue altogether with the CRD project. But I'll share anyways.

App registration flow

1. Gateway validates the app source code and discovers unit tasks.
2. Gateway checks which unit tasks are already deployed and which ones have to be deployed.
   • For now, the gateway should check the usage count of each task stored in-memory inside task_manager (to be renamed).
   • With the CRD-based unit task management project, this will instead be a query to the CRD to check whether there are any existing deployments of that unit task instantiation.
3. Gateway returns the app ID and discovered unit tasks and which ones have to be newly deployed back to the client. The client displays in the table each unit task and which ones are going to be deployed now.
4. Gateway sends a RPC (DeployUnitTask) to the Resource Manager and includes the app ID in the RPC.
5. Resource Manager checks whether a unit task is deployed, and if not, spins up a Task Manager.
6. Resource Manager initializes the newly spawned Task Manager with the unit task info it should handle (RegisterTaskRequest), and includes the app ID in the RPC.
7. Task Manager spins up initial Task Executors. For the first one, it will attach a label cornserve.ai/initially-spawned-by: {app_id}.
8. After step 3, the CLI has been querying the list of pods (/api/v1/namespaces/cornserve/pods) with label filter cornserve.ai/initially-spawned-by: {app_id}. When the pods are created, it can start streaming logs.