Adding v0.4.0 vllm, sglang and dynamo test yamls

[nwangfw]

Pull Request Description

Adding test sglang, vllm, and dynamo yamls

Related Issues

Resolves: #[1336]

Important: Before submitting, please complete the description above and review the checklist below.

---

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

• [Bug]: Corrections to existing functionality
• [CI]: Changes to build process or CI pipeline
• [Docs]: Updates or additions to documentation
• [API]: Modifications to aibrix's API or interface
• [CLI]: Changes or additions to the Command Line Interface
• [Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

• PR title includes appropriate prefix(es)
• Changes are clearly explained in the PR description
• New and existing tests pass successfully
• Code adheres to project style and best practices
• Documentation updated to reflect changes (if applicable)
• Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

[gemini-code-assist]

Summary of Changes

Hello @nwangfw, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the testing infrastructure by adding new Kubernetes YAML configurations for deploying and testing disaggregated large language model (LLM) inference systems across Dynamo, SGLang, and VLLM frameworks. It also updates existing SGLang configurations to use a new disaggregation transfer backend, ensuring broader compatibility and improved performance testing.

Highlights

• New Test Configurations: Added new YAML configurations for testing various deployment scenarios for SGLang, VLLM, and Dynamo, expanding the test suite for large language model (LLM) inference systems.
• Dynamo Platform Deployment: Introduced comprehensive Kubernetes manifests (namespace.yaml, platform.yaml) to deploy the core 'dynamo-platform' components, including etcd, NATS, and the dynamo-operator, providing a foundational setup for Dynamo-based LLM deployments.
• Disaggregated LLM Deployments (Dynamo): Included new DynamoGraphDeployment YAMLs (disagg.yaml, disagg_router.yaml) that define disaggregated LLM inference setups with distinct Frontend, Processor, VllmWorker, PrefillWorker, and Router services, specifying resource requirements and model download procedures.
• SGLang Router Configurations: Added multiple StormService YAMLs for SGLang (sglang-router-1p1d-tp2.yaml, sglang-router-1p1d.yaml, sglang-router-4p3d.yaml) to test various prefill/decode replica and tensor parallelism configurations, leveraging RDMA for inter-pod communication.
• SGLang Disaggregation Backend Update: Modified existing SGLang configurations (sglang-router-cache.yaml, sglang-router-random.yaml, aibrix-router.yaml) to switch the disaggregation-transfer-backend from nixl to mooncake, indicating an update in the underlying data transfer mechanism.
• VLLM Disaggregated Deployment: Added a StormService YAML (1p1d.yaml) for VLLM, configuring a 1 prefill + 1 decode replica setup, utilizing the NixlConnector for KV transfer and RDMA for high-performance communication.
• Kubernetes RBAC for SGLang: Introduced Kubernetes RBAC configurations (rbac-sglang-router.yaml) to grant necessary permissions (get, watch, list pods) to the default service account for SGLang router functionality.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds a number of YAML files for testing sglang, vllm, and dynamo. While the additions are comprehensive, I've identified several critical security vulnerabilities, such as hardcoded secrets and overly permissive RBAC roles, that must be addressed. There are also correctness issues that would cause deployment failures and maintainability concerns like using hostPath volumes and installing packages at runtime. Please see the detailed comments for suggestions on how to resolve these issues.

[gemini-code-assist]

This secret contains hardcoded credentials, which is a critical security vulnerability. Secrets, including registry credentials, must never be stored in version control. Please remove this and manage credentials using a secure method, such as creating Kubernetes secrets out-of-band or using a secret management tool.

[gemini-code-assist]

This secret contains a hardcoded RSA private key. This is a critical security vulnerability. Private keys must never be committed to version control. Please remove this and handle key management securely.

[gemini-code-assist]

The container command includes placeholders for access keys. This is a security risk as it encourages hardcoding secrets. These credentials should be provided securely, for example, by mounting them as environment variables from a Kubernetes secret.

This issue is repeated for VllmWorker (line 109) and PrefillWorker (line 142). Consider using envFrom with a secretRef to inject these values securely.

[gemini-code-assist]

This role grants wildcard permissions (*/*/*), which is extremely permissive and violates the principle of least privilege. This poses a significant security risk. The permissions for dynamo-platform-dynamo-operator-manager-role should be scoped down to only what is absolutely necessary for the component to function.

[gemini-code-assist]

The roleRef in this RoleBinding is set to ClusterRole, but the referenced dynamo-platform-dynamo-operator-proxy-role is defined as a Role on line 750. This will cause the binding to fail. Please change the kind to Role.

kind: Role

[gemini-code-assist]

The PrefillWorker service appears to be configured to serve graphs.disagg:PrefillWorker, but this file is for the disagg_router configuration. This seems like a copy-paste error and will likely cause the service to fail or misbehave. It should probably be graphs.disagg_router:PrefillWorker.

dynamo serve graphs.disagg_router:PrefillWorker --system-app-port 5000 --enable-system-app --use-default-health-checks --service-name PrefillWorker

[gemini-code-assist]

Binding a ClusterRole to the default service account in the default namespace is a security risk. Any pod in that namespace that does not specify a serviceAccountName will inherit these permissions. It's much safer to create a dedicated service account for the application that needs these permissions and bind the role to that specific service account.

[gemini-code-assist]

Installing packages at container startup is inefficient and can make the deployment unreliable if package repositories are unavailable. It's better to include necessary tools like wget in the Docker image itself. This applies to VllmWorker (line 105) and PrefillWorker (line 138) as well.

[gemini-code-assist]

Using hostPath volumes ties the pod to a specific node that has this path, which reduces portability and can cause scheduling issues. For tests, it's better to use a solution like emptyDir with an init container to download models, or a PersistentVolumeClaim if the data needs to persist. This applies to the decode role as well.

[gemini-code-assist]

Using hostPath volumes ties the pod to a specific node that has this path, which reduces portability and can cause scheduling issues. For tests, it's better to use a solution like emptyDir with an init container to download models, or a PersistentVolumeClaim if the data needs to persist. This applies to the decode role as well.

[Jeffwan]

rename folder name from dynamo-test to dynamo