[Rust Frontend] Add static HTTPS and mTLS support for HTTP and gRPC

[tahsintunan]

Summary

• Adds TLS termination to the Rust frontend (tracking: [Roadmap] Rust Frontend Feature Parity #44280): HTTPS and mutual TLS, on both the HTTP and gRPC servers.
• Supported: --ssl-certfile, --ssl-keyfile, --ssl-ca-certs, --ssl-cert-reqs (0/1/2), and --ssl-ciphers. A combined cert+key PEM works (the key falls back to the certfile), mirroring uvicorn.
• Built on OpenSSL (openssl/tokio-openssl), reusing the OpenSSL already in the tree via native-tls. Introduces no new rustls.
• gRPC over TLS: the --grpc-port server reuses the same TLS config (plus ALPN h2 for HTTP/2). It now binds 127.0.0.1 instead of 0.0.0.0 on the unix-socket / inherited-fd path, so the unauthenticated service isn't exposed on all interfaces.
• Connection timeouts: TLS handshake deadline (60s, both servers) and HTTP keep-alive idle timeout (VLLM_HTTP_TIMEOUT_KEEP_ALIVE, default 5s, 0 disables). gRPC adds HTTP/2 keepalive (gRPC-core/Python defaults). Stops stalled or idle connections from piling up.

Notes

• --ssl-ciphers is applied with OpenSSL's set_cipher_list (TLS 1.2 and below), the same as Python's ssl.set_ciphers; an invalid value errors at startup. Without it, the default is the openssl crate's mozilla_intermediate_v5 baseline (forward-secret AEAD, TLS 1.2 floor, server cipher preference), slightly stricter than uvicorn and overridable via --ssl-ciphers.
• Crypto backend is a build-time choice: default builds link the system OpenSSL. --features vendored produces a self-contained binary (server + client OpenSSL bundled). No runtime flag.
• Stricter, fail-loud (not regressions): mTLS requires --ssl-ca-certs. A key-without-cert/ unreadable/malformed PEM errors clearly at startup, and the keep-alive timeout also bounds the request-header read from the first byte, dropping silent/slow clients (uvicorn only bounds the wait between requests).
• gRPC TLS has no Python parity counterpart (Python's gRPC server is plaintext). mTLS isn't required, but applies to both endpoints when --ssl-cert-reqs is set. Applying TLS to Python's gRPC is a follow-up.
• DEFERRED: hot-reload (--enable-ssl-refresh) stays unsupported. A follow-up will reload the cert files into the SslContext and add a file watcher. No listener API change needed.
• Also makes a small, unrelated flaky-test fix in engine-core-client (an abort-burst test surfaced by the main merge).

Divergence:

• Default TLS cipher suites: By default, the Rust server uses the openssl crate's mozilla_intermediate_v5 baseline (forward-secret AEAD suites, TLS 1.2 floor, server cipher preference), which is slightly stricter than Uvicorn's default. I chose it because it is the crate's recommended, maintained policy and the stronger default. --ssl-ciphers still overrides it for full control.
• Header-read timeout: the keep-alive timeout also bounds the request-header read from the first byte, so a silent/slow client is dropped. Uvicorn only bounds the wait between requests, so this is stricter.

[tahsintunan]

Hey @BugenZhao, I got the context from #46052 and reworked this to use OpenSSL (openssl/tokio-openssl) instead of rustls for the server TLS. So this PR no longer adds any new rustls.

[njhill]

Thanks @tahsintunan! Could we have the grpc server use the same TLS config?

Also can we have a handshake timeout? (but keeping the handshake async)

[tahsintunan]

Thanks @njhill! Went through your notes.

Done:

• Connection timeouts: TLS handshake (60s, matches uvloop) + HTTP keep-alive idle (matches uvicorn).
• Also took the liberty of switching the gRPC bind from 0.0.0.0 to 127.0.0.1 on the unix-socket / inherited-fd path. It was quietly exposing the (unauthenticated) gRPC service on all interfaces there.

On gRPC over TLS: yes, we can reuse the same TLS config, but I'd like to run through one thing before implementing.

gRPC has no auth today (same as Python's insecure gRPC server), so plain server TLS would just encrypt an otherwise-open endpoint. And when HTTP binds a TCP host, gRPC uses the same host (just a different port), so a public HTTP bind exposes gRPC there too. Right now the only thing that can actually check who's calling is mTLS (client certificates, --ssl-cert-reqs 2).

So we have these options:

• Require mTLS to expose it: if gRPC TLS is reachable beyond localhost without a client cert, refuse to start (I'm leaning towards this). This ties gRPC's mTLS to HTTP's, since there's one --ssl-cert-reqs.
• Stay permissive: just warn instead of refusing.
• Give gRPC its own auth (e.g. reuse the HTTP API key).

Wdyt?

[BugenZhao]

Thanks!

[BugenZhao]

Just realized there's no built-in support for using TLS in axum... Would you help to check it out whether this library can be used to simplify implementation on our side?

Also it would be better we're agnostic about OpenSSL but only target native-tls, which can be backed by security-framework on macOS.

[tahsintunan]

Thanks, good call. I've switched the TLS termination over to tls-listener so the server side is a lot smaller now. (I also looked at axum-server since it's more widely used, but it's HTTP-only and wouldn't cover gRPC, whereas tls-listener wraps any stream, so it handles both.)

On native-tls instead of OpenSSL: it doesn't work for the server, because no mTLS. native-tls's acceptor has no client-cert verification (just cert+key, version, ALPN), so --ssl-cert-reqs can't be expressed on any backend. (rustls could, but we're keeping that out.)

Also, the Python frontend is OpenSSL as well (CPython's ssl is an OpenSSL binding), so staying on OpenSSL won't be a regression.

[BugenZhao]

On native-tls instead of OpenSSL: it doesn't work for the server, because no mTLS. native-tls's acceptor has no client-cert verification (just cert+key, version, ALPN), so --ssl-cert-reqs can't be expressed on any backend. (rustls could, but we're keeping that out.)

Good observation. Add a link to the original issue for reference: rust-native-tls/rust-native-tls#161

[njhill]

Thanks @tahsintunan ... I think for now we should just apply the same TLS configuration to the gRPC endpoint as the http one. In the python case we should also do that. I don't think mTLS should be required. But if configured then it should be enabled on both endpoints.

[tahsintunan]

Thanks @njhill. I've got gRPC using the same TLS config as HTTP now. mTLS isn't required, but if --ssl-cert-reqs is set it's enabled on both endpoints like you suggested. I'll do the Python side as a follow-up.

[njhill]

Thanks @tahsintunan! LGTM.

I will let @BugenZhao also give a final approval.

It would be great to support cert rotation as a follow-on, for parity with the python-side functionality.

[BugenZhao]

Generally LGTM. Thanks for the work!

I just felt we could simplify a bit on current impl, especially that we seem to mix orthogonal layers together (like socket type / TCP_NODELAY / TLS / socket type / HTTP vs gRPC) , making the code a bit hard to read.

I've pushed some commits to simplify it, and probably I'll open some follow-up PRs for further refactoring.

[tahsintunan]

Follow-up cert rotation PR: #47362
cc @njhill @BugenZhao