api-server-rest: retry requests on LocalClosed#1563
Conversation
The current implementation establishes a connection during process startup and reuses it for all subsequent requests. As a result, if the Attestation Agent is restarted, requests can fail with the error `ttrpc err: local stream closed`. When a `LocalClosed` error is returned, clear the cached client, reconnect to the Attestation Agent, and retry the request once. Signed-off-by: Shunsuke Kimura <pbrehpuum@gmail.com>
Xynnn007
left a comment
Xynnn007
left a comment
There was a problem hiding this comment.
I think it makes sense, especially we will use systemd to control guest-components to launch in CoCo. Note that this commit will bring in a change that original ASR establishes connection to AA when launched, but new behavior is that the connection will be postponed to the first request.
Also, can you help to implement the CDH side also?
Another point off the topic of the PR, this raise some potential need for integration tests for guest components. We need to care about this @fitzthum
|
|
||
| pub struct AAClient { | ||
| client: AttestationAgentServiceClient, | ||
| client: Mutex<Option<AttestationAgentServiceClient>>, |
There was a problem hiding this comment.
Can we use RwLock here? As the call functions (like get_token) are all &self than &mut self. The calls will only do "read" upon the lock and updation will do "write".
There was a problem hiding this comment.
Pull request overview
This PR improves the resiliency of api-server-rest’s Attestation Agent (AA) ttrpc client by handling the case where the AA process restarts and existing connections become invalid. It introduces a reconnect-and-retry-once behavior when requests fail with ttrpc::Error::LocalClosed, preventing transient failures caused by reused stale connections.
Changes:
- Replace the single, eagerly-created AA ttrpc client with a cached, lazily re-creatable client stored as
Mutex<Option<...>>. - Add a
call_with_retryhelper that retries exactly once after clearing the cached client and reconnecting whenLocalClosedoccurs. - Route AA RPC methods (
get_token,get_evidence,get_additional_evidence,extend_runtime_measurement,get_tee_type,get_additional_tees) through the retry helper.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The current implementation establishes a connection during process startup and reuses it for all subsequent requests. As a result, if the Attestation Agent is restarted, requests can fail with the error
ttrpc err: local stream closed.When a
LocalClosederror is returned, clear the cached client, reconnect to the Attestation Agent, and retry the request once.