Move waiting logic for host orchestrator settlement to backend by k311093 · Pull Request #521 · google/cloud-android-orchestration

k311093 · 2026-04-15T02:23:16Z

While interacting with host orchestrator right after creating host
API is returned, there is a possibility of getting 503 service
temporarily unavailable error because host it created and booted but
host orchestrator is not executed / ready yet.

Handling logic for waiting host orchestrator readiness is currently
in the client logic. Moving this logic into cloud orchestrator API
endpoint would make the client that uses host chestrator API simpler.

Context: b/501288123

0405ysj · 2026-04-15T05:09:39Z

+	if err != nil {
+		return nil, err
+	}
+	client, err := g.InstanceManager.GetHostClient(g.Zone, host.Name)


From GceIM, GetHostClient internally invokes getHostInstance, which seems to be calling GCE API to retrieve the host. However, this method already has ins which is typed as *compute.Instance.

I think it's better not to get *compute.Instance twice, also not to bring new fields(InstanceManager, Zone) into opResultGetter.

Changed logic to pass *compute.Instance and reuse it if specified.

ser-io

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with. To do this properly, you'd need to add and implement a new http endpoint that tests whether the host is ready, still clients would need to make these calls after CreateHost returned in order to determine whether the host is ready to interact with. However, it's up to every client implementation how they want to handle this, they can always keep doing what the current client is doing.

jemoreira · 2026-04-15T17:35:51Z

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with.

What use is a host that doesn't have a host orchestrator? If the answer is "none", then it would make sense to only mark the operation to create a host as completed once the host orchestrator is ready. It simplifies the api and puts the need for waiting in a single place (the cloud orchestrator) rather than in every client. The CO is already waiting for the host to be created, it might as well wait a bit longer.

jemoreira · 2026-04-15T17:38:00Z

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

k311093 · 2026-04-16T01:29:28Z

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

Updated commit message with descriptions.

0405ysj · 2026-04-16T01:56:20Z

What use is a host that doesn't have a host orchestrator? If the answer is "none"

+1. I expect users are understanding the meaning of host as a host environment to launch Cuttlefish instances, and the route of orchestrating CF instances is Host Orchestrator. I think health of Host Orchestrator is more representative than health of computing unit(e.g. GCE or docker instance).

From API view, I expect users create a host and verify with 3 steps; POST /v1/zones/{zone}/hosts, POST /v1/zones/{zone}/operations/{operation}/:wait, GET /v1/zones/{zone}/hosts/{host}/. Invoking those endpoints should be responsible by clients though. But having another round of wait by setting retry count for requesting GET /v1/zones/{zone}/hosts/{host}/ would be unexpected behavior in users' perspective, I think.

jemoreira · 2026-04-21T23:23:01Z

+	if err != nil {
+		return nil, err
+	}
+	if err := client.WaitForHostReady(); err != nil {


+1

Waiting for the host to be ready (HO is up and running) should be done in the wait-for-create-operation-to-complete endpoint. Also, please take into account the following caveats:

The /:wait endpoint isn't just for waiting for host creation to complete, it's used to wait for all types of operations. You're going to need to differentiate between the different operation types to only add the extra wait to the correct operation. You'll also need a way to figure out the host name from the operation.

Waiting for the HO in the CO should have a timeout, after which the create operation should return failure. The timeout should depend on the actual InstanceManager implementation, as GCE hosts boot times are different than docker containers'.

The returned failure be clear enough to differentiate between a failure to create the host and the HO not becoming reachable before the timeout.

The wait for operation must detect and handle correctly (return failure) if a host was successfully created but deleted by a concurrent call before the create operation succeeded (it's ok to wait until the timeout the first time, but any subsequent wait should return immediately; this means in practices that you should check the host exists before you try to contact the HO).

While the wait-for-create-operation is being called, the list operation may return the existing host, even when the HO isn't ready in it. That's the way it works today, it's fine to keep that behavior for now.

ser-io · 2026-04-21T23:29:12Z

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.
CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with.

What use is a host that doesn't have a host orchestrator? If the answer is "none", then it would make sense to only mark the operation to create a host as completed once the host orchestrator is ready. It simplifies the api and puts the need for waiting in a single place (the cloud orchestrator) rather than in every client. The CO is already waiting for the host to be created, it might as well wait a bit longer.

Right now the backend doesn't care whether the HO is running or not, it just create/list/delete hosts. Then, waiting to check whether you can interact with the HO is passed down to the client implementation, which I think makes things more simple conceptually.
Moving this responsibility to the backend makes the backend now aware of a new WAITING_FOR_HO status.
How do you plan to list hosts: are you listing all hosts or only listing hosts with the HO ready? If a host was created successfully but the OS failed to boot or the HO failed to launch, as an infra solution, I still want to see that host listed for triaging purposes.

To handle this and comment about gce.go, I tried to add task manager, but it make code more complex than expected and it this is not beneficial than introducing complexity. It might be better to handle that in the client client reduces the complexity. How about just add instruction for comment in CreateHostRequest api/v1/instancemanager.go ?

Comment looks good to me.

Let's do #521 (comment)

k311093 · 2026-04-21T23:33:35Z

Let's do #521 (comment)

Got it. Thanks!

k311093 · 2026-05-06T06:51:48Z

Added endpoint :wait-host-availability endpoint and changed cvdr to use it. When :wait-host-availability fails, fallback to the logic used before.

ser-io · 2026-05-11T17:08:12Z

 	// `Get`/`Create`/`Update`, the response should be the relevant resource.
 	router.Handle("/v1/zones/{zone}/operations/{operation}/:wait", c.Authenticate(c.waitOperation)).Methods("POST")
 	router.Handle("/v1/zones/{zone}/hosts/{host}", c.Authenticate(c.deleteHost)).Methods("DELETE")
+	router.Handle("/v1/zones/{zone}/hosts/{host}/:wait-host-availability", c.Authenticate(c.waitCreateHost)).Methods("POST")


The CO API is designed to have only one :wait endpoint /v1/zones/{zone}/operations/{operation}/:wait, there are no specific wait endpoints like: /:wait-for-create or /:wait-for-delete for example.

For the purpose of this PR specifically wondering whether can you can extend the wait logic when the operation was a create host operation and wait a bit longer till the HO is reachable rather than adding a new endpoint.

In that case, creating another endpoint for creating operation for waiting host orchestrator's settlement and wait it using previous :wait would be a proper way?

The purpose of this PR was to simplify the client logic. Creating a new operation for waiting on HO readiness doesn't simplify the client implementation, it makes things even more complicated than what we have right now.

The way to make client logic simpler is for the client to make two calls only:

POST /host POST /operation/foo/:wait

When the operation was a "create host operation". POST /operation/foo/:wait should be resolved when the host was created and the HO is ready. Right now we resolve "create host operation" the moment the host was created without waiting for the HO to be ready, we need to extend this waiting logic for create host operations and not add a new endpoint.

Changed to wait for host ready at :wait like other operations. Moved host wait logic to WaitOperation function, not Get() method like before.

Checked with cloud orchestrator at GCP, docker and confirmed that it successfully created host and subsequent tasks (for GCP side, didn't checked CVD creation completely, but checked that uploading, extracting artifacts worked well).

If the wait logic was moved behind POST /v1/zones/{zone}/operations/{operation}/:wait can we remove POST /v1/zones/{zone}/hosts/{host}/:wait-host-availability?

ser-io · 2026-05-27T14:56:09Z

 	// `Get`/`Create`/`Update`, the response should be the relevant resource.
 	router.Handle("/v1/zones/{zone}/operations/{operation}/:wait", c.Authenticate(c.waitOperation)).Methods("POST")
 	router.Handle("/v1/zones/{zone}/hosts/{host}", c.Authenticate(c.deleteHost)).Methods("DELETE")
+	router.Handle("/v1/zones/{zone}/hosts/{host}/:wait-host-availability", c.Authenticate(c.waitCreateHost)).Methods("POST")


If the wait logic was moved behind POST /v1/zones/{zone}/operations/{operation}/:wait can we remove POST /v1/zones/{zone}/hosts/{host}/:wait-host-availability?

While interacting with host orchestrator right after creating host API is returned, there is a possibility of getting 503 service temporarily unavailable error because host is created and booted but host orchestrator is not executed / ready yet. Handling logic for waiting host orchestrator readiness is currently in the client logic. Moving this logic into cloud orchestrator API endpoint (specifically inside WaitOperation) makes the client simpler and reduces redundant API endpoints. Context: b/501288123 TAG=agy CONV=f5f788b6-7aa2-4614-b441-87f3ad9fed4e

k311093 force-pushed the wait branch 5 times, most recently from deec9a3 to 2091716 Compare April 15, 2026 04:34

k311093 marked this pull request as ready for review April 15, 2026 04:45

k311093 requested review from Databean, adelva1984, jemoreira, jmacnak, rmuthiah and ser-io as code owners April 15, 2026 04:45

k311093 requested review from 0405ysj and ikicha April 15, 2026 04:45

0405ysj reviewed Apr 15, 2026

View reviewed changes

k311093 force-pushed the wait branch 2 times, most recently from 493fffb to b5e8c57 Compare April 15, 2026 05:59

0405ysj reviewed Apr 15, 2026

View reviewed changes

Comment thread pkg/app/instances/gce.go Outdated

ser-io requested changes Apr 15, 2026

View reviewed changes

Comment thread pkg/app/instances/gce.go Outdated

k311093 force-pushed the wait branch from b5e8c57 to c8ef926 Compare April 16, 2026 01:15

k311093 force-pushed the wait branch 2 times, most recently from b3d13e7 to 8d2831f Compare April 16, 2026 03:55

k311093 requested review from 0405ysj and ser-io April 16, 2026 04:47

0405ysj reviewed Apr 16, 2026

View reviewed changes

Comment thread pkg/app/instances/gce.go Outdated

Comment thread pkg/app/instances/gce.go Outdated

jemoreira requested changes Apr 21, 2026

View reviewed changes

k311093 force-pushed the wait branch 4 times, most recently from f8c9f9f to ef76e5e Compare May 6, 2026 06:32

k311093 requested review from jemoreira and ser-io May 6, 2026 06:51

ser-io requested changes May 11, 2026

View reviewed changes

k311093 force-pushed the wait branch 6 times, most recently from 1a22a6e to ec38115 Compare May 26, 2026 06:25

k311093 requested a review from ser-io May 27, 2026 05:18

ser-io requested changes May 27, 2026

View reviewed changes

k311093 force-pushed the wait branch 5 times, most recently from c4d69b0 to ce31cf8 Compare June 4, 2026 06:07

k311093 force-pushed the wait branch 4 times, most recently from 45a38bc to 0d668a2 Compare June 4, 2026 07:26

k311093 force-pushed the wait branch from 0d668a2 to 427a0a1 Compare June 5, 2026 07:11

Conversation

k311093 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ser-io left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jemoreira commented Apr 15, 2026

Uh oh!

jemoreira commented Apr 15, 2026

Uh oh!

k311093 commented Apr 16, 2026

Uh oh!

0405ysj commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ser-io commented Apr 21, 2026

Uh oh!

k311093 commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k311093 commented May 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

k311093 commented Apr 15, 2026 •

edited

Loading

ser-io left a comment •

edited

Loading

k311093 commented Apr 21, 2026 •

edited

Loading