Feature addition: zones-list to help deal with STOCKOUT issues in GCP

[thatderek]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

When GCE has a capacity stockout in the configured zone, Instances.Insert is accepted but the zone operation fails with ZONE_RESOURCE_POOL_EXHAUSTED (or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS), and the build dies:

==> googlecompute: Error creating instance: The zone 'projects/<project>/zones/us-east1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.

The builder currently takes a single zone, so there is no way to express "any zone in this region is fine". Lately we are seeing stockouts routinely persist longer than any reasonable retry window (often hours, regularly half-days), so retrying in place doesn't help; trying the next zone almost always does. Today the only workarounds live outside Packer (wrapper scripts or CI-level retries that rewrite the zone between attempts).

I'd proposing an optional zones list: when instance creation fails with a capacity error, the builder tries the next candidate zone before giving up. Existing single-zone behavior would be unchanged, and non-capacity errors would still fail immediately.

One small prereq that would need an edit: the operation waiters (refreshZoneOp / refreshGlobalOp / refreshRegionOp in lib/common/driver_gce.go) currently flatten compute.Operation.Error.Errors[] into plain errors.New(e.Message), discarding the structured Code field. Preserving the code in a typed error is what makes stockout classification possible without substring-matching the human-readable message, and it would benefit any future error-specific handling (quota, etc.) independently of this feature.

Use Case(s)

Scheduled image-bake pipelines. We run nightly/weekly Packer builds across several image families, and GCE stockouts in a single zone (most recently for newer machine series) fail otherwise-healthy builds several times a month. Any zone in the build region would serve equally well, since the artifact is an image, not a running instance. This seems broadly applicable to anyone running unattended image builds on machine types with tighter capacity (newer generations, GPU shapes, etc.).

Potential configuration

source "googlecompute" "example" {
  project_id = "my-project"
  zone       = "us-east1-b"

  # Optional: additional zones to try, in order, if instance creation
  # fails with a capacity error in the previous zone.
  zones        = ["us-east1-c", "us-east1-d"]
  machine_type = "n4-standard-4"
  # ...
}

Assumptions / judgement calls that might be different from what ya'll might want for something like this

• zones as a supplement to zone (as shown) vs. a mutually-exclusive alternative to it.
• Whether to restrict candidates to the region derived from zone, since region is currently inferred from it.
• Which operation error codes count as "capacity" (ZONE_RESOURCE_POOL_EXHAUSTED, ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, others?).

Potential References

• GCE troubleshooting doc for resource availability errors: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-vm-creation#resource_availability
• compute.OperationErrorErrors carries the machine-readable Code at failure time: https://pkg.go.dev/google.golang.org/api/compute/v1#OperationErrorErrors
• An earlier error-code-specific retry handling in this plugin: Treat ERROR 4047 as retryable. #34 (treated GCE error 4047 as retryable), add retry loop around ImportSshPublicKey #324 (retry loop around ImportSshPublicKey on HTTP 409)
• Prior art for "try candidates in order": source_image_project_id already accepts a list and falls back through it during image lookup.

---

Happy to do the implementation. My plan would be two PRs: first the typed operation-error change then the zones fallback built on it, with mock-driver coverage for the per-zone failure sequencing.