[deployer] Improve Insufficient Capacity Handling

[patrick-ogrady]

Closes: #4522

Problem

The AWS deployer only tried each eligible subnet once on InsufficientInstanceCapacity. If every eligible AZ was temporarily constrained, create failed instead of waiting for capacity to recover. Its fallback retry path also retried fatal RunInstances errors indefinitely, and concurrent launches in one region could amplify the shortage.

Changes

• Classify RunInstances failures from structured AWS metadata instead of display strings.
• Rescan all usable eligible AZs every 15 seconds on capacity exhaustion, without a fixed retry limit.
• Permanently exclude subnets that report InsufficientFreeAddressesInSubnet for the current launch.
• Disable per-operation SDK retries so one deployer loop owns AZ selection, retry cadence, and warnings.
• Reuse one EC2 client token for identical ambiguous or transient retries and create a new token when probe parameters change.
• Preserve standard retryable 5xx and throttling handling while returning fatal authorization, configuration, and quota errors immediately.
• Run one launch at a time per region, with monitoring first in its region, while independent regions continue in parallel.

Grouped deployments remain restricted to their selected AZ so capacity fallback does not violate colocation.

Testing

• just test -p commonware-deployer --features aws (88 passed)
• just check-stability
• just check-fmt
• git diff --check

[github-actions]

Benchmark results

Regressions: 0.

✅ `qmdb::merkleize/v=any::unordered::fixed::mmr k=10000 ch=false s=true cc=true` (2/2 gates passed)

Field	Value
Package	commonware-storage
Benchmark target	qmdb_gungraun
Variant	qmdb::merkleize/v=any::unordered::fixed::mmr k=10000 ch=false s=true cc=true
Filter	*::bench_merkleize::any_unordered_fixed_mmr
Baseline suite	commonware-storage
Gates	EstimatedCycles should decrease; tolerance 10.00%; blob_reads should decrease; tolerance 10.00%
Cargo flags	--features test-traits

Metric	Baseline	Current	Delta	Gate
Ir	13,179,242	13,224,039	+0.34%	-
L1hits	16,855,901	16,923,732	+0.40%	-
LLhits	95,111	102,982	+8.28%	-
RamHits	11,345	12,246	+7.94%	-
TotalRW	16,962,357	17,038,960	+0.45%	-
EstimatedCycles	17,728,531	17,867,252	+0.78%	EstimatedCycles should decrease; tolerance 10.00%
blob_reads	158	158	+0.00%	blob_reads should decrease; tolerance 10.00%

✅ `qmdb::merkleize/v=current::ordered::fixed::mmb chunk=256 k=10000 ch=false s=true cc=true` (2/2 gates passed)

Field	Value
Package	commonware-storage
Benchmark target	qmdb_gungraun
Variant	qmdb::merkleize/v=current::ordered::fixed::mmb chunk=256 k=10000 ch=false s=true cc=true
Filter	*::bench_merkleize::current_ordered_fixed_mmb_chunk_256
Baseline suite	commonware-storage
Gates	EstimatedCycles should decrease; tolerance 10.00%; blob_reads should decrease; tolerance 10.00%
Cargo flags	--features test-traits

Metric	Baseline	Current	Delta	Gate
Ir	15,477,876	15,634,470	+1.01%	-
L1hits	20,339,233	20,569,943	+1.13%	-
LLhits	134,497	153,656	+14.24%	-
RamHits	17,260	20,002	+15.89%	-
TotalRW	20,490,990	20,743,601	+1.23%	-
EstimatedCycles	21,615,818	22,038,293	+1.95%	EstimatedCycles should decrease; tolerance 10.00%
blob_reads	235	235	+0.00%	blob_reads should decrease; tolerance 10.00%

Baseline commit(s): c90c0acdccd2

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	commonware-mcp	3eec41b	Aug 20 2026, 03:30 PM

[cloudflare-workers-and-pages]

Deploying monorepo with    Cloudflare Pages

Latest commit:	3eec41b
Status:	✅  Deploy successful!
Preview URL:	https://8da187a5.monorepo-eu0.pages.dev
Branch Preview URL:	https://deployer-robustness.monorepo-eu0.pages.dev

View logs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.57%. Comparing base (789019f) to head (3eec41b).
⚠️ Report is 4 commits behind head on main.

@@            Coverage Diff             @@
##             main    #4543      +/-   ##
==========================================
+ Coverage   95.52%   95.57%   +0.05%     
==========================================
  Files         610      611       +1     
  Lines      276780   277515     +735     
  Branches     6646     6655       +9     
==========================================
+ Hits       264382   265237     +855     
+ Misses      10210    10109     -101     
+ Partials     2188     2169      -19     

see 49 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 789019f...3eec41b. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.