Health management APIs for switches, powershelves and racks

[Matthias247]

Bare Metal Manager currently manages the health for managed hosts:

• It accepts health report from various sources (both internal and external to carbide) and aggregates them.
• The aggregated health can be queried via Machine::health on gRPC.
• External systems can manage health overrides via the gRPC calls
  • ListHealthReportOverrides
  • InsertHealthReportOverride
  • RemoveHealthReportOverride
• This is described in the documentation on https://github.com/NVIDIA/bare-metal-manager-core/blob/main/book/src/architecture/health_aggregation.md

We need a similar system to manage health for additional components that are managed by Bare Metal Manager:

• NvSwitches
• Powershelves
• Racks

For each of these components

• Carbide should offer a similar looking set of APIs, e.g. InsertRackHealthReportOverrides, InsertSwitchHealthReportOverrides, etc.
• the aggregate health should be visible for callers which load the component state (message Switch/Rack/etc)
• The aggregate health, health history, and health override tools should be visible on admin CLI and admin web UI

In addition that we want to have the ability where the health of components within the rack directly affects all machines in the rack. E.g. "if any nvswitch in the Rack is unhealthy with PreventAllocations classification, then all hosts in the rack should show up as unhealthy with the same classification.

Implementing this mechanism will allow users to only monitor the health of hosts when deciding which hosts to use as instances.

[kfelternv]

---

name: FORGE-7810 rack health
overview: "Implement FORGE-7810: persist rack overrides on the racks table, implement gRPC CRUD, and propagate rack health into host aggregate health at read time. CLI/UI surfaces deferred to follow-up PR."
todos:

• id: schema-model
  content: "DB migration: add health_report_overrides jsonb column to racks table. Update Rack model and FromRow. Update Rack proto message with health/override fields."
  status: pending
• id: db-rack-crud
  content: "Add rack DB helpers: insert_health_report_override, remove_health_report_override (mirror machine.rs pattern but on racks table with RackId)."
  status: pending
• id: grpc-rack-handlers
  content: Replace handler stubs in rack.rs with real implementations for Insert/List/Remove. Log-and-ignore when rack_id does not exist in racks table.
  status: pending
• id: read-time-propagation
  content: In derive_aggregate_health, merge rack-level overrides into host aggregate health at read time. Requires host-to-rack lookup via RackConfig.compute_trays or expected_machines join.
  status: pending
• id: tests
  content: "New rack_health.rs test module: insert/list/remove CRUD, idempotent upsert, missing rack ID, read-time propagation to host aggregate health, PreventAllocations visibility, DSX consumer contract."
  status: pending
  isProject: false

---

FORGE-7810 Rack Health Override API

Locked Decisions

• Read-time propagation only: rack overrides are stored on the racks table. Host aggregate health merges them at read time in derive_aggregate_health. No write-time fan-out to host override rows. No dual-write drift, no reconciliation needed.
• PR scope: rack gRPC CRUD + read-time host propagation + tests. CLI/UI surfaces are a follow-up PR.
• Missing rack_id: when DSX consumer sends a rack_id that does not exist in the racks table, return NotFound error. This aligns with the consumer's retry semantics (errors are not cached, so the next identical value message retries the API call). When the rack eventually gets provisioned, the retry succeeds.
• Proof artifact: cargo test output from the new rack_health test module is the proof. No dedicated script needed.

1) DB Migration + Model + Proto

Migration (new file in crates/api-db/migrations):

ALTER TABLE racks ADD COLUMN health_report_overrides jsonb;

Model (crates/api-model/src/rack.rs):

• Add pub health_report_overrides: HealthReportOverrides to Rack struct (reuse the existing HealthReportOverrides type from crates/api-model/src/machine/health_override.rs).
• Update FromRow impl to try_get("health_report_overrides") with a fallback to HealthReportOverrides::default() for NULL rows.
• Update From<Rack> for rpc::forge::Rack to populate new proto fields.

Proto (crates/rpc/proto/forge.proto):

• Add to message Rack:
  • health.HealthReport health = 12; (rack aggregate health, computed from overrides)
  • repeated HealthOverrideOrigin health_overrides = 13; (active override sources/modes)
• Remove stub comments from the three rack health RPCs (lines 136-141).

2) Rack Override DB Helpers

File: crates/api-db/src/rack.rs

Add two functions mirroring the machine pattern in crates/api-db/src/machine.rs lines 982-1050, but operating on racks table with RackId:

• insert_health_report_override(txn, rack_id, mode, health_report, no_overwrite) -- uses jsonb_set on racks.health_report_overrides.
• remove_health_report_override(txn, rack_id, mode, source) -- uses #- operator on racks.health_report_overrides.

Key difference from machine pattern: these use RackId (VARCHAR) not MachineId, and the RETURNING id returns (RackId,).

3) gRPC Handler Implementation

File: crates/api/src/handlers/rack.rs

Replace the three stubs (lines 114-176) with real implementations:

**insert_rack_health_report_override:**

1. Parse rack_id, validate override/report/mode (same checks as machine handler).
2. Set observed_at if missing, set triggered_by from AuthContext.
3. txn_begin.
4. Load rack via db::rack::get. If not found: return NotFound error (aligns with DSX consumer retry semantics).
5. Remove existing override with same source (ignore NotFound).
6. Insert new override via db::rack::insert_health_report_override.
7. Commit.

**list_rack_health_report_overrides:**

1. Parse rack_id.
2. Load rack via db::rack::get. If not found, return NotFound error.
3. Return overrides from rack.health_report_overrides.into_iter().

**remove_rack_health_report_override:**

1. Parse rack_id and source.
2. txn_begin.
3. Load rack. If not found: return NotFound error.
4. Verify source exists in rack's overrides (check replace source or merges key). If not found, return NotFoundError.
5. Remove via db::rack::remove_health_report_override.
6. Commit.

RBAC already wired correctly in crates/api/src/auth/internal_rbac_rules.rs lines 139-150.

4) Read-Time Propagation into Host Aggregate Health

File: crates/api-model/src/machine/mod.rs -- derive_aggregate_health

The key insight: propagation happens at read time, not write time. When derive_aggregate_health runs, it checks if the host belongs to a rack, and if that rack has health overrides, it merges them into the host's aggregate health.

Implementation approach:

• The ManagedHostStateSnapshot or LoadSnapshotOptions needs access to rack overrides. The cleanest option: add an Option<HealthReportOverrides> field (e.g. rack_health_overrides) to ManagedHostStateSnapshot that gets populated during snapshot loading.
• In the managed host SQL query (crates/api-db/src/sql/managed_hosts.sql.template), join through machine_interfaces -> expected_machines -> racks to pull in racks.health_report_overrides. Or alternatively, use RackConfig.compute_trays which already stores resolved Vec<MachineId>.
• Rack override merging must happen regardless of whether the host has a Replace override. Today derive_aggregate_health returns early at line 250-254 when a host Replace exists. The change: after the early return sets self.aggregate_health = over, also merge rack overrides on top of that result before returning. This ensures a physical rack event (e.g. water leak with PreventAllocations) is never hidden by a per-host administrative Replace.
• Concretely, the logic becomes:
  1. Compute host aggregate health as today (Replace early-return or full merge pipeline).
  2. After the aggregate is set (either path), merge rack overrides on top: for rack_health_overrides.replace, treat as merge; for rack_health_overrides.merges, merge each into output.
• This means host Replace still controls the base health state, but rack-level alerts always layer on top.

Host-to-rack resolution options (choose during implementation):

• Option A: Join through expected_machines.rack_id in the managed host SQL template. This adds a lateral join but is accurate.
• Option B: Preload rack overrides for all racks with overrides set, then match by RackConfig.compute_trays containing the host machine_id. This avoids modifying the complex managed host SQL.
• Option B is likely simpler and avoids touching the perf-critical managed host query.

Rack Replace -> Host Merge normalization: A rack-level Replace override is stored faithfully on the rack row, but when propagated to host aggregate health, it is always treated as a Merge. This prevents rack overrides from silently destroying host-specific Replace overrides set by SREs. The rack List API will still show the original mode.

5) Tests

New file: crates/api/src/tests/rack_health.rs, registered in crates/api/src/tests/mod.rs.

Test cases:

• insert_list_remove_rack_override: create rack, insert merge override, list (verify present), remove, list (verify gone).
• idempotent_insert: insert same source twice, verify only one override present, no error.
• remove_nonexistent_source: verify NotFound error.
• missing_rack_id: insert override for nonexistent rack_id, verify NotFound error returned.
• propagation_to_host_aggregate_health: create rack with hosts, insert override with PreventAllocations, load managed host snapshot, verify aggregate health contains the rack override alert with correct classifications.
• host_allocatability_blocked_by_rack_override: create rack with hosts, insert override with PreventAllocations, verify is_usable_as_instance returns Err(HealthAlert).
• rack_override_applies_even_with_host_replace: set a host-level Replace override (empty alerts, marking host healthy), then set a rack-level Merge override with PreventAllocations. Verify host aggregate health contains both the Replace base AND the rack alert on top. Verify is_usable_as_instance returns Err(HealthAlert).
• rack_replace_normalized_to_merge: set a rack-level Replace override, verify it does not destroy a host-level Replace. Rack Replace is treated as Merge at read time.
• dsx_consumer_contract: call insert_rack_health_report_override and remove_rack_health_report_override with the same request shapes the DSX consumer uses (crates/dsx-exchange-consumer/src/api_client.rs lines 77-104), verify success.

Use existing test fixtures: TestRackDbBuilder for rack creation, create_managed_host + ExpectedMachineData { rack_id: Some(...) } for host-rack association.

Proof artifact for PR: run cargo test -p carbide-api rack_health -- --nocapture 2>&1 and paste the output into the PR description under "Testing > Manual testing performed".

Data Flow

flowchart LR
  dsxConsumer[DsxExchangeConsumer] -->|InsertRackHealthReportOverride| rackHandler[rack handler]
  rackHandler --> rackTable[(racks.health_report_overrides)]
  rackTable -->|read time| deriveHealth[derive_aggregate_health]
  hostSources[host health sources] --> deriveHealth
  deriveHealth --> aggregateHealth[host aggregate health]
  aggregateHealth --> api[gRPC GetMachine / FindMachines]

Loading

Files Changed (Expected)

• crates/api-db/migrations/YYYYMMDD_rack_health_report_overrides.sql (new)
• crates/api-model/src/rack.rs (model + FromRow + proto conversion)
• crates/rpc/proto/forge.proto (Rack message fields, remove stub comments)
• crates/api-db/src/rack.rs (insert/remove override helpers)
• crates/api/src/handlers/rack.rs (replace stubs with implementations)
• crates/api-model/src/machine/mod.rs (derive_aggregate_health rack merge)
• crates/api-db/src/managed_host.rs or crates/api-db/src/sql/managed_hosts.sql.template (rack override loading, depending on approach chosen)
• crates/api/src/tests/rack_health.rs (new)
• crates/api/src/tests/mod.rs (register new test module)

Follow-Up PR (Out of Scope)

• Admin CLI rack health-override show/add/remove commands
• Web UI rack health page, override management forms
• Rack health history tracking
• Switch/powershelf health override APIs (GitHub issue Health management APIs for switches, powershelves and racks #240 broader scope)

[ajf]

can you review @Matthias247 @akorobkov-nvda @nvity @zhaozhongn