feat: wire hub OTel metrics + pipeline health observability (Phases 1-3)

[ptone]

Summary

Phase 1 — Wire Hub OTel Recorders

• Create hubmetrics.NewMeterProvider() with GCP Cloud Monitoring exporter
• Wire dbmetrics and dispatchmetrics recorders in server_foreground.go
• Add metric group toggles via env vars

Phase 2 — Bridge Hub In-Memory Counters to OTel

• Add OTelMetricsRecorder implementing MetricsRecorder with OTel instruments + atomic snapshots
• Add OTelGCPTokenMetrics wrapping GCP token methods with OTel instruments
• Wire in server_foreground.go with graceful fallback

Phase 3 — Pipeline Health Observability

• Add scion.telemetry.pipeline.status gauge that self-reports via cloud exporter every 60s
• Add scion.telemetry.export.errors counter with signal/error_type attributes
• Upgrade pipeline credential logging to structured slog fields
• Fix slog handler to render key=value attributes

Closes #241

Test plan

• Hub metrics export enabled with valid GCPProjectID
• Graceful degradation when GCPProjectID is empty
• OTelMetricsRecorder dual-writes to OTel + atomic snapshot
• /api/metrics JSON endpoint backward compatible
• Pipeline health gauge registers with cloud exporter, cancels on Stop()
• Export error counter increments on failure with correct error_type
• classifyError buckets timeout/auth/quota/other correctly
• All telemetry package tests pass (53 tests)
• Verify scion.telemetry.pipeline.status appears in Cloud Monitoring

[gemini-code-assist]

Code Review

This pull request integrates OpenTelemetry metrics export to GCP Cloud Monitoring for the hub server, introducing database and dispatch metrics recorders. It adds a new hubmetrics package to manage the MeterProvider, which supports disabling specific metric groups via environment variables. The review feedback suggests two important improvements: utilizing a context with a timeout during MeterProvider shutdown to prevent potential hangs, and refactoring the TestIsGroupDisabled test to correctly use its defined table fields instead of hardcoded checks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using context.Background() directly in mp.Shutdown can cause the application shutdown to hang indefinitely if the OpenTelemetry exporter is unable to flush metrics (e.g., due to network issues or GCP API latency). It is recommended to use a context with a reasonable timeout (e.g., 5 seconds) to ensure graceful degradation and prevent hanging container terminations.

				defer func() {
					shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
					defer cancel()
					if err := mp.Shutdown(shutdownCtx); err != nil {
						log.Printf("WARNING: failed to shutdown meter provider: %v", err)
					}
				}()

[gemini-code-assist]

The table-driven test TestIsGroupDisabled defines a disabled field in its test cases but completely ignores it during execution. Instead, it hardcodes the expected outcomes based on tc.value == "false" || tc.value == "0". Additionally, the test cases for "false" and "0" incorrectly set disabled to false in the struct definition. Refactoring the test to use the disabled field directly makes the test cases accurate and the test logic cleaner.

func TestIsGroupDisabled(t *testing.T) {
	tests := []struct {
		value    string
		disabled bool
	}{
		{"", false},
		{"true", false},
		{"1", false},
		{"false", true},
		{"0", true},
	}

	for _, tc := range tests {
		t.Run(tc.value, func(t *testing.T) {
			envVar := "SCION_METRICS_TEST_GROUP"
			if tc.value != "" {
				os.Setenv(envVar, tc.value)
				t.Cleanup(func() { os.Unsetenv(envVar) })
			} else {
				os.Unsetenv(envVar)
			}
			got := isGroupDisabled(envVar)
			if got != tc.disabled {
				t.Errorf("isGroupDisabled(%q=%q) = %t, want %t", envVar, tc.value, got, tc.disabled)
			}
		})
	}
}