Overview
Implement controlled emergency mechanisms that allow maintainers to pause specific money-moving operations or isolate failing providers without taking the entire ChainMove application offline.
Problem
During a payment-provider outage, reconciliation incident, suspected duplicate-credit bug, compromised key, or database inconsistency, maintainers need to stop risky writes quickly while preserving login, support visibility, repayment history, and operational investigation. Ad-hoc environment changes and redeployments are too slow and can disable unrelated functionality.
Scope
- Define versioned operational controls for wallet funding, wallet debit, investments, down payments, repayments, payouts, KYC uploads, account linking, admin adjustments, and provider-specific integrations.
- Support states such as enabled, degraded, paused, and read-only with safe user-facing messages.
- Add provider circuit breakers based on failure rate, consecutive failures, timeout rate, and recovery probes.
- Store control changes with actor, reason, incident/reference ID, scope, start/expiry time, and approval requirements.
- Apply controls centrally before domain mutation, not only in frontend buttons.
- Allow safe in-flight/idempotent completion rules so retries do not create ambiguous states.
- Add emergency activation with strict authorization and maker-checker for high-impact/global controls where practical.
- Expose admin status APIs/UI and a public-safe service-status summary without internal details.
- Emit audit events, metrics, and notifications for activation, expiry, bypass, and recovery.
- Add automatic expiry for temporary controls plus explicit recovery checklist.
- Provide local/mock behavior and incident runbooks.
Safety requirements
- Kill switches must fail closed for the protected mutation when configuration state is unavailable.
- Webhooks may still need durable receipt while downstream processing is paused; document per-flow behavior.
- No switch may silently discard accepted payment events.
- Bypass requires elevated authorization, reason, and audit record.
- Pausing one provider must not unnecessarily disable mock mode or unrelated providers.
Test requirements
- pause each operation category
- provider failure threshold and half-open recovery
- concurrent control updates
- automatic expiry
- in-flight idempotent retry during pause
- webhook receipt while processing is paused
- unauthorized bypass
- safe read-only API behavior
- audit/metric emission
Acceptance criteria
Difficulty
Hard
Overview
Implement controlled emergency mechanisms that allow maintainers to pause specific money-moving operations or isolate failing providers without taking the entire ChainMove application offline.
Problem
During a payment-provider outage, reconciliation incident, suspected duplicate-credit bug, compromised key, or database inconsistency, maintainers need to stop risky writes quickly while preserving login, support visibility, repayment history, and operational investigation. Ad-hoc environment changes and redeployments are too slow and can disable unrelated functionality.
Scope
Safety requirements
Test requirements
Acceptance criteria
Difficulty
Hard