A comprehensive debugging guide and diagnostic toolkit for Midnight Network proof server issues. Covers the 5 most common failure modes with specific diagnosis steps, code fixes, and an automated diagnostic script.
- How Midnight's proof server architecture works
- 5 failure modes with symptoms, diagnosis, and fixes
- Diagnostic workflow (6-step checklist)
- Prevention checklist
- Quick reference table
scripts/diagnose_proof_server.sh— Automated bash diagnostic script (container, health, port, memory, disk, network)scripts/version-check.ts— TypeScript version alignment checker (proof server vs SDK vs Compact compiler)
examples/prewarm-proof-server.ts— Proof server warm-up patternexamples/health-check-docker-compose.yml— Docker Compose with health checksexamples/sdk-timeout-config.ts— Proper timeout configuration
# Run the diagnostic script
chmod +x scripts/diagnose_proof_server.sh
./scripts/diagnose_proof_server.sh localhost 6300 --verbose
# Check version alignment
npx ts-node scripts/version-check.tsBefore debugging, you need to understand the architecture. Midnight's proof server is a separate service (typically running in Docker) that:
- Receives proof generation requests from your application
- Downloads and caches ZK parameters on first use (~30MB)
- Generates zero-knowledge proofs using your circuit's witness data
- Returns the proof to your application for transaction submission
The proof server exposes two main endpoints: /check (health check) and /prove (proof generation). Your SDK communicates with it via HTTP.
Your App → SDK → HTTP → Proof Server (Docker) → ZK Parameters → Proof → Transaction Submission → Node
A failure at any point in this chain can manifest as a timeout, rejection, or silent invalid proof.
- Application hangs on proof generation
ECONNREFUSEDorETIMEDOUTerrors in your logs- SDK retry mechanism keeps retrying with no success
Start by checking if the Docker container is running:
# Check container status
docker ps | grep proof-server
# If not running, check why it stopped
docker logs midnight-proof-server --tail 50Common log messages and what they mean:
# Port conflict — another service is using the same port
Error: bind EADDRINUSE 0.0.0.0:6300
# Missing ZK parameters — first run without pre-downloaded params
Error: ENOENT: no such file or directory, open '/params/circuit_0.params'
# Memory limit — container OOM killed
Killed
Port conflict: Change the host port mapping in your Docker run command:
docker run -d --name midnight-proof-server \
-p 6301:6300 \
midnightntwrk/proof-server:latestMissing parameters: Pre-download the ZK parameters or mount a persistent volume:
docker run -d --name midnight-proof-server \
-v ./proof-params:/params \
-p 6300:6300 \
midnightntwrk/proof-server:latestMemory limits: Increase the container memory allocation:
docker run -d --name midnight-proof-server \
--memory=2g \
-p 6300:6300 \
midnightntwrk/proof-server:latestcurl http://localhost:6300/check
# Should return: {"status":"ok"} or similar- First proof generation takes 2-5 minutes
- Subsequent proofs are fast (under 10 seconds)
- SDK may throw a timeout error before the proof completes
The first proof triggers a download of ZK parameters (~30MB). These parameters are specific to your circuit and are cached after the first download. On a slow connection or with large circuits, this can take several minutes.
Increase SDK timeout for first proof:
const proofConfig = {
timeout: 300_000, // 5 minutes for first proof
retries: 1, // Don't retry — if it fails, diagnose
};Pre-warm the proof server: Generate a dummy proof during application startup:
async function warmUpProofServer() {
console.log('Warming up proof server (first proof may take 2-5 min)...');
try {
await contract.circuits.dummyCircuit(ctx);
console.log('Proof server warmed up.');
} catch (e) {
console.log('Warm-up complete (error expected):', e.message);
}
}- Proof generates successfully (no error in proof generation)
- Transaction submission fails with Error 1010 or similar
- Proof bytes look valid in logs but node rejects them
The proof server generates a proof in a specific wire format. If the format expected by the node doesn't match what the proof server produces, the node rejects it. This is almost always a version mismatch.
Pin all versions to the same release:
{
"dependencies": {
"@midnight-ntwrk/sdk": "0.29.0",
"@midnight-ntwrk/compact": "0.29.0"
}
}docker pull midnightntwrk/proof-server:0.29.0
docker run -d --name midnight-proof-server \
-p 6300:6300 \
midnightntwrk/proof-server:0.29.0- Proofs generate but fail validation
- Error messages reference unexpected proof format versions
- Works on one network (testnet) but fails on another (devnet)
Use the proof server version that matches your target network:
# For testnet
docker pull midnightntwrk/proof-server:testnet-0.29.0
# For devnet
docker pull midnightntwrk/proof-server:devnet-0.29.0Check the Midnight Network status page for current version requirements.
/checkendpoint returns non-200 status- SDK throws "proof server unavailable" before attempting proof generation
- Intermittent failures — sometimes works, sometimes doesn't
| Response | Meaning | Action |
|---|---|---|
{"status":"ok"} |
Healthy | No action needed |
{"status":"degraded","message":"low memory"} |
Under pressure | Increase container memory |
{"status":"error","message":"params not found"} |
Missing parameters | Re-download or remount volume |
| Connection refused | Server not running | Check Docker logs |
| 503 Service Unavailable | Server overloaded | Reduce concurrent proof requests |
Add a health check to your Docker Compose:
services:
proof-server:
image: midnightntwrk/proof-server:0.29.0
ports:
- "6300:6300"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:6300/check"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60sWhen your proof server misbehaves, run through this sequence:
- Check Container Status —
docker ps | grep proof-server - Verify Health —
curl http://localhost:6300/check - Check Version Alignment —
docker exec midnight-proof-server cat /version.txt - Test with Minimal Circuit — Use the simplest possible circuit
- Check Resources —
docker stats midnight-proof-server --no-stream - Check Network Connectivity —
docker exec midnight-proof-server curl -v https://testnet.midnight.network/status
Or run the automated script: ./scripts/diagnose_proof_server.sh
- Pin Docker image tags. Never use
:latestin production. - Mount persistent volumes for ZK parameters. The 30MB download only happens once if you persist the parameters.
- Set appropriate timeouts. First proof: 5 minutes. Subsequent proofs: 30 seconds.
- Monitor proof server health. Add Docker health checks and alert on non-200 responses.
- Keep versions synchronized. Proof server, SDK, Compact compiler, and target network must all be on compatible versions.
- Pre-warm in development. Generate a dummy proof after starting the proof server to trigger parameter download.
| Symptom | Likely Cause | First Action |
|---|---|---|
| Connection refused | Container not running | docker ps, check logs |
| First proof slow (2-5 min) | ZK parameter download | Wait or pre-download |
| Timeout on all proofs | Resource exhaustion | Check memory/disk |
| Proof generates but rejected | Version mismatch | Align versions |
| Intermittent failures | Resource pressure or network | Check health, add retries |
| Health check returns error | Misconfiguration | Check logs, verify params |
- Midnight Developer Docs
- Midnight Discord — #dev-help
- Proof Server Docker Images
- Compact Language Reference
This guide is based on real proof server debugging sessions. If you've hit a proof server failure mode not listed here, share it on the Midnight Discord.