-
Notifications
You must be signed in to change notification settings - Fork 5
Description
Describe the bug
Provider crashes with a nil pointer panic in serialBroadcaster.syncSequence (tx.go:541). The crash is triggered by spam tenants creating ghost leases — leases that are closed on-chain but remain in the provider's memory as active deployments. After repeated failed deployment health checks (~10+ attempts with "available replicas below target"), the provider panics and exits with code 1 or 2.
account address = akash1chnhnu50f6hv98xl0m7xm95vel457ysp32uwpj
deployment sequence = 25857450 (example, recurs with new ghost leases)
group sequence = 1
To Reproduce
Spam tenant creates a lease with the provider
Tenant closes deployment on-chain but provider retains it in memory
Provider deployment-monitor repeatedly checks service replicas, gets available=0 for 10+ attempts
Provider panics with nil pointer dereference in serialBroadcaster.syncSequence
Expected behavior
Provider should handle closed/invalid leases gracefully without panicking. Ghost leases should be detected and cleaned up without requiring a full provider restart.
Crash log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x5e8fffd]
goroutine 56 [running]:
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).syncSequence(...)
pkg.akt.dev/[email protected]/node/client/v1beta3/tx.go:541 +0xfd
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).broadcaster(...)
pkg.akt.dev/[email protected]/node/client/v1beta3/tx.go:572 +0xa0c
Additional context
Provider version: 0.10.7 (latest)
Helm chart: 14.0.8
Kubernetes: K3s
Hardware: Dell PowerEdge R640, 80 vCPU, 384GB RAM
The crash recurs every few hours as new ghost leases appear
Workaround:
automated script that closes ghost leases on-chain hourly and restarts provider, but the underlying panic is not resolved