Skip to content

[Provider Bug] Provider crashes with nil pointer panic in serialBroadcaster.syncSequence caused by ghost leases #434

@drape76

Description

@drape76

Describe the bug
Provider crashes with a nil pointer panic in serialBroadcaster.syncSequence (tx.go:541). The crash is triggered by spam tenants creating ghost leases — leases that are closed on-chain but remain in the provider's memory as active deployments. After repeated failed deployment health checks (~10+ attempts with "available replicas below target"), the provider panics and exits with code 1 or 2.

account address = akash1chnhnu50f6hv98xl0m7xm95vel457ysp32uwpj
deployment sequence = 25857450 (example, recurs with new ghost leases)
group sequence = 1

To Reproduce
Spam tenant creates a lease with the provider
Tenant closes deployment on-chain but provider retains it in memory
Provider deployment-monitor repeatedly checks service replicas, gets available=0 for 10+ attempts
Provider panics with nil pointer dereference in serialBroadcaster.syncSequence

Expected behavior
Provider should handle closed/invalid leases gracefully without panicking. Ghost leases should be detected and cleaned up without requiring a full provider restart.

Crash log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x5e8fffd]

goroutine 56 [running]:
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).syncSequence(...)
pkg.akt.dev/[email protected]/node/client/v1beta3/tx.go:541 +0xfd
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).broadcaster(...)
pkg.akt.dev/[email protected]/node/client/v1beta3/tx.go:572 +0xa0c

Additional context
Provider version: 0.10.7 (latest)
Helm chart: 14.0.8
Kubernetes: K3s
Hardware: Dell PowerEdge R640, 80 vCPU, 384GB RAM
The crash recurs every few hours as new ghost leases appear

Workaround:
automated script that closes ghost leases on-chain hourly and restarts provider, but the underlying panic is not resolved

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions