DB close race: goroutines still running cause sql: database is closed errors

[sekulicd]

Description
During shutdown, the database connection is closed before all background goroutines have finished their work. As a result, in-flight operations continue to use the DB and fail with errors like:
time="2026-04-23T11:57:47+02:00" level=error msg="failed to process commitment tx" error="sql: database is closed"
This indicates a race condition between DB shutdown and ongoing async processing (e.g. commitment tx handling).
Expected behavior
All goroutines that depend on the DB should gracefully finish (or be canceled) before the database is closed.

[arkanaai]

Codebase context

Root cause location: client.go lines 289–309 (Stop()) and init.go lines 119–141 (goroutine launch)

The race

Stop() calls a.stopFn() (cancels bgCtx) and then immediately calls a.store.Close() — with no wait in between:

// client.go:299-307
if a.stopFn != nil {
    a.stopFn()   // signals goroutines to stop
}
// ...
a.store.Close()  // closes DB right now — goroutines may still be mid-query

The goroutines launched in init.go — listenForArkTxs, listenForOnchainTxs, listenDbEvents, periodicRefreshDb — react to context cancellation asynchronously. They can be mid-flight on a DB operation when store.Close() runs, producing sql: database is closed.

The store-level sync.WaitGroup in each SQL/KV store (e.g. store/sql/vtxo_store.go, store/sql/tx_store.go) only guards event-emission goroutines spawned inside the stores themselves — it does not cover the top-level client goroutines.

Same issue exists in Lock() (init.go:153): it cancels the context but doesn't close or wait on the store.

Relevant files

File	What it does
client.go:289–309	Stop() — cancels context, closes store without waiting
init.go:119–141	Launches all background DB-using goroutines
init.go:146–178	Lock() — same cancel-without-wait pattern
store/sql/{utxo,vtxo,tx}_store.go	Per-store WaitGroups (event goroutines only)
store/service.go:151–164	Close() delegates to each store

Acceptance criteria

• Stop() waits for all background goroutines (listenForArkTxs, listenForOnchainTxs, listenDbEvents, periodicRefreshDb, and the outer go func in init.go) to return before calling store.Close()
• No sql: database is closed errors appear in logs during normal shutdown
• Lock() does not leave goroutines running that could race against a subsequent Unlock() reinitialisation
• Shutdown completes in bounded time (goroutines respect context cancellation promptly)

Suggested approach

Add a sync.WaitGroup at the arkClient level (e.g. a.bgWg) alongside the existing stopFn:

// init.go — wrap each top-level goroutine
a.bgWg.Add(1)
go func() {
    defer a.bgWg.Done()
    a.listenForArkTxs(ctx)
}()
// ...repeat for the others

// client.go Stop()
if a.stopFn != nil {
    a.stopFn()
}
a.bgWg.Wait()   // drain goroutines first
a.store.Close() // safe to close now

Each listening goroutine already checks ctx.Done(), so they will exit promptly once the context is cancelled — Wait() should return quickly in practice.

[Dunsin-cyber]

happy to pick this up.