Performance Improvements Proposal for Kokaq Core Heap

[iwannabebot]

Summary

We’re exploring targeted performance optimizations to scale the core peek-lock heap in Kokaq, especially under high-concurrency, high-volume workloads. This thread summarizes proposed enhancements across memory usage, disk I/O, and network efficiency.

Goals

• Enable multi-lock concurrency without degrading performance.
• Reduce RAM footprint with smart data layout and lazy access.
• Minimize disk I/O pressure for ack/unlock persistence.
• Optimize network bandwidth for large-scale deployments.

Observations

The core priority queue with peek-lock currently:

• Maintains all items in-memory.
• Supports single active lock (recently extended to multi-lock).
• Writes/reads lock states synchronously.
• Returns full payload in PeekLock() responses.

As Kokaq usage scales, especially in queue-heavy scenarios, this model may:

• Waste memory with stale or expired lock data.
• Generate excessive disk writes on frequent Ack()/Unlock().
• Consume unnecessary network bandwidth when pushing full payloads.

Proposed Improvements

1. Memory Usage

• Store only metadata in heap (id, priority, lock state)
• Lazy-load payloads from an external store (e.g., Redis, BoltDB, Badger)
• Use []*Item (pointers) to avoid slice reallocation and copying
• Periodic cleanup of expired locks to avoid heap bloat
• Use sync.Pool for object reuse to reduce GC pressure

2. I/O Efficiency

• Append-only log for writes to reduce random I/O
• Batch persistence of Ack()/Unlock() with WAL
• Consider memory-mapped files (mmap) to reduce syscall cost and leverage OS page caching
• Avoid syncing to disk on every operation unless explicitly requested (e.g., durable acks)

3. Network Bandwidth

• Return only minimal metadata in PeekLock():

  { "id": "...", "priority": 1, "lockToken": "...", "expiresAt": "..." }

• Introduce optional GET /item/{id} endpoint to retrieve full payload
• Support batch API for PeekLockBatch(n), AckBatch, UnlockBatch
• Add gzip/zstd compression for API responses if enabled by client

4. Multi-Lock TTL Expiry Strategy

• Every locked item gets:
  • lockToken: uuid
  • lockedUntil: timestamp
• If lockedUntil < now, the item is automatically considered "unlocked"
• Background sweeper (optional) can proactively clean up expired locks
• Consider adding a min-heap by lockedUntil for quick expiry scans

5. Optional: Horizontal Scalability Hooks

• Add sharding strategy (by hash(itemID)) to distribute workload across heap instances
• Token-to-item index via in-memory map to reduce linear scans for Ack() and Unlock()

Expected Gains

Area	Current	After Improvement	Trade-offs / Complexity
RAM usage	Grows linearly with items	Flat (metadata only in heap)	Requires external payload store and lazy loading
Disk writes	Per operation	Batched + append-only	Needs WAL management and flush control
Bandwidth	Full payload in every call	Metadata-only + optional fetch	Requires separate data retrieval endpoint
Concurrency	Single lock allowed	Multiple concurrent locks with TTL	Needs lock-token tracking, TTL enforcement, cleanup

Request for Feedback

We're inviting input from the community on:

1. Any real-world use cases where payload size, memory use, or TTL expiration has caused pressure?
2. Preference for pluggable backends for lazy payload fetch (e.g., Redis, disk, DB)?
3. Would mmap be more acceptable than paged heap under scale?
4. Thoughts on extending lock semantics to distributed coordination?