Skip to content

fix: warehouse graceful termination and avoid marshalling proc error job payloads#2

Closed
deepshekhardas wants to merge 1 commit into
masterfrom
fix/pr-6767-6872-graceful-termination-proc-error
Closed

fix: warehouse graceful termination and avoid marshalling proc error job payloads#2
deepshekhardas wants to merge 1 commit into
masterfrom
fix/pr-6767-6872-graceful-termination-proc-error

Conversation

@deepshekhardas

@deepshekhardas deepshekhardas commented May 5, 2026

Copy link
Copy Markdown
Owner

This PR combines fixes from two key PRs:

PR rudderlabs#6767 - Warehouse Graceful Termination:

  • Replace panic with proper error handling in warehouse router
  • Files changed: warehouse/router/identities.go, warehouse/router/state.go

PR rudderlabs#6872 - Avoid Marshalling Proc Error Job Payloads:

  • Introduce procErrorJob type to avoid expensive marshal/unmarshal round-trip
  • Update processor to use new type for error job handling
  • Files changed: processor/pipeline_worker.go, processor/processor.go

Changes:

  • 4 files changed, 22 insertions(+), 12 deletions(-)
  • Added procErrorJob struct wrapping jobsdb.JobT with parsed events
  • Removed unnecessary jobsdb import from pipeline_worker.go
  • Improved error handling in warehouse graceful termination

Summary by cubic

Gracefully handles warehouse router termination by replacing panics with proper errors, and reduces processor overhead by avoiding marshal/unmarshal for error jobs. Prevents crashes and cuts CPU/GC in error paths.

  • Bug Fixes

    • Replaced panics with error returns and logs in warehouse/router (manager init, schema marshal, upload ID scan); state.go now returns an empty state on invalid input instead of panicking.
  • Performance

    • Introduced procErrorJob wrapping *jobsdb.JobT with parsed events to avoid marshal/unmarshal; updated maps to map[string][]procErrorJob in processor and pipeline_worker. Removed an unused jobsdb import.

Written for commit 0d05b4e. Summary will update on new commits.

@deepshekhardas

Copy link
Copy Markdown
Owner Author

Closing this combined PR. Will create separate PRs for rudderlabs#6767 and rudderlabs#6872 for easier review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant