Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-16505: Add source raw bytes in processorContex #17960

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

Dabz
Copy link

@Dabz Dabz commented Nov 27, 2024

Pull request to implement KIP-1034, it adds the records source byte in the processor context

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch from 7cd809b to 33a294b Compare January 28, 2025 18:44
@loicgreffier
Copy link
Contributor

@cadonna @Dabz Just linking the following PR #16684 that has been reverted.

We faced some issues with store caches. I suppose these issues are still relevant in this PR. Going to check and keep this feed updated.

@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch from 36f7e88 to 6c5bc40 Compare January 29, 2025 10:34
@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch from 132aaab to 82d61c7 Compare January 29, 2025 10:37
@loicgreffier
Copy link
Contributor

loicgreffier commented Jan 29, 2025

From my tests, the following topology:

builder
  .stream()
  .groupByKey()
  .windowedBy(...) // Does not really matter. NPE thrown windowing or not
  .aggregate(...)
  .mapValues(value -> throw new RuntimeException(...))

Is not throwing a NPE anymore (compared to the previous PR #16684), since we're using sourceRawKey and sourceRawValue in ProcessorNode instead of rawRecord().key and rawRecord().value.

However, sourceRawKey and sourceRawValue are lost and not forwarded to the ProcessingExceptionHandler. Opening a new PR to address this case: #18739.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants