Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: enable write ahead buffer for streaming service #39549

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

enhance: enable write ahead buffer for streaming service #39549

wants to merge 3 commits into from
+3,783 −2,279

Conversation

chyezh
Copy link
Contributor

@chyezh chyezh commented Jan 23, 2025

issue: #38399

  • Make a timetick-commit-based write ahead buffer at write side.
  • Add a switchable scanner at read side to transfer the state between catchup and tailing read

@sre-ci-robot sre-ci-robot added area/internal-api area/test sig/testing test/integration integration test size/XXL Denotes a PR that changes 1000+ lines. labels Jan 23, 2025
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign liliu-z after the PR has been reviewed.
You can assign the PR to them by writing /assign @liliu-z in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jan 23, 2025
@chyezh chyezh mentioned this pull request Jan 23, 2025
Open
10 tasks
@chyezh chyezh added this to the 2.6.0 milestone Jan 23, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the make_wal_write_ahead_buffer branch from b5aa2c5 to e718daf Compare January 23, 2025 10:04
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Jan 23, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.39%. Comparing base (05ac404) to head (eb55568).
Report is 3 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #39549      +/-   ##
==========================================
- Coverage   69.40%   69.39%   -0.02%     
==========================================
  Files         302      302              
  Lines       27065    27070       +5     
==========================================
  Hits        18785    18785              
- Misses       8280     8285       +5
Components Coverage Δ
Client ∅ <ø> (∅)
Core 69.39% <ø> (-0.02%) ⬇️
Go ∅ <ø> (∅)

see 2 files with indirect coverage changes

@chyezh chyezh force-pushed the make_wal_write_ahead_buffer branch from e718daf to d17833b Compare February 5, 2025 09:15
@chyezh
enhance: erase the rpc level when wal is located at same node
d208c22 
- Make the wal scanner interface same with streaming scanner.
- Use wal if the wal is located at current node.
- Otherwise fallback the old logic.

Signed-off-by: chyezh <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh chyezh force-pushed the make_wal_write_ahead_buffer branch 2 times, most recently from d1c52c3 to ac391ba Compare February 5, 2025 11:14
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh
enhance: make pchannel level flusher
35c53a9 
- Add a pchannel level checkpoint for flush processing
- Refactor the recovery of flushers of wal
- make a shared wal scanner first, then make multi datasyncservice on it

Signed-off-by: chyezh <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh
enhance: enable write ahead buffer for streaming service
eb55568 
- Make a timetick-commit-based write ahead buffer at write side.
- Add a switchable scanner at read side to transfer the state between
  catchup and tailing read

Signed-off-by: chyezh <[email protected]>
@chyezh chyezh force-pushed the make_wal_write_ahead_buffer branch from ac391ba to eb55568 Compare February 5, 2025 12:54
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@congqixia congqixia Awaiting requested review from congqixia

Assignees
No one assigned
Labels
area/internal-api area/test dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement sig/testing size/XXL Denotes a PR that changes 1000+ lines. test/integration integration test
Projects
None yet
Milestone
2.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@chyezh @sre-ci-robot