Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DST horizontal partitioning #556

Merged
merged 13 commits into from
Mar 11, 2025
Merged

DST horizontal partitioning #556

merged 13 commits into from
Mar 11, 2025

Conversation

avillega
Copy link
Contributor

@avillega avillega commented Mar 6, 2025

This PR contains several related changes to be able to make the DST run faster and enable validations between promises and tasks.

  • Make DST do horizontal partitions based on the rootPromiseId
  • Add many validations between tasks and promises as well as adjusting some others
  • Make a complete promise operation also complete all related tasks including the current claimed task
  • General small changes to the way we print and report DST results to make it easier to debug.
  • Make trying to complete a task that is already completed return a StatusOk response
  • Create a single CreatePromiseAndTask command that is aware of not creating a task in case a promise was not being able to be created.

@avillega avillega requested a review from dfarr March 6, 2025 23:50
Comment on lines +310 to +319
if isCreatePromiseAndTask {
promiseAndTaskResult := completion.Store.Results[0].CreatePromiseAndTask
util.Assert(promiseAndTaskResult.PromiseRowsAffected == 0 || promiseAndTaskResult.PromiseRowsAffected == 1, "Creating promise result must return 0 or 1 rows")
if promiseAndTaskResult.PromiseRowsAffected == 0 {
util.Assert(promiseAndTaskResult.TaskRowsAffected == 0, "If not promise was created a task must have not been created")
}
} else {
createPromiseResult := completion.Store.Results[0].CreatePromise
util.Assert(createPromiseResult.RowsAffected == 0 || createPromiseResult.RowsAffected == 1, "CreatePromise result must return 0 or 1 rows")
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general note, I might want to come back to this code and refactor it in a cleaner way. the isCreatePromiseAndTask every where makes me believe there is a cleaner approach to it.

Copy link

codecov bot commented Mar 7, 2025

Codecov Report

Attention: Patch coverage is 51.76909% with 259 lines in your changes missing coverage. Please review.

Project coverage is 52.38%. Comparing base (7195f7f) to head (d9e5337).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
test/dst/dst.go 54.27% 88 Missing and 3 partials ⚠️
test/dst/bc_validator.go 0.00% 62 Missing ⚠️
test/dst/validator.go 32.00% 50 Missing and 1 partial ⚠️
...rnal/app/subsystems/aio/store/postgres/postgres.go 62.96% 13 Missing and 7 partials ⚠️
internal/app/subsystems/aio/store/sqlite/sqlite.go 68.42% 8 Missing and 4 partials ⚠️
internal/app/subsystems/aio/sender/sender_dst.go 0.00% 9 Missing ⚠️
test/dst/generator.go 75.00% 5 Missing ⚠️
cmd/dst/run.go 0.00% 4 Missing ⚠️
internal/kernel/t_aio/store.go 0.00% 2 Missing ⚠️
internal/kernel/t_api/status.go 0.00% 2 Missing ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #556      +/-   ##
==========================================
- Coverage   55.21%   52.38%   -2.84%     
==========================================
  Files         133      133              
  Lines       14163    14408     +245     
==========================================
- Hits         7820     7547     -273     
- Misses       5881     6400     +519     
+ Partials      462      461       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@dfarr dfarr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! This ended up being quite a chunk of work

avillega and others added 2 commits March 10, 2025 15:20
@avillega
Copy link
Contributor Author

This PR fixes #526 and #514

@avillega avillega merged commit f188e8b into main Mar 11, 2025
4 of 6 checks passed
@avillega avillega deleted the avillega/task-stable-id branch March 11, 2025 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants