Skip to content

♻️ preparing TasksManager's interface to be extracted into a common interface #7884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Jul 18, 2025

Conversation

GitHK
Copy link
Contributor

@GitHK GitHK commented Jun 12, 2025

What do these changes do?

Completely separated the interface of the TaskManager form the "public interface" which the user facing API expose placed in packages/service-library/src/servicelib/long_running_tasks/lrt_api.py. No more internals exposed form the TaskManager to the outside world.

  • unique tasks are handled in a simpler manner, they are simple determined by their task_id and are no longer kept track via complex data structures
  • start_task is now part of the "public interface" and no longer a utility bundled with the TaskManager
  • tasks are now started by name and have to be registered in the TasksRegistry before starting them via the TaskManager

In future developments, these changes will allow, without altering the current "public interface" to:

  • solve the issues we are facing with sticky sessions (sticky sessions will no longer be required)
  • start tasks that are defined and run in separate processes
  • unify the long_running_tasks with the celery workers

Related issue/s

How to test

Dev-ops

@GitHK GitHK self-assigned this Jun 12, 2025
Copy link

codecov bot commented Jun 12, 2025

Codecov Report

Attention: Patch coverage is 97.88360% with 4 lines in your changes missing coverage. Please review.

Project coverage is 88.27%. Comparing base (e805af6) to head (ca4949d).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7884      +/-   ##
==========================================
+ Coverage   88.14%   88.27%   +0.13%     
==========================================
  Files        1869     1677     -192     
  Lines       72052    67620    -4432     
  Branches     1279      947     -332     
==========================================
- Hits        63507    59692    -3815     
+ Misses       8165     7638     -527     
+ Partials      380      290      -90     
Flag Coverage Δ
integrationtests 64.27% <32.85%> (+0.03%) ⬆️
unittests 86.78% <97.35%> (+0.02%) ⬆️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library 93.15% <100.00%> (+<0.01%) ⬆️
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration 70.17% <ø> (ø)
pkg_service_library 71.49% <99.05%> (-0.02%) ⬇️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 84.99% <ø> (-0.06%) ⬇️
agent 93.81% <ø> (∅)
api_server 93.02% <ø> (ø)
autoscaling 95.88% <ø> (ø)
catalog 92.34% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 92.37% <ø> (+0.56%) ⬆️
datcore_adapter 97.94% <ø> (ø)
director 76.14% <ø> (-0.19%) ⬇️
director_v2 91.12% <90.00%> (+0.12%) ⬆️
dynamic_scheduler 96.27% <ø> (ø)
dynamic_sidecar 90.07% <97.43%> (+<0.01%) ⬆️
efs_guardian 89.76% <ø> (ø)
invitations 91.44% <ø> (ø)
payments 92.60% <ø> (ø)
resource_usage_tracker 92.55% <ø> (ø)
storage 86.77% <ø> (+0.37%) ⬆️
webclient ∅ <ø> (∅)
webserver 88.57% <100.00%> (+0.04%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e805af6...ca4949d. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@GitHK GitHK changed the title ♻️ WIP: long_running_tasks refactor ♻️ preparing TasksManager's interface to be extracted into a common interface Jun 13, 2025
@GitHK GitHK added this to the Engage milestone Jun 13, 2025
@GitHK GitHK requested review from Copilot June 13, 2025 09:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the TasksManager interface to simplify unique task handling and prepares it to be extracted into a common interface. Key changes include simplifying progress update calls by removing explicit ProgressPercent wrappers, replacing direct start_task calls with lrt_api.start_task using function names, and updating tests and API endpoints to align with these changes.

Reviewed Changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated no comments.

Show a summary per file
File Description
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/long_running_tasks.py Replaced ProgressPercent wrapped values with numeric literals and added TaskRegistry import.
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/api/rest/containers_long_running_tasks.py Updated start_task calls to await lrt_api.start_task and pass task function names.
services/director-v2/src/simcore_service_director_v2/api/routes/dynamic_scheduler.py Refactored task starting logic with lrt_api and adjusted TaskRegistry registration/unregistration.
packages/service-library/* Refactored multiple tests and tasks to use lrt_api, updated HTTP status expectations, and removed deprecated task_name usage in models.
packages/service-library/src/servicelib/long_running_tasks/task.py Updated start_task implementation and unique task ID logic using TaskRegistry.
packages/models-library/src/models_library/api_schemas_long_running_tasks/tasks.py Removed task_name from TaskBase to align with the refactored interface.
Comments suppressed due to low confidence (2)

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/long_running_tasks.py:152

  • Since the ProgressPercent wrapper has been removed in favor of passing numeric literals directly, consider updating the function's docstring and any related documentation to clearly indicate that the 'percent' parameter now expects a float value.
progress.update(message="started pulling user services", percent=0)

packages/models-library/src/models_library/api_schemas_long_running_tasks/tasks.py:21

  • The removal of the 'task_name' field from TaskBase could affect clients that expect this information. Please ensure that the API documentation and any client integrations are updated accordingly to reflect this change.
class TaskBase(BaseModel):

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments. Thanks!

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx. careful with this refactoring towards the coming release. Perhaps avoid it gets into staging?

@GitHK
Copy link
Contributor Author

GitHK commented Jun 19, 2025

⚠️ Please avoid merging for now

@GitHK GitHK added the 🤖-do-not-merge (optional) blocks Mergify from merging the PR label Jun 19, 2025
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Just a few comments. Thanks

@GitHK GitHK added 🤖-automerge marks PR as ready to be merged for Mergify and removed 🤖-do-not-merge (optional) blocks Mergify from merging the PR labels Jul 16, 2025
@GitHK
Copy link
Contributor Author

GitHK commented Jul 16, 2025

@Mergifyio queue

Copy link
Contributor

mergify bot commented Jul 16, 2025

queue

🛑 The pull request has been removed from the queue default

The following conditions don't match anymore:

  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • any of: [🛡 GitHub branch protection]
        • check-neutral = deploy to dockerhub
        • check-skipped = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-neutral = system-tests
        • check-skipped = system-tests
        • check-success = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-neutral = unit-tests
        • check-skipped = unit-tests
        • check-success = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
        • check-success = check OAS' are up to date
      • any of: [🛡 GitHub branch protection]
        • check-neutral = integration-tests
        • check-skipped = integration-tests
        • check-success = integration-tests
      • any of: [🛡 GitHub branch protection]
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
        • check-success = build-test-images (frontend) / build-test-images

Copy link
Contributor

mergify bot commented Jul 18, 2025

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

@GitHK
Copy link
Contributor Author

GitHK commented Jul 18, 2025

@Mergifyio requeue

Copy link
Contributor

mergify bot commented Jul 18, 2025

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request.

Copy link
Contributor

mergify bot commented Jul 18, 2025

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at ae5f894

Copy link

@mergify mergify bot merged commit ae5f894 into ITISFoundation:master Jul 18, 2025
94 of 97 checks passed
@GitHK GitHK deleted the pr-osparc-long-running-refactor-4 branch July 18, 2025 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖-automerge marks PR as ready to be merged for Mergify
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants