Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline should prioritize save=True before subsequent tasks #283

Closed
ljgray opened this issue Dec 9, 2024 · 1 comment
Closed

Pipeline should prioritize save=True before subsequent tasks #283

ljgray opened this issue Dec 9, 2024 · 1 comment
Assignees

Comments

@ljgray
Copy link
Contributor

ljgray commented Dec 9, 2024

In certain cases, it's possible that the dataset produced by a task will be modified by another task before the task save method is run, resulting in the wrong dataset state being saved.

This happens when task runs process, passes the output on to a subsequent task, and the subsequent tasks runs its process before the finish method of the first task. If the second task modifies the data in-place, both tasks reference the same data and the file saved by the first task will include the modifications made by the second task. In practice, this has primarily been seen when a mask is applied.

If save=True for a task, the pipeline should figure out that it needs to save that dataset before it is used by any other task.

@ljgray ljgray self-assigned this Dec 9, 2024
@ljgray
Copy link
Contributor Author

ljgray commented Mar 10, 2025

I've been thinking about this one on and off, and I don't think it's realistic to include this sort of logic in the pipeline. Once #278 is finished and merged, the logic to avoid this issue will be straightforward.

@ljgray ljgray closed this as completed Mar 10, 2025
@ljgray ljgray reopened this Mar 10, 2025
@ljgray ljgray closed this as not planned Won't fix, can't repro, duplicate, stale Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant