You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In certain cases, it's possible that the dataset produced by a task will be modified by another task before the task save method is run, resulting in the wrong dataset state being saved.
This happens when task runs process, passes the output on to a subsequent task, and the subsequent tasks runs its processbefore the finish method of the first task. If the second task modifies the data in-place, both tasks reference the same data and the file saved by the first task will include the modifications made by the second task. In practice, this has primarily been seen when a mask is applied.
If save=True for a task, the pipeline should figure out that it needs to save that dataset before it is used by any other task.
The text was updated successfully, but these errors were encountered:
I've been thinking about this one on and off, and I don't think it's realistic to include this sort of logic in the pipeline. Once #278 is finished and merged, the logic to avoid this issue will be straightforward.
In certain cases, it's possible that the dataset produced by a task will be modified by another task before the task
save
method is run, resulting in the wrong dataset state being saved.This happens when task runs
process
, passes the output on to a subsequent task, and the subsequent tasks runs itsprocess
before thefinish
method of the first task. If the second task modifies the data in-place, both tasks reference the same data and the file saved by the first task will include the modifications made by the second task. In practice, this has primarily been seen when a mask is applied.If
save=True
for a task, the pipeline should figure out that it needs to save that dataset before it is used by any other task.The text was updated successfully, but these errors were encountered: