Allow to regenerate memo files for images imported after a specific date #155
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to #148
The changes made in #150 aimed to improve the distribution of the memo file regeneration to make better use of the resources allocated to the process. This used the initialization time as measured at import time and stored in the database as a proxy to assess the processing time for each fileset.
While reasonable, these assumptions might be challenged by the storage realities e.g. deployments applying data management policies that move data through different tiered storages. For these cases, regenerating all the memo files involves reading all the data from archived/slow storage/.... which might have real implications both in terms of time and billing.
This PR proposes to mitigate this scenario by specifying an import time cut-off and regenerating only memo files for images imported after this cut-off.
The logic use the image creation time retrieved from the database as well as PSQL variable assignment to filter entries based on the import date. The default behavior is unmodified and should return a CSL files for all images.
To test this PR, compare the outcome of the process with different variants of
--since
argumentsFuture extensions might take advantage of the
image.archived
flag which has been proposed to communicate the archival state of a particular image - see ome/openmicroscopy#6390./cc @erindiel @atTODO