feat: deduplicate shared URL downloads across test suites#338
Conversation
|
@ylatuya ping |
|
@rsanchez87 can you have a look to this PR as well? The idea would be to fast up the build of docker images containing all the test suites |
| return (url, local_path) | ||
|
|
||
| max_workers = max(1, min(jobs, len(unique_source_list))) | ||
| with ThreadPoolExecutor(max_workers=max_workers) as dl_pool: |
There was a problem hiding this comment.
We use Pool from multiprocessing
There was a problem hiding this comment.
The idea is to move all the logic for download pool inside the download manager which handled the dedup, the thread lock, the extraction and everything to download through ThreadPoolExecutor.
Do you have a preference for multiprocessing for a reason ?
@dabrain34, tested with Also regression tests ✔️ I’ll test again once the requested changes by @ylatuya are implemented. Thanks! |
|
thanks for the test, indeed this is even better on low speed lines as we dont redownload all the time the AV1 zip file. I'm currently addressing comments from ylatuya. When this is ready I will come back to you |
dabrain34
force-pushed
the
dab_duplication_download
branch
from
dc31a68 to
a3b3d5f
Compare
Introduce a centralized DownloadManager so each URL is downloaded at most once, both within and across selected suites. Saves re-fetching multi-GB archives like AV1-ARGON shared by 12 suites. DownloadManager (fluster/utils.py): - Thread-safe per-URL caching at resources/.cache/; concurrent get() calls on the same URL block on the in-flight download. - BoundedSemaphore caps HTTP concurrency at 8. - Per-URL retry budget; ChecksumMismatchError poisons immediately. - invalidate(url) lets consumers drop a corrupt cached archive. - Context manager: cleanup() runs via __exit__, honoring keep_file. - filename_from_url() strips query strings for safe on-disk names. TestSuite.download() (fluster/test_suite.py): - Requires a DownloadManager (keyword-only). All three download paths consume pre-downloaded archives. - Multi-TV branch pre-downloads unique URLs in parallel before the multiprocessing extraction pool. - Raw source files are moved out of the cache (no double storage). CLI (fluster/fluster.py): - Three-phase: collect URLs across selected suites, parallel pre-download, per-suite extraction. Cross-suite parallelism is the main user-visible win. - All callers (CLI + 7 scripts/gen_*.py) use the with-statement form.
dabrain34
force-pushed
the
dab_duplication_download
branch
from
a3b3d5f to
2bfdac4
Compare
3 participants
Introduce a centralized DownloadManager that ensures each URL is downloaded at most once, eliminating duplicate downloads both across test suites and within a single test suite.
This feature allows to fast up considerably the download of AV1-ARGON* which was downloading each time the 6GB archive for every test vector.
Fix #309