feat(scans): skip cdxgen when a commit's dependency set is unchanged (S8) - #225
Merged
Merged
Conversation
…(S8) Fingerprints a scan's manifest/lockfile set, cdxgen scanner version, and scan-time config; when it matches the project's prior succeeded scan on the same ref, the pipeline reuses that scan's preserved SBOM instead of re-running cdxgen (5-30 min), while vulnerability matching and license classification always re-run against current data. A scanner-version bump or config change invalidates the fingerprint automatically. Schema and the pure fingerprint function were built in a prior session; this adds the reuse decision, the extraction/fallback wiring in tasks/scan_source.py, and the cdxgen_scanner_version() config accessor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scans.dependency_fingerprint) and the purecompute_scan_fingerprint()function were built in a prior session (db-designer). This PR adds the reuse decision, the extraction/fallback wiring intasks/scan_source.py, andcore.config.cdxgen_scanner_version()(readsCDXGEN_VERSION, set byDockerfile.workerat image-build time).(project, ref)invalidates the fingerprint automatically (full pipeline re-runs); a failed extraction of the prior scan's preserved SBOM falls back to the full cdxgen path transparently, never aborting the scan.Design
The reuse path only replaces the cdxgen SBOM-generation call. Everything else in the pipeline (fetch, scancode first-party license detection, cosign signing, license classification, Trivy matching, source preservation, finalize) runs unchanged on both paths, through the same
_mark_succeeded/_set_stagewriters, so a reused scan's row state and notifications are indistinguishable from a full-pipeline scan's from the outside.The extracted document is the prior scan's FINAL, fully-processed SBOM (cocoapods-merged, scope-filtered, metadata-stamped), the same bytes
_preserve_source_treealready archives and the weekly rematch beat already reads (extract_preserved_sbom/preserved_tarball_has_sbom). No new extraction machinery.Test plan
tests/unit/test_scan_fingerprint.py(db-designer, 28 cases, 100% line coverage oncompute_scan_fingerprint)tests/integration/test_scan_dependency_fingerprint_migration.py(db-designer, migration round-trip)tests/unit/tasks/test_scan_source_dependency_fingerprint_fallback.py(new, 12 cases: reuse extraction success/failure modes, corrupt/non-object JSON, DB-write failures, full-pipeline fallback when extraction fails)tests/integration/scan/test_scan_source_dependency_fingerprint_reuse.py(new, 4 cases against a real Postgres: same-fingerprint reuse skips cdxgen but Trivy still re-runs, scanner-version bump forces a full re-run, a different ref never reuses another ref's SBOM, idempotent retry of an already-succeeded reused scan)tests/unit(6593 passed, 1 pre-existing environment-only failure unrelated to this change, 8 skipped)tests/integration/scan/+ the migration test (109 passed, 1 deselected, a pre-existing subprocess-timing flake on this local worker-shutdown-grace test, confirmed to fail identically onorigin/mainbefore this change)diff-coveragainstorigin/main: 100% on all four changed production files (108/108 lines)ruff check .andmypy .(full backend, both clean)node tools/em-dash/lint.mjs --base origin/mainclean