Skip to content

feat(scans): skip cdxgen when a commit's dependency set is unchanged (S8) - #225

Merged
haksungjang merged 1 commit into
mainfrom
feat/s8-dependency-fingerprint-scan-reuse
Aug 24, 2026
Merged

feat(scans): skip cdxgen when a commit's dependency set is unchanged (S8)#225
haksungjang merged 1 commit into
mainfrom
feat/s8-dependency-fingerprint-scan-reuse

Conversation

@haksungjang

@haksungjang haksungjang commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • S8 from the concurrency-scaling plan (§3.2, unit 29). When a commit's manifest/lockfile set, cdxgen scanner version, and scan-time config all match the project's prior succeeded scan on the same ref, the pipeline reuses that scan's preserved SBOM instead of re-running cdxgen (5-30 min per scan). Vulnerability matching and license classification always re-run against current Trivy DB / policy state, regardless of reuse.
  • Schema (scans.dependency_fingerprint) and the pure compute_scan_fingerprint() function were built in a prior session (db-designer). This PR adds the reuse decision, the extraction/fallback wiring in tasks/scan_source.py, and core.config.cdxgen_scanner_version() (reads CDXGEN_VERSION, set by Dockerfile.worker at image-build time).
  • Accuracy safety net: a scanner-version bump or a different (project, ref) invalidates the fingerprint automatically (full pipeline re-runs); a failed extraction of the prior scan's preserved SBOM falls back to the full cdxgen path transparently, never aborting the scan.

Design

The reuse path only replaces the cdxgen SBOM-generation call. Everything else in the pipeline (fetch, scancode first-party license detection, cosign signing, license classification, Trivy matching, source preservation, finalize) runs unchanged on both paths, through the same _mark_succeeded/_set_stage writers, so a reused scan's row state and notifications are indistinguishable from a full-pipeline scan's from the outside.

The extracted document is the prior scan's FINAL, fully-processed SBOM (cocoapods-merged, scope-filtered, metadata-stamped), the same bytes _preserve_source_tree already archives and the weekly rematch beat already reads (extract_preserved_sbom / preserved_tarball_has_sbom). No new extraction machinery.

Test plan

  • tests/unit/test_scan_fingerprint.py (db-designer, 28 cases, 100% line coverage on compute_scan_fingerprint)
  • tests/integration/test_scan_dependency_fingerprint_migration.py (db-designer, migration round-trip)
  • tests/unit/tasks/test_scan_source_dependency_fingerprint_fallback.py (new, 12 cases: reuse extraction success/failure modes, corrupt/non-object JSON, DB-write failures, full-pipeline fallback when extraction fails)
  • tests/integration/scan/test_scan_source_dependency_fingerprint_reuse.py (new, 4 cases against a real Postgres: same-fingerprint reuse skips cdxgen but Trivy still re-runs, scanner-version bump forces a full re-run, a different ref never reuses another ref's SBOM, idempotent retry of an already-succeeded reused scan)
  • Full tests/unit (6593 passed, 1 pre-existing environment-only failure unrelated to this change, 8 skipped)
  • tests/integration/scan/ + the migration test (109 passed, 1 deselected, a pre-existing subprocess-timing flake on this local worker-shutdown-grace test, confirmed to fail identically on origin/main before this change)
  • diff-cover against origin/main: 100% on all four changed production files (108/108 lines)
  • ruff check . and mypy . (full backend, both clean)
  • node tools/em-dash/lint.mjs --base origin/main clean

…(S8)

Fingerprints a scan's manifest/lockfile set, cdxgen scanner version, and
scan-time config; when it matches the project's prior succeeded scan on
the same ref, the pipeline reuses that scan's preserved SBOM instead of
re-running cdxgen (5-30 min), while vulnerability matching and license
classification always re-run against current data. A scanner-version
bump or config change invalidates the fingerprint automatically.

Schema and the pure fingerprint function were built in a prior session;
this adds the reuse decision, the extraction/fallback wiring in
tasks/scan_source.py, and the cdxgen_scanner_version() config accessor.
@haksungjang
haksungjang merged commit 57acb95 into main Aug 24, 2026
24 checks passed
@haksungjang
haksungjang deleted the feat/s8-dependency-fingerprint-scan-reuse branch August 24, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant