fix(scan): read the lockfiles the scanner already claimed to support - #143
Merged
Conversation
`pnpm-lock.yaml`, `yarn.lock` and `Pipfile.lock` were listed in `LOCKFILES`, matched by filename, and then handed to a parser that only implemented `package-lock.json` and `requirements.txt`. They fell through it and resolved to zero dependencies, so OSV was never asked a single question about them — and the scan reported no dependency findings, which is indistinguishable from a clean result. This repository is pnpm; it has been scanning itself this way. Every entry in `LOCKFILES` now names its parser, so a filename cannot be added without one. Added: - pnpm v5, v6 and v9 key shapes, normalised rather than version-detected, with peer suffixes stripped from the version only — `_` is legal in a name. - yarn v1 and berry, skipping `__metadata` and anything resolved by a non-registry protocol (`workspace:`, `file:`, `link:`). - `Pipfile.lock`, both sections, skipping VCS entries that carry a ref rather than a version. - `requirements.txt` now keeps PEP 440 pre/post-release pins (`5.0rc1`, `2.2.2.post1`), which the old `([0-9.]+)` pattern silently dropped, and handles extras, markers and flag lines. Wildcard pins stay out: `==1.4.*` is a range wearing an equals sign. - `package-lock.json` v2+ nested paths resolve to the package name instead of keeping an inner `node_modules/` prefix. Two silences also become output, per PRD 0002 R6. A lockfile that parses to nothing now reports that it did, and the 50-package cap now says when it truncated — the operator was previously told nothing either way. Queries are deduplicated by name@version so repeated entries do not spend the cap re-asking. On this repository's own lockfile the parser returns 1876 packages where it previously returned none; every name and version passes the existing OSV validators.
ThreatCrush Security Scan70 finding(s) HIGH/CRITICAL: 12 | MEDIUM: 57 | LOW: 1
…and 20 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
packages/scan/src/node/dependencies.tslisted five lockfiles as supported. Its parser implemented two.pnpm-lock.yaml,yarn.lockandPipfile.lockmatched by filename, fell throughparseDependencies, and resolved to zero dependencies. OSV was never queried, so the scan produced no dependency findings — which looks exactly like a clean result. ThreatCrush is a pnpm repository, so it has been scanning itself this way.This is the half of the product the outreach leads with, and it was inert for the most common JS lockfile in our own target market.
What changed
Each
LOCKFILESentry now carries its parser, so a filename cannot be registered without one.package-lock.jsonnode_modules/prefixes on nested pathspnpm-lock.yamlyarn.lock__metadataand local protocolsPipfile.lockrequirements.txt==pins matching[0-9.]+onlyTwo silences become output, per PRD 0002 R6 ("parse failures are first-class output"):
MAX_DEPS_PER_LOCKFILE = 50cap now reports when it truncated, instead of quietly checking 50 of 1876Queries are also deduplicated by
name@version, so repeated lockfile entries don't spend the cap re-asking the same question.Verification
@threatcrush/scantsc --noEmitclean inpackages/scanandapps/clipnpm-lock.yaml: 1876 packages parsed, previously 0, with every name and version passing the existing OSV input validatorsDeliberately not in scope
prd/0002R4 lists the latter five but omits Java entirely. Most Android repos have no Gradle lockfile at all (dependency locking is opt-in), so that work is SBOM ingestion rather than another file parser.POST /v1/querybatch. OSV takes up to 1000 queries per request, which would make the 50-package cap unnecessary rather than merely visible. Bigger change to network behaviour; better on its own.