Description
Static analysis tools are designed to be conservative, which often results in false positives—findings that are technically flagged but do not represent real security issues. These false positives create unnecessary noise, slow down remediation efforts, and reduce developer confidence in scan results.
This issue introduces the False-Positive Classifier, the third major component of Machine Learning Tier 1 – Intelligent Triage. Rather than using a trained machine learning model initially, the classifier will employ a configurable set of deterministic heuristics to estimate the probability that a finding is a false positive.
The output of this classifier will:
- Reduce noise in the dashboard.
- Help contributors prioritize genuine vulnerabilities.
- Collect structured feedback for future ML model training.
- Establish the data collection pipeline required for Tier 2 intelligent prediction models.
Note: This issue only covers the heuristic bootstrap and feedback storage. Model retraining is out of scope.
Problem Statement
PatchPilot currently displays every finding returned by scanners such as:
- Semgrep
- OSV-Scanner
- Gitleaks
While scanners prioritize finding potential issues, they also generate many findings that developers intentionally ignore, including:
- Security rules triggered inside unit tests.
- Example or demo code.
- Sample credentials used in documentation.
- Low-confidence static analysis results.
- Rules known to generate frequent false positives.
Since PatchPilot currently treats all findings equally:
- Developers waste time reviewing harmless findings.
- Dashboard signal-to-noise ratio decreases.
- Users lose trust in scan accuracy.
- Future ML models have no labeled false-positive dataset.
Proposed Solution
Create a new module:
backend/
└── app/
└── ml/
├── fp_classifier.py
└── fp_rules.yaml
Expose the public API:
def classify(finding: dict) -> float:
"""
Returns the probability that a finding is a false positive.
Returns:
float in [0.0, 1.0]
"""The classifier should analyze each finding using deterministic heuristics and return a normalized probability.
Example:
The caller should then enrich the finding with:
{
"fp_score": 0.83,
"likely_false_positive": true
}
Initial Heuristics
The classifier should combine multiple signals rather than relying on a single rule.
1. Test File Detection
Many findings inside test files are intentional.
Common path patterns include:
tests/
test/
__tests__/
spec/
fixtures/
examples/
sample/
demo/
mock/
Example:
Increase FP probability.
2. Low Severity
If:
increase FP probability.
3. Missing CVE Information
If a finding:
- has no CVE
- has no OSV advisory
- has no vulnerability identifier
increase FP probability.
4. Scanner Type
Certain scanners produce fewer actionable findings.
Example:
Conversely:
- Gitleaks secrets
- OSV vulnerabilities
should generally receive lower FP probabilities.
Example heuristic:
| Scanner |
FP Bias |
| Semgrep |
High |
| Gitleaks |
Low |
| OSV |
Very Low |
5. Known Noisy Semgrep Rules
Create:
backend/app/ml/fp_rules.yaml
Example:
semgrep:
- python.lang.correctness.useless-comparison
- python.lang.best-practice.debug-print
- python.lang.performance.inefficient-loop
- javascript.console.log
- generic.unused-variable
If a finding's rule ID exists in this configuration, increase the FP probability.
The classifier should load this configuration dynamically instead of hardcoding rule IDs.
Probability Calculation
Heuristics may be combined using weighted scoring.
Example:
| Signal |
Weight |
| Test path |
+0.35 |
| Low severity |
+0.20 |
| Missing CVE |
+0.15 |
| Known noisy rule |
+0.30 |
Clamp the final score to:
UI Integration
Do not remove findings.
Instead:
If:
attach:
{
"likely_false_positive": true
}The frontend should display these findings with:
- muted styling
- reduced opacity
- "Likely False Positive" badge
- optional tooltip explaining why the finding was classified
Users should still be able to inspect and remediate them.
Feedback API
Introduce a feedback endpoint:
Request:
{
"finding_id": "abc123",
"is_false_positive": true
}Purpose:
- Allow users to confirm or reject classifier predictions.
- Store labels for future supervised ML training.
No retraining should occur in this issue.
Database Changes
Create a new table:
CREATE TABLE feedback (
id INTEGER PRIMARY KEY AUTOINCREMENT,
finding_id TEXT NOT NULL,
is_false_positive BOOLEAN NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);The endpoint should simply persist user feedback.
Configuration
Create:
backend/app/ml/fp_rules.yaml
This file should:
- list known noisy Semgrep rules
- be easy to extend
- be loaded during classifier initialization
Avoid embedding rule IDs directly inside Python code.
Unit Tests
Create:
backend/tests/test_fp_classifier.py
Tests should verify:
✅ Test File Heuristic
Input:
Expected:
✅ Low Severity
Expected:
✅ Missing CVE
Finding without:
Expected:
✅ Known Rule ID
Rule exists in:
Expected:
✅ Secret Finding
Example:
Expected:
✅ High Severity CVE
Critical OSV vulnerability.
Expected:
✅ Threshold
Verify:
adds:
likely_false_positive = True
✅ Feedback Endpoint
Submit:
{
"finding_id":"123",
"is_false_positive":true
}Verify:
- HTTP 200/201 response
- Record inserted into the
feedback table
- Stored values match the request
Acceptance Criteria
Expected Impact
- 📉 Reduces noise from static analysis tools without hiding findings.
- 🎯 Improves vulnerability prioritization and developer focus.
- 🧠 Begins collecting high-quality labeled data for future ML models.
- 📊 Establishes the feedback infrastructure required for supervised learning.
- 🔄 Completes the third core pillar of the Machine Learning Tier 1 – Intelligent Triage pipeline.
Difficulty
Medium
Estimated Time: 5–7 hours
Skills Required
- Python
- FastAPI
- SQLite
- YAML configuration
- Pytest
- Basic machine learning concepts
- Static analysis and security tooling
Description
Static analysis tools are designed to be conservative, which often results in false positives—findings that are technically flagged but do not represent real security issues. These false positives create unnecessary noise, slow down remediation efforts, and reduce developer confidence in scan results.
This issue introduces the False-Positive Classifier, the third major component of Machine Learning Tier 1 – Intelligent Triage. Rather than using a trained machine learning model initially, the classifier will employ a configurable set of deterministic heuristics to estimate the probability that a finding is a false positive.
The output of this classifier will:
Problem Statement
PatchPilot currently displays every finding returned by scanners such as:
While scanners prioritize finding potential issues, they also generate many findings that developers intentionally ignore, including:
Since PatchPilot currently treats all findings equally:
Proposed Solution
Create a new module:
Expose the public API:
The classifier should analyze each finding using deterministic heuristics and return a normalized probability.
Example:
0.83The caller should then enrich the finding with:
{ "fp_score": 0.83, "likely_false_positive": true }Initial Heuristics
The classifier should combine multiple signals rather than relying on a single rule.
1. Test File Detection
Many findings inside test files are intentional.
Common path patterns include:
Example:
Increase FP probability.
2. Low Severity
If:
increase FP probability.
3. Missing CVE Information
If a finding:
increase FP probability.
4. Scanner Type
Certain scanners produce fewer actionable findings.
Example:
Conversely:
should generally receive lower FP probabilities.
Example heuristic:
5. Known Noisy Semgrep Rules
Create:
Example:
If a finding's rule ID exists in this configuration, increase the FP probability.
The classifier should load this configuration dynamically instead of hardcoding rule IDs.
Probability Calculation
Heuristics may be combined using weighted scoring.
Example:
Clamp the final score to:
UI Integration
Do not remove findings.
Instead:
If:
attach:
{ "likely_false_positive": true }The frontend should display these findings with:
Users should still be able to inspect and remediate them.
Feedback API
Introduce a feedback endpoint:
Request:
{ "finding_id": "abc123", "is_false_positive": true }Purpose:
No retraining should occur in this issue.
Database Changes
Create a new table:
The endpoint should simply persist user feedback.
Configuration
Create:
This file should:
Avoid embedding rule IDs directly inside Python code.
Unit Tests
Create:
Tests should verify:
✅ Test File Heuristic
Input:
Expected:
✅ Low Severity
Expected:
✅ Missing CVE
Finding without:
Expected:
✅ Known Rule ID
Rule exists in:
Expected:
✅ Secret Finding
Example:
Expected:
✅ High Severity CVE
Critical OSV vulnerability.
Expected:
✅ Threshold
Verify:
adds:
✅ Feedback Endpoint
Submit:
{ "finding_id":"123", "is_false_positive":true }Verify:
feedbacktableAcceptance Criteria
backend/app/ml/fp_classifier.pyclassify(finding: dict) -> float[0.0, 1.0]backend/app/ml/fp_rules.yamlfor configurable noisy rule IDslikely_false_positivewhenfp_score > 0.7POST /feedbackendpoint to persist user labelsfeedbackdatabase tableExpected Impact
Difficulty
Medium
Estimated Time: 5–7 hours
Skills Required