Problem
The _calc_safety method in gitgalaxy/metrics/signal_processor.py (lines 1350-1388) computes risk exposure for the safety_score dimension by building a small fraction called net_exposure — roughly (danger_hits * 4.0 + other_attack_terms) / (loc + 20) - defense — which for real files is normally well under 0.25.
For any language NOT in tier1 (i.e., languages where fc < 1.0, which includes tier2: Python, JavaScript, C++, C, Ruby, Kotlin, PHP; and tier3: all other languages), the code at lines 1372-1373 subtracts a flat constant systems_buffer = t.get("systems_buffer", 0.25) directly from that already-small fraction:
systems_buffer = t.get("systems_buffer", 0.25) if fc < 1.0 else 0.0
net_exposure = (attack - defense) - systems_buffer
Because 0.25 is larger than the typical net_exposure value it's subtracted from, this doesn't act as a modest tolerance adjustment (which seems to be the intent) — it wipes out nearly the entire attack signal for tier2/tier3 languages.
Empirical Proof
Direct benchmark of SignalProcessor._calc_safety() with controlled synthetic inputs (same high_risk_execution hit count and same file LOC across all three tiers, isolating exactly what systems_buffer does):
| loc |
danger hits |
tier1 (rust) score |
tier2 (python) score |
tier3 (bash) score |
| 150 |
1 |
57.0 |
7.1 |
8.6 |
| 500 |
1 |
52.3 |
5.4 |
5.8 |
| 500 |
3 |
56.9 |
6.4 |
6.9 |
Same single dangerous-execution hit (e.g., one eval() / os.system() / shell execution call), same file size — a Rust file scores 52-66 (correctly flagged as moderate-to-high risk), while a Python file with the identical evidence scores 5-7 (effectively reads as no risk). This is not a subtle miscalibration: for the languages most commonly associated with dynamic/dangerous execution (Python, JS, Ruby, PHP, C, C++, Kotlin), the safety_score dimension is close to blind to real attack signal in typical-sized files (100-500 LOC), because the flat subtraction dominates the signal.
Secondary Artifact
There's also a discontinuous jump in scores once danger_density crosses the vulnerability_density_min (0.03) breach-floor threshold (e.g., tier2 at loc=150 jumps from 11.8 to 46.7 between 3 and 5 danger hits). This creates a step-function rather than a smooth curve, which is worth noting separately but is secondary to the primary flat-subtraction bug.
Root Cause
The systems_buffer is meant to suppress false positives in languages where dynamic/unsafe patterns are common and less risky (tier2/tier3 languages). However, subtracting a fixed density value from a small fraction is dimensionally inconsistent and doesn't scale with the signal:
- Small attack density (e.g., 0.05) minus 0.25 floor = often negative (gets clamped/dampened)
- Large attack density (e.g., 0.50) minus 0.25 floor = still visible but proportionally dampened less than the small signal
This causes tier2/tier3 languages to have their real dangerous-execution signals (eval(), shell execution, etc.) collapsed to noise, while tier1 languages see those same signals clearly.
Proposed Fix
Convert systems_buffer from an absolute subtraction into a proportional discount applied to the attack density itself. For example:
systems_buffer = t.get("systems_buffer", 0.25) if fc < 1.0 else 0.0
# Instead of: net_exposure = (attack - defense) - systems_buffer
# Use: net_exposure = ((attack - defense) * (1.0 - systems_buffer))
Or more explicitly, apply a leniency ratio to the raw attack signal before it's used:
systems_buffer_ratio = t.get("systems_buffer_ratio", 0.75) if fc < 1.0 else 1.0
attack = ((attack_hits + irc) / smoothed_loc) * mp * systems_buffer_ratio
A proportional discount scales down with the signal instead of overwhelming it:
- Small attack density gets a small absolute reduction
- Large attack density gets a proportionally large reduction
- Neither is ever erased outright the way the current flat subtraction can erase a small-but-real signal
The exact discount ratio would need calibration against the golden-master corpus, but the shape of the fix (proportional, not absolute; dimensionally consistent with what it's adjusting) is the core recommendation.
Golden-Master & Verification
Per the Differential Scan protocol in CLAUDE.md, any change to signal_processor.py's risk equations requires:
- Verification via
python tests/tools/crucible_check.py before pushing
- Golden-master re-bless via
python tests/tools/update_golden_master.py with explanation in the PR description
- This will shift
safety_score (and cumulative_risk) scores for a large fraction of non-tier1-language files in the corpus
References
Problem
The
_calc_safetymethod ingitgalaxy/metrics/signal_processor.py(lines 1350-1388) computes risk exposure for the safety_score dimension by building a small fraction callednet_exposure— roughly(danger_hits * 4.0 + other_attack_terms) / (loc + 20) - defense— which for real files is normally well under 0.25.For any language NOT in tier1 (i.e., languages where
fc < 1.0, which includes tier2: Python, JavaScript, C++, C, Ruby, Kotlin, PHP; and tier3: all other languages), the code at lines 1372-1373 subtracts a flat constantsystems_buffer = t.get("systems_buffer", 0.25)directly from that already-small fraction:Because
0.25is larger than the typicalnet_exposurevalue it's subtracted from, this doesn't act as a modest tolerance adjustment (which seems to be the intent) — it wipes out nearly the entire attack signal for tier2/tier3 languages.Empirical Proof
Direct benchmark of
SignalProcessor._calc_safety()with controlled synthetic inputs (samehigh_risk_executionhit count and same file LOC across all three tiers, isolating exactly whatsystems_bufferdoes):Same single dangerous-execution hit (e.g., one
eval()/os.system()/ shell execution call), same file size — a Rust file scores 52-66 (correctly flagged as moderate-to-high risk), while a Python file with the identical evidence scores 5-7 (effectively reads as no risk). This is not a subtle miscalibration: for the languages most commonly associated with dynamic/dangerous execution (Python, JS, Ruby, PHP, C, C++, Kotlin), thesafety_scoredimension is close to blind to real attack signal in typical-sized files (100-500 LOC), because the flat subtraction dominates the signal.Secondary Artifact
There's also a discontinuous jump in scores once
danger_densitycrosses thevulnerability_density_min(0.03) breach-floor threshold (e.g., tier2 at loc=150 jumps from 11.8 to 46.7 between 3 and 5 danger hits). This creates a step-function rather than a smooth curve, which is worth noting separately but is secondary to the primary flat-subtraction bug.Root Cause
The
systems_bufferis meant to suppress false positives in languages where dynamic/unsafe patterns are common and less risky (tier2/tier3 languages). However, subtracting a fixed density value from a small fraction is dimensionally inconsistent and doesn't scale with the signal:This causes tier2/tier3 languages to have their real dangerous-execution signals (eval(), shell execution, etc.) collapsed to noise, while tier1 languages see those same signals clearly.
Proposed Fix
Convert
systems_bufferfrom an absolute subtraction into a proportional discount applied to the attack density itself. For example:Or more explicitly, apply a leniency ratio to the raw attack signal before it's used:
A proportional discount scales down with the signal instead of overwhelming it:
The exact discount ratio would need calibration against the golden-master corpus, but the shape of the fix (proportional, not absolute; dimensionally consistent with what it's adjusting) is the core recommendation.
Golden-Master & Verification
Per the Differential Scan protocol in CLAUDE.md, any change to
signal_processor.py's risk equations requires:python tests/tools/crucible_check.pybefore pushingpython tests/tools/update_golden_master.pywith explanation in the PR descriptionsafety_score(and cumulative_risk) scores for a large fraction of non-tier1-language files in the corpusReferences
gitgalaxy/metrics/signal_processor.py(lines 1350-1388, specifically 1372-1373)fc(feature-count ratio) incalculate_risk_vectorcognitive_load)