Skip to content

Contiguous 'dead zones' of real functions silently lost from extraction on production files, independent of mid-file language drift #1184

Description

@squid-protocol

What

On real (non-test-fixture, non-drift-flagged) production source files, Python function
extraction (_slice_by_indentation, Mode C) loses contiguous ranges of otherwise ordinary
functions while functions immediately before and after the range are found correctly. This is
distinct from #1183 (mid-file language misrouting) — none of the three files below trigger
_partition_segments's language switch; every segment stays python throughout.

Root cause not yet isolated — filing this as a confirmed, reproducible pattern with concrete
evidence, not a diagnosed fix. It needs real debugging time in _slice_by_indentation /
_build_indentation_safe_stream.

Concrete repro (ast ground truth vs. extracted, in source order)

gitgalaxy/core/guidestar_lens.py — first 11 functions all missing, last 4 all found:

line   54  MISSED  __init__
line   92  MISSED  scan_project_config
...  (9 more missed) ...
line  306  MISSED  _parse_toml_style_manifest
line  321  FOUND   _extract_execution_triggers
line  345  FOUND   _scan_gitattributes
line  403  FOUND   _scan_gitignore_evasion
line  452  FOUND   _calculate_documentation_coverage

gitgalaxy/galaxyscope.py — first 3 found, then everything from line 682 to EOF (20 functions,
including main) missing:

line   88  FOUND   execution_timeout_failsafe
line   99  FOUND   _init_worker
line  178  FOUND   _process_file_worker
line  682  MISSED  __init__
...  (19 more missed, through line 3013) ...

gitgalaxy/core/prism.py — a missing block sandwiched between two found blocks (lines 68-313
found, 425-718 missing, 731 found again) — rules out a simple "everything after one bad match
cascades to EOF" theory; whatever the mechanism is, it can apparently recover.

None of these are naming collisions or truncation (#1182) — the missed names are short,
ordinary, and unique within their file (__init__, main, cleanup, unmask, callback).
Precision stays high throughout (~99.7% corpus-wide once #1182's truncation is corrected for) —
this is a pure recall/loss problem, not a hallucination problem.

Scale

Measured against GitGalaxy's own 228-file Python self-scan corpus, restricted to the 217 files
that do not trigger #1183's language-drift bug: 2,183 real functions (ast ground truth) vs.
1,375 extracted (name-matched, truncation-normalized) — 63% recall on files with no other
known confounding bug
. Class extraction on the same corpus is 100% (61/61), and the dedicated
per-rule unit test suite (tests/extraction/languages/test_python.py,
tests/extraction/languages/test_python_strict.py) already proves func_start's regex is
correct on isolated snippets — so this is specifically a scope/boundary-resolution problem in
_slice_by_indentation's dedent-scan, not a regex-correctness problem.

Related observations, likely downstream of the same root cause (not separately filed)

  • Args-count accuracy: for functions that are found, args count matches ast ground
    truth exactly only ~46% of the time (944 compared, 433 exact matches). Some mismatches are
    clean 0-counts on multi-line typed signatures that verify correctly when the args regex is
    tested against the captured block in isolation (e.g. _slice_by_braces's own 6-parameter
    signature) — suggesting the block text handed to the args regex in production differs from
    what a standalone reproduction receives, possibly the same underlying mechanism as this issue.
  • file_data.struct_branch (the self-scan DB's per-file keyword-hit aggregate) undercounts
    badly for some files (e.g. gitgalaxy/security/manifest_parser.py reports 0, vs. 271 real
    if/for/while/... occurrences confirmed via tokenize and via running the branch regex
    directly against the file's shielded code_stream, which found 301). The raw regex is fine in
    isolation; the aggregation layer that rolls hits up appears to only credit hits attributed to
    successfully-extracted functions, so it inherits this issue's loss.

How this was found

Diffing GitGalaxy's own self-scan of its own repo against Python's ast module as ground
truth, file by file, then confirming specific missed names weren't a naming/truncation artifact
by checking exact (non-truncated) name matches and manually reading the source at the
found/missed boundary lines.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugUnintended behavior or logic failure in the enginecore-engineModifications to the central physics and parsing enginehelp wantedExtra attention or domain expertise is neededpriority: highCore feature broken, but workarounds exist

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions