Skip to content

fix(code): pin tree-sitter-language-pack <1.0.0 in [code] extra#1219

Open
TUTU244 wants to merge 3 commits into
headroomlabs-ai:mainfrom
TUTU244:fix/1216-code-extra-pin-tree-sitter-language-pack-below-1
Open

fix(code): pin tree-sitter-language-pack <1.0.0 in [code] extra#1219
TUTU244 wants to merge 3 commits into
headroomlabs-ai:mainfrom
TUTU244:fix/1216-code-extra-pin-tree-sitter-language-pack-below-1

Conversation

@TUTU244

@TUTU244 TUTU244 commented Jun 20, 2026

Copy link
Copy Markdown

Description

Closes #

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Performance improvement
  • Code refactoring (no functional changes)

Changes Made

Testing

  • Unit tests pass (pytest)
  • Linting passes (ruff check .)
  • Type checking passes (mypy headroom)
  • New tests added for new functionality
  • Manual testing performed

Test Output

# Paste relevant command output or artifact links here

Real Behavior Proof

  • Environment:
  • Exact command / steps:
  • Observed result:
  • Not tested:

Review Readiness

  • I have performed a self-review
  • This PR is ready for human review

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the CHANGELOG.md if applicable

Screenshots (if applicable)

Add screenshots to help explain your changes.

Additional Notes

@TUTU244
fix(code): pin tree-sitter-language-pack <1.0.0 in [code] extra (head…
4084807 
…roomlabs-ai#1216)

tree-sitter-language-pack >=1.0.0 removed the bundled tree-sitter package
and introduced an incompatible node API (.kind instead of .type, callable
root_node(), parse_bytes() vs parse(), etc.).  The existing Python code in
code_compressor.py relies on the 0.x API throughout, so a fresh install of
headroom-ai[code] silently fails: is_tree_sitter_available() returns True
(tree_sitter_language_pack imports fine) but every parse falls back to
passthrough because `from tree_sitter import Parser` then fails with
ImportError when tree-sitter-language-pack >=1.x is installed.

Fix:
- Pin `tree-sitter-language-pack>=0.10.0,<1.0.0` in pyproject.toml so that
  pip resolves to 0.13.0 (latest 0.x), which declares tree-sitter>=0.25.2
  as a dependency and keeps the existing node-walk code working.
- Also add `from tree_sitter import Parser` to `_check_tree_sitter_available()`
  so that environments where only the 1.x language-pack is present receive a
  clear "not available" signal (and log the install hint) instead of silently
  passing content through unchanged.

Confirmed working: Python 3.14 / tree-sitter 0.25.2 / language-pack 0.13.0.
52% token reduction on a 593-token Python module in manual test.
Pre-existing test_parser_usable_in_thread_pool failure is unrelated to this
change (it fails on main before and after the patch).

Closes headroomlabs-ai#1216
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR governance

This PR does not yet satisfy the required template fields:

  • Fill in Description with a real summary of the change.
  • Replace the placeholder bullets in Changes Made with the actual changes.
  • Check at least one box in Type of Change.
  • Check at least one verification item in Testing.
  • Paste real command output or artifact links in Testing → Test Output.
  • Fill in Real Behavior Proof → Environment.
  • Fill in Real Behavior Proof → Exact command / steps.
  • Fill in Real Behavior Proof → Observed result.
  • Fill in Real Behavior Proof → Not tested.
  • Check I have performed a self-review before requesting human review.
  • Check This PR is ready for human review or convert the PR back to draft.

Please update the PR body, or move the PR back to draft while it is still in progress.

@github-actions github-actions Bot added the status: needs author action Pull request body or readiness checklist still needs author updates label Jun 21, 2026
JerrettDavis
JerrettDavis approved these changes Jun 22, 2026
View reviewed changes

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change is the right direction: pinning tree-sitter-language-pack below the incompatible 1.x API and checking that tree_sitter.Parser is importable prevents the code compressor from silently accepting an unusable tree-sitter install.

Before merge, please refresh the branch because it is currently merge-conflicted (DIRTY), and please replace the empty PR template with the real summary/testing details from the commit message. The implementation itself looks good from my side once those housekeeping issues are fixed.

@JerrettDavis

JerrettDavis commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Maintainer refresh: merged current main and resolved the pyproject.toml conflict by keeping the PR's tree-sitter-language-pack>=0.10.0,<1.0.0 cap together with main's explicit tree-sitter>=0.25.2,<0.26 dependency. Local validation: parsed the code optional extra and ran tests/test_compression/test_code_handler.py plus tests/test_transforms/test_detect_fallback_1123.py (32 passed).

@github-actions github-actions Bot added the status: has conflicts Pull request has merge conflicts with the base branch label Jun 25, 2026
JerrettDavis
JerrettDavis approved these changes Jun 25, 2026
View reviewed changes

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the tree-sitter pin refresh. I pushed one test-only follow-up (d2d15f0f) so the thread-safety regression test passes bytes to Parser.parse(), matching the API the production compressor already uses.

Local validation:

  • uv run --extra code --with pytest --with pytest-asyncio python -m pytest tests/test_transforms/test_code_compressor.py tests/test_transforms/test_tree_sitter_thread_safety.py tests/test_code_compressor_thread_safety.py -q (77 passed)
  • uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_tree_sitter_thread_safety.py pyproject.toml

Approval stands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JerrettDavis JerrettDavis JerrettDavis approved these changes

Assignees

No one assigned

Labels

status: has conflicts Pull request has merge conflicts with the base branch status: needs author action Pull request body or readiness checklist still needs author updates

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@TUTU244 @JerrettDavis