Skip to content

feat(cli): concurrent builds of the same grammar on different locations on disk - #5174

Merged
WillLillis merged 3 commits into
tree-sitter:masterfrom
stackmystack:concurrent
Dec 29, 2025
Merged

feat(cli): concurrent builds of the same grammar on different locations on disk#5174
WillLillis merged 3 commits into
tree-sitter:masterfrom
stackmystack:concurrent

Conversation

@stackmystack

@stackmystack stackmystack commented Dec 26, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Hello,

The test suite of tsdl runs many builds of the same grammars (same name ofc) in parallel, and the tests were flaky, which required me to put a --retries 2 defensively, and it's not always guaranteed to work.

I skimmed through the cli, and I figured there shouldn't be a reason to disallow concurrent builds like this.

So I did the fix, and it works for me now.

The choices I made with the hashing and lock file names are ofc arbitrary :)

WillLillis
WillLillis requested changes Dec 27, 2025
View reviewed changes

@WillLillis WillLillis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this! I have a few suggestions/questions.

Comment thread crates/loader/src/loader.rs Outdated
Comment thread crates/loader/src/loader.rs Outdated
Comment thread crates/loader/src/loader.rs
WillLillis
WillLillis reviewed Dec 27, 2025
View reviewed changes
Comment thread crates/loader/src/loader.rs Outdated
@WillLillis WillLillis added the ci:backport release-0.26 Backport label label Dec 27, 2025
@stackmystack

stackmystack commented Dec 27, 2025

Copy link
Copy Markdown
Contributor Author

But on a more general note, I am not quite sure why the locking is necessary: to protect output path or the parser path ? If the former, then this PR addresses it, but if it's the latter, shouldn't we hash config.parser_path ?

@WillLillis

WillLillis commented Dec 27, 2025

Copy link
Copy Markdown
Member

But on a more general note, I am not quite sure why the locking is necessary: to protect output path or the parser path ? If the former, then this PR addresses it, but if it's the latter, shouldn't we hash config.parser_path ?

My understanding is that the locking is protecting the output path. This PR addresses that issue and (presumably) fixes the problem you were running into with tsdl, so I think it's good as is :)

@stackmystack
fix(cli): report library load failure
004c3fc 
Instead of panicking somehere else.

This happens on concurrent builds of the the same grammar.
@stackmystack
feat(cli): concurrent build of same grammar on different paths
310d0fb
WillLillis
WillLillis approved these changes Dec 29, 2025
View reviewed changes

@WillLillis WillLillis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@WillLillis
WillLillis merged commit 5d9605a into tree-sitter:master Dec 29, 2025
19 checks passed
@tree-sitter-ci-bot

tree-sitter-ci-bot Bot commented Dec 29, 2025

Copy link
Copy Markdown

Successfully created backport PR for release-0.26:

@tree-sitter-ci-bot tree-sitter-ci-bot Bot mentioned this pull request Dec 29, 2025
Merged
@WillLillis WillLillis mentioned this pull request Dec 29, 2025
Merged
stackmystack added a commit to stackmystack/tsdl that referenced this pull request Feb 19, 2026
@stackmystack
test: --retries 0 after bumping tree-sitter
c0aa480 
My PR fixed the flakiness issue, so it's time to be strict

tree-sitter/tree-sitter#5174
stackmystack added a commit to stackmystack/tsdl that referenced this pull request Feb 20, 2026
@stackmystack
test: --retries 0 after bumping tree-sitter
1216d2d 
My PR fixed the flakiness issue, so it's time to be strict

tree-sitter/tree-sitter#5174
@mbelyayev mbelyayev mentioned this pull request Aug 30, 2026
Closed
1 task
@scottames-github-bot scottames-github-bot Bot mentioned this pull request Sep 4, 2026
Merged
1 task
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
HarmonybrewBot HarmonybrewBot
!18138 merge bump-tree-sitter-0.27.0 into main
bc0b817 
tree-sitter 0.27.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>## What's Changed
* feat: start working on v0.27 by @clason in tree-sitter/tree-sitter#5127
* docs(cli): include information on generated files by @clason in tree-sitter/tree-sitter#5129
* fix(docs): remove conflicting --release cargo flag in contributing.md by @arkriny in tree-sitter/tree-sitter#5132
* fix(cli): trailing whitespace after multiline text nodes in CST by @arkriny in tree-sitter/tree-sitter#5133
* ci: bump the actions group with 2 updates by @dependabot[bot] in tree-sitter/tree-sitter#5137
* fix(cli): correct discrepancy with cst for `--no-ranges` by @WillLillis in tree-sitter/tree-sitter#5134
* ci: bump korthout/backport-action from 3 to 4 in the actions group by @dependabot[bot] in tree-sitter/tree-sitter#5155
* build(deps): bump the cargo group with 2 updates by @dependabot[bot] in tree-sitter/tree-sitter#5154
* Fix Playground Query by @kevin-hua-kraken in tree-sitter/tree-sitter#5150
* fix(cli): cst printing improvements by @arkriny in tree-sitter/tree-sitter#5161
* docs(cli): better description of files generated by `init` by @clason in tree-sitter/tree-sitter#5152
* fix(cli): set language in cwd for all usages of `highlight` command by @WillLillis in tree-sitter/tree-sitter#5180
* feat(cli): concurrent builds of the same grammar on different locations on disk by @stackmystack in tree-sitter/tree-sitter#5174
* build(deps): bump the cargo group with 2 updates by @dependabot[bot] in tree-sitter/tree-sitter#5184
* fix(cli): canonicalize build `--output` path by @WillLillis in tree-sitter/tree-sitter#5183
* feat(cli): make `test --update` rewrite all corpus files by @arkriny in tree-sitter/tree-sitter#5179
* build(deps): update rquickjs to 0.11.0 by @clason in tree-sitter/tree-sitter#5186
* fix(cli): increase verbosity of `tree-sitter init -u` updates by @WillLillis in tree-sitter/tree-sitter#5178
* fix(cli): restore test summary output for `tree-sitter test` by @arkriny in tree-sitter/tree-sitter#5197
* feat(wasm): add some common used defs to wasm stdlib by @trim21 in tree-sitter/tree-sitter#5199
* fix(wasm): update wasm-stdlib.h by @clason in tree-sitter/tree-sitter#5209
* build(deps): bump clap from 4.5.53 to 4.5.54 in the cargo group by @dependabot[bot] in tree-sitter/tree-sitter#5211
* Fix python free-threading compatibility by @kevmo314 in tree-sitter/tree-sitter#5212
* fix(build): define `_BSD_SOURCE` by @WillLillis in tree-sitter/tree-sitter#5219
* build(deps): bump the cargo group with 3 updates by @dependabot[bot] in tree-sitter/tree-sitter#5227
* docs: various updates by @WillLillis in tree-sitter/tree-sitter#5228
* fix(docs): appease clippy regarding spacing in README by @WillLillis in tree-sitter/tree-sitter#5236
* fix(wasm): regenerate stdlib with wasm-opt by @clason in tree-sitter/tree-sitter#5237
* fix: Clarify lifetime parameters; Rust 2018 idioms by @DanikVitek in tree-sitter/tree-sitter#5223
* feat: free memory automatically by @theanarkh in tree-sitter/tree-sitter#5225
* fix(cli): warn user when `nm` can't be run to verify the symbols inside the parser being built by @WillLillis in tree-sitter/tree-sitter#5245
* feat(ci): ensure `wasm-stdlib.h` is regenerated when wasm stdlib source files are modified by @WillLillis in tree-sitter/tree-sitter#5229
* docs(playground):  highlight full row for highlighted nodes by @Tam1SH in tree-sitter/tree-sitter#5207
* ci: bump actions/github-script from 7 to 8 in the actions group by @dependabot[bot] in tree-sitter/tree-sitter#5248
* build(deps): bump cc from 1.2.52 to 1.2.53 in the cargo group by @dependabot[bot] in tree-sitter/tree-sitter#5247
* wasm: give binaryen the wasi-sdk treatment by @WillLillis in tree-sitter/tree-sitter#5241
* fix(loader): search correct cache directory for wasi-sdk clang by @WillLillis in tree-sitter/tree-sitter#5251
* Fix tree-sitter-ruby crate failing to build to Wasm by @kossnocorp in tree-sitter/tree-sitter#5158
* fix(wasm): add `iswpunct` to stdlib by @clason in tree-sitter/tree-sitter#5130
* fix(lib): address strict aliasing violations with `Array` type by @WillLillis in tree-sitter/tree-sitter#5242
* misc Windows fixes by @WillLillis in tree-sitter/tree-sitter#5256
* docs: document parser library symbol checks for `build` command by @WillLillis in tree-sitter/tree-sitter#5258
* build(deps): bump the cargo group with 2 updates by @dependabot[bot] in tree-sitter/tree-sitter#5263
* fix(wasm): correct several bugs in `realloc` by @WillLillis in tree-sitter/tree-sitter#5261
* fix(ci): add wasm-stdlib.h to cache key by @WillLillis in tree-sitter/tree-sitter#5266
* fix(generate): error if a supertype is defined as a terminal by @WillLillis in tree-sitter/tree-sitter#5267
* Action overflow by @WillLillis in tree-sitter/tree-sitter#5273
* fix(wasm): don't wasm-opt twice by @clason in tree-sitter/tree-sitter#5253
* build(deps): cargo update by @clason in tree-sitter/tree-sitter#5281
* fix(loader): account for nm/ld fix on newer powerpc linux toolchains by @WillLillis in tree-sitter/tree-sitter#5268
* wasm: fix `realloc` optimization by @WillLillis in tree-sitter/tree-sitter#5276
* fix(cli): use --edits value for fuzz tests by @WillLillis in tree-sitter/tree-sitter#5287
* feat(cli): add `--verbose` flag to `build` command by @WillLillis in tree-sitter/tree-sitter#5285
* rust: bump language crate version by @WillLillis in tree-sitter/tree-sitter#5291
* fix(rust): place `std::env::set_var` inside `unsafe` blocks by @WillLillis in tree-sitter/tree-sitter#5293
* build(deps): bump bytes from 1.11.0 to 1.11.1 by @dependabot[bot] in tree-sitter/tree-sitter#5297
* fix(lib): correct ts_parser_parse docstring by @bertie-wheen in tree-sitter/tree-sitter#5295
* fix(test): acquire lock before downloading wasm tools by @WillLillis in tree-sitter/tree-sitter#5296
* fix(cli): allow for both debug logs and graphs by @WillLillis in tree-sitter/tree-sitter#5306
* lib: clean up strict aliasing fixes in array.h by @amaanq in tree-sitter/tree-sitter#5309
* fix(cli): require tree-sitter CLI in CMake template by @WillLillis in tree-sitter/tree-sitter#5310
* fix(rust)!: return `u32` from `Node::child_count` by @WillLillis in tree-sitter/tree-sitter#5313
* feat: allow `-` in grammar names by @jfly in tree-sitter/tree-sitter#5315
* fix(query): prevent cross-branch capture contamination in alternations with quantifiers by @WillLillis in tree-sitter/tree-sitter#5317
* web: add default export to CJS bundle by @chadxz in tree-sitter/tree-sitter#5305
* ci: speed it up by @amaanq in tree-sitter/tree-sitter#5322
* build(deps): bump eslint from 9.11.0 to 9.39.2 in /crates/cli/eslint by @dependabot[bot] in tree-sitter/tree-sitter#5283
* build(deps): bump the cargo group with 5 updates by @dependabot[bot] in tree-sitter/tree-sitter#5316
* build(deps): bump the cargo group with 4 updates by @dependabot[bot] in tree-sitter/tree-sitter#5338
* ci: bump the actions group with 2 updates by @dependabot[bot] in tree-sitter/tree-sitter#5339
* rust: edition 2024, MSRV 1.90, lint cleanup, and dep updates by @amaanq in tree-sitter/tree-sitter#5326
* feat(highlight)!: utf16 support by @WillLillis in tree-sitter/tree-sitter#5324
* use provided by RedoxOS byte orders implementations by @greyshaman in tree-sitter/tree-sitter#5342
* build(deps): bump wasmtime to v36.0.5 LTS by @clason in tree-sitter/tree-sitter#5350
* fix(generate): `unsafe extern` -> `extern` in rendered C parser by @WillLillis in tree-sitter/tree-sitter#5352
* fix(lib): cast NULL in ts_subtree_children macro by @lucasew in tree-sitter/tree-sitter#5343
* ci: add a dry-run workflow to test rust releases by @WillLillis in tree-sitter/tree-sitter#5292
* build(deps): bump ajv from 6.12.6 to 6.14.0 in /crates/cli/eslint by @dependabot[bot] in tree-sitter/tree-sitter#5355
* build(deps): bump minimatch, eslint and eslint-plugin-jsdoc in /crates/cli/eslint by @dependabot[bot] in tree-sitter/tree-sitter#5354
* ci: retrigger crates check on PR updates by @cl

See merge request: Harmonybrew/homebrew-core!18138
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WillLillis WillLillis WillLillis approved these changes

Assignees

No one assigned

Labels

ci:backport release-0.26 Backport label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stackmystack @WillLillis