tree-sitter-wasms resolution fails in monorepos – falls back to chonkie's local node_modules

[connorblack]

tree-sitter-wasms resolution fails in monorepos – falls back to chonkie's local node_modules

Expected Behavior

When using chonkie in a monorepo setup, the tree-sitter-wasms dependency should be resolved using Node.js's standard module resolution algorithm. This means it should:

• Check the local node_modules directory first
• Traverse up the directory tree to find hoisted dependencies in parent node_modules directories
• Properly resolve packages installed at the monorepo root level
• Work seamlessly with package managers that use hoisting strategies (npm workspaces, yarn workspaces, pnpm, etc.)

Current Behavior & Error

When attempting to use chonkie in a monorepo where dependencies are hoisted to the root, the following error occurs:

Error: Cannot find module 'tree-sitter-wasms'
Require stack:
- /path/to/monorepo/packages/my-package/node_modules/chonkie/dist/chunkers/CodeChunker.js
- /path/to/monorepo/packages/my-package/src/index.js

    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/path/to/monorepo/packages/my-package/node_modules/chonkie/dist/chunkers/CodeChunker.js:10:25)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)

The module resolution fails even though tree-sitter-wasms is properly installed at the monorepo root and accessible to other packages.

Root Cause

The issue stems from hard-coded path logic in the CodeChunker module. Instead of relying on Node.js's module resolution algorithm, the code appears to be making assumptions about the location of tree-sitter-wasms relative to chonkie's installation directory. This breaks in environments where:

• Dependencies are hoisted to a parent directory (common in monorepos)
• Package managers use alternative node_modules structures (e.g., pnpm's symlinked structure)
• The consuming package and chonkie are not in a standard nested node_modules hierarchy

The problematic code likely uses a pattern like:

// Incorrect approach - assumes specific directory structure
const wasmPath = path.join(__dirname, '../node_modules/tree-sitter-wasms/...');

// Should instead use:
require('tree-sitter-wasms');
// or
import 'tree-sitter-wasms';

Affected Environments

This issue affects:

• Monorepo setups using:

  • npm workspaces
  • Yarn workspaces (v1, v2, v3)
  • pnpm workspaces
  • Lerna-managed monorepos
  • Rush monorepos
  • Nx monorepos

• Package managers with hoisting:

  • npm v7+ (with automatic hoisting)
  • Yarn with default hoisting behavior
  • pnpm (even with its unique node_modules structure)

• Non-standard installations:

  • Global installations that reference local packages
  • Linked packages during development
  • Docker containers with optimized dependency layers

Proposed Solution

PR #20 addresses this issue by updating the module resolution logic to use Node.js's standard require() mechanism, which properly handles all the above scenarios.

Workaround

Until the fix is merged, users can work around this issue by:

1. Installing tree-sitter-wasms directly in the package that uses chonkie:

   cd packages/my-package
   npm install tree-sitter-wasms

2. Or, if using pnpm, creating a .pnpmfile.cjs to ensure the dependency is not hoisted:

   module.exports = {
     hooks: {
       readPackage(pkg) {
         if (pkg.name === 'chonkie') {
           pkg.dependenciesMeta = {
             'tree-sitter-wasms': {
               injected: true
             }
           };
         }
         return pkg;
       }
     }
   };

Additional Context

• This issue is particularly problematic for teams adopting monorepo architectures for better code sharing and dependency management
• The fix in PR fix: Refactor WASM file resolution in CodeChunker for improved compatibility #20 should be backwards compatible and not affect users with standard installations
• Similar issues might exist with other dependencies if they use hard-coded path resolution

References

• PR fix: Refactor WASM file resolution in CodeChunker for improved compatibility #20: [Link to PR with the fix]
• Related issue: [Any related issues if they exist]
• Node.js Module Resolution Documentation: https://nodejs.org/api/modules.html#modules_all_together