[Tracking] tree-sitter

[ironm00n]

This is a tracking issue for integrating a tree-sitter based parser into various parts of Pyret. Tree-sitter provides a nice opportunity to reduce duplication and fragmentation of parsing and highlighting across different platforms.

Currently there are:

• a custom stream parser defined for CodeMirror 5
• a simple TextMate grammar for the VSCode extension
• Emacs mode + basic syntax highlighting
• limited regex keyword highlighting for (n)vim
• the official pyret-lang parser

Currently code.pyret.org cannot upgrade to CodeMirror 6 since streaming based parsers are no longer supported. CodeMirror 6 officially supports Lezer which is inspired by tree-sitter, but doesn't support a custom scanner (which tree-sitter does support, and is needed for Pyret's whitespace sensitivity).

Steps:

• Basic tree-sitter parser - https://github.com/ironm00n/tree-sitter-pyret
  • adjust the scanner to support nested multi-line comments
• Replace CodeMirror highlighting
  • Upgrade to CodeMirror 6
  • Adapt highlighting to use tree-sitter rather than Lezer (ex see: discussion post)
  • Re-implement indentation, keyword matching, and code folding
• Replace VSCode highlighting
  • Either:
    • wait for official support Support syntax highlighting with tree-sitter microsoft/vscode#50140
    • (ab)use the existing semantic tokens API
• Write tree-sitter queries for Emacs
• Write tree-sitter queries for Neovim
  • wip
• Replace the pyret-lang parser with tree-sitter
  • ensure all parsing tests still pass
  • ensure span information is treated the same way as the current parser (especially regarding whitespace)