Allow for custom parsing solutions

[eoyoa]

The Parsons Problems group in Glimmer is looking to support Parsons Problem generation based on template solutions.
For example, say we have the template Scamper solution:

(reduce `+ `(list 1 2))

We hope for this to generate the following blocks:

- (reduce _ _)
- +
- (list 1 2)

In order to accomplish our goals, we need a custom Scamper parser to support our special syntax:

• backticks (`) to denote the following expression should be popped out as a block, and
• carets (^) to denote all of a block's arguments to be automatically popped out.

This PR:

1. refactors some of the parsing code to allow for easy extensibility, and
2. allows for custom extensions to be passed into tokenizer/parser functions.

Thus, no functionality/Parsons Problem-specific code gets added to this repository, but it allows us to reuse the existing tokenizer/parser.

[eoyoa]

Depends on https://github.com/jakerss2/scamperAccessibility/tree/better-tree

[Copilot]

Pull Request Overview

This PR refactors the parsing code to support custom extensions while preserving existing functionality. The refactoring enables the Parsons Problems group to add custom parsing logic for template solutions without modifying core repository code.

Key changes:

• Refactors the monolithic parser into modular token and parse handlers with extensible interfaces
• Introduces handler-based architecture allowing custom tokenizer and parser extensions to be injected
• Adds comprehensive test coverage for the tokenizer functionality

Reviewed Changes

Copilot reviewed 11 out of 30 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/parser/parser.ts	Main parser logic extracted from original parser.ts with extensibility support
src/parser/tokenhandler.ts	Token handling logic with pluggable handler interface
src/parser/parsehandler.ts	Parse handling logic with pluggable handler interface
test/parser.test.ts	New test suite covering tokenizer functionality
src/web/ide.ts	Updated import path and commented out AST description functionality
src/scamper.ts	Updated import path to new parser location
src/ast.ts	Enhanced AST node handling and commented out description method
package.json	Added Jest globals dependency and updated test script
jest.config.ts	Added ESM support configuration
pnpm-workspace.yaml	Added workspace configuration
src/parser.ts	Removed original monolithic parser file

[Copilot]

Instead of commenting out code, consider removing it entirely or using a feature flag if this functionality might be restored later.

[Copilot]

Remove commented-out code rather than leaving it in the codebase. If this property might be needed later, consider using a feature flag or documenting the decision.

Suggested change

	//name: string = ''
	//parentname: string | null = null;

[Copilot]

The comment has an incorrect escape sequence. It should be "advance past '\'" with proper escaping.

Suggested change

	tokenizer.advance() // advance past '\\
	tokenizer.advance() // advance past '\\'

[Copilot]

[nitpick] The comment refers to skipping "the extra \" but should clarify it's skipping the escaped character, not just the backslash.

Suggested change

	// NOTE: skip the extra \ that we parsed in this case. If/when we support
	// extended escape codes, the size of the jump will obviously grow!
	// NOTE: skip the escaped character (including the backslash) that we parsed in this case.
	// If/when we support extended escape codes, the size of the jump will obviously grow!

[Copilot]

[nitpick] Consider importing only the specific functions needed rather than destructuring from a large module, or use a barrel export for better organization.