refactor(es/parser): detach swc_ecma_parser from swc_emca_lexer

[CPunisher]

Background:

In #10377 and #10399, we create another crate swc_ecma_lexer from swc_ecma_parser.

1. The main goal is that we want to split TokenKind and TokenValue to make lexer and parser run faster. This has been done in swc_ecma_parser, whose Token is only 1 byte. This also means we should refactor the lexer and the parser.
2. SWC always made Lexer and Token public, so the change of Token will introduce a large breaking for rust users. So for compatibility, we have to keep the legacy set of lexer and parser to produce compatible Tokens.
3. However, I made a wrong decision in refactor(es/parser): Split parser into also-lex/parse-only #10399. That is, we separate the common lexer/parser functions as much as possible in the swc_ecma_lexer/common by introducing a set of complexy ParseTrait, LexerTrait, etc, which makes the project chaotic and less comprehensive. You can see that swc_ecma_parser depends on swc_ecma_lexer and calls the common functions everywhere.

Motivation:

1. Apparently it makes DX better. You don't need to jump to empty trait functions before jump to their implementations. You don't need to jump cross two crates forward and back.
2. It also improve the space for performance optimization because you won't be restricted by the trait or the legacy lexer and parser.

Description:

Now it's time to correct the decision. This pr makes swc_ecma_parser self-contained and doesn't depends swc_ecma_lexer any more. On the contrary, this pr makes swc_ecma_lexer depends on some swc_ecma_parser instead such as some common simple data structure like Syntax. For compatibility, I also move and import legacy Token in swc_ecma_parser.

After this pr the swc_ecma_lexer is nearly marked as no longer maintained. All the bug fixes and performance optimization should only be applied in swc_ecma_parser.

Specifically, what I do in this pr is only copy all common functions from swc_ecma_lexer/common to swc_ecma_parser and eliminate the trait-based generics. For example:

// Before
// crates/swc_ecma_lexer/common/...
pub trait Parser {
   fn xxx();
   fn yyy() { ... }
}

pub fn parse_xx<P: Parser>(p: &mut P) { ... }

// Impl for both legacy Parser and new performant Parser
impl common::Parser for Parser { ... }

// After
// crates/swc_ecma_parser/...
impl Parser {
   pub fn parse_xx(&mut self) { // Copy and paste the code }
}

Note that I nearly doesn't change anything in swc_ecma_lexer so the lexer and parser in that crate are still based on trait and common function.

Breaking Changes:

If you don't use Token API, then there's no breaking changes, which means for most rust api users, there's no breaking change. Otherwise you may need to remove the dependencies of swc_ecma_lexer and related imports of traits.

Test in community crates:

• rspack: Remove swc_ecma_lexer and related imports of traits.
• deno_ast: No breaking changes
• dprint-swc-ext: No breaking changes

Future Works:

Actually when I finished copying all the code, the performance got regression. It takes me lots of time to figure it out but I finally keep the regression around -1%. So I have to do some complex optimization ahead of time such as refactor of parse_subscripts. I think it's better to merge the optimization in other PRs. So after this pr is ready I will split the that off.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 20f5d1d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codspeed-hq]

CodSpeed Performance Report

Merging #11148 will degrade performances by 5.32%

_{Comparing CPunisher:10-11-refactor/parser (20f5d1d) with main (146c77c)}

Summary

❌ 1 regression
✅ 110 untouched
🆕 11 new
⏩ 29 skipped¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
🆕	es/lexer/angular	N/A	7.7 ms	N/A
🆕	es/lexer/backbone	N/A	1 ms	N/A
🆕	es/lexer/cal-com	N/A	13.5 ms	N/A
🆕	es/lexer/colors	N/A	28.7 µs	N/A
🆕	es/lexer/jquery	N/A	5.6 ms	N/A
🆕	es/lexer/jquery mobile	N/A	8.7 ms	N/A
🆕	es/lexer/mootools	N/A	4.4 ms	N/A
🆕	es/lexer/three	N/A	20.2 ms	N/A
🆕	es/lexer/typescript	N/A	110.4 ms	N/A
🆕	es/lexer/underscore	N/A	883.8 µs	N/A
🆕	es/lexer/yui	N/A	4.7 ms	N/A
❌	es2020_nullish_coalescing	298.8 µs	315.6 µs	-5.32%

Footnotes

1. 29 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​babel/​traverse@​7.22.1
	@​types/​terser@​3.12.0
	has@​1.0.3
	@​swc/​counter@​0.1.3
	node-releases@​2.0.13
	hasown@​2.0.2
	path-parse@​1.0.7
	is-core-module@​2.12.1
	function-bind@​1.1.2
	@​swc/​plugin-jest@​1.5.117
	@​types/​parse-json@​4.0.0
	is-arrayish@​0.2.1
	axios@​0.21.4
	@​babel/​plugin-transform-react-jsx-development@​7.18.6
	jest-regex-util@​25.2.6 ⏵ 29.6.3
	jest-get-type@​25.2.6 ⏵ 29.6.3			+1
	escape-string-regexp@​2.0.0 ⏵ 1.0.5			-5
	@​jest/​globals@​25.5.2 ⏵ 29.7.0			+1
	@​babel/​plugin-transform-dotall-regex@​7.18.6
	@​babel/​plugin-transform-unicode-regex@​7.18.6
	@​jest/​source-map@​25.5.0 ⏵ 29.6.3
	@​babel/​plugin-transform-unicode-sets-regex@​7.22.3
	@​babel/​plugin-transform-exponentiation-operator@​7.18.6
	@​babel/​plugin-transform-reserved-words@​7.18.6
	@​babel/​plugin-transform-class-properties@​7.22.3
	@​babel/​plugin-transform-private-methods@​7.22.3
	@​babel/​plugin-transform-sticky-regex@​7.18.6
	core-util-is@​1.0.2 ⏵ 1.0.3			-1
	collect-v8-coverage@​1.0.1
	@​babel/​plugin-transform-named-capturing-groups-regex@​7.22.3
	jest-resolve-dependencies@​25.5.4 ⏵ 29.7.0	+1		+1
	@​babel/​plugin-transform-property-literals@​7.18.6
	@​babel/​plugin-transform-optional-catch-binding@​7.22.3
See 381 more rows in the dashboard

View full report