Skip to content

Make most lexer errors recoverable #204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

mikmart
Copy link
Contributor

@mikmart mikmart commented Jul 7, 2025

The idea is that when the lexer encounters an error it can tell the compiler whether compilation can continue or not based on the Err variant of Result rather than a simple Option:

pub type Result = core::result::Result<(), ErrorKind>;

#[derive(Clone, Copy)]
pub enum ErrorKind {
    Error, // An (invalid) token is available; compilation can continue.
    Fatal, // No token is available; compilation must be terminated.
}

However there was a bad interaction with the backtracking we do in the compiler, which caused duplicate errors being reported when tokens were re-lexed. To avoid that, I implemented peek_token() and used it instead where possible.

Closes #205.

@mikmart
Copy link
Contributor Author

mikmart commented Jul 8, 2025

Before:

$ build/b tests/lexer_errors.b
INFO: Compiling files tests/lexer_errors.b ./libb/all.b ./libb/gas-x86_64-linux.b
tests/lexer_errors.b:5:5: LEXER ERROR: Empty character literal

After, without peeking:

$ build/b tests/lexer_errors.b
INFO: Compiling files tests/lexer_errors.b ./libb/all.b ./libb/gas-x86_64-linux.b
tests/lexer_errors.b:5:5: LEXER ERROR: Empty character literal
tests/lexer_errors.b:5:5: LEXER ERROR: Empty character literal
tests/lexer_errors.b:5:5: LEXER ERROR: Empty character literal
tests/lexer_errors.b:6:5: LEXER ERROR: Character literal contains more than two characters
tests/lexer_errors.b:6:5: LEXER ERROR: Character literal contains more than two characters
tests/lexer_errors.b:6:5: LEXER ERROR: Character literal contains more than two characters
tests/lexer_errors.b:9:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:9:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:9:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:10:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:10:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:10:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:13:7: LEXER ERROR: Unknown escape sequence starting with `f`
tests/lexer_errors.b:13:11: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:15: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:7: LEXER ERROR: Unknown escape sequence starting with `f`
tests/lexer_errors.b:13:11: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:15: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:7: LEXER ERROR: Unknown escape sequence starting with `f`
tests/lexer_errors.b:13:11: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:15: LEXER ERROR: Unknown escape sequence starting with `b`

After, with peeking:

$ build/b tests/lexer_errors.b
INFO: Compiling files tests/lexer_errors.b ./libb/all.b ./libb/gas-x86_64-linux.b
tests/lexer_errors.b:5:5: LEXER ERROR: Empty character literal
tests/lexer_errors.b:6:5: LEXER ERROR: Character literal contains more than two characters
tests/lexer_errors.b:9:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:10:5: LEXER ERROR: Integer literal overflow
tests/lexer_errors.b:13:7: LEXER ERROR: Unknown escape sequence starting with `f`
tests/lexer_errors.b:13:11: LEXER ERROR: Unknown escape sequence starting with `b`
tests/lexer_errors.b:13:15: LEXER ERROR: Unknown escape sequence starting with `b`

@mikmart mikmart marked this pull request as ready for review July 8, 2025 21:29
@mikmart

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Recoverable lexer errors and backtracking
1 participant