Internal: Implement tokenization. #4
Description
The tokenizer should return a iterator of Token structs that can be iterated over in parsing.
let tokens = tokenize(&"someCode");
for token in tokens {
// parse
}{{ message }}
The tokenizer should return a iterator of Token structs that can be iterated over in parsing.
let tokens = tokenize(&"someCode");
for token in tokens {
// parse
}