- Scan the input string *linearly*, without backtracking
- Peek at most one character (unicode code-point) ahead
- Store data (unescaped string literals and chars, identifiers, integers, floats) inside Token
- This unfortunately makes tokens non-Copy
- Refactor Parser to accommodate these changes
- On the bright side, Parser no longer needs a reference to the text!
- Write a new set of lexer tests
- TODO: write a new set of token tests using tokendata
Every day, we get closer to parsing `dummy.cl`!
- Renamed literal Types to reflect their literal nature
- This allows for consistent naming across future non-literal Types
- Complicated lexer Rules have been split into composable sub-rules,
and moved into the Rule struct.
- This improves modularity, and allows sharing of sub-rules across rules.
- Documented each lexer rule with (at least) a one-line blurb
describing its function
