- Scan the input string *linearly*, without backtracking
- Peek at most one character (unicode code-point) ahead
- Store data (unescaped string literals and chars, identifiers, integers, floats) inside Token
- This unfortunately makes tokens non-Copy
- Refactor Parser to accommodate these changes
- On the bright side, Parser no longer needs a reference to the text!
- Write a new set of lexer tests
- TODO: write a new set of token tests using tokendata
Every day, we get closer to parsing `dummy.cl`!
- Unified math operations into a single self-referential enum
- Walk now visits the children of a node, rather than the node itself
- The old behavior was super confusing, and led to numerous stack overflows.
