Things left to consider:
- token::Type enum is getting fairly large.
breaking it up could invoke substantial code bloat
- Compound operators might make more sense at the parser level
- Compound-assign operators are ripe for syntactic desugaring,
but there must be some reason it's done separately in other languages.
- Operators like FatArrow may still make sense at the tokenizer level, regardless.
- What is a lexer? A miserable pile of parsers!
- Operator overloading, or user-defined operators? Hmm...
- Renamed literal Types to reflect their literal nature
- This allows for consistent naming across future non-literal Types
- Complicated lexer Rules have been split into composable sub-rules,
and moved into the Rule struct.
- This improves modularity, and allows sharing of sub-rules across rules.
- Documented each lexer rule with (at least) a one-line blurb
describing its function
- Lexer now ignores leading whitespace
- Rule now has shorter, clearer function names
- Tests for comment lexing are now consolidated into a module
- Tests using the assert_has_type_and_len wrapper can now specify
an expected length
