Commit Graph

4 Commits

Author SHA1 Message Date
1f0725d2c8 examples/identify_tokens.rs: Overhaul user experience 2023-10-16 22:26:32 -05:00
545483dae6 token: Add Display impl for Type
This improves readability in identify_tokens
2023-09-28 01:34:56 -05:00
097e2c4f11 conlang: Rename literals; split, compose, and document Rules
- Renamed literal Types to reflect their literal nature
  - This allows for consistent naming across future non-literal Types
- Complicated lexer Rules have been split into composable sub-rules,
  and moved into the Rule struct.
  - This improves modularity, and allows sharing of sub-rules across rules.
- Documented each lexer rule with (at least) a one-line blurb
  describing its function
2023-09-27 18:13:01 -05:00
46e72e4889 conlang: add toy program to interact with the tokenizer 2023-09-25 18:06:03 -05:00