b1d9e4fd03
ast: Move module into file
2023-10-17 13:30:16 -05:00
5d9c45c26a
token: Move module into file
2023-10-17 13:23:34 -05:00
b707bcef81
examples: Add parse_input REPL for fun!
2023-10-16 22:51:59 -05:00
8d1107cb57
Conlang: Add a pretty printer for the AST
2023-10-16 22:51:07 -05:00
b89ed307a2
parser: Add recursive descent parser for AST. TODO: Error recovery and synchronization.
2023-10-16 22:50:15 -05:00
0a99a37bd8
ast: Disallow unused imports
2023-10-16 22:47:16 -05:00
c7d1aa4d2c
lexer: Make the Lexer IntoIter'able
2023-10-16 22:46:38 -05:00
2ce89eec82
lexer: unmatched delimiters? in my commits? Shame! Shame for a thousand years!
2023-10-16 22:42:32 -05:00
4870ff17ba
Token: whoops, these fields need to be public (for now)
2023-10-16 22:37:04 -05:00
3866a2e9f2
libconlang: Define the AST (TODO: clean up the AST)
2023-10-16 22:36:26 -05:00
1f0725d2c8
examples/identify_tokens.rs: Overhaul user experience
2023-10-16 22:26:32 -05:00
50bb03ae18
token/lexer: Keywords are identifiers. Add missing operators. Fix <<=/>>=.
2023-10-16 22:25:48 -05:00
9c993b31a1
token: Rename literal tokens & assoc. functions
2023-10-16 22:14:08 -05:00
c15490847a
token & lexer: add line & col to tokens
2023-10-13 13:05:14 -05:00
545483dae6
token: Add Display impl for Type
...
This improves readability in identify_tokens
2023-09-28 01:34:56 -05:00
48f5e5248c
conlang: Add break, true, false keywords, and example file
2023-09-28 01:31:46 -05:00
f25685560a
conlang: Tokenize punctuation ( resolves #9 )
...
Things left to consider:
- token::Type enum is getting fairly large.
breaking it up could invoke substantial code bloat
- Compound operators might make more sense at the parser level
- Compound-assign operators are ripe for syntactic desugaring,
but there must be some reason it's done separately in other languages.
- Operators like FatArrow may still make sense at the tokenizer level, regardless.
- What is a lexer? A miserable pile of parsers!
- Operator overloading, or user-defined operators? Hmm...
2023-09-28 00:11:31 -05:00
d3b1e65e35
lexer: Add documentation blurbs to token-class functions
2023-09-28 00:09:47 -05:00
b1621f2522
tests: Add tests for token-class functions
2023-09-28 00:08:51 -05:00
0661789d42
Lexer: reduce code duplication in Rule mapping
...
produce_token renamed to map_rule
- Skips leading whitespace before any Rule evaluations
- Handles creation and destruction of Rule
2023-09-27 22:51:51 -05:00
d4245844ce
lexer: Add Invalid token to aid in implementing features
2023-09-27 21:17:35 -05:00
335fea8d73
conlang: add delimiter types, lexer rules, and tests
2023-09-27 18:26:38 -05:00
bed21941e8
conlang: Add keyword types, rules, and tests
2023-09-27 18:21:19 -05:00
097e2c4f11
conlang: Rename literals; split, compose, and document Rules
...
- Renamed literal Types to reflect their literal nature
- This allows for consistent naming across future non-literal Types
- Complicated lexer Rules have been split into composable sub-rules,
and moved into the Rule struct.
- This improves modularity, and allows sharing of sub-rules across rules.
- Documented each lexer rule with (at least) a one-line blurb
describing its function
2023-09-27 18:13:01 -05:00
46e72e4889
conlang: add toy program to interact with the tokenizer
2023-09-25 18:06:03 -05:00
09f22d6bf3
lexer: Tokenize float literals ( resolves #4 )
2023-09-25 16:51:18 -05:00
8ddf73dc76
lexer: refactor string escape to separate rule (fixes '\\"')
2023-09-25 16:49:18 -05:00
fe2b9880d6
lexer: Fix infinite loop while trimming whitespace
2023-09-25 16:46:06 -05:00
247bb1f8f8
integer: Match based on base, if present
2023-09-25 16:43:50 -05:00
03660fd641
tests: fix whitespace
2023-09-25 14:31:57 -05:00
e49a3e9fec
lexer: Tokenize string literals ( resolves #5 )
2023-09-25 14:26:56 -05:00
6f6d8ec916
lexer: Tokenize integer literals ( resolves #3 )
2023-09-25 14:25:00 -05:00
8f07b29ff3
lexer: Tokenize identifiers (resolves #2.)
2023-09-25 14:22:27 -05:00
443cd11803
lexer: Refactor rule and assoc. tests
...
- Lexer now ignores leading whitespace
- Rule now has shorter, clearer function names
- Tests for comment lexing are now consolidated into a module
- Tests using the assert_has_type_and_len wrapper can now specify
an expected length
2023-09-25 14:13:28 -05:00
71053f1992
lerox: Consolidate traits
...
The auto-implementation scheme doesn't really fit the application
2023-09-25 13:52:21 -05:00
06597fa1e9
libconlang: implement comment tokenization (for #1 )
2023-09-22 23:56:10 -05:00
90ef1f542d
lerox: Add lerox, a silly little combinator library
2023-09-22 23:54:09 -05:00
8f66a64924
Initial Commit
2023-09-21 15:41:12 -05:00
