j/Conlang
1
1
Fork 0
Code Issues 5 Pull Requests Packages Projects 2 Releases Wiki Activity
142 Commits 2 Branches 6 Tags 2.3 MiB
69f5035a8b
Commit Graph

113 Commits

Author SHA1 Message Date
John
aead97e357 interpreter: Implement ranges and for loops 2023-10-26 21:51:18 -05:00
John
55070bcc41 ast: Improve documentation linking 2023-10-26 17:56:02 -05:00
John
ed6917070c interpreter: fix comment 2023-10-26 14:51:13 -05:00
John
15a3d265df interpreter: Prototype interpreter 
TODO: Type-checking, floats, variables & scope
TODO Later: A bytecode interpreter
 2023-10-26 14:48:44 -05:00
John
9ab9583a5b parser: Reorder functions, add doc comments 2023-10-26 14:41:59 -05:00
John
0445598ae8 grammar: Remove Ignore expression to avoid parsing ambiguity :'( 2023-10-26 14:36:55 -05:00
John
b1f90ca4e9 conlang: Add statements 
TODO: Parse `let` statements
 2023-10-26 14:33:56 -05:00
John
5c4c8bcb80 lexer: Consume invalid characters 2023-10-25 19:28:54 -05:00
John
32bde2f749 Minor documentation fixes 2023-10-23 23:43:11 -05:00
John
0e1beca43d Cargos.toml: wrong key ("publish") 2023-10-23 23:42:39 -05:00
John
9196d3b1da Add package metadata 2023-10-23 21:38:06 -05:00
John
0e917837c0 identify_tokens: Use Display on token Data 2023-10-23 21:02:09 -05:00
John
8b1a1534f3 docs: Improve and update documentation 2023-10-23 21:01:32 -05:00
John
c43ecf00d1 token::token_data::Data: Renamed from token::TokenData 
& Moved into its own module
+ Added token::preamble; common imports when working with Token
 2023-10-23 19:43:16 -05:00
John
ccfddcc09a Conlang: Add range operators 
TODO: Limit range operators to at most 2 operands
 2023-10-22 18:30:00 -05:00
John
b5abd2bff1 Lexer rewrite: 
- Scan the input string *linearly*, without backtracking
  - Peek at most one character (unicode code-point) ahead
- Store data (unescaped string literals and chars, identifiers, integers, floats) inside Token
  - This unfortunately makes tokens non-Copy
- Refactor Parser to accommodate these changes
  - On the bright side, Parser no longer needs a reference to the text!
- Write a new set of lexer tests
  - TODO: write a new set of token tests using tokendata

Every day, we get closer to parsing `dummy.cl`!
 2023-10-22 18:28:20 -05:00
John
feb5cc5dd0 AST: Refactor binary operations, fix Walk trait 
- Unified math operations into a single self-referential enum
- Walk now visits the children of a node, rather than the node itself
  - The old behavior was super confusing, and led to numerous stack overflows.
 2023-10-21 12:24:52 -05:00
John
1e5f7149d9 Token: Add range operators, rename for consistency 2023-10-20 17:09:14 -05:00
John
34525ec77a lexer: Only allow lines up to 4GB in size. Saves 8 bytes per token 2023-10-20 15:34:54 -05:00
John
a26a01fc17 constr: Misuse iterators to parse tokens 2023-10-20 15:33:02 -05:00
John
7f7393d2c6 conlang: Grammatical cleanup; merge control flow expressions 2023-10-19 14:40:03 -05:00
John
331141356a tests: Move module into file 2023-10-17 13:39:05 -05:00
John
73f7337fd5 pretty_printer: Move module into file 2023-10-17 13:38:30 -05:00
John
59617d805a parser: Move module into file 2023-10-17 13:36:21 -05:00
John
3d0698ba60 lexer: Move module into file 2023-10-17 13:33:07 -05:00
John
b1d9e4fd03 ast: Move module into file 2023-10-17 13:30:16 -05:00
John
5d9c45c26a token: Move module into file 2023-10-17 13:23:34 -05:00
John
b707bcef81 examples: Add parse_input REPL for fun! 2023-10-16 22:51:59 -05:00
John
8d1107cb57 Conlang: Add a pretty printer for the AST 2023-10-16 22:51:07 -05:00
John
b89ed307a2 parser: Add recursive descent parser for AST. TODO: Error recovery and synchronization. 2023-10-16 22:50:15 -05:00
John
0a99a37bd8 ast: Disallow unused imports 2023-10-16 22:47:16 -05:00
John
c7d1aa4d2c lexer: Make the Lexer IntoIter'able 2023-10-16 22:46:38 -05:00
John
2ce89eec82 lexer: unmatched delimiters? in my commits? Shame! Shame for a thousand years! 2023-10-16 22:42:32 -05:00
John
4870ff17ba Token: whoops, these fields need to be public (for now) 2023-10-16 22:37:04 -05:00
John
3866a2e9f2 libconlang: Define the AST (TODO: clean up the AST) 2023-10-16 22:36:26 -05:00
John
1f0725d2c8 examples/identify_tokens.rs: Overhaul user experience 2023-10-16 22:26:32 -05:00
John
50bb03ae18 token/lexer: Keywords are identifiers. Add missing operators. Fix <<=/>>=. 2023-10-16 22:25:48 -05:00
John
9c993b31a1 token: Rename literal tokens & assoc. functions 2023-10-16 22:14:08 -05:00
John
c15490847a token & lexer: add line & col to tokens 2023-10-13 13:05:14 -05:00
John
545483dae6 token: Add Display impl for Type 
This improves readability in identify_tokens
 2023-09-28 01:34:56 -05:00
John
48f5e5248c conlang: Add break, true, false keywords, and example file 2023-09-28 01:31:46 -05:00
John
f25685560a conlang: Tokenize punctuation (resolves #9) 
Things left to consider:
- token::Type enum is getting fairly large.
  breaking it up could invoke substantial code bloat
- Compound operators might make more sense at the parser level
  - Compound-assign operators are ripe for syntactic desugaring,
    but there must be some reason it's done separately in other languages.
  - Operators like FatArrow may still make sense at the tokenizer level, regardless.
  - What is a lexer? A miserable pile of parsers!
- Operator overloading, or user-defined operators? Hmm...
 2023-09-28 00:11:31 -05:00
John
d3b1e65e35 lexer: Add documentation blurbs to token-class functions 2023-09-28 00:09:47 -05:00
John
b1621f2522 tests: Add tests for token-class functions 2023-09-28 00:08:51 -05:00
John
0661789d42 Lexer: reduce code duplication in Rule mapping 
produce_token renamed to map_rule
- Skips leading whitespace before any Rule evaluations
- Handles creation and destruction of Rule
 2023-09-27 22:51:51 -05:00
John
d4245844ce lexer: Add Invalid token to aid in implementing features 2023-09-27 21:17:35 -05:00
John
335fea8d73 conlang: add delimiter types, lexer rules, and tests 2023-09-27 18:26:38 -05:00
John
bed21941e8 conlang: Add keyword types, rules, and tests 2023-09-27 18:21:19 -05:00
John
097e2c4f11 conlang: Rename literals; split, compose, and document Rules 
- Renamed literal Types to reflect their literal nature
  - This allows for consistent naming across future non-literal Types
- Complicated lexer Rules have been split into composable sub-rules,
  and moved into the Rule struct.
  - This improves modularity, and allows sharing of sub-rules across rules.
- Documented each lexer rule with (at least) a one-line blurb
  describing its function
 2023-09-27 18:13:01 -05:00
John
46e72e4889 conlang: add toy program to interact with the tokenizer 2023-09-25 18:06:03 -05:00
John
09f22d6bf3 lexer: Tokenize float literals (resolves #4) 2023-09-25 16:51:18 -05:00
John
8ddf73dc76 lexer: refactor string escape to separate rule (fixes '\\"') 2023-09-25 16:49:18 -05:00
John
fe2b9880d6 lexer: Fix infinite loop while trimming whitespace 2023-09-25 16:46:06 -05:00
John
247bb1f8f8 integer: Match based on base, if present 2023-09-25 16:43:50 -05:00
John
03660fd641 tests: fix whitespace 2023-09-25 14:31:57 -05:00
John
e49a3e9fec lexer: Tokenize string literals (resolves #5) 2023-09-25 14:26:56 -05:00
John
6f6d8ec916 lexer: Tokenize integer literals (resolves #3) 2023-09-25 14:25:00 -05:00
John
8f07b29ff3 lexer: Tokenize identifiers (resolves #2.) 2023-09-25 14:22:27 -05:00
John
443cd11803 lexer: Refactor rule and assoc. tests 
- Lexer now ignores leading whitespace
- Rule now has shorter, clearer function names
- Tests for comment lexing are now consolidated into a module
- Tests using the assert_has_type_and_len wrapper can now specify
  an expected length
 2023-09-25 14:13:28 -05:00
John
71053f1992 lerox: Consolidate traits 
The auto-implementation scheme doesn't really fit the application
 2023-09-25 13:52:21 -05:00
John
06597fa1e9 libconlang: implement comment tokenization (for #1) 2023-09-22 23:56:10 -05:00
John
90ef1f542d lerox: Add lerox, a silly little combinator library 2023-09-22 23:54:09 -05:00
John
8f66a64924 Initial Commit 2023-09-21 15:41:12 -05:00