11 Commits

Author	SHA1	Message	Date
John	6e1d5af134	cl-token: Move token definition into its own crate	2024-02-29 19:36:06 -06:00
John	69f5035a8b	span: Break out into its own crate, to make room for future expansion	2024-02-29 18:31:41 -06:00
John	e01c71f9a7	lexer: implement std::error:;Error for Error (whoops!)	2024-02-28 05:00:39 -06:00
John	9b7cf9c017	lexer: Add :: and #! tokens	2023-10-30 19:17:03 -05:00
John	5c4c8bcb80	lexer: Consume invalid characters	2023-10-25 19:28:54 -05:00
John	8b1a1534f3	docs: Improve and update documentation	2023-10-23 21:01:32 -05:00
John	c43ecf00d1	token::token_data::Data: Renamed from token::TokenData ... & Moved into its own module + Added token::preamble; common imports when working with Token	2023-10-23 19:43:16 -05:00
John	b5abd2bff1	Lexer rewrite: ... - Scan the input string *linearly*, without backtracking - Peek at most one character (unicode code-point) ahead - Store data (unescaped string literals and chars, identifiers, integers, floats) inside Token - This unfortunately makes tokens non-Copy - Refactor Parser to accommodate these changes - On the bright side, Parser no longer needs a reference to the text! - Write a new set of lexer tests - TODO: write a new set of token tests using tokendata Every day, we get closer to parsing `dummy.cl`!	2023-10-22 18:28:20 -05:00
John	1e5f7149d9	Token: Add range operators, rename for consistency	2023-10-20 17:09:14 -05:00
John	34525ec77a	lexer: Only allow lines up to 4GB in size. Saves 8 bytes per token	2023-10-20 15:34:54 -05:00
John	3d0698ba60	lexer: Move module into file	2023-10-17 13:33:07 -05:00