Feature: Lexer #7

New Issue

j · 2023-09-23T05:39:06Z

j commented

2023-09-23 05:39:06 +00:00

Feature description

The lexer takes a block of text, and breaks it up into identifiable parts (tokens) with lexical significance.

It should be able to tokenize:

Keywords (fn, let, if, else, for, etc.)
Identifiers (foo, bar, baz, _foo, _bar, _baz, _9, etc.) (matching XID_START ~ XID_CONTINUE*)
Literals (10. 1_000, 0x41, 0b100, 4.5, .5, "Hello, world!", '0')
Punctuation ((, [, {, }, ], ), ,, <, >, =, +, -, *, &, ^, etc.)

# Feature description The lexer takes a block of text, and breaks it up into identifiable parts (tokens) with lexical significance. It should be able to tokenize: - Keywords (`fn`, `let`, `if`, `else`, `for`, etc.) - Identifiers (`foo`, `bar`, `baz`, `_foo`, `_bar`, `_baz`, `_9`, etc.) (matching `XID_START` ~ `XID_CONTINUE`*) - Literals (`10`. `1_000`, `0x41`, `0b100`, `4.5`, `.5`, `"Hello, world!"`, `'0'`) - Punctuation (`(`, `[`, `{`, `}`, `]`, `)`, `,`, `<`, `>`, `=`, `+`, `-`, `*`, `&`, `^`, etc.)

j added the

enhancement

label 2023-09-23 05:39:06 +00:00

j self-assigned this 2023-09-23 05:39:06 +00:00

j added this to the Basic Syntax project 2023-09-23 05:39:06 +00:00

j added a new dependency 2023-09-23 05:39:31 +00:00

#2 Feature: Tokenize Identifier

j added a new dependency 2023-09-23 05:39:39 +00:00

#3 Feature: Tokenize Integer literals

j added a new dependency 2023-09-23 05:39:49 +00:00

#4 Feature: Tokenize Floating Point Literals

j removed a dependency 2023-09-23 05:40:25 +00:00