Feature: Tokenize Punctuation #9

New Issue

j · 2023-09-25T22:21:48Z

j commented

2023-09-25 22:21:48 +00:00

Feature Progress

Create operator/punctuation types
- Semi ;
- Dot .
- Star *
- Slash /
- Plus +
- Minus -
- Rem %
- Bang !
- Eq =
- Lt <
- Gt >
- And &
- Or |
- Xor ^
Decide if operators should compound at token level?
- Lsh <<
- Rsh >>
- AndAnd &&
- OrOr ||
- CatEar ^^
- EqEq ==
- NotEq !=
- StarEq *=
- DivEq /=
- AddEq +=
- SubEq -=
- LshEq <<=
- RshEq >>=
- etc.
Create tests for tokenizing each type
Define rules for tokenizing each type

Feature description

Operators are the heart of any programming language. Conlang, being an expression language, has a variety:

Ignore operator: ;
Dereference operator: *
Member operator: .
Arithmetic operators: *, /, +, -
- Mul-precedence: *, /
- Add-precedence: +, -
Bitwise operators
- Not: ! (unary)
- Shift: << >>
- Logic: &, |, ^
Boolean operators
- Not: ! (unary)
- Equality: == !=
- Logic: &&, ||
Assignment operators
- Plain: =
- Arithmetic: *=, /=, +=, -=
- Bitwise: <<=, >>=, &=, |=, ^=
etc.

Example

let x = {
    let a = arr.len() - 1; let b = 10;
    while (b < a) { if arr[b] == arr[a] { break b } }
    else a
}

# Feature Progress  - [x] Create operator/punctuation types - [x] Semi `;` - [x] Dot `.` - [x] Star `*` - [x] Slash `/` - [x] Plus `+` - [x] Minus `-` - [x] Rem `%` - [x] Bang `!` - [x] Eq `=` - [x] Lt `<` - [x] Gt `>` - [x] And `&` - [x] Or `|` - [x] Xor `^` - [x] Decide if operators should compound at token level? - [x] Lsh `<<` - [x] Rsh `>>` - [x] AndAnd `&&` - [x] OrOr `||` - [x] CatEar `^^` - [x] EqEq `==` - [x] NotEq `!=` - [x] StarEq `*=` - [x] DivEq `/=` - [x] AddEq `+=` - [x] SubEq `-=` - [x] LshEq `<<=` - [x] RshEq `>>=` - [x] etc. - [x] Create tests for tokenizing each type - [x] Define rules for tokenizing each type # Feature description  Operators are the heart of any programming language. Conlang, being an expression language, has a variety: - Ignore operator: `;` - Dereference operator: `*` - Member operator: `.` - Arithmetic operators: `*`, `/`, `+`, `-` - Mul-precedence: `*`, `/` - Add-precedence: `+`, `-` - Bitwise operators - Not: `!` (unary) - Shift: `<<` `>>` - Logic: `&`, `|`, `^` - Boolean operators - Not: `!` (unary) - Equality: `==` `!=` - Logic: `&&`, `||` - Assignment operators - Plain: `=` - Arithmetic: `*=`, `/=`, `+=`, `-=` - Bitwise: `<<=`, `>>=`, `&=`, `|=`, `^=` - etc. # Example  ```rust let x = { let a = arr.len() - 1; let b = 10; while (b < a) { if arr[b] == arr[a] { break b } } else a } ```

j added the

enhancement

label 2023-09-25 22:21:48 +00:00

j added this to the Basic Syntax project 2023-09-25 22:21:48 +00:00

j changed title from ~~Feature: Tokenize Operators and Punctuation~~ to Feature: Tokenize Punctuation

2023-09-25 23:39:57 +00:00

j commented

2023-09-28 05:17:25 +00:00

This feature has been implemented. Things left to consider:

Operator overloading vs. user-defined operators
- Are operators a language feature?
How are operators defined?
- Syntactic sugar approach?
- AST substitution?
- Parser problem.

This feature has been implemented. Things left to consider: - Operator overloading vs. user-defined operators - Are operators a language feature? - How are operators defined? - Syntactic sugar approach? - AST substitution? - Parser problem.

j referenced this issue from a commit

2023-09-28 05:23:21 +00:00

conlang: Tokenize punctuation (resolves #9)

j closed this issue

2023-09-28 05:23:22 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: j/Conlang#9