src: tokenize: Don't join whitespaces when splitting

Fork: 0

LuminaSensum / NewLang

Browse code src: tokenize: Don't join whitespaces when splitting This will be useful later when we need to track line numbers. main
1 parent 6b4f999 commit b25df30de3a9c4706634d05f04c44ab8777c3b2b Jookia authored on 4 Feb 2022

Patch

Unified Split

Showing 2 changed files

Ignore Space Show notes View src/tokenize.py

Ignore Space Show notes View tests/test_tokenize.py

Show line notes below