tokenize: Add classify_tokens to set token types

Fork: 0

LuminaSensum / NewLang

Browse code tokenize: Add classify_tokens to set token types main
1 parent 51a6744 commit c996681783def62dc422c769886e967dbd3f72b2 Jookia authored on 7 Feb 2022

Patch

Unified Split

Showing 2 changed files

Ignore Space Show notes View src/tokenize.py

Ignore Space Show notes View tests/test_tokenize.py

Show line notes below