tokenize: Add tokenize wrapper function

Fork: 0

LuminaSensum / NewLang

Browse code tokenize: Add tokenize wrapper function main
1 parent 451c6f5 commit 31ae0720eea5e88a212d2bd5dbe9feab60c76f5c Jookia authored on 7 Feb 2022

Patch

Unified Split

Showing 2 changed files

Ignore Space Show notes View src/tokenize.py

Ignore Space Show notes View tests/test_tokenize.py

Show line notes below