tokenize: Use type for locating newlines

Fork: 0

LuminaSensum / NewLang

Browse code tokenize: Use type for locating newlines This allows us to test locating with a broader variety of inputs, not just what we pass to split_tokens. main
1 parent 40e09b7 commit 4564cdb158ee807ed6e26c0ba088418d1b04b7c3 Jookia authored on 8 Feb 2022

Patch

Unified Split

Showing 2 changed files

Ignore Space Show notes View src/tokenize.py

Ignore Space Show notes View tests/test_tokenize.py

Show line notes below