Fork: 0
LuminaSensum / NewLang
Browse code
tests: tokenize: Give garbled tokens to the classifier
This helps prove that the classifier only classifies full tokens, not
substrings.
main
1 parent 5f2eb82 commit 1e2291373f8b1a840c82e667a571bd04d2614d88
@Jookia Jookia authored on 15 Feb 2022
Patch
Unified Split
Showing 1 changed file
View
tests/test_tokenize.py