tokenize: Use type for locating newlines ...

This allows us to test locating with a broader variety of inputs, not
just what we pass to split_tokens.

tokenize: Add NEWLINE token type ...

This isn't used yet but is intended to make location handling easier.

tests: tokenize: Add more accurate comments

tests: tokenize: Have draw_token_* use draw_token_random

tests: tokenize: Put generic draws at top of file

tokenize: Add tokenize wrapper function

tests: tokenize: Add comments

tests: tokenize: Run classification with random input

tokenize: Add classify_tokens to set token types

tests: tokenize: Rename draw_token to draw_token_random

tokenize: Add WHITESPACE TokenType ...

This will be used later.

tests: Use random token types for tokens

tokenize: Add type value for token type

src: tokenize: Rename symbol to token

tests: Generate random locations for symbols ...

This requires adding a new higher-order strategy to set the location of
a symbol.

tokenize: Add locate_symbols function to add symbol locations ...

This isn't the best soltuion yet, but it's a start.

tests: tokenize: Test symbol structure

tokenize: Always specify a location for symbols ...

This gets rid of the default symbol location. The 'default' used now is
just the result of split_symbols.

tests: tokenize: Test location structure ...

I didn't think to test this and it was giving off mutmut errors.
In retrospect it's kind of vital we test the equality check if we're
going to use it to validate tests.

tests: tokenize: Fix test only drawing whitespace

src: tokenize: Use symbol objects for symbols ...

This will be useful for when we add line numbers.

src: tokenize: Don't join whitespaces when splitting ...

This will be useful later when we need to track line numbers.

tests: Add SPDX license headers

src: tokenizer: Add beginning of new tokenizer

tests: parse: Remove unused integers include ...

Found by flake8.

tests: lexer: Drop error testing code ...

This code is awful:

- It reaches across multiple layers of abstractions
- It does a lot of string splitting and joining to rebuild code
- It only tests very specific cases because the main code is broken

Just dump it for now and start over.

tests: lexer: Add test for inserting an EndText in to test ...

This is BROKEN and fundamentally flawed, but I'm keeping it here for
reference in Git history.

tests: Clarify purpose of draw_error_endtext_remove

tests: Rename EndText test to EndText stray test

tests: parse: Use repr() to print string values