tests: Fix test_tokenize not testing whitespace ...

This relied on draw_token_random which now doesn't include whitespace

tests: Move whitespace code to test_tokenize ...

The parser doesn't have to worry about whitespace, so it's not worth
worrying about it here.

tests: Add garbled strategy for draw_token_random

src: Rename syntax to token

parse: Remove SyntaxType ...

Once again I find myself in the pattern of having a useless 'type' value
that just complicates my life. In this case the correct appproach is to
output the AST directly.

syntax: Refactor draw_syntax_random and add draw_syntax_texttoken ...

I realized draw_syntax_classified is the same as draw_syntax_random, so
just merge them.

Also add draw_syntax_texttoken explicitly meant to weed out cases where
token type is not set.

tests: Clean up syntax and parse code

tests: Clarify function documentation

tokenize: Support CR LF new line sequences

tokenize: Add more new line symbols

parse: Ignore whitespace in text, instead join words with spaces ...

This cuts down on complexity as now only the tokenizer deals with
whitespace and makes whitespace no longer a factor in the language.

syntax: Rename SyntaxLocation's column to offset ...

We don't know anything about columns or displaying the code points in
our source file, so be clear it's an offset and not a column.

tests: tokenize: Move syntax tests to new file

tokenize: Move syntax to its own module

tests: parse: Move syntax tests to tokenize ...

This helps with mutmut.

tests: tokenize: Remove reference to NEWLINE

tests: tokenize: Draw classified tokens for testing locations ...

This way things are actually likely to draw newlines.

parse: Move Syntax to tokenize and replace Token ...

These are basically the same classes so merge them

tokenize: Remove TokenType enum ...

In practice this wasn't used: The parse code always checks the value AND
the type. So just check the value only.

tests: tokenize: Fix typo

testss: tokenize: Add documentation on the expected behaviours

tests: tokenize: Give garbled tokens to the classifier ...

This helps prove that the classifier only classifies full tokens, not
substrings.

tokenize: Add shebang types

tokenize: Classify language keywords

tokenize: Add booleans

test: tokenize: Move draw_tokens_list to draw_tokens_to_split ...

While we're at it include the source flattening code.

tests: tokenize: Add a draw_token_classified strategy

tests: tokenize: Use random tokens again for locations ...

By using random tokens and passing that set to the test we now have both
random values, random locations and random types to help prove that the
location pass only uses types and length for locating.

tests: tokenize: Test location using types only

tokenize: Use type for locating newlines ...

This allows us to test locating with a broader variety of inputs, not
just what we pass to split_tokens.