tests: parse: Draw text syntax tokens ...

This requires fixing a bug for comparing values of different types.

parse: tests: Clarify errors and tests

parse: Check token types when parsing text

parse: Add parse_text to parse a text syntax node ...

No error handling tests are added yet.

parse: Rename TokenStream to SyntaxStream

parse: Require the caller to copy the TokenStream list ...

This is more Pythonic I think.

parse: Add TokenStream class ...

This will be used for stateful parsing, with lookahead in future.

parse: Don't strip non-token syntax ...

This bug was picked up by mutmut.

parse: Use syntax for strip_whitespace

parse: Add import_tokens to import tokens to syntax

parse: Add Syntax node ...

Currently this just holds a token, but will eventually hold parsed
syntax tree values.

parse: Rename remove_whitespace to strip_whitespace

parse: Add parse wrapper function

parse: Add remove_whitespace pass

parse: Move to oldparse ...

This is in preparation of writing a new parser.
Once the new parser is ready then this code can be junked.

Change license to LGPL-2.1-only

src: parse: Fix uppercase repr value

parse: Use __str__ for ParseError printing ...

Exceptions don't use __repr__ but __str__ for printing

tests: lexer: Check error location

src: parse: Don't assign unused variables ...

Found by flake8.

parse: Strip whitespace in text tokens

parse: Replace 'Begin' with 'Start' in keywords ...

Okay, it's to the point I'm writing bugs in tests due to this.
Just change it already.

parse: Read text literal up to beginning of EndText ...

Using the end of the last word before EndText misses any trailing
whitespace, and it's a lost cause trying to figure out the offset
from after EndText given it might be whitespace or an end of life.

Instead use an offset from the start of EndText.

parse: Add code position to word value ...

This will be useful for finding the start of a word.

parse: Add Word class for lexer tokens ...

Instead of passing around a tuple, use a class.
Rename untokenized tokens to 'words' while we're at it.

parse: Fix tracing in lexer next() ...

This makes the line and column numbers accurate and prints
whitespace properly.

parse: Have lexer position be current instead of future ...

When running next(), self.pos and friends would actually be in reference
to the next call to next(), not the current one.

This is confusing, so change it to make sense.

parse: Fix text literals being cut off when placed at end of file

parse: Account for EOF when checking for whitespace ...

When parsing a file that doesn't end with whitespace (like a new line),
the parser would choke on having no symbols left. Fix that.

Fixes: 1519cef

parse: Decode file as UTF-8 ...

I was testing the parser with binary input.
Use Unicode instead and make sure we pass it that.