parse: Handle end of file better
Instead of throwing an error, treat end of files as a specific type of token that's added to the end of the tokenizer. This also means we can always return EOF once we reach the end of the tokens just by sticking to the last token. |
---|
|
parse.py |
---|