mirror of
https://github.com/zulip/zulip.git
synced 2025-11-03 21:43:21 +00:00
a744e38e67
We now create tokens for whitespace and text, such that you could rebuild the template file with "".join(token.s for token in tokens). I also fixed a few bugs related to not parsing whitespace-control tokens. We no longer ignore template variables, although we could do a lot better at validating them. The most immediate use case for the more thorough parser is to simplify the pretty printer, but it should also make it less likely for us to skip over new template constructs (i.e. the tool will fail hard rather than acting strange). Note that this speeds up the tool by almost 3x, which may be slightly surprising considering we are building more tokens. The reason is that we are now munching efficiently through big chunks of whitespace and text at a time, rather than checking each individual character to see if it starts one of the N other token types. The changes to the pretty_print module here are a bit ugly, but they should mostly be made irrelevant in subsequent commits.