check-templates: Make parser more thorough (and faster).

We now create tokens for whitespace and text, such that you
could rebuild the template file with "".join(token.s for
token in tokens).

I also fixed a few bugs related to not parsing
whitespace-control tokens.

We no longer ignore template variables, although we could do
a lot better at validating them.

The most immediate use case for the more thorough parser is
to simplify the pretty printer, but it should also make it
less likely for us to skip over new template constructs
(i.e. the tool will fail hard rather than acting strange).

Note that this speeds up the tool by almost 3x, which may be
slightly surprising considering we are building more tokens.
The reason is that we are now munching efficiently through
big chunks of whitespace and text at a time, rather than
checking each individual character to see if it starts one
of the N other token types.

The changes to the pretty_print module here are a bit ugly,
but they should mostly be made irrelevant in subsequent
commits.

tools/lib/pretty_print.py

@@ -45,11 +45,12 @@ def else_token(token: Token) -> bool:
 
 
 def pop_unused_tokens(tokens: List[Token], row: int) -> bool:
+    was_closed = False
     while tokens and tokens[-1].line <= row:
         token = tokens.pop()
         if close_token(token):
-            return True
-    return False
+            was_closed = True
+    return was_closed
 
 
 def indent_pref(row: int, tokens: List[Token], line: str) -> str:
@@ -146,10 +147,19 @@ def pretty_print_html(html: str) -> str:
                 next_offset = open_offsets.pop()
             return tag_continuation_offset
 
+        while tokens and tokens[-1].line < row:
+            token = tokens.pop()
+
         offset = next_offset
         if tokens:
             token = tokens[-1]
-            if token.line == row and token.line_span > 1:
+            if token.kind == "indent":
+                token = tokens[-2]
+            if (
+                token.line == row
+                and token.line_span > 1
+                and token.kind not in ("template_var", "text")
+            ):
                 if token.kind in ("django_comment", "handlebar_comment", "html_comment"):
                     tag_continuation_offset = offset
                 else: