Tokens are defined by their regular expressions (see 6.2). TPG builds a regular expression by assembling each
regular expression in a or structure. For example to recognize int ([0 - 9]+) and word ([a - zA - Z]+),
TPG builds this composite expression: (?P < int > [0 - 9]+)|(?P < word > [a - zA - Z]+) This
expression is then compiled using the re module.
For each token we save its name, its text, its value (i.e. the result of its action applied to its
text), the line number and the position of the start and the end of the token in the input
string.
There is a special token named EOF used as the erroneous token when a lexical error appears near
the end of the input.