Primitive Tokens

   Token                Token Type      Flags

   identifier           IDENT            b e
   integer-literal      INTLIT           b e
   real-literal         REALLIT          b e
   string-literal       STRINGLIT        b e
   cset-literal         CSETLIT          b e
   end-of-file          EOFX

Reserved Words

   Token                Token Type      Flags

   break                BREAK            b e
   by                   BY
   case                 CASE             b
   create               CREATE           b
   default              DEFAULT          b
   do                   DO
   dynamic              DYNAMIC          b
   else                 ELSE
   end                  END              b
   every                EVERY            b
   fail                 FAIL             b e
   global               GLOBAL
   if                   IF               b
   initial              INITIAL          b
   link                 LINK
   local                LOCAL            b
   next                 NEXT             b e
   not                  NOT              b
   of                   OF
   procedure            PROCEDURE
   record               RECORD
   repeat               REPEAT           b
   return               RETURN           b e
   static               STATIC           b
   suspend              SUSPEND          b e
   then                 THEN
   to                   TO
   until                UNTIL            b
   while                WHILE            b

Operators

   Token                Token Type      Flags

   :=                   ASSIGN
   @                    AT               b
   @:=                  AUGACT
   &:=                  AUGAND
   =:=                  AUGEQ
   ===:=                AUGEQV
   >=:=                 AUGGE
   >:=                  AUGGT
   <=:=                 AUGLE
   <:=                  AUGLT
   ~=:=                 AUGNE
   ~===:=               AUGNEQV
   ==:=                 AUGSEQ
   >>=:=                AUGSGE
   >>:=                 AUGSGT
   <<=:=                AUGSLE
   <<:=                 AUGSLT
   ~==:=                AUGSNE
   \\                   BACKSLASH        b
   !                    BANG             b
   |                    BAR              b
   ^                    CARET            b
   ^:=                  CARETASGN
   :                    COLON
   ,                    COMMA
   ||                   CONCAT           b
   ||:=                 CONCATASGN
   &                    CONJUNC          b
   .                    DOT              b
   --                   DIFF             b
   --:=                 DIFFASGN
   ===                  EQUIV            b
   **                   INTER            b
   **:=                 INTERASGN
   {                    LBRACE           b
   [                    LBRACK           b
   |||                  LCONCAT          b
   |||:=                LCONCATASGN      b
   ==                   LEXEQ            b
   >>=                  LEXGE
   >>                   LEXGT
   <<=                  LEXLE
   <<                   LEXLT
   ~==                  LEXNE            b
   (                    LPAREN           b
   -:                   MCOLON
   -                    MINUS            b
   -:=                  MINUSASGN
   %                    MOD
   %:=                  MODASGN
   ~===                 NOTEQUIV         b
   =                    NUMEQ            b
   >=                   NUMGE
   >                    NUMGT
   <=                   NUMLE
   <                    NUMLT
   ~=                   NUMNE            b
   +:                   PCOLON
   +                    PLUS             b
   +:=                  PLUSASGN
   ?                    QMARK            b
   <-                   REVASSIGN
   <->                  REVSWAP
   }                    RBRACE             e
   ]                    RBRACK             e
   )                    RPAREN             e
   ;                    SEMICOL
   ?:=                  SCANASGN
   /                    SLASH            b
   /:=                  SLASHASGN
   *                    STAR             b
   *:=                  STARASGN
   :=:                  SWAP
   ~                    TILDE            b
   ++                   UNION            b
   ++:=                 UNIONASGN

Explanation of Flags

   b    indicates that the token may begin an expression.
   e    indicates that the token may end an expression.

   These two flags are used for semicolon insertion.  If a line
   ends with an "e" token, and the next token is a "b" token,
   a semicolon is inserted between the two tokens.