-
Book Overview & Buying
-
Table Of Contents
Practical Systems Programming in Go
By :
A lexer (short for lexical analyzer), also known as a tokenizer, is the first concrete processing stage of a programming language implementation. Building on its role of supplying the parser with tokens, the lexer operates by scanning the source code character by character, skipping over whitespace and comments. Based on the current character, it decides which token to construct: letters trigger the reading of identifiers or keywords, digits form numeric literals, and quotation marks initiate string literals with proper handling of escape sequences. For operators, the lexer performs limited lookahead to distinguish between single-character symbols and multicharacter operators, such as ==, !=, <=, or >=. Each recognized element is emitted as a structured token, providing the parser with a clean, unambiguous stream of syntactic input.
While the machinery of a lexer is generic — reading input, tracking positions, and skipping whitespace — the definitions are...