Article by Ayman Alheraki on January 11 2026 10:37 AM
After preprocessing completes macro expansion, includes, and directive handling, the assembler proceeds to lexical analysis, also known as tokenizing. This phase is essential to convert the raw text of the assembly source code into structured elements called tokens that the assembler’s parser can efficiently process.
The primary goal of lexical analysis is to break down the linear stream of characters from the preprocessed source into meaningful atomic units — tokens — while discarding irrelevant characters such as whitespace and comments. This transformation simplifies syntax analysis and enables precise error detection and recovery.
In the context of x86-64 assembly, tokens commonly include:
Mnemonics: Instruction operation codes such as MOV, ADD, JMP.
Registers: Identifiers representing CPU registers like RAX, RCX, XMM0.
Immediate values: Numeric constants, both decimal and hexadecimal (e.g., 123, 0x7F).
Memory operands: Address expressions, often enclosed in brackets, e.g., [RBP-8].
Labels and symbols: User-defined names for code or data locations.
Directives: Assembler pseudo-operations such as SECTION, DB.
Operators and punctuation: Symbols like commas, colons, plus +, minus -, parentheses (), brackets [].
Comments: Portions of the line following a comment delimiter, usually ignored beyond lexical analysis.
The lexical analyzer scans the source code character-by-character, applying the following rules:
Whitespace Handling: Spaces, tabs, and newline characters separate tokens but are not themselves tokens. Newlines can be significant for error reporting or directive termination.
Identifier Recognition: Sequences matching alphabetic or alphanumeric patterns are recognized as identifiers, mnemonics, or directives based on symbol tables maintained by the assembler.
Numeric Parsing: Numeric literals are parsed with awareness of formats (decimal, hexadecimal, binary, octal) and suffixes that indicate size or type.
String Literals: If supported, string tokens enclosed in quotes are recognized as single tokens.
Comment Skipping: Comments are identified and discarded or stored separately for listing generation but do not generate tokens.
Symbol Resolution Preparation: Tokens representing symbols or labels are recorded for later phases to resolve addresses or values.
Lexical analysis in an assembler involves some complexities unique to assembly language:
Context Sensitivity: Some tokens can be ambiguous without syntactic context. For example, an identifier can be a label, a macro name, or a register alias. The lexer typically tokenizes uniformly, while the parser resolves ambiguities.
Variable-Length Tokens: Registers, mnemonics, and directives can vary in length and must be matched against a dictionary or symbol table efficiently.
Complex Memory Operand Syntax: Memory references can include base registers, index registers, scale factors, and displacement values inside brackets, requiring careful tokenization of nested structures.
Line Continuations: Assemblers often support multi-line statements using continuation characters; the lexer must join such lines properly.
Case Sensitivity: Most assemblers treat mnemonics and registers case-insensitively but preserve original case for symbols; lexical analysis must normalize tokens accordingly.
The output of lexical analysis is a sequential stream or array of tokens, each tagged with:
Token type (mnemonic, register, immediate, etc.)
Literal value or identifier text
Source location information (line number, column) for error reporting
Any relevant flags or attributes (e.g., immediate size hints)
This structured token stream forms the input for the parsing phase, which analyzes instruction formats, operand types, and statement semantics.
Recent assembler implementations have improved lexical analysis by:
Utilizing state machines and finite automata optimized for speed and minimal memory use.
Supporting Unicode and UTF-8 encoding to allow extended character sets in symbol names.
Enabling incremental tokenization for interactive development environments and real-time error detection.
Integrating with syntax highlighting engines and tooling via token stream exports.
Applying robust error recovery strategies to continue processing despite lexical errors, facilitating better diagnostic messages.
Lexical analysis and tokenizing represent a critical early step in assembling x86-64 code, transforming raw source into atomic components that downstream phases can reliably interpret. Mastery of this phase ensures efficient and accurate parsing, robust error handling, and smooth support for complex assembly language features.