x86-64 Machine Code Encoding : ModR/M Encoding with Examples

The ModR/M byte is a crucial component of x86-64 machine code encoding that specifies the addressing mode, the register operand, and the memory addressing details for many instructions. It acts as a flexible encoding scheme to select registers or memory operands, allowing the CPU to interpret instruction operands efficiently. Mastering ModR/M encoding is essential when designing an assembler, as it directly impacts the accuracy of generated machine code.

1. Structure of the ModR/M Byte

The ModR/M byte is a single 8-bit value divided into three fields:

• Mod (2 bits): Determines whether the operand is a register or memory and the addressing displacement size.

• Reg (3 bits): Usually encodes a register operand or an opcode extension field.

• R/M (3 bits): Specifies the register or memory operand.

2. Meaning of Each Field

2.1 Mod Field

Special case:

• When Mod = 00 and R/M = 101, the R/M field encodes a 32-bit displacement-only address (i.e., no base register).

2.2 Reg Field

• Encodes one of the registers, often serving as the source or destination register.

• Alternatively, it can extend the opcode when certain instructions use it as an opcode extension.

• Values range from 0 to 7, representing registers: RAX, RCX, RDX, RBX, RSP, RBP, RSI, RDI (in legacy 3-bit form), extended by REX prefixes.

2.3 R/M Field

• Specifies either a register operand or a memory addressing mode depending on the Mod field.

• When Mod ≠ 11, the R/M value specifies the base register or special addressing modes, often combined with a Scale-Index-Base (SIB) byte for complex addressing.

3. Interaction with SIB Byte and Displacement

• When R/M = 100 (binary 100), a SIB (Scale-Index-Base) byte follows to enable more complex addressing modes involving scaled index registers.

• Displacement bytes (8-bit or 32-bit) follow the ModR/M (and SIB if present) byte depending on the Mod field value.

4. REX Prefix and ModR/M Extension

• The introduction of x86-64 and the REX prefix expands the Reg and R/M fields by adding an extra high bit, allowing access to registers R8 through R15.

• The REX prefix bits R, X, and B respectively extend the Reg, Index (in SIB), and R/M fields from 3 bits to 4 bits.

5. Detailed Examples

Example 1: Register to Register Move — MOV RAX, RBX

• Instruction: MOV r64, r64

• Encoding requires ModR/M byte with Mod=11 (register-direct), Reg and R/M fields specifying registers.

• RAX register code: 000

• RBX register code: 011

• Mod = 11 (register addressing)

• Reg = RBX (011) — source

• R/M = RAX (000) — destination

ModR/M byte: 11 011 000 binary = 0xD8 hex

• Complete encoding includes the REX prefix to specify 64-bit operand size (REX.W = 1).

Example 2: Memory Operand with 8-bit Displacement — MOV EAX, [RBX + 5]

• Instruction: Move a 32-bit value from memory at [RBX + 5] into EAX.

• Mod field = 01 (8-bit displacement)

• Reg = EAX (000)

• R/M = RBX (011)

• Displacement: 0x05 (8-bit signed integer)

ModR/M byte: 01 000 011 binary = 0x43 hex Displacement: 0x05

Example 3: Memory Operand with SIB — MOV EAX, [RAX + RDX*4 + 16]

• Mod = 10 (32-bit displacement)

• Reg = EAX (000)

• R/M = 100 (indicates SIB byte follows)

• SIB byte specifies: scale=4 (10), index=RDX (010), base=RAX (000)

• Displacement: 16 (0x10) 32-bit

ModR/M byte: 10 000 100 binary = 0x84 hex SIB byte: 10 010 000 binary = 0x90 hex Displacement: 0x00000010

Example 4: RIP-Relative Addressing — MOV EAX, [RIP + offset]

• In x86-64, RIP-relative addressing is used with Mod = 00 and R/M = 101 indicating a displacement relative to the instruction pointer (RIP).

• Displacement is a signed 32-bit value relative to the next instruction.

ModR/M byte: 00 000 101 binary = 0x05 hex Displacement: 32-bit signed offset

6. Summary of Key Points

• ModR/M byte encodes operand addressing modes flexibly in 8 bits.

• Mod field defines if the operand is register or memory and size of displacement.

• Reg field encodes a register operand or opcode extension.

• R/M field encodes register or memory operand; combined with SIB byte for complex addressing.

• REX prefix extends these fields to support more registers.

• Correct ModR/M encoding is mandatory for correct operand decoding by the CPU.

7. Assembler Implementation Considerations

• The assembler must decode instruction operands and determine the appropriate ModR/M byte based on operand types and addressing modes.

• Displacement size (none, 8-bit, 32-bit) must be chosen carefully based on the target address.

• SIB bytes must be generated when necessary to support scaled-index addressing modes.

• REX prefix bits must be set to extend register fields when encoding registers R8–R15.

• Edge cases such as RIP-relative addressing and special Mod/RM combinations require precise handling.