Introduction to x86_64 Assembly

Before diving further into the study of crackmes, it seemed appropriate to write a short introduction to x86_64 assembly language.
It is a complex architecture, mainly due to the vast number of instructions in its instruction set.

1. Registers

On the x86_64 architecture, there are 16 general-purpose 64-bit registers. Each register has a full 64-bit name and can be divided into sub-registers to manipulate 32-bit, 16-bit, or 8-bit data.

The 16 General-Purpose Registers (64 bits)

Special and Control Registers

• RIP (Instruction Pointer): Points to the next instruction to execute.
• RFLAGS: Status register containing flags:
• ZF (Zero Flag): Set to 1 if the result of the last operation is zero.
• SF (Sign Flag): Set to 1 if the result is negative.
• CF (Carry Flag): Unsigned arithmetic overflow.
• OF (Overflow Flag): Signed arithmetic overflow.

Vector and Floating-Point Registers

(SIMD)XMM0 to XMM15 (128 bits) / YMM0 to YMM15 (256 bits) / ZMM0 to ZMM31 (512 bits): Used for floating-point calculations (float/double) and vector processing (AVX/SSE).

2. Stack Mechanics

Understanding how the stack operates is essential for binary analysis.

In the x86_64 architecture, the stack follows a LIFO (Last-In, First-Out) structure and grows downwards toward lower memory addresses. When data is pushed onto the stack (via PUSH or a CALL instruction), the Stack Pointer (RSP) is decremented by 8 bytes; conversely, when data is popped (via POP or RET), RSP is incremented.

The stack is heavily used by binaries to store local variables, pass function parameters beyond the 6th argument, and keep track of function return addresses.

3. Instructions

There is no fixed, exact count for the x86_64 instruction set, as the architecture has continuously evolved by accumulating extensions (SSE, AVX, AVX-512, AMX, VMX, etc.).Counting basic mnemonics alone (without advanced vector variants), there are roughly 1,000 to 1,500 distinct instructions.
Including all modern SIMD instruction sets, that number exceeds 2,000 to 3,000 opcodes.However, in reverse engineering, fewer than 50 instructions account for about 90% of the assembly code encountered in crackmes.

1. Data Transfer (1–7)

MOV (Move) — Copies the source operand to the destination operand.

• Example: mov rax, 0x1 (Copies 1 into RAX)

LEA (Load Effective Address) — Calculates the memory address of the source or performs fast arithmetic without accessing memory.

• Example: lea rsi, rbx + 0x10

MOVZX (Move with Zero-Extend) — Copies a smaller register into a larger one, padding the higher bits with zeros.

• Example: movzx eax, byte ptr rsi

MOVSX (Move with Sign-Extend) — Copies a smaller register into a larger one while preserving the sign bit.

• Example: movsx eax, al (Extends AL to EAX while maintaining its sign)

CMOVE / CMOVZ (Conditional Move if Equal) — Copies the source into the destination only if ZF = 1.

• Example: cmove eax, ebx (Copies EBX to EAX if the comparison was equal)

CMOVNE / CMOVNZ (Conditional Move if Not Equal) — Copies source to destination only if ZF = 0.

• Example: cmovne ax, r8w (Copies R8W to AX if not equal)

XCHG (Exchange) — Swaps the contents of two registers or a register and a memory location.

• Example: xchg rax, rbx (Swaps RAX and RBX)

2. Stack Management (8–9)

PUSH — Places a value onto the top of the stack (RSP decreases by 8).

• Example: push rbp (Saves RBP onto the stack)

POP — Pops the top value from the stack into a register (RSP increases by 8).

• Example: pop rsi (Pops the top stack value into RSI)

3. Basic Arithmetic (10–18)

ADD — Adds two values.

• Example: add rax, rbx (RAX = RAX + RBX)

SUB — Subtracts the source from the destination.

• Example: sub rsp, 0x20 (Allocates 32 bytes on the stack)

INC (Increment) — Adds 1 to the destination.

• Example: inc rcx (RCX = RCX + 1)

DEC (Decrement) — Subtracts 1 from the destination.

• Example: dec rcx (RCX = RCX - 1)

MUL (Unsigned Multiply) — Unsigned multiplication (RAX * source, result stored in RDX:RAX).

• Example: mul rbx (RAX = RAX * RBX)

IMUL (Signed Multiply) — Signed multiplication.

• Example: imul eax, ebx, 0x5 (EAX = EBX * 5)

DIV (Unsigned Divide) — Unsigned division of RDX:RAX by the source.

• Example: div rbx (RAX = Quotient, RDX = Remainder)

IDIV (Signed Divide) — Signed division of RDX:RAX by the source.

• Example: idiv rbx (Signed division by RBX)

NEG (Two's Complement Negation) — Inverts the sign of a number (two's complement).

• Example: neg eax (EAX = -EAX)

4. Bitwise Logic & Masking (19–23)

XOR — Bitwise Exclusive OR (commonly used to clear a register or for light encryption).

• Example: xor eax, eax (Sets EAX to 0)

AND — Bitwise AND (used for masking).

• Example: and al, 0x0f (Isolates the 4 lower bits)

OR — Bitwise OR (used to set specific bits).

• Example: or eax, 0xa (Forces specific bits to 1)

NOT — Bitwise NOT (one's complement, flips all bits).

• Example: not eax (Flips all 0s to 1s and vice versa)

NOP (No Operation) — Does nothing (frequently used for patching instructions).

• Example: nop (Advances to the next instruction)

5. Shifts & Rotations (Crypto/Hash) (24–29)

• SHL / SAL (Shift Left) — Logical left shift (multiplies by 2^n). Example: shl rax, 4 (Multiplies RAX by 16)

SHR (Shift Right) — Logical right shift (divides unsigned integer by 2^n).

• Example: shr eax, 4 (Shifts 4 bits to the right)

SAR (Shift Arithmetic Right) — Arithmetic right shift (preserves the sign bit).

• Example: sar eax, 2 (Signed division by 4)

ROL (Rotate Left) — Rotates bits to the left.

• Example: rol rbx, 8 (Rotates bits 8 positions to the left)

ROR (Rotate Right) — Rotates bits to the right.

• Example: ror rbx, 8 (Rotates bits 8 positions to the right)

SETcc (Set Byte on Condition: SETE, SETNE, etc.) — Sets a byte to 1 or 0 based on RFLAGS.

• Example: setz al (Sets AL to 1 if the Zero Flag is set)

6. Comparisons & Tests (30–32)

CMP (Compare) — Subtracts the source from the destination internally and updates RFLAGS without modifying the operands.

• Example: cmp ax, 0xf (Compares AX with 15)

TEST — Performs a bitwise AND internally and updates RFLAGS (notably setting ZF if the result is zero).

• Example: test al, al (Checks if AL is zero or negative)

BT (Bit Test) — Copies the value of a specific bit into the Carry Flag (CF).

• Example: bt eax, 3 (Tests the 3rd bit of EAX)

7. Unconditional & Conditional Jumps (33–44)

JMP (Unconditional Jump) — Jumps to the specified target address unconditionally.

• Example: jmp 0x4001f9 (Jumps directly to address 0x4001f9)

JE / JZ (Jump if Equal / Zero) — Jumps if ZF = 1.

• Example: je 0x400235 (Jumps if the previous test resulted in equality)

JNE / JNZ (Jump if Not Equal / Not Zero) — Jumps if ZF = 0.

• Example: jne 0x400235 (Jumps to the failure block if non-equal)

JG / JNLE (Jump if Greater) — Signed jump if target > source.

• Example: jg 0x401000 (Jumps if A > B)

JGE / JNL (Jump if Greater or Equal) — Signed jump if target >= source.

• Example: jge 0x401000 (Jumps if A >= B)

JL / JNGE (Jump if Less) — Signed jump if target < source.

• Example: jl 0x401000 (Jumps if A < B)

JLE / JNG (Jump if Less or Equal) — Signed jump if target <= source.

• Example: jle 0x401000 (Jumps if A <= B)

JA / JNBE (Jump if Above) — Unsigned jump if target > source.

• Example: ja 0x40013f (Jumps if A > B in unsigned comparison)

JAE / JNB (Jump if Above or Equal) — Unsigned jump if target >= source.

• Example: jae 0x40013f (Jumps if A >= B)

JB / JNAE (Jump if Below) — Unsigned jump if target < source.

• Example: jb 0x40013f (Jumps if A < B in unsigned comparison)

JBE / JNA (Jump if Below or Equal) — Unsigned jump if target <= source.

• Example: jbe 0x40013f (Jumps if A <= B)

JS / JNS (Jump if Sign / No Sign) — Jumps if the result is negative (SF = 1).

• Example: js 0x400235 (Jumps if the sign flag is set)

8. Function Calls & Control Flow (45–47)

CALL — Pushes the return address onto the stack and jumps to the target procedure.

• Example: call 0x4000fe (Calls the validation function)

RET (Return) — Pops the return address off the stack and jumps to it.

• Example: ret (Returns from the current function)

SYSCALL — Triggers a kernel system call specified by the number in RAX.

• Example: syscall (Executes the configured system call)

9. String Operations & Loops (48–50)

LOOP — Decrements RCX and jumps to the target address if RCX != 0.

• Example: loop 0x401020 (Loops until RCX reaches zero)

REP MOVSB — Moves a block of memory from RSI to RDI for RCX bytes.

• Example: rep movsb (Copies a string byte by byte)

STOSB — Stores the byte in AL into the memory address pointed to by RDI.

• Example: rep stosb (Fills a memory region with a constant byte / memset)

4. Epilogue

In this introductory post, I aimed to provide the essential foundations needed to approach a crackme with confidence.
As we progress through upcoming articles, you will notice that these 50 instructions are the ones that appear repeatedly.
In the next post, we will tackle a relatively simple crackme, and I will walk you through the exact methodology I use to save time when analyzing binaries.