Reverse Engineering: Tools and Techniques

Reverse engineering is the process of analyzing a binary executable or firmware to understand its structure, behavior, and logic — using disassemblers, debuggers, and analysis techniques without access to the original source code.

What You’ll Learn

You’ll use industry-standard disassemblers (IDA Pro, Ghidra, radare2), debuggers (x64dbg, GDB), understand static vs dynamic analysis approaches, set up a malware analysis sandbox, unpack compressed binaries, and evade signature-based detection.

Why It Matters

Reverse engineering is essential for malware analysis, vulnerability research, competitive analysis, and intellectual property protection. Durga Antivirus Pro’s malware detection engine uses reverse engineering techniques daily to identify new malware variants and unpack suspicious binaries before they execute.

Real-World Use

A security analyst receives a suspicious email attachment. Instead of running it (which could infect the network), they execute it in a sandbox, trace API calls with x64dbg, identify the command-and-control (C2) URL in the unpacked binary, and block it at the firewall — all without executing the malware on real hardware.

Reverse Engineering Approach

flowchart TD
    A[Binary Sample] --> B{Approach}
    B --> C[Static Analysis]
    B --> D[Dynamic Analysis]
    C --> E[Disassemble with IDA/Ghidra]
    E --> F[Analyze Control Flow]
    F --> G[Identify Key Functions]
    D --> H[Run in Sandbox]
    H --> I[Monitor API Calls]
    H --> J[Trace Execution]
    I --> K[Dump Memory/Unpack]
    G --> L[Complete Analysis]
    K --> L
    style C fill:#2563eb,color:#fff
    style D fill:#dc2626,color:#fff

Step 1: Static Analysis with Disassemblers

Ghidra (NSA, free) is the best starting point:

# Install Ghidra
wget https://github.com/NationalSecurityAgency/ghidra/releases/download/ghidra_11.1.2/ghidra_11.1.2_PUBLIC.zip
unzip ghidra_*.zip
cd ghidra_*/ && ./ghidraRun

Key analysis in Ghidra:

# Ghidra Python script: find suspicious API calls
from ghidra.program.model.symbol import RefType

def find_suspicious_apis():
    suspicious = [
        "CreateRemoteThread", "WriteProcessMemory", "VirtualAllocEx",
        "URLDownloadToFileA", "WinExec", "ShellExecuteA",
        "RegSetValueA", "socket", "connect", "send", "recv"
    ]

    fm = currentProgram.getFunctionManager()
    funcs = fm.getFunctions(True)

    for func in funcs:
        name = func.getName()
        if name in suspicious:
            print(f"[!] Found: {name} at {func.getEntryPoint()}")

find_suspicious_apis()

Expected output: Ghidra lists each suspicious API call with its memory address. CreateRemoteThread + WriteProcessMemory together suggest process injection. URLDownloadToFileA suggests a downloader trojan. These are strong indicators of malicious behavior.

radare2 Command-Line Analysis

# Open file for analysis
r2 -A suspicious.exe

# List all imports
[0x10001234]> iI

# Find cross-references to a function
[0x10001234]> axt sym.WriteProcessMemory

# Decompile current function
[0x10001234]> pdc

# Graph view of control flow
[0x10001234]> ag

# Strings (often reveal C2 URLs, encryption keys)
[0x10001234]> iz
[0x10001234]> izz  # All strings (including data sections)

Expected output: radare2’s izz reveals embedded strings — HTTP URLs, IP addresses, registry keys, and error messages. These are often the quickest way to determine what a binary does.

Step 2: Dynamic Analysis with Debuggers

x64dbg (Windows) for live debugging:

# Open sample.exe in x64dbg
# First, set breakpoint at entry point (system breakpoint)
# Press F9 to run until Entry Point

# Set breakpoint on specific API calls
bp kernel32!VirtualAlloc
bp wininet!InternetOpenA

# Run: F9
# Trace into: F7
# Step over: F8

# View call stack
# View → Call Stack (Alt+K)

# Memory map view for unpacking
# View → Memory Map (Alt+M)
# Look for regions marked RWX (Read-Write-Execute) — suspicious!

Expected output: Execution stops at each API call. You can inspect arguments in the stack, return values in EAX/RAX, and memory buffers. Watching URLDownloadToFileA with its parameters reveals the download URL and save path.

GDB for Linux ELF Analysis

# Disassemble main function
gdb ./sample
(gdb) disassemble main

# Break on system calls
(gdb) catch syscall write
(gdb) catch syscall execve

# Run with command-line arguments
(gdb) run --flag evil_input

# Examine registers
(gdb) info registers
(gdb) x/10x $rip  # examine 10 hex values at current instruction

# Break at specific address
(gdb) break *0x401234

Expected output: GDB catches the execve syscall — you see the program trying to execute a second process. The x/10x dump shows raw bytes at the current instruction pointer, confirming the opcodes you saw in the disassembler.

Step 3: Malware Analysis Sandbox

# floss.py — FireEye Labs Obfuscated String Solver
from floss import Floss

# Analyze strings in a sample
floss = Floss()
results = floss.scan("sample.exe")

print("=== Static Strings ===")
for s in results.strings.static:
    print(s.string)

print("\n=== Decoded Strings (stack strings) ===")
for s in results.strings.decoded:  # Some require FLOSS (FireEye Labs Obfuscated String Solver)
    print(s.string)

print("\n=== Tight Strings (API decodes) ===")
for s in results.strings.tight:
    print(s.string)

Expected output: FLOSS reveals hidden strings that standard strings utility misses. Malware authors often construct strings at runtime by XORing bytes on the stack — FLOSS emulates these decoders automatically.

Sandbox Setup (Cuckoo/CAPE)

# docker-compose.yml for CAPE sandbox
version: "3"
services:
  cape:
    image: cape/cape:latest
    privileged: true
    volumes:
      - ./samples:/samples
      - ./reports:/reports
    environment:
      - ANALYSIS_TIME=120  # seconds
    network_mode: "none"   # no network — prevent C2 communication

Expected output: CAPE runs the sample, monitors all API calls, file system changes, registry modifications, network connections, and memory allocations. The report shows: “Created process: cmd.exe /c ping 8.8.8.8” or “Wrote file: C:\Users\victim\AppData\Roaming\svchost.exe”.

Step 4: Unpacking Compressed Binaries

Most malware is packed (compressed/obfuscated) to evade signature detection:

# Simple XOR unpacker
def xor_decode(data: bytes, key: bytes) -> bytes:
    return bytes(data[i] ^ key[i % len(key)] for i in range(len(data)))

with open("packed.bin", "rb") as f:
    packed = f.read()

# Try common XOR keys
for key in [b"\x01", b"\xFF", b"ABCD"]:
    unpacked = xor_decode(packed, key)
    if b"MZ" in unpacked[:2]:  # PE header
        with open(f"unpacked_{key.hex()}.exe", "wb") as out:
            out.write(unpacked)
        print(f"Found PE header with key: {key}")

Expected output: The script finds the original PE executable (MZ header) embedded inside the packed binary. Advanced packers like UPX, Themida, and VMProtect require more sophisticated unpacking — often involving dumping process memory at the Original Entry Point (OEP).

Finding the OEP (Original Entry Point)

# Using x64dbg to find OEP:
# 1. Open packed.exe in x64dbg
# 2. Go to Memory Map (Alt+M)
# 3. Set breakpoint on all memory accesses
# 4. Run (F9) — execution stops frequently
# 5. When you see a "push ebp; mov ebp, esp" sequence (function prologue)
# 6. That's likely the OEP — dump the memory with Scylla plugin

Expected output: The dumped executable at OEP shows the original imports. Before unpacking, the import table might show only LoadLibraryA and GetProcAddress. After dumping, you see the real imports: CreateFile, WriteFile, RegOpenKeyEx, etc.

Common Errors

1. Not analyzing in a sandbox first Running unknown binaries on your host machine — even in a VM without snapshots — is dangerous. Always use a sandbox (CAPE, Joe Sandbox, Hybrid Analysis) for initial analysis. Set up a dedicated VM with snapshots restored after each analysis.

2. Confusing compiler artifacts with malicious code Modern C++ executables include thousands of functions from standard libraries. std::sort looks nothing like a suspicious function. Learn to distinguish compiler-generated code (C runtime, exception handling, vtable dispatch) from actual application logic.

3. Giving up at packed binaries Packing is the default, not the exception. Learn basic unpacking: single-byte XOR, UPX (upx -d), and API monitoring to find the OEP. Most packers decompress to memory — dumping at the right moment reveals the original code.

4. Over-relying on automated tools Ghidra’s decompiler sometimes produces incorrect output, especially with obfuscated code. Cross-reference with the raw disassembly, check register values, and trace execution manually for critical functions.

5. Missing anti-analysis tricks Malware detects debuggers (IsDebuggerPresent, NtQueryInformationProcess), virtual machines (registry keys, MAC addresses), and sandbox environments (sleep calls, user interaction checks). Learn to patch these checks or use a custom sandbox that evades detection.

Practice Questions

1. What is the difference between static and dynamic analysis? Static analysis examines the binary without executing it — disassembling, string extraction, control flow graph analysis. Dynamic analysis executes the binary in a controlled environment and monitors behavior — API calls, memory changes, network traffic.

2. How do disassemblers like Ghidra and IDA Pro differ? Ghidra (NSA, free) has a powerful decompiler, collaborative analysis, and Python scripting. IDA Pro (Hex-Rays, paid) has superior decompilation quality, more processor architectures, the IDAPython ecosystem, and better commercial support. Ghidra is better for collaborative teams; IDA is better for single advanced analysts.

3. What is the Original Entry Point (OEP) and why does it matter? OEP is the address where the original program code starts executing after the unpacker/loader finishes. Finding the OEP lets you dump the unpacked binary in memory, revealing the original imports, code structure, and strings.

4. How do anti-analysis techniques work in malware? Anti-debug: IsDebuggerPresent(), NtGlobalFlag. Anti-VM: checking for VMware/VirtualBox processes, MAC addresses, hardware identifiers. Anti-sandbox: sleep calls (Sandboxie has 1-second time limit bypass), checking user interaction (mouse clicks, uptime).

5. Challenge: Write a YARA rule for the unpacked sample Analyze an unpacked malware sample. Identify unique strings, section names, import patterns, and byte sequences. Write a YARA rule that detects this family without false positives.

FAQ

What’s Next

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.