An Analytical Approach to Modern Binary Deobfuscation


Arnau Gàmez i Montolio

For Reverse Engineers and Malware Analysts


Code obfuscation has become one of the most prevalent mechanisms aiming to complicate the process of software reverse engineering. It plays a major role on a wide range of domains: from malware threats to protection of intellectual property and digital rights management.

“An Analytical approach to Modern Binary Deobfuscation” is a curated training that provides an intensive jump-start into the field of code (de)obfuscation. Over the course of this training, students will receive a comprehensive introduction to the most relevant software obfuscation mechanisms as well as existing deobfuscation techniques to analyze, confront and defeat obfuscated code.

Key learning objectives

  • Obtain a high-level overview of the context and scenarios where code obfuscation is used
  • Gain an in-depth understanding of code obfuscation mechanisms
  • Build obfuscated code, both from scratch and through available tooling
  • Develop an understanding of the main code deobfuscation techniques
  • Learn tooling for analyzing obfuscated code and apply deobfuscation techniques
  • Become familiar with state of the art (de)obfuscation research literature


Part 1: Code obfuscation

Introduction, context and motivation

Data-flow based obfuscation

  • Constant unfolding
  • Dead code insertion
  • Encodings
  • Pattern-based obfuscation

Control-flow based obfuscation

  • Function inlining/outlining
  • Opaque predicates
  • Control-flow flattening

Mixing data-flow and control-flow obfuscation

  • VM-based obfuscation
  • Hardening VM-based obfuscation

Mixed Boolean-Arithmetic

  • Preliminary concepts
  • MBA rewriting
  • Insertion of identities
  • Opaque constants

Part 2: Code deobfuscation

Data-flow analysis

  • Reaching definition analysis
  • Liveness analysis

Dynamic binary Instrumentation

  • Tracing code execution
  • Hooking
  • Extracting deobfuscated data

SMT-based analysis

  • Semantic equivalence checking
  • Translating code conditions into SMT solver constraints
  • Proving code properties
  • Attacking simple MBA and weak cryptography

Symbolic execution

  • Reasoning about code in a symbolic way
  • Working with native code
  • Working with intermediate representations
  • Plugging an SMT solver
  • Attacking opaque predicates
  • Attacking MBA obfuscation
  • Attacking VM-based obfuscation

Program synthesis

  • Code syntax vs code semantics
  • Oracle-based program synthesis
  • Describing semantics through I/O behavior
  • Generating I/O pairs
  • Attacking MBA obfuscation
  • Attacking VM-based obfuscation

Conclusions and research directions

Tools used

  • Disassemblers
    • IDA Pro
    • radare2
  • Obfuscation
    • Manual obfuscation
    • O-LLVM
    • Tigress
  • Dynamic Binary Instrumentation
    • Frida
    • QBDI
  • Symbolic execution
    • Miasm
    • Triton
    • Radius
  • Program synthesis
    • syntia
    • msynth
    • qsynthesis
  • Other tools
    • Z3
    • MBA-Solver
    • Custom tooling

Teaching methodology

Live classes are designed to be dynamic and engaging, making the students get the most out of the training materials and instructor expertise. A clear presentation of the concepts, accompanied by illustrative examples and demos. For each section, there will be practice time allocated. The students will be provided with several exercises to work on, with the continuous support of the instructor.


  • Understanding of basic programming concepts
  • Familiarity with x86 assembly, C and Python
  • Knowledge of reverse engineering fundamentals

System requirements

  • A working desktop/laptop capable of running virtual machines
  • 40 GB free hard disk space

Students will be provided with

  • A Virtual Machine image with all tools, examples and exercises