x86-64 Assembly


Xeno Kovah


This class teaches you how to disassemble binaries, read x86-64 assembly language, and debug black-box binaries in WinDbg and GDB. This knowledge of assembly is the fundamental skill which is required to learn reverse engineering and vulnerability exploitation. Reverse engineering is in turn a fundamental skill which is required for malware analysis and vulnerability hunting.

Prerequisite Knowledge

Complexity: Beginner

This class has minimal prerequisites. It just requires that you are comfortable with reading small (< 20 line) C programs, and have debugged C source code in the past.

Key Learning Objectives

  • Learn the most common assembly instructions, which cover > 96%+ of all code found in most programs.
  • Learn about the 16 Intel x86-64 general purpose registers + RFLAGS.
  • Understand the at time confusing or counter-intuitive compiler-isms of both Microsoft Visual Studio, and GCC which lead to particular patterns in executables' assembly.
  • Learn to debug and analyze executables which you don't have the source code for, in both WinDbg and GDB.
  • Learning how to write C code and disassemble it to see what instructions were generated. But also learning how to write assembly to see how it behaves, or even raw bytes to see how the assembler and processor interprets it.
  • Being comfortable with Reading The Fun Manual (RTFM!) to go seek out the most accurate details of how things work.
  • Reverse engineer the black box Carnegie Mellon “Binary Bomb Lab”, which has changed the lives of so many students (the instructor included!) This is a major hands-on reverse engineering exercise (which can take anywhere from 2 hours to 2 weeks!) which has been shared the world over by thousands of students. This gives you something substantive to chew on even after class to really reinforce your understanding and capability to read assembly.

One-of-a-kind Class Format!

This class is run a little different from most classes. We provide you purpose-built recorded lectures instead of trapping you in realtime with live-lectures. But fear not, the instructor is always right there eagerly waiting to mingle with the students and answer any questions you have. (The instructor really likes being asked questions. It shows you're paying attention ;)). One of many benefits is that you can watch lectures at 2x speed and zoom ahead of the other students and get to the hands on labs quicker. Or if there's bits of material you already know, you can just skip them and move on to the bits you don't know! Another big benefit is that you get to take the full lectures and labs with you! That means if you forget stuff and then need it in 6 months, you can quickly re-bootstrap yourself! Or you can watch the class twice, to really grow those neural connections and cement it in your brain! And unlike live lectures, our lectures are always getting more factually accurate, by having any accidental errors edited out.

Because we give you all the lecture and lab materials and videos after class, what you're really paying for is support from the instructor! So you'll be entitled to keep asking up to 20 questions after class, with 1-2 hour turnaround answers (after accounting for time-zone differences.) This lets you keep productively working through the material if you run out of time at the conference. If you'd like to learn more about the benefits of this style of class delivery, please read this blog post.

If you'd like to take this class, the follow on x86-64 OS Internals class, and the follow on to that, the x86-64 Intel Firmware Attack & Defense class all sequentially, you can sign up for the x86-64 All You Can Learn Buffet class instead. Keep in mind that this is more than 4 days of content, so you'd need to pick and choose which stuff you want to focus on.

Course Agenda

  • Introduction
    • About this class
    • Optional refresher: binary to hex to decimal
    • Optional refresher: two's complement negative numbers
    • Optional refresher: C data type sizes
    • Background: Endianness
  • Computer Registers
    • Memory hierarchy
    • x86-64 general purpose registers
    • Intel recommended register conventions
  • Your First Instruction
    • No-Operation (not)
  • The Stack
    • Overview
    • New instructions: push & pop
    • RoX0r Arcade: The RSP Game:
      • Level 1: Canonical orientation, rbp at top, rsp at bottom
      • Level 2: Canonical orientation, rbp and rsp move around
      • Level 3: Canonical or flipped orientation, rbp and rsp move around
      • Level 4: Four possible orientations, rbp and rsp move around
  • Calling Functions
    • CallASubroutine1.c: New Instructions: call, ret, mov, add, sub
    • Stack frame single-step slideware walkthrough
    • Stack frame single-step VisualStudio walkthrough
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: Basic mov, add, sub w/o calculator
      • Level 2: mov, add, sub w/ calculator
  • Local Variables
    • SingleLocalVariable.c
    • Mystery Listery 3 Solved!
    • ArrayLocalVariable.c, New Instructions: imul, movsx, movzx
    • StructLocalVariable.c
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: movzx
      • Level 2: movsx
  • Function Parameter Passing
    • Pass1Parameter.c
    • TooManyParameters.c
    • Mystery Listery 2 Solved!
    • 64-bit Calling Conventions
    • 32-bit Calling Conventions
    • Mystery Listery 1 Solved!
    • SpecialMaths.c, New instructions: lea
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: lea
  • Control Flow
    • GotoExample.c, New instructions: jmp
    • IfExample.c, New instructions, “jcc”, cmp
    • IfExample2.c
    • SwitchExample.c
    • RoX0r Arcade: 1 step forward, 3 steps back: Jumps
      • Level 1: jz/je, jnz/jne
      • Level 2: ja/jnbe, jna/jbe, jb/jnae, jnb/jae
      • Level 3: jg/jnle, jng/jle, jl/jnge, jnl/jge
  • Boolean Logic
    • Refresher: Boolean logic
    • BooleanBecause.c, New instructions: and, or, not, xor
    • ForLoopNoRet.c, New instructions: inc, dec
    • BitmaskExample.c, New instructions: test
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: and, or, xor, not
      • Level 2: inc, dec
    • RoX0r Arcade: 1 step forward, 3 steps back: Jumps w/ test
      • Level 1: test, js, jns
  • Bit Shifting
    • ShiftExample1.c, New instructions: shl, shr
    • ShiftExample2Unsigned.c
    • ShiftExample3Signed.c, New instructions: cdq, sar
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: shl/sal, shr, sar
  • Multiply and Divide
    • MulDivExample.c, New instructions: div, idiv
    • RoX0r Arcade: Dark Mathemagic:
      • Level 1: mul + imul, 1-operand form
      • Level 2: mul + 1, 2, and 3-operand forms of imul
      • Level 3: div + idiv
  • CISC Delight: REPeatable Instructions
    • ArrayLocalVariable2.c, New instructions: rep stos
    • ThereWillBe0xb100d.c
    • JourneyToTheCenterOfMemcpy.c: New instructions: rep movs
    • RoX0r Arcade: DarkMathemagic: BOSS LEVEL!!!
  • Choose Your Own Adventure!
    • Here, if you mostly care about Windows, GOTO Windows Binary Debugging
    • Else, if you care about *NIX systems, you can learn about AT&T syntax and see all of the previous code again as compiled by GCC instead
  • Windows Binary Debugging
    • Windbg tutorial
    • GOTO Learning to Fish: Read The F*n Intel Manual!
  • Looking at all those examples on Linux!
    • Intel vs. AT&T assembly syntax
    • Separate videos for all of the following in Linux:
      • CallAFunction1.c, SingleLocalVariable.c, ArrayLocalVariable.c, StructLocalVariable.c, Pass1Parameter.c, TooManyParameters.c, SpecialMaths.c, GotoExample.c, IfExample.c, IfExample2.c, SwitchExample.c, BooleanBecause.c, ForLoopNoRet.c BitmaskExample.c, ShiftExample1.c, ShiftExample2Unsigned.c, ShiftExample3Signed.c, MulDivExample.c, ArrayLocalVariable2.c, ThereWillBe0xb100d.c, JourneyToTheCenterOfMemcpy.c
    • RoX0r Arcade: DarkMathemagic: BOSS LEVEL!!!
  • Learning to Fish: Read The F*n Intel Manual!
    • Learn to read the FUN manual!
  • Learning to Fish: Writing Assembly
    • Writing x86-64 assembly in Visual Studio
    • Writing x86-64 assembly in GCC
  • The Most Important Assembly Exercise You'll Ever Do: Binary Bomb Lab
    • Bomb Lab Intro
    • Do the Bomb Lab in either WinDbg or GDB
  • Conclusion

Hardware Requirements

A PC or an x86 Mac (class won't work with an M1 Mac!) capable of running 2 VMs at a time with ideally 4 GB of dedicated RAM per VM.

Headphones for watching videos, (preferably over-ear so you're not disturbed as the instructor is walking around the class answering individuals' questions).

Software Requirements

  • Administrator privileges to install virtualization software on your machine.
  • A PC with VMWare Workstation or an x86 Mac with VMWare Fusion (the free “Player” versions are fine).
  • ISO for installing 1 instance of x86-64 Windows 10 (30 day trial version is fine).
  • ISO for installing Ubuntu Linux 20.04 (if you choose to learn the optional AT&T assembly syntax material).
  • A link to a software setup guide will be sent before class, and the student should install before class to maximize time available for interaction with the instructor.
  • Other software includes Visual Studio 2019, the Windows Software Development Kit (SDK), the Windows Driver Development Kit (WDK), and WinDbg.
  • For the optional Linux material, it includes gcc and gdb.