RTOS Reverse Engineering

4 DAY U_LONG 32 CPE HOUR TRAINING: JANUARY 2021 * FEB 2-7

Evan Jensen

Abstract

This training will teach students how to analyze real time operating systems deployments. The training will focus on concepts from open source operating systems, where the most context is available, then transition the lessons learned to closed source third-party deployments with an emphasis on VXWORKS based products.

The training will focus predominately on the ARM architecture with cameos from others as circumstances allow.

Students will learn about challenges solved by forward engineering teams and use that information to make informed conclusions when reversing.

Students will learn about security technology in embedded products including Cryptographic Security Modules (CSMs) and Memory Protection Units (MPUs).

The course is primarily hands-on-keyboard exercises rather than lecturing, but will introduce diagrams and theory as needed. The entire class will regularly sync up as a group to discuss concepts, problems, and solutions.

Course Topics

Real Time Operating System Concepts

with examples from:

Forward Development Concepts

  • Embedded development life-cycle
  • Chip capabilities and selection
  • Memory management

QEMU

  • Emulating "full" computer systems
  • Debugging and run-time introspection
  • Challenges associated with "re-hosting" deployments found in the wild

Reverse Engineering Challenges

  • Separating operating system code from application code
  • Data reconstruction
  • Reversing unknown APIs
  • Automatically identifying standard library functions
  • Static identification of ABIs

Tooling

  • Writing Loader and Analyzer plugins for Ghidra to create a more familiar analysis environment and accelerate reverse engineering.

Prerequisites

Students are expected to have experience programming in C or C++, and basic knowledge of the Linux command line. Prior experience with reverse-engineering is nice to have, but not required.

System Requirements

Computer capable of running a virtual machine. Recommended minimum 16GB RAM with quad-core processor. VMWare or VirtualBox to run a Linux VM (all exercises will be in the Linux VM)