Awesome Virtualization

A curated list of awesome resources about virtualization.

Table of Contents

• Chronology
• Documentation
• Books
• Courses
• Papers
• Research Projects
• Mainstream Hypervisors Documentation
  • KVM
  • Xen
  • QEMU
  • VMware
  • VirtualBox
  • Hyper-V
• Hypervisor Development
• Virtual Machine Introspection
• Attacking Hypervisors
• Malware Analysis

Chronology

• 2005-November-13: Intel VT-x released on Pentium 4 (Model 662 and 672) processors
• 2006-May-23: AMD AMD-V released on Orleans and Windsor processors
• 2007-September-10 : AMD Barcelona adds RVI (Rapid Virtualization Indexing) a.k.a (Nested Page Tables) a.k.a (SLAT)
• 2008-November: Intel Nehalem
  • EPT (Extended Page Tables) a.k.a (SLAT)
  • VPID (Virtual Processor ID)
• 2010-January-7: Intel Westmere adds unrestricted guests a.k.a (Real Mode Support)
• 2013-June-4: Intel haswell:
  • VMCS Shadowing
  • VMFUNC, #VE and EPTP switching
• 2017
  • June-21: AMD EPYC adds suport for Secure Encrypted Virtualization (SEV)
  • AMD documents Encrypted State (SEV-ES)
• 2019
  • AMD documents Secure Nested Paging (SEV-SNP)
  • August-1: Intel Ice Lake
    • EPT SPP (EPT-Based Sub-Page Write Protection)
    • Virtualizing Intel Processor Trace output buffer using EPT
• 2020-March: Intel documents Hypervisor-Managed Linear Address Translation (HLAT)

Documentation

Intel

• Intel® 64 and IA-32 architectures software developer's manual volume 3C
• VMCS Layout

AMD

• Secure Encrypted Virutalization

Books

• Virtual Machines: Versatile Platforms for Systems and Processes
• Mastering KVM Virtualization

Courses

• Memory Virtualization playlist by Udacity
• Full Virtualization by Geoffrey Challen
• Xen and the Art of Virtualization by Geoffrey Challen
• Container Virtualization by Geoffrey Challen
• Open Security Training Advanced VT-x course
• From Kernel to VMM

Papers

• A comparison of software and hardware techniques for x86 virtualization by K. Adams and O. Agesen (2006)
• Bringing Virtualization to the x86 Architecture with the Original VMware Workstation by Edouard Bugnion, Scott Devine, Mendel Rosenblum, Jeremy Sugerman, And Edward Y. Wang
• The evolution of an x86 virtual machine monitor by O. Agesen, A. Garthwaite, J. Sheldon, and P. Subrahmanyam
• Formal Requirements for Virtualizable Third Generation Architectures by Gerald J. Popek & Robert P. Goldberg
• Modern Operating System 4th Edition (Chapter: Virtualization and the cloud) by Andrew Tanembaum
• Xen and the Art of Virtualization by Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew Warfield
• Understanding Full Virtualization, Paravirtualization and Hardware Assisted Virtualization by VMWare
• Dynamic Binary Translation from x86-32 code to x86-64 code for Virtualization by Yu-hsin Chen.
• MemoryRanger Prevents Hijacking FILE_OBJECT Structures in Windows Kernel by Igor Korkin (2019)

Research Projects

• 2008: BitVisor
• 2010: Xvisor
• 2011:
  • ramooflax
  • TinyVM
• 2013: jailhouse
• 2014: HOSS
• 2015: Bareflank
• 2016:
  • SimpleVisor
  • HyperPlatform
  • kHypervisor
  • rustyvisor
  • HyperBone
  • VivienneVMM
• 2017:
  • hypervisor-for-beginners
  • Intel HAXM
  • ksm
  • crosvm
  • Firecracker
  • SimpleSvm
  • GiantVM
• 2018:
  • hvpp
  • ACRN
  • gbhv
  • applepie
  • boxy
  • nemu
  • gvisor
• 2019:
  • rust-vmm
  • MemoryRanger
  • ZeldaOS.x86_64
  • vbh
  • HypSec
  • zpp_hypervisor
  • orange_slice
  • cloud-hypervisor
  • uhyve
  • mythril
• 2020:
  • MiniVisorPkg
  • MicroV
  • Zelda.RISCV

Mainstream Hypervisors Documentation

KVM

• KVM website
• KVM forum
• set of KVM documentations
• How VT-x, KVM and QEMU Work Together

Xen

• Xen website

QEMU

• QEMU website

VMware

• The Architecture of VMware ESXi

VirtualBox

• VirtualBox website
• VirtualBox documentation

Hyper-V

• Hyper-V internals researches (2006-2019)
• 2015:
  • Battle of SKM and IUM
  • Ring 0 to Ring -1 Attacks
• 2017:
  • Virtualization Based Security - Part 1: The boot process
  • Virtualization Based Security - Part 2: kernel communications
  • Hyper-V and its Memory Manager
• 2018:
  • A Dive in to Hyper-V Architecture & Vulnerabilities
  • Hardening Hyper-V through offensive security research - Black Hat
  • First Steps in Hyper-V Research
• 2019:
  • Growing Hypervisor 0day with Hyperseed
  • VBS and VSM Internals
• 2020:
  • Hyper-V #0x1 - Hypercalls part 1
  • Hyper-V LIS
• Virtualization Documentation
• Hyper-V technet
• Hyper-V Internals

Hypervisor Development

Hypervisor From Scratch

• Part 1: Basic Concepts & Configure Testing Environment
• Part 2: Entering VMX Operation
• Part 3: Setting up Our First Virtual Machine
• Part 4: Address Translation Using Extended Page Table (EPT)
• Part 5: Setting up VMCS & Running Guest Code
• Part 6: Virtualizing An Already Running System
• Part 7: Using EPT & Page-Level Monitoring Features
• Part 8: How To Do Magic With Hypervisor!

5 Days to Virtualization

• Day 0: Virtual Environment Setup, Scripts, and WinDbg
• Day 1: Introduction to Virtualization, Type Definitions, and Support Testing
• Day 2: Entering VMX Operation, Explaining Implementation Requirements
• Day 3: The VMCS, Component Encoding, and Multiprocessor Initialization
• Day 4: VMCS Initialization, Segmentation, and Operation Visualization
• Day 5: The VM-exit Handler, Event Injection, Context Modifications, and CPUID Emulation

Virtual Machine Introspection

• Zero-Footprint Guest Memory Introspection from Xen by Mihai Dontu - [Slides] [Update]
• Hypervisor memory introspection at the next level
• Bringing Commercial Grade Virtual Machine Introspection to KVM by Mihai Donțu - [Slides]
• Hypervisor-based, hardware-assisted system monitoring
• Virtual Machine Introspection to Detect and Protect
• Hypervisor Memory Forensics - [Slides]
• Who Watches The Watcher? Detecting Hypervisor Introspection from Unprivileged Guests
• DRAKVUF Black-box Binary Analysis for in-depth execution tracing of arbitrary binaries
• Patchguard: Detection of Hypervisor Based Introspection - P1
• Patchguard: Detection of Hypervisor Based Introspection - P2

Attacking Hypervisors

• Blackhat 2010 - Hacking the Hypervisor
• Software Attacks on Hypervisor Emulation of Hardware - [Slides]
• Lessons Learned from Eight Years of Breaking Hypervisors - [Slides]
• Attacking Hypervisors Using Firmware And Hardware - [Slides]
• The Arms Race Over Virtualization - [Slides]
• Glitches in the Matrix – Escape via NMI

KVM

• Virtualization under attack: Breaking out of KVM - [Slides]
• Performant Security Hardening of KVM by Steve Rutherford - [Slides]

Xen

• Ouroboros: Tearing Xen Hypervisor With the Snake
• Subverting the Xen hypervisor
• Preventing and Detecting Xen Hypervisor Subversions
• Bluepilling the Xen Hypervisor
• XenPwn: Breaking paravirtualized devices - [Slide]
• Advanced Exploitation: Xen Hypervisor VM Escape
• Xen exploitation part 1: XSA-105, from nobody to root
• Xen exploitation part 2: XSA-148, from guest to host

VMware

• Cloudburst: Hacking 3D And Breaking Out Of Vmware
• The Great Escapes Of Vmware: A Retrospective Case Study Of VMWare Guest-To-Host Escape Vulnerabilities
• Out of the Truman Show: VM Escape in VMware Gracefully
• Control Register Access Exiting and Crashing VMware

VirtualBox

• Unboxing your virtualBox - [Slides]
• Breaking Out of VirtualBox through 3D Acceleration - [Slides]
• VirtualBox VMSVGA VM Escape
• VirtualBox NAT DHCP/BOOTP server vulnerabilities

Hyper-V

• Ring 0 to Ring -1 Exploitation with Hyper-V IPC
• Hardening Hyper-V through offensive security research - Black Hat
• Growing Hypervisor 0day with Hyperseed
• A Dive in to Hyper-V Architecture & Vulnerabilities
• Security Assessment of Microsoft Hyper-V
• VBS and VSM Internals
• Fuzzing para-virtualized devices in Hyper-V
• Writing a Hyper-V Bridge for Fuzzing
• Awesome Hyper-V Exploitation

CVEs

• Wandering through the Shady Corners of VMware Workstation/Fusion
• CVE-2018-2844: From Compiler Optimization to Code Execution - VirtualBox VM Escape
• CVE-2017-3558: Oracle VM VirtualBox - Guest-to-Host Privilege Escalation via Broken Length Handling in slirp Copy
• Better slow than sorry - VirtualBox 3D acceleration considered harmful
• Analyzing a Patch of a Virtual Machine Escape on VMware
• VirtualBox 3D Acceleration: An Acceleration Attack Surface
• A bunch of Red Pills: VMware Escapes
• SSD Advisory – Oracle VirtualBox Multiple Guest to Host Escape Vulnerabilities
• Pandavirtualization: Exploiting the Xen hypervisor

Malware analysis

• DEFCON 17: Reverse Engineering By Crayon: Hypervisor Based Malware Analysis and Visualization
• Hypervisors In Ur Toolbox: Monitoring N Controlling System Events With HyperPlatform
• How to hide a hook: A hypervisor for rootkits