Unikraft Confidential Compute

This project sets up a unikernel with AMD SEV-SNP enabled hardware memory encryption and validation. It uses the Unikraft elfloader project in combination with the SEV patch to securely load and execute binaries into the unikernel. Furthermore, a rudimentary port of open62541, a UA protocol C library is added. Client/Server binaries are provided as an example.

Requirements

This project was developed and tested on host kernel 6.5.0-rc2-1-snp-host-latest-ad9c0bf (with SNP-specific configuration). To achieve compatibility, the host kernel, configuration, QEMU version, and firmware version must be compatible with each other. The project was developed with Docker version 24.0.6, build ed223bc and is recommended for building.

QEMU

This specific QEMU version was used to run with SNP on this host. Note that due to AMD force-pushing commits to their repository, this version can no longer be downloaded. It is kernel and hardware specific; use the right release for your specific hardware. The scripts expect a locally compiled qemu version in qemu/build/qemu-system-x86_64

OVMF

This OVMF (Open Virtual Machine Firmware) is a specific UEFI version designed to work for the SEV-SNP version that this project was tested with. Variables and code of the VM's UEFI are separated into two files (variables are only needed for the VM) and are expected in OVMF/OVMF_CODE.fd and OVMF/OVMF_VARS.fd

(Optional) VM Image

To execute the benchmarks and or run a comparable setup inside a linux VM, a debian image is required, stored in linux-vm/debian.qcow2.

Setup

make install_starter_pack # (internal only)
make container
make 
make run

With make native and make run_native you can disable SEV. To communicate with the server you can launch the UA-control client wit make run_client.

The make command compiles the provided Unikraft image and the binary to be loaded (see the Makefile on how to set paths). Afterwards, it creates a kernel.iso image which contains the Unikraft image as an EFI boot file, a .cmdl file which provides the start arguments, and an initrd file which contains the files in the UA-control/build/rootfs folder. The following scripts are all included in the Makefile as a rules (see Makefile for rule names).

./start-unikraft-sev.sh [native] launches the kernel.iso in EFI mode with the provided network interface and OVMF firmware. It looks for OVMF files in an OVMF folder in the same repository. Since a specific QEMU version is used, this script (and the make command) expect a locally available QEMU binary. The script accepts "native" as an argument, which disables the SEV-SNP configuration. Note: the Unikraft image needs to be compiled for the specific launch parameters, either with or without SEV support; otherwise, either the SEV driver will complain, or KVM will crash because of missing instructions.

./start-vm-sev.sh [native] launches a VM with the same configuration as the unikernel. This script expects a debian.qcow2 image inside the linux-vm folder. To enable networking, the linux-vm/setup.sh script must be run. It configures the same bridge network that Unikraft uses.

./create-docker-environment [cca] launches the Docker environment required for building (and running) the kernel.iso. Project build is untested outside of the container. Running the kernel.iso is possible without the container. Optionally, the cca argument can be provided, which sets up an environment to build the Unikraft kernel with CCA support. Currently, the elfloader is not compatible with the CCA environment, the container is available for future work.

If you want to build without docker, ensure all the dependencies are installed; they are listed in container-files/amd-sev-x86_64. Bridge networking has to be set up outside of a container. By default, the container creates an unconnected bridge that has to be connected manually to a network interface. To create the same setup as in the container run the following commands:

ip link add dev virbr0 type bridge && \
ip address add 172.44.0.1/24 dev virbr0 && \
ip link set dev virbr0 up && \
mkdir /etc/qemu && \
echo "allow virbr0" > /etc/qemu/bridge.conf

UA-control

UA-control contains the sources for open62541 server/client binaries which are able to communicate whith each other and demonstrate the potential use of this project in industrial settings. The Makefile outputs the binaries in the build folder. Additionally it creates the following files:

build/encrypt encrypts and decrypts a provided file with AES 256 Bit CBC encryption with a provided keyfile. build/keyfile is the 32 Byte randomly generated key that is used to encrypt the binaries.

build/negotiate_key is an executable that uses OpenSSL to establish a secure connection either as a server or a client and it exchanges the provided keys to the container for decryption of the client/server.

build/rootfs contains all the files that will be passed to the unikernel as an initrd. It contains the encrypted formats of the client/server binaries named server.enc and client.enc. Additional files can be moved in manually and by rerunning the toplevel Makefile they will be compiled into the initrd.

Benchmarks

The following benchmarks are suggested:

• Http request bandwidth: Compare the httreply throughput provided by Mihnea Popeanga
• Memory speed: Use the memory benchmark provided by Joe Huang
• Boot time: Read the boot time that the VM images report
• Latency comparison: Perform a ping test and compute the average latency
• File size comparison:
  • 101M kernel.iso
  • 3.5M OVMF_CODE.fd
  • 400M Debian Cloud Image

Note: The kernel.iso may be reduced in size, 100M is just the recommended standart size. See here. Unikraft kernel has ~1.3M in size and the loaded binary is ~6.2M in size (because of statically linked glibc).

Results are analysed in the thesis. To run the benchmarks you can run the python script benchmarks/run_benchmarks.py. Command line options are documented in the code or use -h.

Error Handling

• The ./start-unikraft-sev.sh and ./start-vm-sev.sh scripts need the qemu version to be installed in a specific directory, specifically ./qemu/build/. This is intended, because to run SEV-SNP features a specific qemu version is required. The same goes for OVMF firmware which is expected in ./OVMF
• ./create-docker-environment.sh does attach to a docker container with the same name if there already exists one, rather than creating a new one, and it closes the container with exit. These may not get deleted if the user does not exit them properly and need to be manually stopped with docker stop.
• The toplevel Makefile automatically sets the .config to either .config_sev_reference or .config_native_reference. This has to be disabled in two of the Makefile rules manually in order to be able to work with the .config files again in the elfloader folder. Otherwise the reference configs need to be changed.
• qemu-bridge-helper is needed and not always provided, especially if built locally from QEMU source. apt install libvirt-clients-qemu provides it. It is located in /usr/lib/qemu/ and can be set with the helper= attribute as a qemu command line argument for -netdev
• If unikraft shuts down immediately after booting, configuration variables in the toplevel Makefile may need to be changed in order to load the right executable. Check in efi-files if the .cmdl and .cpio files were created and that they contain the right configuration. cpio archives can be checked with cpio -it < rootfs.cpio
• If unikraft hangs during the memory setup, the kernel is either configured for native execution and is launched in an SEV environment or VGA is enabled in the kernel config and causes the kernel to get stuck (please disable)
• UA-control binaries depend on open62541 and openssl which can be build with the Makefile with two seperate Makefile rules. Inside it there are also instructions on how to get the right library versions.
• If benchmarks fail due to VM connection issues try launching the VM manually before exeuting the benchmarks again
• unikraft-0.16.3-sev needs CONFIG_LIBUKDEBUG_PRINTK_INFO=y to enable networking.
• elfloader needs to be built with compiled in initrd . Source
• To change the network interface, change the address in the toplevel Makefile
• unikraft-0.16.3-sev is patched in efi.c, lwip/cc.h, virtio_pci.c to compile
• unikraft-0.16.3-sev has ukrandom from the latest unikraft.
• unikraft-0.16.3-sev is patched in mmap.c and similar functionalities with PAGE_ATTR_ENCRYPT.
• unikraft-0.16.3-sev is patched in nolibc to be able to compile openssl
• unikraft-0.16.3-sev is patched in uksev and virtio to disable prints to get slightly better performance

Limitations:

• SMP is currently not supported by unikraft. Because of this everything is only tested on a single core
• getaddrinfo does currently not work because statically linked applications require the shared libraries from the glibc version used for linking at runtime. open62541 is patched to use a static network interface.

Commit hashes:

- unikraft-sev    | 1e0e5403f90ff0bc24ec5dd66ca56803019b3b69
- elfloader       | 4cd9074fc83f309eb038fb1e592f5d828075db16
- httreply        | 36c07fd506b8d1a9b3e031f76b31852616838a84
- lwip            | e20459c47a6b5ab16967c15b220686c5be50d4d7
- libelf          | 0f1476d5f7fbc1a8fb44776066ab13e7403cfcae
- libzydis        | f47c083da795ff822ae1a4d538626ec552495ae6
- libopenssl      | 20ab0db15d62562a68d1bfbc3a5ae81ea4389918
- open62541       | e6bbfd312a4c2d1786349eea3127422c5f23afd3