Security Benefits of AWS Nitro-based Architecture

At reInforce 2019, AWS's inaugural security conference held in Boston, I was fortunate enough to attend many interesting sessions, workshops and chalk talks on improving security in AWS, leveraging AWS and third-party services to help secure my own and customer workloads, and have a lot of fun!

One of the sessions that really piqued my interest was around something that has been readily available on AWS for some time, but has a deeper story to it. I am referring to the architecture platform that AWS uses for its virtualization offerings: Nitro. To understand the benefits of using this architecture, and how it fits into your Enterprise Architecture and Cloud Strategy, it's important to understand the history of how we arrived here.

So, how did we get here?

I will not spend too long going over what hardware virtualization is as it has been around for quite some time now and continually improves at a rapid rate. Most employees in the modern workforce use some type of virtualization, whether it is a virtual PC, virtual desktop or virtualized application.

Virtualization's journey began prior to the 2000s, but took off when VMware introduced ESX and GSX in 2001. This was prior to the era that chip manufacturers had hardware-assisted virtualization (added by Intel VT-x and AMD-V around 2005 and 2006). The first x86 hypervisors had to utilize emulation for nearly everything, including binary translations and privileged operations. This had a tremendous impact on performance, and although virtualization was revolutionary, many considered it "slow."

In 2006, AWS started to utilize a Type 1 virtualization project/solution known as Xen. As a Type-1 hypervisor, this allowed AWS to run multiple operating systems on the same computer hardware concurrently (known as bare metal). The two types of virtualization methods achievable with Xen that AWS utilizes are paravirtual (PV) and hardware assisted virtualization (otherwise referred to as Hardware Virtual Machine or HVM).

PV vs HVM to KVM (Nitro)

Before we get to what Nitro is, and why it is awesome, we need to understand a bit more about what AWS used to utilize for its hardware services. There is a lot of information on these topics, so I will provide a high-level overview of what they are, what their issues are and what AWS did to optimize this space.

Paravirtualization

PV refers to a type of virtualization that runs on guests with a modified OS. Through utilizing PV, Xen can achieve a higher performance on the x86 architecture, even on CPUs that do not have any explicit support for virtualization. Previously, PV used to be faster than HVM, and was recommended if you needed near native speed of the hardware. However, that has changed with improvements to HVM, which include improvements to the Xen Hypervisor, newer generation CPUs and new instruction sets, and EC2 driver enhancements. In fact, as of this writing, the main method to choose PV type AMIs are to search in the community AMIs!

Hardware virtual machine

HVM alternatively, are guests that are fully virtualized without requiring a modified OS.  The guests running on top of the hypervisor are not aware that they are sharing processing time or resources with other clients on the same hardware. This means, that to the guest, it is as if it is the only OS running on the hardware and hardware extensions allow fast access to underlying hardware of the system.

Xen hypervisor model

Nevertheless, virtualization still had issues with system interrupts and various device models would compete for CPU and System resources, causing unnecessary jitter between the guest and the host. AWS thought they could do better. As such, sticking with a development mindset, they thought about approaching the problem from a microservices perspective, and use building blocks to simplify deployment of the architecture.

Enter Nitro!

Utilizing KVM, a different framework than Xen, AWS created Nitro to provide direct cards and controllers for block storage, networking and security, CPU and Memory isolation to Guest OSs, increase performance consistency, and provide larger instance sizes. It is important to note that as of this writing, AWS does not plan to completely remove Xen, and plans to continue to use and develop the Xen project for the foreseeable future. However, AWS has stated that all new instance types will utilize the Nitro architecture in the future.

To delve a little deeper, think of Nitro as having an intermediary system to process data sent to the Hypervisor. To expand, when an API call is made to an AWS EC2 (such as attach EBS), this call makes its way to a Nitro Controller, which is a separate component than the guest. Taking the example of attaching an EBS to EC2, the Nitro Controller then sends a command to an EBS Nitro Computer, and this in turn sends a hot-plug event for the PCIe device which is emulated by NVMe and shows up on the instance. As we will explore later, the communication design between the AWS fabric, the Nitro Controller and Nitro Computers, and the Hypervisor is a passive communication mechanism. Only the Nitro Controller has access to the physical Amazon EC2 network, the mainboard of the Hypervisor does not.

As such, software device models (known as Dom0, or the controller software) that were present in Xen have been replaced by software-defined hardware devices on a system bus, allowing system changes to be modeled as hardware events. This includes NVMe and ENA hot-plug events, as well as utilizing ACPI power state changes. Because the hardware devices have their own dedicated cards and act as software defined "microservices," they are all dynamically update-able. In other words, firmware updates no longer require virtual machine/guest downtime even for major updates.

Nitro-based architecture with guests (EC2 Instances)

What makes this exciting is that because of this, Nitro can also be used to utilize bare metal architectures as well! In other words, Nitro provides the same benefits whether the guest is emulated or runs on bare metal and enables technologies such as AWS Outposts, and VMWare Cloud.

Nitro-based architecture bare metal

Security benefits of Nitro

A large majority of the workload that was once leveraged by a heavily integrated Dom0 software component has been offloaded from the stack to the Nitro Controllers, this allows AWS greater flexibility for improved design and security. Each Nitro Component talks on a private network that only communicates with the Nitro Controller, and the Controller itself boots from a completely private SSD. This enables the Controller to be a single root of trust for the other components through the Nitro Security Component and supports instance monitoring functionality.

Nitro also provides a huge benefit for encryption. Because of the ability to utilize Hardware Acceleration, AWS allows for line-rate AES-256 encryption of EBS, instance storage and network without a performance penalty. In fact, for all "N" type instances, Nitro allows encryption between N type instances to be done at up to 100Gb/s within the same VPC and same region!  Note that this has to be done between N instance types, and is not available with other instance types (yet).

Encryption is further expanded upon with EBS; AWS now has the option to allow customers to specify that all EBS drives must be encrypted upon creation. Previously, one would need to set this at launch, and in order to convert drives to be encrypted, one would need to create a snapshot and recreate the drive as an encrypted volume. When utilizing AWS KMS, Nitro cache's the plaintext key within the Nitro EBS Component memory space. Because the Nitro Controller and Component is isolated from the Hypervisor by way of PCI bus, this protects customer workload co-processing.

As eluded to, Nitro enforces a passive communications design; there is a one way trust relationship with the Nitro Controller and the Hypervisor. The Hypervisor will only ever expect messages initiated from a Nitro Controller, and likewise, the Nitro Controller will never expect initiated communications from the Hypervisor. Likewise, all commands sent to the Nitro Controller are initiated from the external control plane (such as when a customer sends an EC2 API call). This means that the Controller never initiates outbound connections; it is always in a listening state, awaiting encrypted, authenticated API calls. This allows AWS to quickly and easily identify compromised resources.

Also, speaking of communication, Nitro does not allow SSH or any other interactive modes anywhere; there is no direct human access to these systems. All access is via AuthN/AuthZ API's with logging and auditing. In addition, there are no APIs for memory access.

Upgrading to Nitro

Just a quick word of warning for those that are running non-Nitro-based architectures that there is an upgrade path that must be followed in order to migrate to Nitro, if you want to use existing non-Nitro-based images. This includes installing drivers for NVMe (if the OS does not already have a driver present, say for custom images), ENA drivers, chipset drivers, and serial port drivers for Bare Metal instances. If you do not do this, you will encounter issues after you try to boot the instance. It is not as simple as selecting a new instance type from the drop-down unfortunately for these legacy built AMIs. But once you do update the instance, you will enable the ability to enjoy all of the performance and security benefits Nitro has to offer!

As such, if you are using a management tool to push out configurations, or IaC, then you can simply specify the supported AMI type and you are good to go!

Conclusions

AWS has applied the builders mentality to its underlying virtualization platform, and migrated away from a large and hard to manage controller to a "microservices" concept of system design. This has enabled them to remove unnecessary functionality off of the underlying hardware and move it to a separate controller and component model. This has enabled a stronger root of trust (the Nitro Controller) and a separation of concerns for the underlying virtualized system. It has also enabled AWS to create a more secure hardware environment for its customers. AWS views Nitro as the future and will continue to develop it for EC2 and beyond (check out Firecracker)!

If you found this article interesting, I highly recommend watching the reInforce Session SEP401-R by Mark Ryland. And as always, if you would like to learn more about how WWT can assist you with migrating your systems to Nitro, or AWS Consulting in general, please feel free to reach out!  Thank you for reading.