Run Nitro Sandboxes with Firecracker

Launch the parent instance with Nitro Enclaves enabled. Use the run-instances command with the --enclave-options Enabled=true option.

aws ec2 run-instances \
    --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.18-x86_64 \
    --instance-type m7i.4xlarge \
    --key-name my-key-pair \
    --iam-instance-profile Name=my-instance-profile \
    --enclave-options Enabled=true

To use attestation with AWS Key Management Service (AWS KMS), the parent instance needs an instance profile with an AWS Identity and Access Management (IAM) role. The role provides the credentials for the AWS KMS requests, and its ARN is part of the attestation document.

Connect to the instance. For more information, see Connect to your EC2 instance.

Check that the kernel version is 6.18.51-120.162.amzn2023 or later.

uname -r

Add the memory pool to the kernel command line. Size the pool for the memory of all the Nitro Sandboxes that you plan to run at the same time, plus their overhead. For the overhead, see Considerations and limitations. On an instance with more than one NUMA node, see Nitro Sandboxes on instances with multiple NUMA nodes.

sudo grubby --update-kernel=ALL --args="nitro_enclaves.mempool=8G@0"

Reboot the instance.

sudo reboot

Check that the parent instance reserved the pool.

sudo dmesg | grep "nitro_enclaves: CMA pool"

The following is example output.

nitro_enclaves: CMA pool: 8192 MiB reserved in 16 region(s), 8192 MiB requested

Download Firecracker from the Firecracker releases on the GitHub website, and install it in /usr/local/bin. Replace version with version 1.18.0 or later.

VERSION=version
ARCH=$(uname -m)
curl -sSfL -o firecracker.tgz \
    https://github.com/firecracker-microvm/firecracker/releases/download/v${VERSION}/firecracker-v${VERSION}-${ARCH}.tgz
tar -xzf firecracker.tgz
sudo install -m 755 release-v${VERSION}-${ARCH}/firecracker-v${VERSION}-${ARCH} /usr/local/bin/firecracker

Check the installed version.

firecracker --version

View the current number of slots and the maximum for the instance.

cat /sys/class/misc/nitro_enclaves/pcie_slots /sys/class/misc/nitro_enclaves/pcie_slots_total

Raise the number of slots before you launch the additional Nitro Sandboxes.

echo 12 | sudo tee /sys/class/misc/nitro_enclaves/pcie_slots

Create a file named sandbox-init with the following content. The Nitro Sandbox runs it as its first process.

#!/bin/bash
export PATH=/usr/local/bin:/usr/sbin:/usr/bin
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev
modprobe -a virtio_pci virtio_net vmw_vsock_virtio_transport nsm

# Configure eth0 with a static address and the TAP device of the parent
# instance as the default gateway.
ip link set lo up
ip addr add 192.0.2.2/24 dev eth0
ip link set eth0 up
ip route add default via 192.0.2.1
echo "nameserver 169.254.169.253" > /etc/resolv.conf

# Start your workload here.
exec sleep infinity

Make the file executable. The kernel does not start an init script without the execute permission.

chmod +x sandbox-init

Copy the kernel of the parent instance.

cp /boot/vmlinuz-$(uname -r) vmlinuz

Build the initial RAM disk. The --install option lists the programs that the init script calls, and dracut adds the libraries that they need.

sudo dracut --force --no-hostonly --reproducible --no-early-microcode \
    --modules bash \
    --add-drivers "virtio_pci virtio_net vmw_vsock_virtio_transport nsm" \
    --install "mount modprobe ip sleep" \
    --include "$PWD/sandbox-init" /init \
    --kver $(uname -r) initrd.img
sudo chown "$(id -un):" initrd.img

Do not add the base dracut module. It installs its own /init, which replaces your init script. Builds from the same files produce the same initrd.img.

Create a TAP device for the network interface of the Nitro Sandbox, give it the address of the default gateway of the guest, and turn on IP forwarding.

sudo ip tuntap add dev tap0 mode tap
sudo ip addr add 192.0.2.1/24 dev tap0
sudo ip link set tap0 up
sudo sysctl -w net.ipv4.ip_forward=1

Install nftables.

sudo dnf install -y nftables

Route the traffic of the Nitro Sandbox through the primary network interface of the parent instance with NAT. The reject rule blocks the access of the Nitro Sandbox to the instance metadata service (IMDS) of the parent instance. Without it, the Nitro Sandbox can read the instance metadata and the IAM role credentials of the parent instance.

IFACE=$(ip route show default | awk '{print $5; exit}')
sudo nft -f - <<EOF
table ip sandbox {
    chain forward {
        type filter hook forward priority filter; policy accept;
        iifname "tap0" ip daddr 169.254.169.254 reject
    }
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        iifname "tap0" oifname "$IFACE" masquerade
    }
}
EOF

With reject, a request to the IMDS fails immediately. With drop, it waits for a timeout. Keep 169.254.169.253 reachable if the guest uses the Amazon DNS server.

The TAP device, the IP forwarding setting, and the nftables rules do not persist after a reboot of the parent instance. In the Nitro Sandbox, configure an address in the same subnet with the address of the TAP device as the default gateway, as the init script in Prepare a guest image does.

Create the directory for the vsock socket, and remove a socket that an earlier launch left behind. Firecracker does not create the directory and does not remove the socket when it stops. Use a different socket path and a different TAP device for each Nitro Sandbox that runs at the same time.

sudo mkdir -p /run/sandbox
sudo rm -f /run/sandbox/v.sock

Save the following configuration as sandbox.json.

{
    "boot-source": {
        "kernel_image_path": "vmlinuz",
        "initrd_path": "initrd.img"
    },
    "machine-config": {
        "vcpu_count": 2,
        "mem_size_mib": 2048
    },
    "drives": [],
    "network-interfaces": [
        {
            "iface_id": "eth0",
            "host_dev_name": "tap0"
        }
    ],
    "vsock": {
        "guest_cid": 3,
        "uds_path": "/run/sandbox/v.sock"
    },
    "enclave": {
        "cpu_overcommit": true
    }
}

The configuration must contain the drives key, even if the Nitro Sandbox has no block devices. For the other settings, see Nitro Sandbox settings in the Firecracker configuration.

Run the following command in the directory that contains the guest image files.

sudo firecracker --no-api --config-file sandbox.json

The --no-api option runs Firecracker without an API socket. Every Firecracker process otherwise uses the same default socket path. If you need the API socket, for example for hot plug or to resize a block device, give each process its own path with the --api-sock option.

In another terminal, check that the Nitro Sandbox runs.

cat /sys/class/misc/nitro_enclaves/pcie_slots_in_use

The file shows the number of running Nitro Sandboxes. With one Nitro Sandbox, the output is 1. Firecracker does not show the guest console unless you set debug_mode.

If the volume has no file system yet, create one on the parent instance. Then make sure that the parent instance does not mount the volume.

sudo dnf install -y e2fsprogs
sudo mkfs.ext4 /dev/nvme1n1

In sandbox.json, replace the drives array with the following.

"drives": [
    {
        "drive_id": "data",
        "path_on_host": "/dev/nvme1n1",
        "is_root_device": false,
        "is_read_only": false
    }
]

Add the block device driver and the file system to the guest image from Prepare a guest image. In sandbox-init, add the following lines before the line that starts your workload. The guest sees the first drive as /dev/vda.

modprobe -a virtio_blk ext4
mkdir -p /data
mount /dev/vda /data

Build the initial RAM disk again, with the additional drivers and programs.

sudo dracut --force --no-hostonly --reproducible --no-early-microcode \
    --modules bash \
    --add-drivers "virtio_pci virtio_net vmw_vsock_virtio_transport nsm virtio_blk ext4" \
    --install "mount modprobe ip sleep mkdir" \
    --include "$PWD/sandbox-init" /init \
    --kver $(uname -r) initrd.img
sudo chown "$(id -un):" initrd.img