Attest a Nitro Sandbox

Install a C linker and the Rust toolchain, with the target for static binaries. For other ways to install Rust, see Install Rust on the Rust website.

sudo dnf install -y gcc
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
. "$HOME/.cargo/env"
rustup target add $(uname -m)-unknown-linux-musl

Create a directory for the program, and save the following content as nsm-attest/Cargo.toml.

mkdir -p nsm-attest/src

[package]
name = "nsm-attest"
version = "0.1.0"
edition = "2021"

[dependencies]
aws-nitro-enclaves-nsm-api = "0.4"
serde_bytes = "0.11"

Save the following content as nsm-attest/src/main.rs.

// Requests an attestation document from the NSM and writes it to a file.
// Usage: nsm-attest attest --public-key key.der --output attestation.bin
use aws_nitro_enclaves_nsm_api::api::{Request, Response};
use aws_nitro_enclaves_nsm_api::driver::{nsm_exit, nsm_init, nsm_process_request};
use serde_bytes::ByteBuf;
use std::{env, fs, process::exit};

fn usage() -> ! {
    eprintln!("usage: nsm-attest attest --public-key FILE --output FILE");
    exit(2);
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.get(1).map(String::as_str) != Some("attest") {
        usage();
    }
    let (mut public_key, mut output) = (None, None);
    let mut i = 2;
    while i + 1 < args.len() {
        match args[i].as_str() {
            "--public-key" => public_key = Some(fs::read(&args[i + 1]).expect("read public key")),
            "--output" => output = Some(args[i + 1].clone()),
            _ => usage(),
        }
        i += 2;
    }
    let (Some(public_key), Some(output)) = (public_key, output) else {
        usage()
    };

    let fd = nsm_init();
    if fd < 0 {
        eprintln!("cannot open /dev/nsm");
        exit(1);
    }
    let request = Request::Attestation {
        user_data: None,
        nonce: None,
        public_key: Some(ByteBuf::from(public_key)),
    };
    let response = nsm_process_request(fd, request);
    nsm_exit(fd);
    match response {
        Response::Attestation { document } => fs::write(&output, document).expect("write document"),
        other => {
            eprintln!("NSM error: {:?}", other);
            exit(1);
        }
    }
}

Build the program as a static binary, so that it runs from the initial RAM disk, and install it in /usr/local/bin.

cd nsm-attest
cargo build --release --target $(uname -m)-unknown-linux-musl
sudo install -m 755 target/$(uname -m)-unknown-linux-musl/release/nsm-attest /usr/local/bin/
cd ..

Install socat. The guest image and the parent instance use it to exchange files over vsock.

sudo dnf install -y socat

In sandbox-init, add the following lines before the line that starts your workload.

# Request an attestation document that carries a new public key.
cd /tmp
openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -pubout -outform DER -out key.der
nsm-attest attest --public-key key.der --output attestation.bin

# Send the document to the parent instance over vsock port 5005, and
# receive the KMS response on port 5006.
until socat -u OPEN:attestation.bin VSOCK-CONNECT:2:5005; do sleep 1; done
until socat -u VSOCK-CONNECT:2:5006 CREATE:response.cms; do sleep 1; done
openssl cms -decrypt -inform DER -in response.cms -inkey key.pem -out secret.bin

The script waits until the parent instance completes the exchange in Use attestation with AWS KMS. Your workload can then read the secret from /tmp/secret.bin.

Build the initial RAM disk again, with the additional programs.

sudo dracut --force --no-hostonly --reproducible --no-early-microcode \
    --modules bash \
    --add-drivers "virtio_pci virtio_net vmw_vsock_virtio_transport nsm" \
    --install "mount modprobe ip sleep openssl socat /usr/local/bin/nsm-attest /etc/pki/tls/openssl.cnf" \
    --include "$PWD/sandbox-init" /init \
    --kver $(uname -r) initrd.img
sudo chown "$(id -un):" initrd.img

Install eif_build from the aws-nitro-enclaves-image-format project. The command uses the Rust toolchain from Add attestation to a guest image and installs eif_build in ~/.cargo/bin.

sudo dnf install -y openssl-devel
cargo install --git https://github.com/aws/aws-nitro-enclaves-image-format eif_build

Build the image from the files of Add attestation to a guest image, with the default command line of Firecracker.

eif_build --kernel vmlinuz \
    --cmdline "panic=1 initcall_blacklist=i8042_init" \
    --ramdisk initrd.img \
    --pcie-virtio \
    --output sandbox.eif

The --pcie-virtio option builds the image for a Nitro Sandbox in confidential mode without CPU overcommit or hot plug.

Copy the PCR0, PCR1, and PCR2 values from the output. PCR0 depends on the files, the command line, and the settings. PCR1 and PCR2 depend only on the files and the command line.

The following is example output.

Output file: sandbox.eif
{
  "HashAlgorithm": "Sha512_48 { .. }",
  "PCR0": "294af39e266a15be97de5d99b210f81fb9147044d9eb456ad808eeb8b2c2498cfebd4cf57e669228a2a8cde356722f25",
  "PCR1": "61f2b585b0994827830d1dbc2f0011cc30cdc9fd65e1e24eb99666442020676a1b9adf63008df638ca0a228e342d953c",
  "PCR2": "21b9efbc184807662e966d34f390821309eeac6802309798826296bf3e8bec7c10edb30948c90ba67310f7b964fc500a",
  "PCR6": "fb9718a5397717ed57cb42c2a6c4dea98e22f4bc21df1808ad8dde4ed8c1c450c123fdab9f2fa3c065e005357a768be6"
}

In sandbox.json from Launch a Nitro Sandbox, set kernel_image_path to the image file, remove initrd_path, and remove cpu_overcommit from the enclave object. On instances with two threads per core, use an even number of vCPUs.

"boot-source": {
    "kernel_image_path": "sandbox.eif"
}

Save the key policy of your AWS KMS key as policy.json, and add the following statement to its Statement array.

aws kms get-key-policy \
    --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
    --policy-name default \
    --query Policy \
    --output text > policy.json

{
    "Sid": "AllowDecryptFromAttestedNitroSandbox",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/parent-instance-role"
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEqualsIgnoreCase": {
            "kms:RecipientAttestation:PCR0": "PCR0_value",
            "kms:RecipientAttestation:PCR3": "PCR3_value"
        }
    }
}

The statement allows the role of the parent instance to decrypt and to create data keys only for a Nitro Sandbox that runs the expected image on a parent instance with the expected IAM role. Replace the PCR values with the values from Compute the expected PCR values. Use the kms:RecipientAttestation:PCRN condition keys with the StringEqualsIgnoreCase operator. The kms:RecipientAttestation:ImageSha384 condition key compares PCR0. In a key policy, "Resource": "*" refers to the AWS KMS key that the policy is attached to, not to all keys.

Always include PCR0, PCR1, or kms:RecipientAttestation:ImageSha384 in the condition. PCR3 and PCR4 identify only the parent instance, which also chooses the image that it launches, and they keep their values in debug mode. Make sure that no other policy allows the principal to use the key without the attestation condition.

Apply the key policy with the put-key-policy command. The command replaces the whole policy, so keep the statements that give your key administrators access.

aws kms put-key-policy \
    --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
    --policy-name default \
    --policy file://policy.json

Encrypt the secret with the AWS KMS key. Use a principal that has kms:Encrypt permission, for example on your workstation, and use the encrypt command. Then copy secret.enc to the parent instance.

aws kms encrypt \
    --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
    --plaintext fileb://secret.txt \
    --query CiphertextBlob \
    --output text | base64 --decode > secret.enc

Launch the image that you built in Compute the expected PCR values, with the sandbox.json that you changed there. If the Nitro Sandbox from Launch a Nitro Sandbox still runs, stop it first, so that it releases the TAP device, and remove its socket file. Run the command in the directory that contains sandbox.eif.

sudo rm -f /run/sandbox/v.sock
sudo firecracker --no-api --config-file sandbox.json

Run the commands in the following steps in another terminal on the parent instance.

Receive the attestation document. When the Nitro Sandbox connects to port N of CID 2, Firecracker forwards the connection to the Unix socket uds_path_N on the parent instance. The following command returns after the Nitro Sandbox has sent the document. The attestation document is not secret.

sudo socat -u UNIX-LISTEN:/run/sandbox/v.sock_5005 CREATE:attestation.bin

Use the decrypt command with the attestation document. The parent instance uses its own credentials.

aws kms decrypt \
    --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
    --ciphertext-blob fileb://secret.enc \
    --recipient KeyEncryptionAlgorithm=RSAES_OAEP_SHA_256,AttestationDocument@=fileb://attestation.bin \
    --query CiphertextForRecipient \
    --output text | base64 --decode > response.cms

To create a data key instead, use the generate-data-key command. The following example keeps both parts of the response. Store data-key.enc, so that you can decrypt the data key again later with the decrypt command.

aws kms generate-data-key \
    --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
    --key-spec AES_256 \
    --recipient KeyEncryptionAlgorithm=RSAES_OAEP_SHA_256,AttestationDocument@=fileb://attestation.bin \
    --query '[CiphertextBlob,CiphertextForRecipient]' \
    --output text > data-key.txt
cut -f1 data-key.txt | base64 --decode > data-key.enc
cut -f2 data-key.txt | base64 --decode > response.cms

Send the response to the Nitro Sandbox on port 5006. The init script decrypts it with its private key into secret.bin, and then starts your workload. The parent instance can't decrypt the response, because AWS KMS encrypts it to the public key in the attestation document.

sudo socat -u OPEN:response.cms UNIX-LISTEN:/run/sandbox/v.sock_5006