Encrypting Amazon EMR on EKS logs with managed storage
The sections that follow show you how to configure encryption for logs.
Enable encryption
To encrypt logs in managed storage with your own KMS key, use the following configuration when you submit a job run.
"monitoringConfiguration": { "managedLogs": { "allowAWSToRetainLogs":"ENABLED", "encryptionKeyArn":"
KMS key arn
" }, "persistentAppUI": "ENABLED" }
The allowAWSToRetainLogs
configuration allows AWS to retain system namespace logs when running a job using Native FGAC. The persistentAppUI
configuration
allows AWS to save event logs which are used to generate the Spark UI. The encryptionKeyArn
is used to specify the KMS key ARN you want to use to encrypt the logs stored by AWS.
Required permissions for log encryption
The user who submits the job or views the Spark UI must be allowed the actions kms:DescribeKey
, kms:GenerateDataKey
, and kms:Decrypt
for the
encryption key. These permissions are used to verify the validity of the key and check that the user has the necessary permissions to read and write
logs encrypted with the KMS key. If the user who submits the job lacks the necessary key permissions, Amazon EMR on EKS rejects the job run submission.
Example IAM Policy for Role Used to Call StartJobRun
You must also configure the KMS key to allow the persistentappui.elasticmapreduce.amazonaws.com
and elasticmapreduce.amazonaws.com
Service Principals to kms:GenerateDataKey
and kms:Decrypt
. This allows EMR to read and write logs encrypted with the KMS key to managed storage.
Example KMS Key Policy
As a security best practice, we recommend that you add the kms:viaService
, kms:EncryptionContext
, and aws:SourceArn
conditions. These conditions
help ensure the key is only used by Amazon EMR on EKS and only used for logs generated from jobs running in a specific virtual cluster.