# Security in AWS Glue Security Cloud security at AWS is the highest priority. As an AWS customer, you benefit from a data center and network architecture that is built to meet the requirements of the most security-sensitive organizations. Security is a shared responsibility between AWS and you. The [shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) describes this as security *of* the cloud and security *in* the cloud: + **Security of the cloud** – AWS is responsible for protecting the infrastructure that runs AWS services in the AWS Cloud. AWS also provides you with services that you can use securely. Third-party auditors regularly test and verify the effectiveness of our security as part of the [AWS compliance programs](https://aws.amazon.com/compliance/programs/). To learn about the compliance programs that apply to AWS Glue, see [AWS Services in Scope by Compliance Program](https://aws.amazon.com/compliance/services-in-scope/). + **Security in the cloud** – Your responsibility is determined by the AWS service that you use. You are also responsible for other factors including the sensitivity of your data, your company’s requirements, and applicable laws and regulations. This documentation helps you understand how to apply the shared responsibility model when using AWS Glue. The following topics show you how to configure AWS Glue to meet your security and compliance objectives. You also learn how to use other AWS services that help you to monitor and secure your AWS Glue resources. **Topics** + [ # Data protection in AWS Glue ](data-protection.md) + [ # Identity and access management for AWS Glue ](security-iam.md) + [ # AWS Lake Formation access control models ](lake-formation-access-control-models.md) + [ # Using Amazon S3 Access Grants with AWS Glue ](security-s3-access-grants.md) + [ # Trusted Identity Propagation with AWS Glue ETL ](security-trusted-identity-propagation.md) + [ # Logging and monitoring in AWS Glue ](logging-and-monitoring.md) + [ # Compliance validation for AWS Glue ](compliance.md) + [ # Resilience in AWS Glue ](disaster-recovery-resiliency.md) + [ # Infrastructure security in AWS Glue ](infrastructure-security.md) # Data protection in AWS Glue Data protection AWS Glue offers several features that are designed to help protect your data. **Topics** + [ # Encrypting data at rest ](encryption-at-rest.md) + [ # Encrypting data in transit ](encryption-in-transit.md) + [ # FIPS compliance ](fips-compliance.md) + [ # Key management ](key-management.md) + [ # AWS Glue dependency on other AWS services ](dependency-on-other-services.md) + [ # Development endpoints ](dev-endpoints.md) # Encrypting data at rest Encrypting data at rest AWS Glue supports data encryption at rest for [Building visual ETL jobs](author-job-glue.md) and [Developing scripts using development endpoints](dev-endpoint.md). You can configure extract, transform, and load (ETL) jobs and development endpoints to use [AWS Key Management Service (AWS KMS)](https://aws.amazon.com/kms/) keys to write encrypted data at rest. You can also encrypt the metadata stored in the [AWS Glue Data Catalog](components-overview.md#data-catalog-intro) using keys that you manage with AWS KMS. Additionally, you can use AWS KMS keys to encrypt job bookmarks and the logs generated by [crawlers](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html) and ETL jobs. You can encrypt metadata objects in your AWS Glue Data Catalog in addition to the data written to Amazon Simple Storage Service (Amazon S3) and Amazon CloudWatch Logs by jobs, crawlers, and development endpoints. When you create jobs, crawlers, and development endpoints in AWS Glue, you can provide encryption settings by attaching a security configuration. Security configurations contain Amazon S3-managed server-side encryption keys (SSE-S3) or customer master keys (CMKs) stored in AWS KMS (SSE-KMS). You can create security configurations using the AWS Glue console. You can also turn on encryption of the entire Data Catalog in your account. You do so by specifying CMKs stored in AWS KMS. **Important** AWS Glue supports only symmetric customer managed keys. For more information, see [Customer Managed Keys (CMKs)](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys) in the *AWS Key Management Service Developer Guide*. With encryption turned on, when you add Data Catalog objects, run crawlers, run jobs, or start development endpoints, SSE-S3 or SSE-KMS keys are used to write data at rest. In addition, you can configure AWS Glue to only access Java Database Connectivity (JDBC) data stores through a trusted Transport Layer Security (TLS) protocol. In AWS Glue, you control encryption settings in the following places: + The settings of your Data Catalog. + The security configurations that you create. + The server-side encryption setting (SSE-S3 or SSE-KMS) that is passed as a parameter to your AWS Glue ETL (extract, transform, and load) job. For more information about how to set up encryption, see [Setting up encryption in AWS Glue](set-up-encryption.md). **Topics** + [ # Encrypting your Data Catalog ](encrypt-glue-data-catalog.md) + [ # Encrypting connection passwords ](encrypt-connection-passwords.md) + [ # Encrypting data written by AWS Glue ](encryption-security-configuration.md) # Encrypting your Data Catalog Encrypting your Data Catalog AWS Glue Data Catalog encryption provides enhanced security for your sensitive data. AWS Glue integrates with AWS Key Management Service (AWS KMS) to encrypt metadata that's stored in the Data Catalog. You can enable or disable encryption settings for resources in the Data Catalog using the AWS Glue console or the AWS CLI. When you enable encryption for your Data Catalog, all new objects that you create will be encrypted. When you disable encryption, the new objects you create will not be encrypted, but existing encrypted objects will remain encrypted. You can encrypt your entire Data Catalog using AWS managed encryption keys or customer managed encryption keys. For more information on key types and states, see [AWS Key Management Service concepts](https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html#key-state-cmk-type) in the AWS Key Management Service Developer Guide. **Note** When you use the encrypted Data Catalog with a crawler, you must maintain the encryption settings. Removing encryption settings after a crawler has processed an encrypted catalog results in errors. If you need to remove encryption settings, create a new crawler instead of modifying the existing one. ## AWS managed keys AWS managed keys are KMS keys in your account that are created, managed, and used on your behalf by an AWS service that's integrated with AWS KMS. You can view the AWS managed keys in your account, view their key policies, and audit their use in AWS CloudTrail logs. However, you can't manage these keys or change their permissions. Encryption at rest automatically integrates with AWS KMS for managing the AWS managed keys for AWS Glue that are used to encrypt your metadata. If an AWS managed key doesn't exist when you enable metadata encryption, AWS KMS automatically creates a new key for you. For more information, see [AWS managed keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk). ## Customer managed keys Customer managed keys are KMS keys in your AWS account that you create, own, and manage. You have full control over these KMS keys. You can: + Establish and maintain their key policies, IAM policies, and grants + Enable and disable them + Rotate their cryptographic material + Add tags + Create aliases that refer to them + Schedule them for deletion For more information about managing the permissions of a customer managed key, see [Customer managed keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk). **Important** AWS Glue supports only symmetric customer managed keys. The KMS key list displays only symmetric keys. However, if you select **Choose a KMS key ARN**, the console lets you enter an ARN for any key type. Ensure that you enter only ARNs for symmetric keys. To create a symmetric customer managed key, follow the steps for [creating symmetric customer managed keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html#create-symmetric-cmk) in the AWS Key Management Service Developer Guide. When you enable Data Catalog encryption at rest, the following resource types are encrypted using KMS keys: + Databases + Tables + Partitions + Table versions + Column statistics + User-defined functions + Data Catalog views ## AWS Glue encryption context An [encryption context](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) is an optional set of key-value pairs that contain additional contextual information about the data. AWS KMS uses the encryption context as [additional authenticated data](https://docs.aws.amazon.com/crypto/latest/userguide/cryptography-concepts.html#term-aad) to support [authenticated encryption](https://docs.aws.amazon.com/crypto/latest/userguide/cryptography-concepts.html#define-authenticated-encryption). When you include an encryption context in a request to encrypt data, AWS KMS binds the encryption context to the encrypted data. To decrypt data, you include the same encryption context in the request. AWS Glue uses the same encryption context in all AWS KMS cryptographic operations, where the key is `glue_catalog_id` and the value is the `catalogId`. ``` "encryptionContext": { "glue_catalog_id": "111122223333" } ``` When you use an AWS managed key or a symmetric customer managed key to encrypt your Data Catalog, you can also use the encryption context in audit records and logs to identify how the key is being used. The encryption context also appears in logs that are generated by AWS CloudTrail or Amazon CloudWatch logs. ## Enabling encryption You can enable encryption for your AWS Glue Data Catalog objects in the **Data Catalog settings** in the AWS Glue console or by using the AWS CLI. ------ #### [ Console ] **To enable encryption using the console** 1. Sign in to the AWS Management Console and open the AWS Glue console at [https://console.aws.amazon.com/glue/](https://console.aws.amazon.com/glue/). 1. Choose **Data Catalog** in the navigation pane. 1. On the **Data Catalog settings** page, select the **Metadata encryption** check box, and choose an AWS KMS key. When you enable encryption, if you don’t specify a customer managed key, the encryption settings use an AWS managed KMS key. 1. (Optional) When you use a customer managed key to encrypt your Data Catalog, the Data Catalog provides an option to register an IAM role to encrypt and decrypt resources. You need to grant your IAM role permissions that AWS Glue can assume on your behalf. This includes AWS KMS permissions to encrypt and decrypt data. When you create a new resource in the Data Catalog, AWS Glue assumes the IAM role that's provided to encrypt the data. Similarly, when a consumer accesses the resource, AWS Glue assumes the IAM role to decrypt data. If you register an IAM role with the required permissions, the calling principal no longer requires permissions to access the key and decrypt the data. **Important** You can delegate KMS operations to an IAM role only when you use a customer managed key to encrypt the Data Catalog resources. KMS role delegation feature doesn't support using AWS managed keys for encrypting Data Catalog resources at this time. **Warning** When you enable an IAM role to delegate KMS operations, you can no longer access the Data Catalog resources that were encrypted previously with an AWS managed key. 1. To enable an IAM role that AWS Glue can assume to encrypt and decrypt data on your behalf, select the **Delegate KMS operations to an IAM role** option. 1. Next, choose an IAM role. To create an IAM role, see [Create an IAM role for AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/create-an-iam-role.html). The IAM role that AWS Glue assumes to access the Data Catalog must have the permissions to encrypt and decrypt metadata in the Data Catalog. You can create an IAM role, and attach the following inline policies: + Add the following policy to include AWS KMS permissions to encrypt and decrypt the Data Catalog. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey" ], "Resource": "arn:aws:kms:us-east-1:111122223333:key/" } ] } ``` ------ + Next, add the following trust policy to the role for AWS Glue service to assume the IAM role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ------ + Next, add the `iam:PassRole` permission to the IAM role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::111122223333:role/" ] } ] } ``` ------ When you enable encryption, if you haven't specified an IAM role for AWS Glue to assume, the principal accessing the Data Catalog must have permissions to perform the following API operations: + `kms:Decrypt` + `kms:Encrypt` + `kms:GenerateDataKey` ------ #### [ AWS CLI ] **To enable encryption using the SDK or AWS CLI** + Use the `PutDataCatalogEncryptionSettings` API operation. If no key is specified, AWS Glue uses AWS managed encryption key for the customer account to encrypt the Data Catalog. ``` aws glue put-data-catalog-encryption-settings \ --data-catalog-encryption-settings '{ "EncryptionAtRest": { "CatalogEncryptionMode": "SSE-KMS-WITH-SERVICE-ROLE", "SseAwsKmsKeyId": "arn:aws:kms:::key/", "CatalogEncryptionServiceRole":"arn:aws:iam:::role/" } }' ``` ------ When you enable encryption, all objects that you create in the Data Catalog objects are encrypted. If you clear this setting, the objects you create in the Data Catalog are no longer encrypted. You can continue to access the existing encrypted objects in the Data Catalog with the required KMS permissions. | | | --- | | The AWS KMS key must remain available in the AWS KMS key store for any objects that are encrypted with it in the Data Catalog. If you remove the key, the objects can no longer be decrypted. You might want this in some scenarios to prevent access to Data Catalog metadata. | ## Monitoring your KMS keys for AWS Glue When you use KMS keys with your Data Catalog resources, you can use AWS CloudTrail or Amazon CloudWatch Logs to track requests that AWS Glue sends to AWS KMS. AWS CloudTrail monitors and records KMS operations that AWS Glue calls to access data that’s encrypted by your KMS keys. The following examples are AWS CloudTrail events for the `Decrypt` and `GenerateDataKey` operations. ------ #### [ Decrypt ] ``` { "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "principalId": "AROAXPHTESTANDEXAMPLE:Sampleuser01", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/Sampleuser01", "accountId": "111122223333", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROAXPHTESTANDEXAMPLE", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "111122223333", "userName": "Admin" }, "webIdFederationData": {}, "attributes": { "creationDate": "2024-01-10T14:33:56Z", "mfaAuthenticated": "false" } }, "invokedBy": "glue.amazonaws.com" }, "eventTime": "2024-01-10T15:18:11Z", "eventSource": "kms.amazonaws.com", "eventName": "Decrypt", "awsRegion": "eu-west-2", "sourceIPAddress": "glue.amazonaws.com", "userAgent": "glue.amazonaws.com", "requestParameters": { "encryptionContext": { "glue_catalog_id": "111122223333" }, "encryptionAlgorithm": "SYMMETRIC_DEFAULT" }, "responseElements": null, "requestID": "43b019aa-34b8-4798-9b98-ee968b2d63df", "eventID": "d7614763-d3fe-4f84-a1e1-3ca4d2a5bbd5", "readOnly": true, "resources": [ { "accountId": "111122223333", "type": "AWS::KMS::Key", "ARN": "arn:aws:kms::111122223333:key/" } ], "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "111122223333", "eventCategory": "Management", "sessionCredentialFromConsole": "true" } ``` ------ #### [ GenerateDataKey ] ``` { "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "principalId": "AROAXPHTESTANDEXAMPLE:V_00_GLUE_KMS_GENERATE_DATA_KEY_111122223333", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/V_00_GLUE_KMS_GENERATE_DATA_KEY_111122223333", "accountId": "111122223333", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROAXPHTESTANDEXAMPLE", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "AKIAIOSFODNN7EXAMPLE", "userName": "Admin" }, "webIdFederationData": {}, "attributes": { "creationDate": "2024-01-05T21:15:47Z", "mfaAuthenticated": "false" } }, "invokedBy": "glue.amazonaws.com" }, "eventTime": "2024-01-05T21:15:47Z", "eventSource": "kms.amazonaws.com", "eventName": "GenerateDataKey", "awsRegion": "eu-west-2", "sourceIPAddress": "glue.amazonaws.com", "userAgent": "glue.amazonaws.com", "requestParameters": { "keyId": "arn:aws:kms:eu-west-2:AKIAIOSFODNN7EXAMPLE:key/AKIAIOSFODNN7EXAMPLE", "encryptionContext": { "glue_catalog_id": "111122223333" }, "keySpec": "AES_256" }, "responseElements": null, "requestID": "64d1783a-4b62-44ba-b0ab-388b50188070", "eventID": "1c73689b-2ef2-443b-aed7-8c126585ca5e", "readOnly": true, "resources": [ { "accountId": "111122223333", "type": "AWS::KMS::Key", "ARN": "arn:aws:kms:eu-west-2:111122223333:key/AKIAIOSFODNN7EXAMPLE" } ], "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "111122223333", "eventCategory": "Management" } ``` ------ ``` ``` # Encrypting connection passwords Encrypting connection passwords You can retrieve connection passwords in the AWS Glue Data Catalog by using the `GetConnection` and `GetConnections` API operations. These passwords are stored in the Data Catalog connection and are used when AWS Glue connects to a Java Database Connectivity (JDBC) data store. When the connection was created or updated, an option in the Data Catalog settings determined whether the password was encrypted, and if so, what AWS Key Management Service (AWS KMS) key was specified. On the AWS Glue console, you can turn on this option on the **Data catalog settings** page. **To encrypt connection passwords** 1. Sign in to the AWS Management Console and open the AWS Glue console at [https://console.aws.amazon.com/glue/](https://console.aws.amazon.com/glue/). 1. Choose **Settings** in the navigation pane. 1. On the **Data catalog settings** page, select **Encrypt connection passwords**, and choose an AWS KMS key. **Important** AWS Glue supports only symmetric customer master keys (CMKs). The **AWS KMS key** list displays only symmetric keys. However, if you select **Choose a AWS KMS key ARN**, the console lets you enter an ARN for any key type. Ensure that you enter only ARNs for symmetric keys. For more information, see [Data Catalog settings](console-data-catalog-settings.md). # Encrypting data written by AWS Glue Encrypting data written by AWS Glue A *security configuration* is a set of security properties that can be used by AWS Glue. You can use a security configuration to encrypt data at rest. The following scenarios show some of the ways that you can use a security configuration. + Attach a security configuration to an AWS Glue crawler to write encrypted Amazon CloudWatch Logs. For more information about attaching security configurations to crawlers, see [Step 3: Configure security settings](define-crawler-configure-security-settings.md). + Attach a security configuration to an extract, transform, and load (ETL) job to write encrypted Amazon Simple Storage Service (Amazon S3) targets and encrypted CloudWatch Logs. + Attach a security configuration to an ETL job to write its jobs bookmarks as encrypted Amazon S3 data. + Attach a security configuration to a development endpoint to write encrypted Amazon S3 targets. **Important** Currently, a security configuration overrides any server-side encryption (SSE-S3) setting that is passed as an ETL job parameter. Thus, if both a security configuration and an SSE-S3 parameter are associated with a job, the SSE-S3 parameter is ignored. For more information about security configurations, see [Managing security configurations on the AWS Glue console](console-security-configurations.md). **Topics** + [ ## Setting Up AWS Glue to use security configurations ](#encryption-setup-Glue) + [ ## Creating a route to AWS KMS for VPC jobs and crawlers ](#encryption-kms-vpc-endpoint) + [ # Managing security configurations on the AWS Glue console ](console-security-configurations.md) ## Setting Up AWS Glue to use security configurations Follow these steps to set up your AWS Glue environment to use security configurations. 1. Create or update your AWS Key Management Service (AWS KMS) keys to grant AWS KMS permissions to the IAM roles that are passed to AWS Glue crawlers and jobs to encrypt CloudWatch Logs. For more information, see [Encrypt Log Data in CloudWatch Logs Using AWS KMS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html) in the *Amazon CloudWatch Logs User Guide*. In the following example, *"role1"*, *"role2"*, and *"role3"* are IAM roles that are passed to crawlers and jobs. ``` { "Effect": "Allow", "Principal": { "Service": "logs.region.amazonaws.com", "AWS": [ "role1", "role2", "role3" ] }, "Action": [ "kms:Encrypt*", "kms:Decrypt*", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:Describe*" ], "Resource": "*" } ``` The `Service` statement, shown as `"Service": "logs.region.amazonaws.com"`, is required if you use the key to encrypt CloudWatch Logs. 1. Ensure that the AWS KMS key is `ENABLED` before it is used. **Note** If you are using Iceberg as your data lake framework, Iceberg tables have their own mechanisms to enable server-side encryption. You should enable these configuration in addition to AWS Glue's security configurations. To enable server-side encryption on Iceberg tables, review the guidance from [Iceberg documentation](https://iceberg.apache.org/docs/latest/aws/#s3-server-side-encryption). ## Creating a route to AWS KMS for VPC jobs and crawlers Creating a Route to AWS KMS You can connect directly to AWS KMS through a private endpoint in your virtual private cloud (VPC) instead of connecting over the internet. When you use a VPC endpoint, communication between your VPC and AWS KMS is conducted entirely within the AWS network. You can create an AWS KMS VPC endpoint within a VPC. Without this step, your jobs or crawlers might fail with a `kms timeout` on jobs or an `internal service exception` on crawlers. For detailed instructions, see [Connecting to AWS KMS Through a VPC Endpoint](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html) in the *AWS Key Management Service Developer Guide*. As you follow these instructions, on the [VPC console](https://console.aws.amazon.com//vpc), you must do the following: + Select **Enable Private DNS name**. + Choose the **Security group** (with self-referencing rule) that you use for your job or crawler that accesses Java Database Connectivity (JDBC). For more information about AWS Glue connections, see [Connecting to data](glue-connections.md). When you add a security configuration to a crawler or job that accesses JDBC data stores, AWS Glue must have a route to the AWS KMS endpoint. You can provide the route with a network address translation (NAT) gateway or with an AWS KMS VPC endpoint. To create a NAT gateway, see [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) in the *Amazon VPC User Guide*. # Managing security configurations on the AWS Glue console Managing security configurations on the AWS Glue console **Warning** AWS Glue security configurations are not currently supported in Ray jobs. A *security configuration* in AWS Glue contains the properties that are needed when you write encrypted data. You create security configurations on the AWS Glue console to provide the encryption properties that are used by crawlers, jobs, and development endpoints. To see a list of all the security configurations that you have created, open the AWS Glue console at [https://console.aws.amazon.com/glue/](https://console.aws.amazon.com/glue/) and choose **Security configurations** in the navigation pane. The **Security configurations** list displays the following properties about each configuration: **Name** The unique name you provided when you created the configuration. The name may contain letters (A-Z), numbers (0-9), hypens (-), or underscores (\$1), and be up to 255 characters long. **Enable Amazon S3 encryption** If turned on, the Amazon Simple Storage Service (Amazon S3) encryption mode such as `SSE-KMS` or `SSE-S3` is enabled for metadata store in the data catalog. **Enable Amazon CloudWatch logs encryption** If turned on, the Amazon S3 encryption mode such as `SSE-KMS` is enabled when writing logs to Amazon CloudWatch. **Advanced settings: Enable job bookmark encryption** If turned on, the Amazon S3 encryption mode such as `CSE-KMS` is enabled when jobs are bookmarked. You can add or delete configurations in the **Security configurations** section on the console. To see more details for a configuration, choose the configuration name in the list. Details include the information that you defined when you created the configuration. ## Adding a security configuration Adding a Security Configuration To add a security configuration using the AWS Glue console, on the **Security configurations** page, choose **Add security configuration**. ![\[The screenshot shows the Add security configuration page.\]](http://docs.aws.amazon.com/glue/latest/dg/images/add_security_configuration.png) **Security configuration properties** Enter a unique security configuration name. The name may contain letters (A-Z), numbers (0-9), hyphens (-), or underscores (\$1), and can be up to 255 characters long. **Encryption settings** You can enable at-rest encryption for metadata stored in the Data Catalog in Amazon S3 and logs in Amazon CloudWatch. To set up encryption of data and metadata with AWS Key Management Service (AWS KMS) keys on the AWS Glue console, add a policy to the console user. This policy must specify the allowed resources as key Amazon Resource Names (ARNs) that are used to encrypt Amazon S3 data stores, as in the following example. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt", "kms:Encrypt" ], "Resource": "arn:aws:kms:us-east-1:111122223333:key/key-id" } } ``` ------ **Important** When a security configuration is attached to a crawler or job, the IAM role that is passed must have AWS KMS permissions. For more information, see [Encrypting data written by AWS Glue](encryption-security-configuration.md). When you define a configuration, you can provide values for the following properties: **Enable S3 encryption** When you are writing Amazon S3 data, you use either server-side encryption with Amazon S3 managed keys (SSE-S3) or server-side encryption with AWS KMS managed keys (SSE-KMS). This field is optional. To allow access to Amazon S3, choose an AWS KMS key, or choose **Enter a key ARN** and provide the ARN for the key. Enter the ARN in the form `arn:aws:kms:region:account-id:key/key-id`. You can also provide the ARN as a key alias, such as `arn:aws:kms:region:account-id:alias/alias-name`. If you enable Spark UI for your job, the Spark UI log file uploaded to Amazon S3 will be applied with the same encryption. AWS Glue supports only symmetric customer master keys (CMKs). The **AWS KMS key** list displays only symmetric keys. However, if you select **Choose a AWS KMS key ARN**, the console lets you enter an ARN for any key type. Ensure that you enter only ARNs for symmetric keys. **Enable CloudWatch Logs encryption** Server-side (SSE-KMS) encryption is used to encrypt CloudWatch Logs. This field is optional. To turn it on, choose an AWS KMS key, or choose **Enter a key ARN** and provide the ARN for the key. Enter the ARN in the form `arn:aws:kms:region:account-id:key/key-id`. You can also provide the ARN as a key alias, such as `arn:aws:kms:region:account-id:alias/alias-name`. **Advanced settings: Job bookmark encryption** Client-side (CSE-KMS) encryption is used to encrypt job bookmarks. This field is optional. The bookmark data is encrypted before it is sent to Amazon S3 for storage. To turn it on, choose an AWS KMS key, or choose **Enter a key ARN** and provide the ARN for the key. Enter the ARN in the form `arn:aws:kms:region:account-id:key/key-id`. You can also provide the ARN as a key alias, such as `arn:aws:kms:region:account-id:alias/alias-name`. For more information, see the following topics in the *Amazon Simple Storage Service User Guide*: + For information about `SSE-S3`, see [Protecting Data Using Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html). + For information about `SSE-KMS`, see [Protecting Data Using Server-Side Encryption with AWS KMS keys](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). + For information about `CSE-KMS`, see [ Using a KMS key stored in AWS KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncryption.html#client-side-encryption-kms-managed-master-key-intro). # Encrypting data in transit Encrypting data in transit AWS provides Transport Layer Security (TLS) encryption for data in motion. You can configure encryption settings for crawlers, ETL jobs, and development endpoints using [security configurations](https://docs.aws.amazon.com/glue/latest/dg/console-security-configurations.html) in AWS Glue. You can turn on AWS Glue Data Catalog encryption via the settings for the Data Catalog. As of September 4, 2018, AWS KMS (*bring your own key* and *server-side encryption*) for AWS Glue ETL and the AWS Glue Data Catalog is supported. # FIPS compliance If you require FIPS 140-2 validated cryptographic modules when accessing AWS through a command line interface or an API, use a FIPS endpoint. For more information about the available FIPS endpoints, see [Federal Information Processing Standard (FIPS) 140-2](https://aws.amazon.com/compliance/fips/). # Key management Key management You can use AWS Identity and Access Management (IAM) with AWS Glue to define users, AWS resources, groups, roles and fine-grained policies regarding access, denial, and more. You can define the access to the metadata using both resource-based and identity-based policies, depending on your organization’s needs. Resource-based policies list the principals that are allowed or denied access to your resources, allowing you to set up policies such as cross-account access. Identity policies are specifically attached to users, groups, and roles within IAM. For a step-by-step example, see [Restrict access to your AWS Glue Data Catalog with resource-level IAM permissions and resource-based policies](https://aws.amazon.com/blogs/big-data/restrict-access-to-your-aws-glue-data-catalog-with-resource-level-iam-permissions-and-resource-based-policies/) on the AWS Big Data Blog. The fine-grained access portion of the policy is defined within the `Resource` clause. This portion defines both the AWS Glue Data Catalog object that the action can be performed on, and what resulting objects get returned by that operation. A *development endpoint* is an environment that you can use to develop and test your AWS Glue scripts. You can add, delete, or rotate the SSH key of a development endpoint. As of September 4, 2018, AWS KMS (*bring your own key* and *server-side encryption*) for AWS Glue ETL and the AWS Glue Data Catalog is supported. # AWS Glue dependency on other AWS services For a user to work with the AWS Glue console, that user must have a minimum set of permissions that allows them to work with the AWS Glue resources for their AWS account. In addition to these AWS Glue permissions, the console requires permissions from the following services: + Amazon CloudWatch Logs permissions to display logs. + AWS Identity and Access Management (IAM) permissions to list and pass roles. + CloudFormation permissions to work with stacks. + Amazon Elastic Compute Cloud (Amazon EC2) permissions to list virtual private clouds (VPCs), subnets, security groups, instances, and other objects (to set up Amazon EC2 items such as VPCs when running jobs, crawlers, and creating development endpoints). + Amazon Simple Storage Service (Amazon S3) permissions to list buckets and objects, and to retrieve and save scripts. + Amazon Redshift permissions to work with clusters. + Amazon Relational Database Service (Amazon RDS) permissions to list instances. # Development endpoints A development endpoint is an environment that you can use to develop and test your AWS Glue scripts. You can use AWS Glue to create, edit, and delete development endpoints. You can list all the development endpoints that are created. You can add, delete, or rotate the SSH key of a development endpoint. You can also create notebooks that use the development endpoint. You provide configuration values to provision the development environments. These values tell AWS Glue how to set up the network so that you can access the development endpoint securely, and so that your endpoint can access your data stores. Then, you can create a notebook that connects to the development endpoint. You use your notebook to author and test your ETL script. Use an AWS Identity and Access Management (IAM) role with permissions similar to the IAM role that you use to run AWS Glue ETL jobs. Use a virtual private cloud (VPC), a subnet, and a security group to create a development endpoint that can connect to your data resources securely. You generate an SSH key pair to connect to the development environment using SSH. You can create development endpoints for Amazon S3 data and within a VPC that you can use to access datasets using JDBC. You can install a Jupyter notebook client on your local machine and use it to debug and test ETL scripts on a development endpoint. Or, you can use a Sagemaker notebook to author ETL scripts in JupyterLab on AWS. See [ Use a SageMaker notebook with your development endpoint ](https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-sage.html). AWS Glue tags Amazon EC2 instances with a name that is prefixed with `aws-glue-dev-endpoint`. You can set up a notebook server on a development endpoint to run PySpark with AWS Glue extensions. # Identity and access management for AWS Glue Identity and access management AWS Identity and Access Management (IAM) is an AWS service that helps an administrator securely control access to AWS resources. IAM administrators control who can be *authenticated* (signed in) and *authorized* (have permissions) to use AWS Glue resources. IAM is an AWS service that you can use with no additional charge. **Note** You can grant access to your data in the AWS Glue Data Catalog using either AWS Glue methods or AWS Lake Formation grants. You can use AWS Identity and Access Management (IAM) policies to set fine-grained access control with AWS Glue methods. Lake Formation uses a simpler `GRANT/REVOKE` permissions model that is similar to the `GRANT/REVOKE` commands in a relational database system. This section includes information about how to use the AWS Glue methods. For information about using Lake Formation grants, see [Granting Lake Formation permissions](https://docs.aws.amazon.com/lake-formation/latest/dg/lake-formation-permissions.html) in the *AWS Lake Formation Developer Guide*. **Topics** + [ ## Audience ](#security_iam_audience) + [ ## Authenticating with identities ](#security_iam_authentication) + [ ## Managing access using policies ](#security_iam_access-manage) + [ # How AWS Glue works with IAM ](security_iam_service-with-iam.md) + [ # Configuring IAM permissions for AWS Glue ](configure-iam-for-glue.md) + [ # AWS Glue access control policy examples ](glue-policy-examples.md) + [ # Granting AWS managed policies for AWS Glue ](security-iam-awsmanpol.md) + [ # Granting dynamically scoped policies for job execution ](dynamically-scoped-policies.md) + [ # Specifying AWS Glue resource ARNs ](glue-specifying-resource-arns.md) + [ # Granting cross-account access ](cross-account-access.md) + [ # Troubleshooting AWS Glue identity and access ](security_iam_troubleshoot.md) ## Audience How you use AWS Identity and Access Management (IAM) differs based on your role: + **Service user** - request permissions from your administrator if you cannot access features (see [Troubleshooting AWS Glue identity and access](security_iam_troubleshoot.md)) + **Service administrator** - determine user access and submit permission requests (see [How AWS Glue works with IAM](security_iam_service-with-iam.md)) + **IAM administrator** - write policies to manage access (see [Identity-based policy examples for AWS Glue](security_iam_id-based-policy-examples.md)) ## Authenticating with identities Authentication is how you sign in to AWS using your identity credentials. You must be authenticated as the AWS account root user, an IAM user, or by assuming an IAM role. You can sign in as a federated identity using credentials from an identity source like AWS IAM Identity Center (IAM Identity Center), single sign-on authentication, or Google/Facebook credentials. For more information about signing in, see [How to sign in to your AWS account](https://docs.aws.amazon.com/signin/latest/userguide/how-to-sign-in.html) in the *AWS Sign-In User Guide*. For programmatic access, AWS provides an SDK and CLI to cryptographically sign requests. For more information, see [AWS Signature Version 4 for API requests](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv.html) in the *IAM User Guide*. ### AWS account root user When you create an AWS account, you begin with one sign-in identity called the AWS account *root user* that has complete access to all AWS services and resources. We strongly recommend that you don't use the root user for everyday tasks. For tasks that require root user credentials, see [Tasks that require root user credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks) in the *IAM User Guide*. ### Federated identity As a best practice, require human users to use federation with an identity provider to access AWS services using temporary credentials. A *federated identity* is a user from your enterprise directory, web identity provider, or Directory Service that accesses AWS services using credentials from an identity source. Federated identities assume roles that provide temporary credentials. For centralized access management, we recommend AWS IAM Identity Center. For more information, see [What is IAM Identity Center?](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) in the *AWS IAM Identity Center User Guide*. ### IAM users and groups An *[IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html)* is an identity with specific permissions for a single person or application. We recommend using temporary credentials instead of IAM users with long-term credentials. For more information, see [Require human users to use federation with an identity provider to access AWS using temporary credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#bp-users-federation-idp) in the *IAM User Guide*. An [https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html) specifies a collection of IAM users and makes permissions easier to manage for large sets of users. For more information, see [Use cases for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/gs-identities-iam-users.html) in the *IAM User Guide*. ### IAM roles An *[IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)* is an identity with specific permissions that provides temporary credentials. You can assume a role by [switching from a user to an IAM role (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-console.html) or by calling an AWS CLI or AWS API operation. For more information, see [Methods to assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_manage-assume.html) in the *IAM User Guide*. IAM roles are useful for federated user access, temporary IAM user permissions, cross-account access, cross-service access, and applications running on Amazon EC2. For more information, see [Cross account resource access in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html) in the *IAM User Guide*. ## Managing access using policies You control access in AWS by creating policies and attaching them to AWS identities or resources. A policy defines permissions when associated with an identity or resource. AWS evaluates these policies when a principal makes a request. Most policies are stored in AWS as JSON documents. For more information about JSON policy documents, see [Overview of JSON policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policies-json) in the *IAM User Guide*. Using policies, administrators specify who has access to what by defining which **principal** can perform **actions** on what **resources**, and under what **conditions**. By default, users and roles have no permissions. An IAM administrator creates IAM policies and adds them to roles, which users can then assume. IAM policies define permissions regardless of the method used to perform the operation. ### Identity-based policies Identity-based policies are JSON permissions policy documents that you attach to an identity (user, group, or role). These policies control what actions identities can perform, on which resources, and under what conditions. To learn how to create an identity-based policy, see [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) in the *IAM User Guide*. Identity-based policies can be *inline policies* (embedded directly into a single identity) or *managed policies* (standalone policies attached to multiple identities). To learn how to choose between managed and inline policies, see [Choose between managed policies and inline policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-choosing-managed-or-inline.html) in the *IAM User Guide*. ### Resource-based policies Resource-based policies are JSON policy documents that you attach to a resource. Examples include IAM *role trust policies* and Amazon S3 *bucket policies*. In services that support resource-based policies, service administrators can use them to control access to a specific resource. You must [specify a principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html) in a resource-based policy. Resource-based policies are inline policies that are located in that service. You can't use AWS managed policies from IAM in a resource-based policy. ### Other policy types AWS supports additional policy types that can set the maximum permissions granted by more common policy types: + **Permissions boundaries** – Set the maximum permissions that an identity-based policy can grant to an IAM entity. For more information, see [Permissions boundaries for IAM entities](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html) in the *IAM User Guide*. + **Service control policies (SCPs)** – Specify the maximum permissions for an organization or organizational unit in AWS Organizations. For more information, see [Service control policies](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) in the *AWS Organizations User Guide*. + **Resource control policies (RCPs)** – Set the maximum available permissions for resources in your accounts. For more information, see [Resource control policies (RCPs)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html) in the *AWS Organizations User Guide*. + **Session policies** – Advanced policies passed as a parameter when creating a temporary session for a role or federated user. For more information, see [Session policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) in the *IAM User Guide*. ### Multiple policy types When multiple types of policies apply to a request, the resulting permissions are more complicated to understand. To learn how AWS determines whether to allow a request when multiple policy types are involved, see [Policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) in the *IAM User Guide*. # How AWS Glue works with IAM Before you use IAM to manage access to AWS Glue, learn what IAM features are available to use with AWS Glue. **IAM features you can use with AWS Glue** | IAM feature | AWS Glue support | | --- | --- | | [Identity-based policies](#security_iam_service-with-iam-id-based-policies) | Yes | | [Resource-based policies](#security_iam_service-with-iam-resource-based-policies) | Partial | | [Policy actions](#security_iam_service-with-iam-id-based-policies-actions) | Yes | | [Policy resources](#security_iam_service-with-iam-id-based-policies-resources) | Yes | | [Policy condition keys (service-specific)](#security_iam_service-with-iam-id-based-policies-conditionkeys) | Yes | | [ACLs](#security_iam_service-with-iam-acls) | No | | [ABAC (tags in policies)](#security_iam_service-with-iam-tags) | Partial | | [Temporary credentials](#security_iam_service-with-iam-roles-tempcreds) | Yes | | [Principal permissions](#security_iam_service-with-iam-principal-permissions) | No | | [Service roles](#security_iam_service-with-iam-roles-service) | Yes | | [Service-linked roles](#security_iam_service-with-iam-roles-service-linked) | No | To get a high-level view of how AWS Glue and other AWS services work with most IAM features, see [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) in the *IAM User Guide*. ## Identity-based policies for AWS Glue Identity-based policies **Supports identity-based policies:** Yes Identity-based policies are JSON permissions policy documents that you can attach to an identity, such as an IAM user, group of users, or role. These policies control what actions users and roles can perform, on which resources, and under what conditions. To learn how to create an identity-based policy, see [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) in the *IAM User Guide*. With IAM identity-based policies, you can specify allowed or denied actions and resources as well as the conditions under which actions are allowed or denied. To learn about all of the elements that you can use in a JSON policy, see [IAM JSON policy elements reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html) in the *IAM User Guide*. AWS Glue supports identity-based policies (IAM policies) for all AWS Glue operations. By attaching a policy, you can grant permissions to create, access, or modify an AWS Glue resource, such as a table in the AWS Glue Data Catalog. ### Identity-based policy examples for AWS Glue To view examples of AWS Glue identity-based policies, see [Identity-based policy examples for AWS Glue](security_iam_id-based-policy-examples.md). ## Resource-based policies within AWS Glue Resource-based policies **Supports resource-based policies:** Partial Resource-based policies are JSON policy documents that you attach to a resource. Examples of resource-based policies are IAM *role trust policies* and Amazon S3 *bucket policies*. In services that support resource-based policies, service administrators can use them to control access to a specific resource. For the resource where the policy is attached, the policy defines what actions a specified principal can perform on that resource and under what conditions. You must [specify a principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html) in a resource-based policy. Principals can include accounts, users, roles, federated users, or AWS services. To enable cross-account access, you can specify an entire account or IAM entities in another account as the principal in a resource-based policy. For more information, see [Cross account resource access in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html) in the *IAM User Guide*. **Note** You can only use an AWS Glue resource policy to manage permissions for Data Catalog resources. You can't attach it to any other AWS Glue resources such as jobs, triggers, development endpoints, crawlers, or classifiers. Only *one* resource policy is allowed per catalog, and its size is limited to 10 KB. In AWS Glue, a resource policy is attached to a *catalog*, which is a virtual container for all the kinds of Data Catalog resources mentioned previously. Each AWS account owns a single catalog in an AWS Region whose catalog ID is the same as the AWS account ID. You cannot delete or modify a catalog. A resource policy is evaluated for all API calls to the catalog where the caller principal is included in the `"Principal"` block of the policy document. To view examples of AWS Glue resource-based policies, see [Resource-based policy examples for AWS Glue](security_iam_resource-based-policy-examples.md). ## Policy actions for AWS Glue Policy actions **Supports policy actions:** Yes Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**. The `Action` element of a JSON policy describes the actions that you can use to allow or deny access in a policy. Include actions in a policy to grant permissions to perform the associated operation. To see a list of AWS Glue actions, see [Actions defined by AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-actions-as-permissions) in the *Service Authorization Reference*. Policy actions in AWS Glue use the following prefix before the action: ``` glue ``` To specify multiple actions in a single statement, separate them with commas. ``` "Action": [ "glue:action1", "glue:action2" ] ``` You can specify multiple actions using wildcards (\$1). For example, to specify all actions that begin with the word `Get`, include the following action: ``` "Action": "glue:Get*" ``` To view example policies, see [AWS Glue access control policy examples](glue-policy-examples.md). ## Policy resources for AWS Glue Policy resources **Supports policy resources:** Yes Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**. The `Resource` JSON policy element specifies the object or objects to which the action applies. As a best practice, specify a resource using its [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html). For actions that don't support resource-level permissions, use a wildcard (\$1) to indicate that the statement applies to all resources. ``` "Resource": "*" ``` For more information about how to control access to AWS Glue resources using ARNs, see [Specifying AWS Glue resource ARNs](glue-specifying-resource-arns.md). To see a list of AWS Glue resource types and their ARNs, see [Resources defined by AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-resources-for-iam-policies) in the *Service Authorization Reference*. To learn which actions you can use to specify the ARN of each resource, see [Actions defined by AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-actions-as-permissions). ## Policy condition keys for AWS Glue Policy condition keys **Supports service-specific policy condition keys:** Yes Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**. The `Condition` element specifies when statements execute based on defined criteria. You can create conditional expressions that use [condition operators](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html), such as equals or less than, to match the condition in the policy with values in the request. To see all AWS global condition keys, see [AWS global condition context keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html) in the *IAM User Guide*. To see a list of AWS Glue condition keys, see [Condition keys for AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-policy-keys) in the *Service Authorization Reference*. To learn which actions and resources you can use a condition key with, see [Actions defined by AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-actions-as-permissions). To view example policies, see [Control settings using condition keys or context keys](security_iam_id-based-policy-examples.md#glue-identity-based-policy-condition-keys). ## ACLs in AWS Glue ACLs **Supports ACLs:** No Access control lists (ACLs) control which principals (account members, users, or roles) have permissions to access a resource. ACLs are similar to resource-based policies, although they do not use the JSON policy document format. ## ABAC with AWS Glue ABAC **Supports ABAC (tags in policies):** Partial Attribute-based access control (ABAC) is an authorization strategy that defines permissions based on attributes called tags. You can attach tags to IAM entities and AWS resources, then design ABAC policies to allow operations when the principal's tag matches the tag on the resource. To control access based on tags, you provide tag information in the [condition element](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html) of a policy using the `aws:ResourceTag/key-name`, `aws:RequestTag/key-name`, or `aws:TagKeys` condition keys. If a service supports all three condition keys for every resource type, then the value is **Yes** for the service. If a service supports all three condition keys for only some resource types, then the value is **Partial**. For more information about ABAC, see [Define permissions with ABAC authorization](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_attribute-based-access-control.html) in the *IAM User Guide*. To view a tutorial with steps for setting up ABAC, see [Use attribute-based access control (ABAC)](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) in the *IAM User Guide*. **Important** The condition context keys apply only to AWS Glue API actions on crawlers, jobs, triggers, and development endpoints. For more information about which API operations are affected, see [Condition keys for AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html#awsglue-policy-keys). The AWS Glue Data Catalog API operations don't currently support the `aws:referer` and `aws:UserAgent` global condition context keys. To view an example identity-based policy for limiting access to a resource based on the tags on that resource, see [Grant access using tags](security_iam_id-based-policy-examples.md#tags-control-access-example-triggers-allow). ## Using temporary credentials with AWS Glue Temporary credentials **Supports temporary credentials:** Yes Temporary credentials provide short-term access to AWS resources and are automatically created when you use federation or switch roles. AWS recommends that you dynamically generate temporary credentials instead of using long-term access keys. For more information, see [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) and [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) in the *IAM User Guide*. ## Cross-service principal permissions for AWS Glue Principal permissions **Supports forward access sessions (FAS):** No Forward access sessions (FAS) use the permissions of the principal calling an AWS service, combined with the requesting AWS service to make requests to downstream services. For policy details when making FAS requests, see [Forward access sessions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_forward_access_sessions.html). ## Service roles for AWS Glue Service roles **Supports service roles:** Yes A service role is an [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) that a service assumes to perform actions on your behalf. An IAM administrator can create, modify, and delete a service role from within IAM. For more information, see [Create a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*. **Warning** Changing the permissions for a service role might break AWS Glue functionality. Edit service roles only when AWS Glue provides guidance to do so. For detailed instructions on creating a service role for AWS Glue, see [Step 1: Create an IAM policy for the AWS Glue service](create-service-policy.md) and [Step 2: Create an IAM role for AWS Glue](create-an-iam-role.md). ## Service-linked roles for AWS Glue Service-linked roles **Supports service-linked roles:** No A service-linked role is a type of service role that is linked to an AWS service. The service can assume the role to perform an action on your behalf. Service-linked roles appear in your AWS account and are owned by the service. An IAM administrator can view, but not edit the permissions for service-linked roles. For details about creating or managing service-linked roles, see [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html). Find a service in the table that includes a `Yes` in the **Service-linked role** column. Choose the **Yes** link to view the service-linked role documentation for that service. # Configuring IAM permissions for AWS Glue You use AWS Identity and Access Management (IAM) to define policies and roles that AWS Glue uses to access resources. The following steps lead you through various options for setting up the permissions for AWS Glue. Depending on your business needs, you might have to add or reduce access to your resources. **Note** To get started with basic IAM permissions for AWS Glue instead, see [Setting up IAM permissions for AWS Glue](set-up-iam.md). 1. [Create an IAM policy for the AWS Glue service](create-service-policy.md): Create a service policy that allows access to AWS Glue resources. 1. [Create an IAM role for AWS Glue](create-an-iam-role.md): Create an IAM role, and attach the AWS Glue service policy and a policy for your Amazon Simple Storage Service (Amazon S3) resources that are used by AWS Glue. 1. [Attach a policy to users or groups that access AWS Glue](attach-policy-iam-user.md): Attach policies to any users or groups that sign in to the AWS Glue console. 1. [Create an IAM policy for notebooks](create-notebook-policy.md): Create a notebook server policy to use in the creation of notebook servers on development endpoints. 1. [Create an IAM role for notebooks](create-an-iam-role-notebook.md): Create an IAM role and attach the notebook server policy. 1. [Create an IAM policy for Amazon SageMaker AI notebooks](create-sagemaker-notebook-policy.md): Create an IAM policy to use when creating Amazon SageMaker AI notebooks on development endpoints. 1. [Create an IAM role for Amazon SageMaker AI notebooks](create-an-iam-role-sagemaker-notebook.md): Create an IAM role and attach the policy to grant permissions when creating Amazon SageMaker AI notebooks on development endpoints. # Step 1: Create an IAM policy for the AWS Glue service For any operation that accesses data on another AWS resource, such as accessing your objects in Amazon S3, AWS Glue needs permission to access the resource on your behalf. You provide those permissions by using AWS Identity and Access Management (IAM). **Note** You can skip this step if you use the AWS managed policy `AWSGlueServiceRole`. In this step, you create a policy that is similar to `AWSGlueServiceRole`. You can find the most current version of `AWSGlueServiceRole` on the IAM console. **To create an IAM policy for AWS Glue** This policy grants permission for some Amazon S3 actions to manage resources in your account that are needed by AWS Glue when it assumes the role using this policy. Some of the resources that are specified in this policy refer to default names that are used by AWS Glue for Amazon S3 buckets, Amazon S3 ETL scripts, CloudWatch Logs, and Amazon EC2 resources. For simplicity, AWS Glue writes some Amazon S3 objects into buckets in your account prefixed with `aws-glue-*` by default. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose **Create Policy**. 1. On the **Create Policy** screen, navigate to a tab to edit JSON. Create a policy document with the following JSON statements, and then choose **Review policy**. **Note** Add any permissions needed for Amazon S3 resources. You might want to scope the resources section of your access policy to only those resources that are required. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:*", "s3:GetBucketLocation", "s3:ListBucket", "s3:ListAllMyBuckets", "s3:GetBucketAcl", "ec2:DescribeVpcEndpoints", "ec2:DescribeRouteTables", "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcAttribute", "iam:ListRolePolicies", "iam:GetRole", "iam:GetRolePolicy", "cloudwatch:PutMetricData" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:PutBucketPublicAccessBlock" ], "Resource": [ "arn:aws:s3:::aws-glue-*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::aws-glue-*/*", "arn:aws:s3:::*/*aws-glue-*/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::crawler-public*", "arn:aws:s3:::aws-glue-*" ] }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:AssociateKmsKey" ], "Resource": [ "arn:aws:logs:*:*:log-group:/aws-glue/*" ] }, { "Effect": "Allow", "Action": [ "ec2:CreateTags", "ec2:DeleteTags" ], "Condition": { "ForAllValues:StringEquals": { "aws:TagKeys": [ "aws-glue-service-resource" ] } }, "Resource": [ "arn:aws:ec2:*:*:network-interface/*", "arn:aws:ec2:*:*:security-group/*", "arn:aws:ec2:*:*:instance/*" ] } ] } ``` ------ The following table describes the permissions granted by this policy. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/create-service-policy.html) 1. On the **Review Policy** screen, enter your **Policy Name**, for example **GlueServiceRolePolicy**. Enter an optional description, and when you're satisfied with the policy, choose **Create policy**. # Step 2: Create an IAM role for AWS Glue You need to grant your IAM role permissions that AWS Glue can assume when calling other services on your behalf. This includes access to Amazon S3 for any sources, targets, scripts, and temporary directories that you use with AWS Glue. Permission is needed by crawlers, jobs, and development endpoints. You provide those permissions by using AWS Identity and Access Management (IAM). Add a policy to the IAM role that you pass to AWS Glue. ****To create an IAM role within the job editor**** 1. When you create a job in the AWS Glue console, locate the role section. 1. Choose **Create new role**. 1. An inline role creation form opens, allowing you to: + Specify **Role name**; for example, `AWSGlueServiceRoleDefault`. + The managed policy `AWSGlueServiceRole` is automatically selected. + Review the trust policy to assume the role. + Add optional tags for metadata. 1. Choose **Create role**. 1. The newly created role is automatically selected for your job. Alternatively, you can use the IAM console to create the role: ****To create an IAM role for AWS Glue using the IAM console**** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. Choose **Create role**. 1. Choose **AWS service** as the trusted entity type. Then, for service or use case, find and choose **AWS Glue**. Choose **Next**. 1. On the **Add permissions** page, choose the policies that contain the required permissions; for example, the AWS managed policy `AWSGlueServiceRole` for general AWS Glue permissions and the AWS managed policy **AmazonS3FullAccess** for access to Amazon S3 resources. Then choose **Next**. **Note** Ensure that one of the policies in this role grants permissions to your Amazon S3 sources and targets. You might want to provide your own policy for access to specific Amazon S3 resources. Data sources require `s3:ListBucket` and `s3:GetObject` permissions. Data targets require `s3:ListBucket`, `s3:PutObject`, and `s3:DeleteObject` permissions. For more information about creating an Amazon S3 policy for your resources, see [Specifying Resources in a Policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-arn-format.html). For an example Amazon S3 policy, see [ Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket](https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/). If you plan to access Amazon S3 sources and targets that are encrypted with SSE-KMS, attach a policy that allows AWS Glue crawlers, jobs, and development endpoints to decrypt the data. For more information, see [Protecting Data Using Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). The following is an example. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": [ "arn:aws:kms:*:111122223333:key/key-id" ] } ] } ``` 1. Name your role and add a description (optional), then review the trust policy and permissions. For **Role name**, enter a name for your role; for example, `AWSGlueServiceRoleDefault`. Create the role with the name prefixed with the string `AWSGlueServiceRole` to allow the role to be passed from console users to the service. AWS Glue provided policies expect IAM service roles to begin with `AWSGlueServiceRole`. Otherwise, you must add a policy to allow your users the `iam:PassRole` permission for IAM roles to match your naming convention. Choose **Create Role**. **Note** When you create a notebook with a role, that role is then passed to interactive sessions so that the same role can be used in both places. As such, the `iam:PassRole` permission needs to be part of the role's policy. Create a new policy for your role using the following example. Replace the account number with your own and the role name. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::090000000210:role/" } ] } ``` 1. Add tags to your role (optional). Tags are key-value pairs that you can add to AWS resources to help identify, organize, or search for resources. Then, choose **Create role**. # Step 3: Attach a policy to users or groups that access AWS Glue The administrator must assign permissions to any users, groups, or roles using the AWS Glue console or AWS Command Line Interface (AWS CLI). You provide those permissions by using AWS Identity and Access Management (IAM), through policies. This step describes assigning permissions to users or groups. When you finish this step, your user or group has the following policies attached: + The AWS managed policy `AWSGlueConsoleFullAccess` or the custom policy **GlueConsoleAccessPolicy** + **`AWSGlueConsoleSageMakerNotebookFullAccess`** + **`CloudWatchLogsReadOnlyAccess`** + **`AWSCloudFormationReadOnlyAccess`** + **`AmazonAthenaFullAccess`** **To attach an inline policy and embed it in a user or group** You can attach an AWS managed policy or an inline policy to a user or group to access the AWS Glue console. Some of the resources specified in this policy refer to default names that are used by AWS Glue for Amazon S3 buckets, Amazon S3 ETL scripts, CloudWatch Logs, CloudFormation, and Amazon EC2 resources. For simplicity, AWS Glue writes some Amazon S3 objects into buckets in your account prefixed with `aws-glue-*` by default. **Note** You can skip this step if you use the AWS managed policy **`AWSGlueConsoleFullAccess`**. **Important** AWS Glue needs permission to assume a role that is used to perform work on your behalf. **To accomplish this, you add the `iam:PassRole` permissions to your AWS Glue users or groups.** This policy grants permission to roles that begin with `AWSGlueServiceRole` for AWS Glue service roles, and `AWSGlueServiceNotebookRole` for roles that are required when you create a notebook server. You can also create your own policy for `iam:PassRole` permissions that follows your naming convention. Per security best practices, it is recommended to restrict access by tightening policies to further restrict access to Amazon S3 bucket and Amazon CloudWatch log groups. For an example Amazon S3 policy, see [Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket](https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/). In this step, you create a policy that is similar to `AWSGlueConsoleFullAccess`. You can find the most current version of `AWSGlueConsoleFullAccess` on the IAM console. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Users** or **User groups**. 1. In the list, choose the name of the user or group to embed a policy in. 1. Choose the **Permissions** tab and, if necessary, expand the **Permissions policies** section. 1. Choose the **Add Inline policy** link. 1. On the **Create Policy** screen, navigate to a tab to edit JSON. Create a policy document with the following JSON statements, and then choose **Review policy**. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:*", "redshift:DescribeClusters", "redshift:DescribeClusterSubnetGroups", "iam:ListRoles", "iam:ListUsers", "iam:ListGroups", "iam:ListRolePolicies", "iam:GetRole", "iam:GetRolePolicy", "iam:ListAttachedRolePolicies", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:DescribeVpcEndpoints", "ec2:DescribeRouteTables", "ec2:DescribeVpcAttribute", "ec2:DescribeKeyPairs", "ec2:DescribeInstances", "rds:DescribeDBInstances", "rds:DescribeDBClusters", "rds:DescribeDBSubnetGroups", "s3:ListAllMyBuckets", "s3:ListBucket", "s3:GetBucketAcl", "s3:GetBucketLocation", "cloudformation:DescribeStacks", "cloudformation:GetTemplateSummary", "dynamodb:ListTables", "kms:ListAliases", "kms:DescribeKey", "cloudwatch:GetMetricData", "cloudwatch:ListDashboards" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*/*aws-glue-*/*", "arn:aws:s3:::aws-glue-*" ] }, { "Effect": "Allow", "Action": [ "tag:GetResources" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:PutBucketPublicAccessBlock" ], "Resource": [ "arn:aws:s3:::aws-glue-*" ] }, { "Effect": "Allow", "Action": [ "logs:GetLogEvents" ], "Resource": [ "arn:aws:logs:*:*:/aws-glue/*" ] }, { "Effect": "Allow", "Action": [ "cloudformation:CreateStack", "cloudformation:DeleteStack" ], "Resource": "arn:aws:cloudformation:*:*:stack/aws-glue*/*" }, { "Effect": "Allow", "Action": [ "ec2:RunInstances" ], "Resource": [ "arn:aws:ec2:*:*:instance/*", "arn:aws:ec2:*:*:key-pair/*", "arn:aws:ec2:*:*:image/*", "arn:aws:ec2:*:*:security-group/*", "arn:aws:ec2:*:*:network-interface/*", "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:volume/*" ] }, { "Action": [ "iam:PassRole" ], "Effect": "Allow", "Resource": "arn:aws:iam::*:role/AWSGlueServiceRole*", "Condition": { "StringLike": { "iam:PassedToService": [ "glue.amazonaws.com" ] } } }, { "Action": [ "iam:PassRole" ], "Effect": "Allow", "Resource": "arn:aws:iam::*:role/AWSGlueServiceNotebookRole*", "Condition": { "StringLike": { "iam:PassedToService": [ "ec2.amazonaws.com" ] } } }, { "Action": [ "iam:PassRole" ], "Effect": "Allow", "Resource": [ "arn:aws:iam::*:role/service-role/AWSGlueServiceRole*" ], "Condition": { "StringLike": { "iam:PassedToService": [ "glue.amazonaws.com" ] } } } ] } ``` ------ The following table describes the permissions granted by this policy. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/attach-policy-iam-user.html) 1. On the **Review policy** screen, enter a name for the policy, for example **GlueConsoleAccessPolicy**. When you're satisfied with the policy, choose **Create policy**. Ensure that no errors appear in a red box at the top of the screen. Correct any that are reported. **Note** If **Use autoformatting** is selected, the policy is reformatted whenever you open a policy or choose **Validate Policy**. **To attach the AWSGlueConsoleFullAccess managed policy** You can attach the `AWSGlueConsoleFullAccess` policy to provide permissions that are required by the AWS Glue console user. **Note** You can skip this step if you created your own policy for AWS Glue console access. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. In the list of policies, select the check box next to the **AWSGlueConsoleFullAccess**. You can use the **Filter** menu and the search box to filter the list of policies. 1. Choose **Policy actions**, and then choose **Attach**. 1. Choose the user to attach the policy to. You can use the **Filter** menu and the search box to filter the list of principal entities. After choosing the user to attach the policy to, choose **Attach policy**. **To attach the `AWSGlueConsoleSageMakerNotebookFullAccess` managed policy** You can attach the `AWSGlueConsoleSageMakerNotebookFullAccess` policy to a user to manage SageMaker AI notebooks created on the AWS Glue console. In addition to other required AWS Glue console permissions, this policy grants access to resources needed to manage SageMaker AI notebooks. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. In the list of policies, select the check box next to the **AWSGlueConsoleSageMakerNotebookFullAccess**. You can use the **Filter** menu and the search box to filter the list of policies. 1. Choose **Policy actions**, and then choose **Attach**. 1. Choose the user to attach the policy to. You can use the **Filter** menu and the search box to filter the list of principal entities. After choosing the user to attach the policy to, choose **Attach policy**. **To attach the CloudWatchLogsReadOnlyAccess managed policy** You can attach the **CloudWatchLogsReadOnlyAccess** policy to a user to view the logs created by AWS Glue on the CloudWatch Logs console. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. In the list of policies, select the check box next to the **CloudWatchLogsReadOnlyAccess**. You can use the **Filter** menu and the search box to filter the list of policies. 1. Choose **Policy actions**, and then choose **Attach**. 1. Choose the user to attach the policy to. You can use the **Filter** menu and the search box to filter the list of principal entities. After choosing the user to attach the policy to, choose **Attach policy**. **To attach the AWSCloudFormationReadOnlyAccess managed policy** You can attach the **AWSCloudFormationReadOnlyAccess** policy to a user to view the CloudFormation stacks used by AWS Glue on the CloudFormation console. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. In the list of policies, select the check box next to **AWSCloudFormationReadOnlyAccess**. You can use the **Filter** menu and the search box to filter the list of policies. 1. Choose **Policy actions**, and then choose **Attach**. 1. Choose the user to attach the policy to. You can use the **Filter** menu and the search box to filter the list of principal entities. After choosing the user to attach the policy to, choose **Attach policy**. **To attach the AmazonAthenaFullAccess managed policy** You can attach the **AmazonAthenaFullAccess** policy to a user to view Amazon S3 data in the Athena console. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. In the list of policies, select the check box next to the **AmazonAthenaFullAccess**. You can use the **Filter** menu and the search box to filter the list of policies. 1. Choose **Policy actions**, and then choose **Attach**. 1. Choose the user to attach the policy to. You can use the **Filter** menu and the search box to filter the list of principal entities. After choosing the user to attach the policy to, choose **Attach policy**. # Step 4: Create an IAM policy for notebook servers If you plan to use notebooks with development endpoints, you must specify permissions when you create the notebook server. You provide those permissions by using AWS Identity and Access Management (IAM). This policy grants permission for some Amazon S3 actions to manage resources in your account that are needed by AWS Glue when it assumes the role using this policy. Some of the resources that are specified in this policy refer to default names used by AWS Glue for Amazon S3 buckets, Amazon S3 ETL scripts, and Amazon EC2 resources. For simplicity, AWS Glue defaults writing some Amazon S3 objects into buckets in your account prefixed with `aws-glue-*`. **Note** You can skip this step if you use the AWS managed policy **`AWSGlueServiceNotebookRole`**. In this step, you create a policy that is similar to `AWSGlueServiceNotebookRole`. You can find the most current version of `AWSGlueServiceNotebookRole` on the IAM console. **To create an IAM policy for notebooks** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose **Create Policy**. 1. On the **Create Policy** screen, navigate to a tab to edit JSON. Create a policy document with the following JSON statements, and then choose **Review policy**. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:CreatePartition", "glue:CreateTable", "glue:DeleteDatabase", "glue:DeletePartition", "glue:DeleteTable", "glue:GetDatabase", "glue:GetDatabases", "glue:GetPartition", "glue:GetPartitions", "glue:GetTable", "glue:GetTableVersions", "glue:GetTables", "glue:UpdateDatabase", "glue:UpdatePartition", "glue:UpdateTable", "glue:GetJobBookmark", "glue:ResetJobBookmark", "glue:CreateConnection", "glue:CreateJob", "glue:DeleteConnection", "glue:DeleteJob", "glue:GetConnection", "glue:GetConnections", "glue:GetDevEndpoint", "glue:GetDevEndpoints", "glue:GetJob", "glue:GetJobs", "glue:UpdateJob", "glue:BatchDeleteConnection", "glue:UpdateConnection", "glue:GetUserDefinedFunction", "glue:UpdateUserDefinedFunction", "glue:GetUserDefinedFunctions", "glue:DeleteUserDefinedFunction", "glue:CreateUserDefinedFunction", "glue:BatchGetPartition", "glue:BatchDeletePartition", "glue:BatchCreatePartition", "glue:BatchDeleteTable", "glue:UpdateDevEndpoint", "s3:GetBucketLocation", "s3:ListBucket", "s3:ListAllMyBuckets", "s3:GetBucketAcl" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::crawler-public*", "arn:aws:s3:::aws-glue*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::aws-glue*" ] }, { "Effect": "Allow", "Action": [ "ec2:CreateTags", "ec2:DeleteTags" ], "Condition": { "ForAllValues:StringEquals": { "aws:TagKeys": [ "aws-glue-service-resource" ] } }, "Resource": [ "arn:aws:ec2:*:*:network-interface/*", "arn:aws:ec2:*:*:security-group/*", "arn:aws:ec2:*:*:instance/*" ] } ] } ``` ------ The following table describes the permissions granted by this policy. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/create-notebook-policy.html) 1. On the **Review Policy** screen, enter your **Policy Name**, for example **GlueServiceNotebookPolicyDefault**. Enter an optional description, and when you're satisfied with the policy, choose **Create policy**. # Step 5: Create an IAM role for notebook servers If you plan to use notebooks with development endpoints, you need to grant the IAM role permissions. You provide those permissions by using AWS Identity and Access Management IAM, through an IAM role. **Note** When you create an IAM role using the IAM console, the console creates an instance profile automatically and gives it the same name as the role to which it corresponds. **To create an IAM role for notebooks** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. Choose **Create role**. 1. For role type, choose **AWS Service**, find and choose **EC2**, and choose the **EC2** use case, then choose **Next: Permissions**. 1. On the **Attach permissions policy** page, choose the policies that contain the required permissions; for example, **AWSGlueServiceNotebookRole** for general AWS Glue permissions and the AWS managed policy **AmazonS3FullAccess** for access to Amazon S3 resources. Then choose **Next: Review**. **Note** Ensure that one of the policies in this role grants permissions to your Amazon S3 sources and targets. Also confirm that your policy allows full access to the location where you store your notebook when you create a notebook server. You might want to provide your own policy for access to specific Amazon S3 resources. For more information about creating an Amazon S3 policy for your resources, see [Specifying Resources in a Policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-arn-format.html). If you plan to access Amazon S3 sources and targets that are encrypted with SSE-KMS, attach a policy that allows notebooks to decrypt the data. For more information, see [Protecting Data Using Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). The following is an example. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": [ "arn:aws:kms:*:111122223333:key/key-id" ] } ] } ``` 1. For **Role name**, enter a name for your role. Create the role with the name prefixed with the string `AWSGlueServiceNotebookRole` to allow the role to be passed from console users to the notebook server. AWS Glue provided policies expect IAM service roles to begin with `AWSGlueServiceNotebookRole`. Otherwise you must add a policy to your users to allow the `iam:PassRole` permission for IAM roles to match your naming convention. For example, enter `AWSGlueServiceNotebookRoleDefault`. Then choose **Create role**. # Step 6: Create an IAM policy for SageMaker AI notebooks If you plan to use SageMaker AI notebooks with development endpoints, you must specify permissions when you create the notebook. You provide those permissions by using AWS Identity and Access Management (IAM). **To create an IAM policy for SageMaker AI notebooks** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose **Create Policy**. 1. On the **Create Policy** page, navigate to a tab to edit the JSON. Create a policy document with the following JSON statements. Edit *bucket-name*, *region-code*, and *account-id* for your environment. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket" ] }, { "Action": [ "s3:GetObject" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket*" ] }, { "Action": [ "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:CreateLogGroup" ], "Effect": "Allow", "Resource": [ "arn:aws:logs:us-east-1:111122223333:log-group:/aws/sagemaker/*", "arn:aws:logs:us-east-1:111122223333:log-group:/aws/sagemaker/*:log-stream:aws-glue-*" ] }, { "Action": [ "glue:UpdateDevEndpoint", "glue:GetDevEndpoint", "glue:GetDevEndpoints" ], "Effect": "Allow", "Resource": [ "arn:aws:glue:us-east-1:111122223333:devEndpoint/*" ] }, { "Action": [ "sagemaker:ListTags" ], "Effect": "Allow", "Resource": [ "arn:aws:sagemaker:us-east-1:111122223333:notebook-instance/*" ] } ] } ``` ------ Then choose **Review policy**. The following table describes the permissions granted by this policy. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/create-sagemaker-notebook-policy.html) 1. On the **Review Policy** screen, enter your **Policy Name**, for example `AWSGlueSageMakerNotebook`. Enter an optional description, and when you're satisfied with the policy, choose **Create policy**. # Step 7: Create an IAM role for SageMaker AI notebooks If you plan to use SageMaker AI notebooks with development endpoints, you need to grant the IAM role permissions. You provide those permissions by using AWS Identity and Access Management (IAM), through an IAM role. **To create an IAM role for SageMaker AI notebooks** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. Choose **Create role**. 1. For role type, choose **AWS Service**, find and choose **SageMaker**, and then choose the **SageMaker - Execution** use case. Then choose **Next: Permissions**. 1. On the **Attach permissions policy** page, choose the policies that contain the required permissions; for example, **AmazonSageMakerFullAccess**. Choose **Next: Review**. If you plan to access Amazon S3 sources and targets that are encrypted with SSE-KMS, attach a policy that allows notebooks to decrypt the data, as shown in the following example. For more information, see [Protecting Data Using Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": [ "arn:aws:kms:*:111122223333:key/key-id" ] } ] } ``` ------ 1. For **Role name**, enter a name for your role. To allow the role to be passed from console users to SageMaker AI, use a name that is prefixed with the string `AWSGlueServiceSageMakerNotebookRole`. AWS Glue provided policies expect IAM roles to begin with `AWSGlueServiceSageMakerNotebookRole`. Otherwise you must add a policy to your users to allow the `iam:PassRole` permission for IAM roles to match your naming convention. For example, enter `AWSGlueServiceSageMakerNotebookRole-Default`, and then choose **Create role**. 1. After you create the role, attach the policy that allows additional permissions required to create SageMaker AI notebooks from AWS Glue. Open the role that you just created, `AWSGlueServiceSageMakerNotebookRole-Default`, and choose **Attach policies**. Attach the policy that you created named `AWSGlueSageMakerNotebook` to the role. # AWS Glue access control policy examples This section contains examples of both identity-based (IAM) access control policies and AWS Glue resource policies. **Contents** + [ # Identity-based policy examples for AWS Glue ](security_iam_id-based-policy-examples.md) + [ ## Policy best practices ](security_iam_id-based-policy-examples.md#security_iam_service-with-iam-policy-best-practices) + [ ## Resource-level permissions only apply to specific AWS Glue objects ](security_iam_id-based-policy-examples.md#glue-identity-based-policy-limitations) + [ ## Using the AWS Glue console ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-console) + [ ## Allow users to view their own permissions ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-view-own-permissions) + [ ## Grant read-only permission to a table ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-read-only-table-access) + [ ## Filter tables by GetTables permission ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-filter-tables) + [ ## Grant full access to a table and all partitions ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-full-access-tables-partitions) + [ ## Control access by name prefix and explicit denial ](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-deny-by-name-prefix) + [ ## Grant access using tags ](security_iam_id-based-policy-examples.md#tags-control-access-example-triggers-allow) + [ ## Deny access using tags ](security_iam_id-based-policy-examples.md#tags-control-access-example-triggers-deny) + [ ## Use tags with list and batch API operations ](security_iam_id-based-policy-examples.md#tags-control-access-example-triggers-list-batch) + [ ## Control settings using condition keys or context keys ](security_iam_id-based-policy-examples.md#glue-identity-based-policy-condition-keys) + [ ### Control policies that control settings using condition keys ](security_iam_id-based-policy-examples.md#glue-identity-based-policy-condition-key-vpc) + [ ### Control policies that control settings using context keys ](security_iam_id-based-policy-examples.md#glue-identity-based-policy-context-key-glue) + [ ## Deny an identity the ability to create data preview sessions ](security_iam_id-based-policy-examples.md#deny-data-preview-sessions-per-identity) + [ # Resource-based policy examples for AWS Glue ](security_iam_resource-based-policy-examples.md) + [ ## Considerations for using resource-based policies with AWS Glue ](security_iam_resource-based-policy-examples.md#security_iam_resource-based-policy-examples-considerations) + [ ## Use a resource policy to control access in the same account ](security_iam_resource-based-policy-examples.md#glue-policy-resource-policies-example-same-account) # Identity-based policy examples for AWS Glue Identity-based policy examples By default, users and roles don't have permission to create or modify AWS Glue resources. To grant users permission to perform actions on the resources that they need, an IAM administrator can create IAM policies. To learn how to create an IAM identity-based policy by using these example JSON policy documents, see [Create IAM policies (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html) in the *IAM User Guide*. For details about actions and resource types defined by AWS Glue, including the format of the ARNs for each of the resource types, see [Actions, resources, and condition keys for AWS Glue](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsglue.html) in the *Service Authorization Reference*. **Note** The examples provided in this section all use the `us-west-2` Region. You can replace this with the AWS Region that you want to use. **Topics** + [ ## Policy best practices ](#security_iam_service-with-iam-policy-best-practices) + [ ## Resource-level permissions only apply to specific AWS Glue objects ](#glue-identity-based-policy-limitations) + [ ## Using the AWS Glue console ](#security_iam_id-based-policy-examples-console) + [ ## Allow users to view their own permissions ](#security_iam_id-based-policy-examples-view-own-permissions) + [ ## Grant read-only permission to a table ](#security_iam_id-based-policy-examples-read-only-table-access) + [ ## Filter tables by GetTables permission ](#security_iam_id-based-policy-examples-filter-tables) + [ ## Grant full access to a table and all partitions ](#security_iam_id-based-policy-examples-full-access-tables-partitions) + [ ## Control access by name prefix and explicit denial ](#security_iam_id-based-policy-examples-deny-by-name-prefix) + [ ## Grant access using tags ](#tags-control-access-example-triggers-allow) + [ ## Deny access using tags ](#tags-control-access-example-triggers-deny) + [ ## Use tags with list and batch API operations ](#tags-control-access-example-triggers-list-batch) + [ ## Control settings using condition keys or context keys ](#glue-identity-based-policy-condition-keys) + [ ## Deny an identity the ability to create data preview sessions ](#deny-data-preview-sessions-per-identity) ## Policy best practices Identity-based policies determine whether someone can create, access, or delete AWS Glue resources in your account. These actions can incur costs for your AWS account. When you create or edit identity-based policies, follow these guidelines and recommendations: + **Get started with AWS managed policies and move toward least-privilege permissions** – To get started granting permissions to your users and workloads, use the *AWS managed policies* that grant permissions for many common use cases. They are available in your AWS account. We recommend that you reduce permissions further by defining AWS customer managed policies that are specific to your use cases. For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) or [AWS managed policies for job functions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html) in the *IAM User Guide*. + **Apply least-privilege permissions** – When you set permissions with IAM policies, grant only the permissions required to perform a task. You do this by defining the actions that can be taken on specific resources under specific conditions, also known as *least-privilege permissions*. For more information about using IAM to apply permissions, see [ Policies and permissions in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) in the *IAM User Guide*. + **Use conditions in IAM policies to further restrict access** – You can add a condition to your policies to limit access to actions and resources. For example, you can write a policy condition to specify that all requests must be sent using SSL. You can also use conditions to grant access to service actions if they are used through a specific AWS service, such as CloudFormation. For more information, see [ IAM JSON policy elements: Condition](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html) in the *IAM User Guide*. + **Use IAM Access Analyzer to validate your IAM policies to ensure secure and functional permissions** – IAM Access Analyzer validates new and existing policies so that the policies adhere to the IAM policy language (JSON) and IAM best practices. IAM Access Analyzer provides more than 100 policy checks and actionable recommendations to help you author secure and functional policies. For more information, see [Validate policies with IAM Access Analyzer](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html) in the *IAM User Guide*. + **Require multi-factor authentication (MFA)** – If you have a scenario that requires IAM users or a root user in your AWS account, turn on MFA for additional security. To require MFA when API operations are called, add MFA conditions to your policies. For more information, see [ Secure API access with MFA](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_configure-api-require.html) in the *IAM User Guide*. For more information about best practices in IAM, see [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the *IAM User Guide*. ## Resource-level permissions only apply to specific AWS Glue objects Resource-level authorization limitations You can only define fine-grained control for specific objects in AWS Glue. Therefore you must write your client's IAM policy so that API operations that allow Amazon Resource Names (ARNs) for the `Resource` statement are not mixed with API operations that don't allow ARNs. For example, the following IAM policy allows API operations for `GetClassifier` and `GetJobRun`. It defines the `Resource` as `*` because AWS Glue doesn't allow ARNs for classifiers and job runs. Because ARNs are allowed for specific API operations such as `GetDatabase` and `GetTable`, ARNs can be specified in the second half of the policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetClassifier*", "glue:GetJobRun*" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "glue:Get*" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/default", "arn:aws:glue:us-east-1:111122223333:table/default/e*1*", "arn:aws:glue:us-east-1:111122223333:connection/connection2" ] } ] } ``` ------ For a list of AWS Glue objects that allow ARNs, see [Specifying AWS Glue Resource ARNs](glue-specifying-resource-arns.md). ## Using the AWS Glue console Using the console To access the AWS Glue console, you must have a minimum set of permissions. These permissions must allow you to list and view details about the AWS Glue resources in your AWS account. If you create an identity-based policy that is more restrictive than the minimum required permissions, the console won't function as intended for entities (users or roles) with that policy. You don't need to allow minimum console permissions for users that are making calls only to the AWS CLI or the AWS API. Instead, allow access to only the actions that match the API operation that they're trying to perform. To ensure that users and roles can still use the AWS Glue console, also attach the AWS Glue `ConsoleAccess` or `ReadOnly` AWS managed policy to the entities. For more information, see [Adding permissions to a user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) in the *IAM User Guide*. For a user to work with the AWS Glue console, that user must have a minimum set of permissions that allows them to work with the AWS Glue resources for their AWS account. In addition to these AWS Glue permissions, the console requires permissions from the following services: + Amazon CloudWatch Logs permissions to display logs. + AWS Identity and Access Management (IAM) permissions to list and pass roles. + AWS CloudFormation permissions to work with stacks. + Amazon Elastic Compute Cloud (Amazon EC2) permissions to list VPCs, subnets, security groups, instances, and other objects. + Amazon Simple Storage Service (Amazon S3) permissions to list buckets and objects, and to retrieve and save scripts. + Amazon Redshift permissions to work with clusters. + Amazon Relational Database Service (Amazon RDS) permissions to list instances. For more information about the permissions that users require to view and work with the AWS Glue console, see [Step 3: Attach a policy to users or groups that access AWS Glue](attach-policy-iam-user.md). If you create an IAM policy that is more restrictive than the minimum required permissions, the console won't function as intended for users with that IAM policy. To ensure that those users can still use the AWS Glue console, also attach the `AWSGlueConsoleFullAccess` managed policy as described in [AWS managed (predefined) policies for AWS Glue](security-iam-awsmanpol.md#access-policy-examples-aws-managed). ## Allow users to view their own permissions This example shows how you might create a policy that allows IAM users to view the inline and managed policies that are attached to their user identity. This policy includes permissions to complete this action on the console or programmatically using the AWS CLI or AWS API. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "ViewOwnUserInfo", "Effect": "Allow", "Action": [ "iam:GetUserPolicy", "iam:ListGroupsForUser", "iam:ListAttachedUserPolicies", "iam:ListUserPolicies", "iam:GetUser" ], "Resource": ["arn:aws:iam::*:user/${aws:username}"] }, { "Sid": "NavigateInConsole", "Effect": "Allow", "Action": [ "iam:GetGroupPolicy", "iam:GetPolicyVersion", "iam:GetPolicy", "iam:ListAttachedGroupPolicies", "iam:ListGroupPolicies", "iam:ListPolicyVersions", "iam:ListPolicies", "iam:ListUsers" ], "Resource": "*" } ] } ``` ## Grant read-only permission to a table Read-only table access The following policy grants read-only permission to a `books` table in database `db1`. For more information about resource Amazon Resource Names (ARNs), see [Data Catalog ARNs](glue-specifying-resource-arns.md#data-catalog-resource-arns). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesActionOnBooks", "Effect": "Allow", "Action": [ "glue:GetTables", "glue:GetTable" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/books" ] } ] } ``` ------ This policy grants read-only permission to a table named `books` in the database named `db1`. To grant `Get` permission to a table, permission to the catalog and database resources is also required. The following policy grants the minimum necessary permissions to create table `tb1` in database `db1`: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:CreateTable" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:table/db1/tbl1", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:catalog" ] } ] } ``` ------ ## Filter tables by GetTables permission Filter tables Assume that there are three tables—`customers`, `stores`, and `store_sales`—in database `db1`. The following policy grants `GetTables` permission to `stores` and `store_sales`, but not to `customers`. When you call `GetTables` with this policy, the result contains only the two authorized tables (the `customers` table is not returned). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesExample", "Effect": "Allow", "Action": [ "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/store_sales", "arn:aws:glue:us-east-1:111122223333:table/db1/stores" ] } ] } ``` ------ You can simplify the preceding policy by using `store*` to match any table names that start with `store`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesExample2", "Effect": "Allow", "Action": [ "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/store*" ] } ] } ``` ------ Similarly, using `/db1/*` to match all tables in `db1`, the following policy grants `GetTables` access to all the tables in `db1`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesReturnAll", "Effect": "Allow", "Action": [ "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/*" ] } ] } ``` ------ If no table ARN is provided, a call to `GetTables` succeeds, but it returns an empty list. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesEmptyResults", "Effect": "Allow", "Action": [ "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1" ] } ] } ``` ------ If the database ARN is missing in the policy, a call to `GetTables` fails with an `AccessDeniedException`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "GetTablesAccessDeny", "Effect": "Allow", "Action": [ "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:table/db1/*" ] } ] } ``` ------ ## Grant full access to a table and all partitions Full table access The following policy grants all permissions on a table named `books` in database `db1`. This includes read and write permissions on the table itself, on archived versions of it, and on all its partitions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "FullAccessOnTable", "Effect": "Allow", "Action": [ "glue:CreateTable", "glue:GetTable", "glue:GetTables", "glue:UpdateTable", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:GetTableVersion", "glue:GetTableVersions", "glue:DeleteTableVersion", "glue:BatchDeleteTableVersion", "glue:CreatePartition", "glue:BatchCreatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:UpdatePartition", "glue:DeletePartition", "glue:BatchDeletePartition" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/books" ] } ] } ``` ------ The preceding policy can be simplified in practice. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "FullAccessOnTable", "Effect": "Allow", "Action": [ "glue:*Table*", "glue:*Partition*" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1", "arn:aws:glue:us-east-1:111122223333:table/db1/books" ] } ] } ``` ------ Notice that the minimum granularity of fine-grained access control is at the table level. This means that you can't grant a user access to some partitions in a table but not others, or to some table columns but not to others. A user either has access to all of a table, or to none of it. ## Control access by name prefix and explicit denial Name prefix with deny In this example, suppose that the databases and tables in your AWS Glue Data Catalog are organized using name prefixes. The databases in the development stage have the name prefix `dev-`, and those in production have the name prefix `prod-`. You can use the following policy to grant developers full access to all databases, tables, UDFs, and so on, that have the `dev-` prefix. But you grant read-only access to everything with the `prod-` prefix. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "DevAndProdFullAccess", "Effect": "Allow", "Action": [ "glue:*Database*", "glue:*Table*", "glue:*Partition*", "glue:*UserDefinedFunction*", "glue:*Connection*" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/dev-*", "arn:aws:glue:us-east-1:111122223333:database/prod-*", "arn:aws:glue:us-east-1:111122223333:table/dev-*/*", "arn:aws:glue:us-east-1:111122223333:table/*/dev-*", "arn:aws:glue:us-east-1:111122223333:table/prod-*/*", "arn:aws:glue:us-east-1:111122223333:table/*/prod-*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/dev-*/*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/*/dev-*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/prod-*/*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/*/prod-*", "arn:aws:glue:us-east-1:111122223333:connection/dev-*", "arn:aws:glue:us-east-1:111122223333:connection/prod-*" ] }, { "Sid": "ProdWriteDeny", "Effect": "Deny", "Action": [ "glue:*Create*", "glue:*Update*", "glue:*Delete*" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:database/prod-*", "arn:aws:glue:us-east-1:111122223333:table/prod-*/*", "arn:aws:glue:us-east-1:111122223333:table/*/prod-*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/prod-*/*", "arn:aws:glue:us-east-1:111122223333:userDefinedFunction/*/prod-*", "arn:aws:glue:us-east-1:111122223333:connection/prod-*" ] } ] } ``` ------ The second statement in the preceding policy uses explicit `deny`. You can use explicit `deny` to overwrite any `allow` permissions that are granted to the principal. This lets you lock down access to critical resources and prevent another policy from accidentally granting access to them. In the preceding example, even though the first statement grants full access to `prod-` resources, the second statement explicitly revokes write access to them, leaving only read access to `prod-` resources. ## Grant access using tags Example access control using tags For example, suppose that you want to limit access to a trigger `t2` to a specific user named `Tom` in your account. All other users, including `Sam`, have access to trigger `t1`. The triggers `t1` and `t2` have the following properties. ``` aws glue get-triggers { "Triggers": [ { "State": "CREATED", "Type": "SCHEDULED", "Name": "t1", "Actions": [ { "JobName": "j1" } ], "Schedule": "cron(0 0/1 * * ? *)" }, { "State": "CREATED", "Type": "SCHEDULED", "Name": "t2", "Actions": [ { "JobName": "j1" } ], "Schedule": "cron(0 0/1 * * ? *)" } ] } ``` The AWS Glue administrator attached a tag value `Tom` (`aws:ResourceTag/Name": "Tom"`) to trigger `t2`. The AWS Glue administrator also gave Tom an IAM policy with a condition statement based on the tag. As a result, Tom can only use an AWS Glue operation that acts on resources with the tag value `Tom`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "glue:*", "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/Name": "Tom" } } } ] } ``` ------ When Tom tries to access the trigger `t1`, he receives an access denied message. Meanwhile, he can successfully retrieve trigger `t2`. ``` aws glue get-trigger --name t1 An error occurred (AccessDeniedException) when calling the GetTrigger operation: User: Tom is not authorized to perform: glue:GetTrigger on resource: arn:aws:glue:us-east-1:123456789012:trigger/t1 aws glue get-trigger --name t2 { "Trigger": { "State": "CREATED", "Type": "SCHEDULED", "Name": "t2", "Actions": [ { "JobName": "j1" } ], "Schedule": "cron(0 0/1 * * ? *)" } } ``` Tom can't use the plural `GetTriggers` API operation to list triggers because this operation doesn't support filtering on tags. To give Tom access to `GetTriggers`, the AWS Glue administrator creates a policy that splits the permissions into two sections. One section allows Tom access to all triggers with the `GetTriggers` API operation. The second section allows Tom access to API operations that are tagged with the value `Tom`. With this policy, Tom is allowed both `GetTriggers` and `GetTrigger` access to trigger `t2`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "glue:GetTriggers", "Resource": "*" }, { "Effect": "Allow", "Action": "glue:*", "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/Name": "Tom" } } } ] } ``` ------ ## Deny access using tags Deny access using tags Another resource policy approach is to explicitly deny access to resources. **Important** An explicit denial policy does not work for plural API operations such as `GetTriggers`. In the following example policy, all AWS Glue job operations are allowed. However, the second `Effect` statement *explicitly* denies access to jobs tagged with the `Team` key and `Special` value. When an administrator attaches the following policy to an identity, the identity can access all jobs *except* those tagged with the `Team` key and `Special` value. ## Use tags with list and batch API operations A third approach to writing a resource policy is to allow access to resources using a `List` API operation to list out resources for a tag value. Then, use the corresponding `Batch` API operation to allow access to details of specific resources. With this approach, the administrator doesn't need to allow access to the plural `GetCrawlers`, `GetDevEndpoints`, `GetJobs`, or `GetTriggers` API operations. Instead, you can allow the ability to list the resources with the following API operations: + `ListCrawlers` + `ListDevEndpoints` + `ListJobs` + `ListTriggers` And, you can allow the ability to get details about individual resources with the following API operations: + `BatchGetCrawlers` + `BatchGetDevEndpoints` + `BatchGetJobs` + `BatchGetTriggers` As an administrator, to use this approach, you can do the following: 1. Add tags to your crawlers, development endpoints, jobs, and triggers. 1. Deny user access to `Get` API operations such as `GetCrawlers`, `GetDevEndponts`, `GetJobs`, and `GetTriggers`. 1. To enable users to find out which tagged resources they have access to, allow user access to `List` API operations such as `ListCrawlers`, `ListDevEndponts`, `ListJobs`, and `ListTriggers`. 1. Deny user access to AWS Glue tagging APIs, such as `TagResource` and `UntagResource`. 1. Allow user access to resource details with `BatchGet` API operations such as `BatchGetCrawlers`, `BatchGetDevEndponts`, `BatchGetJobs`, and `BatchGetTriggers`. For example, when calling the `ListCrawlers` operation, provide a tag value to match the user name. Then the result is a list of crawlers that match the provided tag values. Provide the list of names to `BatchGetCrawlers` to get details about each crawler with the given tag. For example, if Tom should only be able to retrieve details of triggers that are tagged with `Tom`, the administrator can add tags to triggers for `Tom`, deny access to the `GetTriggers` API operation to all users, and allow access to all users to `ListTriggers` and `BatchGetTriggers`. The following is the resource policy that the AWS Glue administrator grants to Tom. In the first section of the policy, AWS Glue API operations are denied for `GetTriggers`. In the second section of the policy, `ListTriggers` is allowed for all resources. However, in the third section, those resources tagged with `Tom` are allowed access with the `BatchGetTriggers` access. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "glue:GetTriggers", "Resource": "*" }, { "Effect": "Allow", "Action": [ "glue:ListTriggers" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "glue:BatchGetTriggers" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceTag/Name": "Tom" } } } ] } ``` ------ Using the same triggers as the previous example, Tom can access trigger `t2`, but not trigger `t1`. The following example shows the results when Tom tries to access `t1` and `t2` with `BatchGetTriggers`. ``` aws glue batch-get-triggers --trigger-names t2 { "Triggers": { "State": "CREATED", "Type": "SCHEDULED", "Name": "t2", "Actions": [ { "JobName": "j2" } ], "Schedule": "cron(0 0/1 * * ? *)" } } aws glue batch-get-triggers --trigger-names t1 An error occurred (AccessDeniedException) when calling the BatchGetTriggers operation: No access to any requested resource. ``` The following example shows the results when Tom tries to access both trigger `t2` and trigger `t3` (which does not exist) in the same `BatchGetTriggers` call. Notice that because Tom has access to trigger `t2` and it exists, only `t2` is returned. Although Tom is allowed to access trigger `t3`, trigger `t3` does not exist, so `t3` is returned in the response in a list of `"TriggersNotFound": []`. ``` aws glue batch-get-triggers --trigger-names t2 t3 { "Triggers": { "State": "CREATED", "Type": "SCHEDULED", "Name": "t2", "Actions": [ { "JobName": "j2" } ], "TriggersNotFound": ["t3"], "Schedule": "cron(0 0/1 * * ? *)" } } ``` ## Control settings using condition keys or context keys Create policies that control settings using condition keys You can use condition keys or context keys when granting permissions to create and update jobs. These sections discuss the keys: + [Control policies that control settings using condition keys](#glue-identity-based-policy-condition-key-vpc) + [Control policies that control settings using context keys](#glue-identity-based-policy-context-key-glue) ### Control policies that control settings using condition keys Control settings using condition keys AWS Glue provides three IAM condition keys `glue:VpcIds`, `glue:SubnetIds`, and `glue:SecurityGroupIds`. You can use the condition keys in IAM policies when granting permissions to create and update jobs. You can use this setting to ensure that jobs or sessions are not created (or updated to) to run outside of a desired VPC environment. The VPC setting information is not a direct input from the `CreateJob` request, but inferred from the job "connections" field that points to an AWS Glue connection. **Example usage** Create an AWS Glue network type connection named "traffic-monitored-connection" with the desired VpcId "vpc-id1234", SubnetIds, and SecurityGroupIds. Specify the condition keys condition for the `CreateJob` and `UpdateJob` action in the IAM policy. ``` { "Effect": "Allow", "Action": [ "glue:CreateJob", "glue:UpdateJob" ], "Resource": [ "*" ], "Condition": { "ForAnyValue:StringLike": { "glue:VpcIds": [ "vpc-id1234" ] } } } ``` You can create a similar IAM policy to prohibit creating an AWS Glue job without specifying connection information. **Restricting sessions on VPCs** To enforce created sessions to run within a specified VPC, you restrict role permission by adding a `Deny` effect on the `glue:CreateSession` action with the condition that the glue:vpc-id not equal to vpc-<123>. For example: ``` "Effect": "Deny", "Action": [ "glue:CreateSession" ], "Condition": { "StringNotEquals" : {"glue:VpcIds" : ["vpc-123"]} } ``` You also can enforce created sessions to run within a VPC by adding a `Deny` effect on the `glue:CreateSession` action with the condition that the `glue:vpc-id` is null. For example: ``` { "Effect": "Deny", "Action": [ "glue:CreateSession" ], "Condition": { "Null": {"glue:VpcIds": true} } }, { "Effect": "Allow", "Action": [ "glue:CreateSession" ], "Resource": ["*"] } ``` ### Control policies that control settings using context keys Control settings using context keys AWS Glue provides a context key (`glue:CredentialIssuingService= glue.amazonaws.com`) to each role session that AWS Glue makes available to the job and developer endpoint. This allows you to implement security controls for the actions taken by AWS Glue scripts. AWS Glue provides another context key (`glue:RoleAssumedBy=glue.amazonaws.com`) to each role session where AWS Glue makes a call to another AWS service on the customer's behalf (not by a job/dev endpoint, but directly by the AWS Glue service). **Example usage** Specify the conditional permission in an IAM policy and attach it to the role to be used by an AWS Glue job. This ensures certain actions are allowed/denied based on whether the role session is used for an AWS Glue job runtime environment. ``` { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*", "Condition": { "StringEquals": { "glue:CredentialIssuingService": "glue.amazonaws.com" } } } ``` ## Deny an identity the ability to create data preview sessions This section contains an IAM policy example used to deny an identity the ability to create data preview sessions. Attach this policy to the identity, which is separate from the role used by the data preview session during its run. ``` { "Sid": "DatapreviewDeny", "Effect": "Deny", "Action": [ "glue:CreateSession" ], "Resource": [ "arn:aws:glue:*:*:session/glue-studio-datapreview*" ] } ``` # Resource-based policy examples for AWS Glue Resource-based policy examples This section contains example resource-based policies, including policies that grant cross-account access. The examples use the AWS Command Line Interface (AWS CLI) to interact with AWS Glue service API operations. You can perform the same operations on the AWS Glue console or using one of the AWS SDKs. **Important** By changing an AWS Glue resource policy, you might accidentally revoke permissions for existing AWS Glue users in your account and cause unexpected disruptions. Try these examples only in development or test accounts, and ensure that they don't break any existing workflows before you make the changes. **Topics** + [ ## Considerations for using resource-based policies with AWS Glue ](#security_iam_resource-based-policy-examples-considerations) + [ ## Use a resource policy to control access in the same account ](#glue-policy-resource-policies-example-same-account) ## Considerations for using resource-based policies with AWS Glue Considerations **Note** Both IAM policies and an AWS Glue resource policy take a few seconds to propagate. After you attach a new policy, you might notice that the old policy is still in effect until the new policy has propagated through the system. You use a policy document written in JSON format to create or modify a resource policy. The policy syntax is the same as for an identity-based IAM policy (see [IAM JSON policy reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html)), with the following exceptions: + A `"Principal"` or `"NotPrincipal"` block is required for each policy statement. + The `"Principal"` or `"NotPrincipal"` must identify valid existing principals. Wildcard patterns (like `arn:aws:iam::account-id:user/*`) are not allowed. + The `"Resource"` block in the policy requires all resource ARNs to match the following regular expression syntax (where the first `%s` is the *region*, and the second `%s` is the *account-id*): ``` *arn:aws:glue:%s:%s:(\*|[a-zA-Z\*]+\/?.*) ``` For example, both `arn:aws:glue:us-west-2:account-id:*` and `arn:aws:glue:us-west-2:account-id:database/default` are allowed, but `*` is not allowed. + Unlike identity-based policies, an AWS Glue resource policy must only contain Amazon Resource Names (ARNs) of resources that belong to the catalog that the policy is attached to. Such ARNs always start with `arn:aws:glue:`. + A policy cannot cause the identity that creates it to be locked out of further policy creation or modification. + A resource-policy JSON document cannot exceed 10 KB in size. ## Use a resource policy to control access in the same account Same-account access In this example, an admin user in Account A creates a resource policy that grants IAM user `Alice` in Account A full access to the catalog. Alice has no IAM policy attached. To do this, the admin user runs the following AWS CLI command. ``` # Run as admin of Account A $ aws glue put-resource-policy --profile administrator-name --region us-west-2 --policy-in-json '{ "Version": "2012-10-17", "Statement": [ { "Principal": { "AWS": [ "arn:aws:iam::account-A-id:user/Alice" ] }, "Effect": "Allow", "Action": [ "glue:*" ], "Resource": [ "arn:aws:glue:us-west-2:account-A-id:*" ] } ] }' ``` Instead of entering the JSON policy document as a part of your AWS CLI command, you can save a policy document in a file and reference the file path in the AWS CLI command, prefixed by `file://`. The following is an example of how you might do that. ``` $ echo '{ "Version": "2012-10-17", "Statement": [ { "Principal": { "AWS": [ "arn:aws:iam::account-A-id:user/Alice" ] }, "Effect": "Allow", "Action": [ "glue:*" ], "Resource": [ "arn:aws:glue:us-west-2:account-A-id:*" ] } ] }' > /temp/policy.json $ aws glue put-resource-policy --profile admin1 \ --region us-west-2 --policy-in-json file:///temp/policy.json ``` After this resource policy has propagated, Alice can access all AWS Glue resources in Account A, as follows. ``` # Run as user Alice $ aws glue create-database --profile alice --region us-west-2 --database-input '{ "Name": "new_database", "Description": "A new database created by Alice", "LocationUri": "s3://amzn-s3-demo-bucket" }' $ aws glue get-table --profile alice --region us-west-2 --database-name "default" --table-name "tbl1"} ``` In response to Alice's `get-table` call, the AWS Glue service returns the following. ``` { "Table": { "Name": "tbl1", "PartitionKeys": [], "StorageDescriptor": { ...... }, ...... } } ``` # Granting AWS managed policies for AWS Glue Granting AWS managed policies An AWS managed policy is a standalone policy that is created and administered by AWS. AWS managed policies are designed to provide permissions for many common use cases so that you can start assigning permissions to users, groups, and roles. Keep in mind that AWS managed policies might not grant least-privilege permissions for your specific use cases because they're available for all AWS customers to use. We recommend that you reduce permissions further by defining [ customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies) that are specific to your use cases. You cannot change the permissions defined in AWS managed policies. If AWS updates the permissions defined in an AWS managed policy, the update affects all principal identities (users, groups, and roles) that the policy is attached to. AWS is most likely to update an AWS managed policy when a new AWS service is launched or new API operations become available for existing services. For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) in the *IAM User Guide*. ## AWS managed (predefined) policies for AWS Glue AWS addresses many common use cases by providing standalone IAM policies that are created and administered by AWS. These AWS managed policies grant necessary permissions for common use cases so that you can avoid having to investigate what permissions are needed. For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) in the *IAM User Guide*. The following AWS managed policies, which you can attach to identities in your account, are specific to AWS Glue and are grouped by use case scenario: + [AWSGlueConsoleFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console. + [AWSGlueServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints. + [AwsGlueSessionUserRestrictedServiceRole](https://console.aws.amazon.com/iam/home#policies/details/arn:aws:iam::aws:policy%2Fservice-role%2FAwsGlueSessionUserRestrictedServiceRole) – Provides full access to all AWS Glue resources except for sessions. It allows users to create and use only the interactive sessions that are associated with the user. This policy includes other permissions needed by AWS Glue to manage AWS Glue resources in other AWS services. The policy also allows adding tags to AWS Glue resources in other AWS services. **Note** To achieve the full security benefits, do not grant this policy to a user that was assigned the `AWSGlueServiceRole`, `AWSGlueConsoleFullAccess`, or `AWSGlueConsoleSageMakerNotebookFullAccess` policy. + [AwsGlueSessionUserRestrictedPolicy](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AwsGlueSessionUserRestrictedPolicy) – Provides access to create AWS Glue interactive sessions using the `CreateSession` API operation only if a tag key “owner” and value that match the assignee's AWS user ID are provided. This identity policy is attached to the IAM user that invokes the `CreateSession` API operation. This policy also permits the assignee to interact with the AWS Glue interactive session resources that were created with an “owner” tag and value that match their AWS user ID. This policy denies permission to change or remove "owner" tags from an AWS Glue session resource after the session is created. **Note** To achieve the full security benefits, do not grant this policy to a user that was assigned the `AWSGlueServiceRole`, `AWSGlueConsoleFullAccess`, or `AWSGlueConsoleSageMakerNotebookFullAccess` policy. + [AwsGlueSessionUserRestrictedNotebookServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AwsGlueSessionUserRestrictedNotebookServiceRole) – Provides sufficient access to the AWS Glue Studio notebook session to interact with specific AWS Glue interactive session resources. These are resources that are created with the “owner” tag value that matches the AWS user ID of the principal (IAM user or role) that creates the notebook. For more information about these tags, see the [Principal key values](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#policy-vars-infotouse) chart in the *IAM User Guide*. This service-role policy is attached to the role that is specified with a magic command within the notebook or is passed as a role to the `CreateSession` API operation. This policy also permits the principal to create an AWS Glue interactive session from the AWS Glue Studio notebook interface only if a tag key “owner” and value match the AWS user ID of the principal. This policy denies permission to change or remove "owner" tags from an AWS Glue session resource after the session is created. This policy also includes permissions for writing and reading from Amazon S3 buckets, writing CloudWatch logs, and creating and deleting tags for Amazon EC2 resources used by AWS Glue. **Note** To achieve the full security benefits, do not grant this policy to a role that was assigned the `AWSGlueServiceRole`, `AWSGlueConsoleFullAccess`, or `AWSGlueConsoleSageMakerNotebookFullAccess` policy. + [AwsGlueSessionUserRestrictedNotebookPolicy](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AwsGlueSessionUserRestrictedNotebookPolicy) – Provides access to create an AWS Glue interactive session from the AWS Glue Studio notebook interface only if there is a tag key “owner” and value that match the AWS user IDof the principal (IAM user or role) that creates the notebook. For more information about these tags, see the [Principal key values](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#policy-vars-infotouse) chart in the *IAM User Guide*. This policy is attached to the principal (IAM user or role) that creates sessions from the AWS Glue Studio notebook interface. This policy also permits sufficient access to the AWS Glue Studio notebook to interact with specific AWS Glue interactive session resources. These are resources that are created with the “owner” tag value that matches the AWS user ID of the principal. This policy denies permission to change or remove "owner" tags from an AWS Glue session resource after the session is created. + [AWSGlueServiceNotebookRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceNotebookRole) – Grants access to AWS Glue sessions started in an AWS Glue Studio notebook. This policy allows listing and getting session information for all sessions, but only permits users to create and use the sessions tagged with their AWS user ID. This policy denies permission to change or remove “owner” tags from AWS Glue session resources tagged with their AWS ID. Assign this policy to the AWS user who creates jobs using the notebook interface in AWS Glue Studio. + [AWSGlueConsoleSageMakerNotebookFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleSageMakerNotebookFullAccess) – Grants full access to AWS Glue and SageMaker AI resources when the identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console who manage SageMaker AI notebooks. + [AWSGlueSchemaRegistryFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueSchemaRegistryFullAccess) – Grants full access to AWS Glue Schema Registry resources when the identity that the policy is attached to uses the AWS Management Console or AWS CLI. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console or AWS CLI who manage the AWS Glue Schema Registry. + [AWSGlueSchemaRegistryReadonlyAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueSchemaRegistryReadonlyAccess) – Grants read-only access to AWS Glue Schema Registry resources when an identity that the policy is attached to uses the AWS Management Console or AWS CLI. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console or AWS CLI who use the AWS Glue Schema Registry. **Note** You can review these permissions policies by signing in to the IAM console and searching for specific policies there. You can also create your own custom IAM policies to allow permissions for AWS Glue actions and resources. You can attach these custom policies to the IAM users or groups that require those permissions. To create a connection with VPC configuration while using a custom IAM role, it must have the following VPC access actions: + secretsmanager:GetSecretValue + secretsmanager:PutSecretValue + secretsmanager:DescribeSecret + ec2:CreateNetworkInterface + ec2:DeleteNetworkInterface + ec2:DescribeNetworkInterfaces + ec2:DescribeSubnets ## AWS Glue updates to AWS managed policies Policy updates View details about updates to AWS managed policies for AWS Glue since this service began tracking these changes. For automatic alerts about changes to this page, subscribe to the RSS feed on the AWS Glue Document history page. | Change | Description | Date | | --- | --- | --- | | AwsGlueSessionUserRestrictedNotebookPolicy – Minor update to an existing policy. | Add allow for glue:TagResource action on the owner tag key. Required for supporting on tag-on-create for sessions with owner tag key. | August 30, 2024 | | AwsGlueSessionUserRestrictedNotebookServiceRole – Minor update to an existing policy. | Add allow for glue:TagResource action on the owner tag key. Required for supporting on tag-on-create for sessions with owner tag key. | August 30, 2024 | | AwsGlueSessionUserRestrictedPolicy – Minor update to an existing policy. | Add allow for glue:TagResource action on the owner tag key. Required for supporting on tag-on-create for sessions with owner tag key. | August 5, 2024 | | AwsGlueSessionUserRestrictedServiceRole – Minor update to an existing policy. | Add allow for glue:TagResource action on the owner tag key. Required for supporting on tag-on-create for sessions with owner tag key. | August 5, 2024 | | AwsGlueSessionUserRestrictedPolicy – Minor update to an existing policy. | Add glue:StartCompletion and glue:GetCompletion to policy. Required for Amazon Q data integration in AWS Glue. | April 30, 2024 | | AwsGlueSessionUserRestrictedNotebookServiceRole – Minor update to an existing policy. | Add glue:StartCompletion and glue:GetCompletion to policy. Required for Amazon Q data integration in AWS Glue. | April 30, 2024 | | AwsGlueSessionUserRestrictedServiceRole – Minor update to an existing policy. | Add glue:StartCompletion and glue:GetCompletion to policy. Required for Amazon Q data integration in AWS Glue. | April 30, 2024 | | AWSGlueServiceNotebookRole – Minor update to an existing policy. | Add glue:StartCompletion and glue:GetCompletion to policy. Required for Amazon Q data integration in AWS Glue. | Jan 30, 2024 | | AwsGlueSessionUserRestrictedNotebookPolicy – Minor update to an existing policy. | Add glue:StartCompletion and glue:GetCompletion to policy. Required for Amazon Q data integration in AWS Glue. | Nov 29, 2023 | | AWSGlueServiceNotebookRole – Minor update to an existing policy. | Add codewhisperer:GenerateRecommendations to policy. Required for a new feature where AWS Glue generates CodeWhisperer recommendations. | Oct 9, 2023 | | AWSGlueServiceRole – Minor update to an existing policy. | Tighten scope of CloudWatch permissions to better reflect AWS Glue logging. | Aug 4, 2023 | | AWSGlueConsoleFullAccess – Minor update to an existing policy. | Add databrew recipe List and Describe permissions to policy. Required to provide full administrative access for new features where AWS Glue can access recipes. | May 9, 2023 | | AWSGlueConsoleFullAccess – Minor update to an existing policy. | Add cloudformation:ListStacks to policy. Preserves existing capabilities after changes to CloudFormation authorization requirements. | March 28, 2023 | | New managed policies added for the interactive sessions feature: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/security-iam-awsmanpol.html) | These policies were designed to provide additional security for interactive sessions and notebooks in AWS Glue Studio. The policies restrict access to the `CreateSession` API operation so that only the owner has access. | November 30, 2021 | | AWSGlueConsoleSageMakerNotebookFullAccess – Update to an existing policy. | Removed a redundant resource ARN (`arn:aws:s3:::aws-glue-*/*`) for the action that grants read/write permissions on Amazon S3 buckets that AWS Glue uses to store scripts and temporary files. Fixed a syntax issue by changing `"StringEquals"` to `"ForAnyValue:StringLike"`, and moved the` "Effect": "Allow"` lines to precede the `"Action":` line in each place where they were out of order. | July 15, 2021 | | AWSGlueConsoleFullAccess – Update to an existing policy. | Removed a redundant resource ARN (arn:aws:s3:::aws-glue-\$1/\$1) for the action that grants read/write permissions on Amazon S3 buckets that AWS Glue uses to store scripts and temporary files. | July 15, 2021 | | AWS Glue started tracking changes. | AWS Glue started tracking changes for its AWS managed policies. | June 10, 2021 | # Granting dynamically scoped policies for job execution Granting dynamically scoped policies AWS Glue offers a powerful new capability: dynamic session policies for job executions. This feature allows you to specify custom, fine-grained permissions for each job run without creating multiple IAM roles. When you start a Glue job using the `StartJobRun` API, you can include an inline session policy. This policy temporarily modifies the permissions of the job's execution role for the duration of that specific job run. It's similar to using temporary credentials with the `AssumeRole` API in other AWS services. + **Enhanced security**: You can limit job permissions to the minimum necessary for each run. + **Simplified management**: Eliminates the need to create and maintain numerous IAM roles for different scenarios. + **Flexibility**: You can be adjust permissions dynamically based on runtime parameters or tenant-specific needs. + **Scalability**: This method excels in multi-tenant environments where you need to isolate resources between tenants. **Examples for granting dynamically scoped policies usage:** The following examples demonstrate granting jobs *read* and *write* access only to a specific Amazon S3 bucket path, where the path is dynamically determined by the job run ID. This illustrates how to implement granular, execution-specific permissions for each job run. **From CLI** ``` aws glue start-job-run \ --job-name "your-job-name" \ --execution-role-session-policy '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::specific-bucket/${JobRunId}/*" ] } ] }' ``` # Specifying AWS Glue resource ARNs Specifying AWS Glue Resource ARNs In AWS Glue, you can control access to resources using an AWS Identity and Access Management (IAM) policy. In a policy, you use an Amazon Resource Name (ARN) to identify the resource that the policy applies to. Not all resources in AWS Glue support ARNs. **Topics** + [ ## Data Catalog ARNs ](#data-catalog-resource-arns) + [ ## ARNs for non-catalog objects in AWS Glue ](#non-catalog-resource-arns) + [ ## Access control for AWS Glue non-catalog singular API operations ](#non-catalog-singular-apis) + [ ## Access control for AWS Glue non-catalog API operations that retrieve multiple items ](#non-catalog-plural-apis) + [ ## Access control for AWS Glue non-catalog BatchGet API operations ](#non-catalog-batch-get-apis) ## Data Catalog ARNs Data Catalog ARNs Data Catalog resources have a hierarchical structure, with `catalog` as the root. ``` arn:aws:glue:region:account-id:catalog ``` Each AWS account has a single Data Catalog in an AWS Region with the 12-digit account ID as the catalog ID. Resources have unique ARNs associated with them, as shown in the following table. | **Resource type** | **ARN format** | | --- | --- | | Catalog | `arn:aws:glue:region:account-id:catalog` For example: `arn:aws:glue:us-east-1:123456789012:catalog` | | Database | `arn:aws:glue:region:account-id:database/database name` For example: `arn:aws:glue:us-east-1:123456789012:database/db1` | | Table | `arn:aws:glue:region:account-id:table/database name/table name` For example: `arn:aws:glue:us-east-1:123456789012:table/db1/tbl1` | | Federated S3 tables catalog (All table buckets) | `arn:aws:glue:region:account-id:catalog/s3tablescatalog` For example: `arn:aws:glue:us-east-1:123456789012:catalog/s3tablescatalog` | | Federated S3 table bucket catalog (child catalog) | `arn:aws:glue:region:account-id:catalog/s3tablescatalog/bucket name` For example: `arn:aws:glue:us-east-1:123456789012:catalog/s3tablescatalog/amzn-s3-demo-bucket1` | | Federated S3 tables database | `arn:aws:glue:region:account-id:database/s3tablescatalog/child catalog name/database name` For example: `arn:aws:glue:us-east-1:123456789012:database/s3tablescatalog/amzn-s3-demo-bucket1/nsdb1` | | Federated S3 table | `arn:aws:glue:region:account-id:table/s3tablescatalog/child catalog name/database name/table name` For example: `arn:aws:glue:us-east-1:123456789012:table/s3tablescatalog/amzn-s3-demo-bucket1/nsdb1/s3tbl1` | | Federated S3 tables catalog (A single table bucket registered with Lake Formation) | `arn:aws:glue:region:account-id:catalog`/*catalog name* For example: `arn:aws:glue:us-east-1:123456789012:catalog/amzn-s3-demo-bucket1` | | Federated S3 tables database | `arn:aws:glue:region:account-id:catalog`/*catalog name*/*database name* For example: `arn:aws:glue:us-east-1:123456789012:database/amzn-s3-demo-bucket1/nsdb1` | | Federated S3 table | `arn:aws:glue:region:account-id:catalog`/*catalog name*/*database name*/*table name* For example: `arn:aws:glue:us-east-1:123456789012:table/amzn-s3-demo-bucket1/nsdb1/s3tbl1` | | Federated or managed catalog (top-level catalog in a multi-catalog) | `arn:aws:glue:region:account-id:catalog/top-level catalog name` For example: `arn:aws:glue:us-east-1:123456789012:catalog/nscatalog` ARN format for a managed catalog follows the same structure. | | Federated multi-level catalog (child catalog in a multi-level catalog) | `arn:aws:glue:region:account-id:catalog/top-level catalog name/child catalog name` For example: `arn:aws:glue:us-east-1:123456789012:catalog/nscatalog/dbcatalog` | | Federated database | `arn:aws:glue:region:account-id:database/name space catalog name/child catalog name/database name` For example: `arn:aws:glue:us-east-1:123456789012:database/nscatalog/dbcatalog/schemadb` | | Federated table | `arn:aws:glue:region:account-id:table/name space catalog name/child catalog name/database name/table name` For example: `arn:aws:glue:us-east-1:123456789012:table/nscatalog/dbcatalog/schemadb/rstbl1` | | Catalog link container | `arn:aws:glue:region:account-id:catalog`/*link container name* For example: `arn:aws:glue:glue:us-east-1:123456789012:catalog`/linkcontainer-example | | Database | `arn:aws:glue:region:account-id:catalog`/*link container name*/*database name* For example: `arn:aws:glue:glue:us-east-1:123456789012:database`/linkcontainer-example/link-db | | User-defined function | `arn:aws:glue:region:account-id:userDefinedFunction/database name/user-defined function name` For example: `arn:aws:glue:us-east-1:123456789012:userDefinedFunction/db1/func1` | | Connection | `arn:aws:glue:region:account-id:connection/connection name` For example: `arn:aws:glue:us-east-1:123456789012:connection/connection1` | | Interactive Session | `arn:aws:glue:region:account-id:session/interactive session id` For example: `arn:aws:glue:us-east-1:123456789012:session/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111` | To enable fine-grained access control, you can use these ARNs in your IAM policies and resource policies to grant and deny access to specific resources. Wildcards are allowed in the policies. For example, the following ARN matches all tables in database `default`. ``` arn:aws:glue:us-east-1:123456789012:table/default/* ``` **Important** All operations performed on a Data Catalog resource require permission on the resource and all the ancestors of that resource. For example, to create a partition for a table requires permission on the table, database, and catalog where the table is located. The following example shows the permission required to create partitions on table `PrivateTable` in database `PrivateDatabase` in the Data Catalog. ``` { "Sid": "GrantCreatePartitions", "Effect": "Allow", "Action": [ "glue:BatchCreatePartitions" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:table/PrivateDatabase/PrivateTable", "arn:aws:glue:us-east-1:123456789012:database/PrivateDatabase", "arn:aws:glue:us-east-1:123456789012:catalog" ] } ``` In addition to permission on the resource and all its ancestors, all delete operations require permission on all children of that resource. For example, deleting a database requires permission on all the tables and user-defined functions in the database, in addition to the database and the catalog where the database is located. The following example shows the permission required to delete database `PrivateDatabase` in the Data Catalog. ``` { "Sid": "GrantDeleteDatabase", "Effect": "Allow", "Action": [ "glue:DeleteDatabase" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:table/PrivateDatabase/*", "arn:aws:glue:us-east-1:123456789012:userDefinedFunction/PrivateDatabase/*", "arn:aws:glue:us-east-1:123456789012:database/PrivateDatabase", "arn:aws:glue:us-east-1:123456789012:catalog" ] } ``` In summary, actions on Data Catalog resources follow these permission rules: Actions on the catalog require permission on the catalog only. Actions on a database require permission on the database and catalog. Delete actions on a database require permission on the database and catalog plus all tables and user-defined functions in the database. Actions on a table, partition, or table version require permission on the table, database, and catalog. Actions on a user-defined function require permission on the user-defined function, database, and catalog. Actions on a connection require permission on the connection and catalog. ## ARNs for non-catalog objects in AWS Glue Non-catalog ARNs Some AWS Glue resources allow resource-level permissions to control access using an ARN. You can use these ARNs in your IAM policies to enable fine-grained access control. The following table lists the resources that can contain resource ARNs. | **Resource type** | **ARN format** | | --- | --- | | Crawler | `arn:aws:glue:region:account-id:crawler/crawler-name` For example: `arn:aws:glue:us-east-1:123456789012:crawler/mycrawler` | | Job | `arn:aws:glue:region:account-id:job/job-name` For example: `arn:aws:glue:us-east-1:123456789012:job/testjob` | | Trigger | `arn:aws:glue:region:account-id:trigger/trigger-name` For example: `arn:aws:glue:us-east-1:123456789012:trigger/sampletrigger` | | Development endpoint | `arn:aws:glue:region:account-id:devEndpoint/development-endpoint-name` For example: `arn:aws:glue:us-east-1:123456789012:devEndpoint/temporarydevendpoint` | | Machine learning transform | `arn:aws:glue:region:account-id:mlTransform/transform-id` For example: `arn:aws:glue:us-east-1:123456789012:mlTransform/tfm-1234567890` | ## Access control for AWS Glue non-catalog singular API operations Access for non-catalog singular APIs AWS Glue non-catalog *singular* API operations act on a single item (development endpoint). Examples are `GetDevEndpoint`, `CreateUpdateDevEndpoint`, and `UpdateDevEndpoint`. For these operations, a policy must put the API name in the `"action"` block and the resource ARN in the `"resource"` block. Suppose that you want to allow a user to call the `GetDevEndpoint` operation. The following policy grants the minimum necessary permissions to an endpoint named `myDevEndpoint-1`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "MinimumPermissions", "Effect": "Allow", "Action": "glue:GetDevEndpoint", "Resource": "arn:aws:glue:us-east-1:111122223333:devEndpoint/myDevEndpoint-1" } ] } ``` ------ The following policy allows `UpdateDevEndpoint` access to resources that match `myDevEndpoint-` with a wildcard (\$1). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PermissionWithWildcard", "Effect": "Allow", "Action": "glue:UpdateDevEndpoint", "Resource": "arn:aws:glue:us-east-1:111122223333:devEndpoint/myDevEndpoint-*" } ] } ``` ------ You can combine the two policies as in the following example. You might see `EntityNotFoundException` for any development endpoint whose name begins with `A`. However, an access denied error is returned when you try to access other development endpoints. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "CombinedPermissions", "Effect": "Allow", "Action": [ "glue:UpdateDevEndpoint", "glue:GetDevEndpoint" ], "Resource": "arn:aws:glue:us-east-1:111122223333:devEndpoint/A*" } ] } ``` ------ ## Access control for AWS Glue non-catalog API operations that retrieve multiple items Access for get[\$1] non-catalog APIs Some AWS Glue API operations retrieve multiple items (such as multiple development endpoints); for example, `GetDevEndpoints`. For this operation, you can specify only a wildcard (\$1) resource, and not specific ARNs. For example, to include `GetDevEndpoints` in the policy, the resource must be scoped to the wildcard (\$1). The singular operations (`GetDevEndpoint`, `CreateDevEndpoint`, and `DeleteDevendpoint`) are also scoped to all (\$1) resources in the example. ``` { "Sid": "PluralAPIIncluded", "Effect": "Allow", "Action": [ "glue:GetDevEndpoints", "glue:GetDevEndpoint", "glue:CreateDevEndpoint", "glue:UpdateDevEndpoint" ], "Resource": [ "*" ] } ``` ## Access control for AWS Glue non-catalog BatchGet API operations Access for BatchGet non-catalog APIs Some AWS Glue API operations retrieve multiple items (such as multiple development endpoints); for example, `BatchGetDevEndpoints`. For this operation, you can specify an ARN to limit the scope of resources that can be accessed. For example, to allow access to a specific development endpoint, include `BatchGetDevEndpoints` in the policy with its resource ARN. ``` { "Sid": "BatchGetAPIIncluded", "Effect": "Allow", "Action": [ "glue:BatchGetDevEndpoints" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:devEndpoint/de1" ] } ``` With this policy, you can successfully access the development endpoint named `de1`. However, if you try to access the development endpoint named `de2`, an error is returned. ``` An error occurred (AccessDeniedException) when calling the BatchGetDevEndpoints operation: No access to any requested resource. ``` **Important** For alternative approaches to setting up IAM policies, such as using `List` and `BatchGet` API operations, see [Identity-based policy examples for AWS Glue](security_iam_id-based-policy-examples.md). # Granting cross-account access Granting access to Data Catalog resources across accounts enables your extract, transform, and load (ETL) jobs to query and join data from different accounts. **Topics** + [ ## Methods for granting cross-account access in AWS Glue ](#cross-account-how-works) + [ ## Adding or updating the Data Catalog resource policy ](#cross-account-adding-resource-policy) + [ ## Making a cross-account API call ](#cross-account-calling) + [ ## Making a cross-account ETL call ](#cross-account-calling-etl) + [ ## Cross-account CloudTrail logging ](#cross-account-ct-logs) + [ ## Cross-account resource ownership and billing ](#cross-account-ownership-and-billing) + [ ## Cross-account access limitations ](#cross-account-limitations) ## Methods for granting cross-account access in AWS Glue Methods for granting cross-account access You can grant access to your data to external AWS accounts by using AWS Glue methods or by using AWS Lake Formation cross-account grants. The AWS Glue methods use AWS Identity and Access Management (IAM) policies to achieve fine-grained access control. Lake Formation uses a simpler `GRANT/REVOKE` permissions model similar to the `GRANT/REVOKE` commands in a relational database system. This section describes using the AWS Glue methods. For information about using Lake Formation cross-account grants, see [Granting Lake Formation Permissions](https://docs.aws.amazon.com/lake-formation/latest/dg/lake-formation-permissions.html) in the *AWS Lake Formation Developer Guide*. There are two AWS Glue methods for granting cross-account access to a resource: + Use a Data Catalog resource policy + Use an IAM role **Granting cross-account access using a resource policy** The following are the general steps for granting cross-account access using a Data Catalog resource policy: 1. An administrator (or other authorized identity) in Account A attaches a resource policy to the Data Catalog in Account A. This policy grants Account B specific cross-account permissions to perform operations on a resource in Account A's catalog. 1. An administrator in Account B attaches an IAM policy to an IAM identity in Account B that delegates the permissions received from Account A. The identity in Account B now has access to the specified resource in Account A. The identity needs permission from *both* the resource owner (Account A) *and* their parent account (Account B) to be able to access the resource. **Granting cross-account access using an IAM role** The following are the general steps for granting cross-account access using an IAM role: 1. An administrator (or other authorized identity) in the account that owns the resource (Account A) creates an IAM role. 1. The administrator in Account A attaches a policy to the role that grants cross-account permissions for access to the resource in question. 1. The administrator in Account A attaches a trust policy to the role that identifies an IAM identity in a different account (Account B) as the principal who can assume the role. The principal in the trust policy can also be an AWS service principal if you want to grant an AWS service permission to assume the role. 1. An administrator in Account B now delegates permissions to one or more IAM identities in Account B so that they can assume that role. Doing so gives those identities in Account B access to the resource in account A. For more information about using IAM to delegate permissions, see [Access management](https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html) in the *IAM User Guide*. For more information about users, groups, roles, and permissions, see [Identities (users, groups, and roles)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html) in the *IAM User Guide*. For a comparison of these two approaches, see [How IAM roles differ from resource-based policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html) in the *IAM User Guide*. AWS Glue supports both options, with the restriction that a resource policy can grant access only to Data Catalog resources. For example, to give the `Dev` role in Account B access to database `db1` in Account A, attach the following resource policy to the catalog in Account A. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetDatabase" ], "Principal": { "AWS": [ "arn:aws:iam::111122223333:role/Dev" ] }, "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1" ] } ] } ``` ------ In addition, Account B would have to attach the following IAM policy to the `Dev` role before it would actually get access to `db1` in Account A. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetDatabase" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/db1" ] } ] } ``` ------ ## Adding or updating the Data Catalog resource policy You can add or update the AWS Glue Data Catalog resource policy using the console, API, or AWS Command Line Interface (AWS CLI). **Important** If you have already made cross-account permission grants from your account with AWS Lake Formation, adding or updating the Data Catalog resource policy requires an extra step. For more information, see [Managing cross-account permissions using both AWS Glue and Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/hybrid-cross-account.html) in the *AWS Lake Formation Developer Guide*. To determine if Lake Formation cross-account grants exist, use the `glue:GetResourcePolicies` API operation or the AWS CLI. If `glue:GetResourcePolicies` returns any policies other than an already existing Data Catalog policy, then Lake Formation grants exist. For more information, see [Viewing all cross-account grants using the GetResourcePolicies API operation](https://docs.aws.amazon.com/lake-formation/latest/dg/cross-account-getresourcepolicies.html) in the *AWS Lake Formation Developer Guide*. **To add or update the Data Catalog resource policy (console)** 1. Open the AWS Glue console at [https://console.aws.amazon.com/glue/](https://console.aws.amazon.com/glue/). Sign in as an AWS Identity and Access Management (IAM) administrative user who has the `glue:PutResourcePolicy` permission. 1. In the navigation pane, choose **Settings**. 1. On the **Data catalog settings** page, under **Permissions**, paste a resource policy into the text area. Then choose **Save**. If the console displays a alert stating that the permissions in the policy will be in addition to any permissions granted using Lake Formation, choose **Proceed**. **To add or update the Data Catalog resource policy (AWS CLI)** + Submit an `aws glue put-resource-policy` command. If Lake Formation grants already exist, ensure that you include the `--enable-hybrid` option with the value `'TRUE'`. For examples of using this command, see [Resource-based policy examples for AWS Glue](security_iam_resource-based-policy-examples.md). ## Making a cross-account API call Cross-account calling All AWS Glue Data Catalog operations have a `CatalogId` field. If the required permissions have been granted to enable cross-account access, a caller can make Data Catalog API calls across accounts. The caller does this by passing the target AWS account ID in `CatalogId` so as to access the resource in that target account. If no `CatalogId` value is provided, AWS Glue uses the caller's own account ID by default, and the call is not cross-account. ## Making a cross-account ETL call Cross-account calling from a job Some AWS Glue PySpark and Scala APIs have a catalog ID field. If all the required permissions have been granted to enable cross-account access, an ETL job can make PySpark and Scala calls to API operations across accounts by passing the target AWS account ID in the catalog ID field to access Data Catalog resources in a target account. If no catalog ID value is provided, AWS Glue uses the caller's own account ID by default, and the call is not cross-account. For PySpark APIs that support `catalog_id`, see [GlueContext class](aws-glue-api-crawler-pyspark-extensions-glue-context.md). For Scala APIs that support `catalogId`, see [AWS Glue Scala GlueContext APIs](glue-etl-scala-apis-glue-gluecontext.md). The following example shows the permissions required by the grantee to run an ETL job. In this example, *grantee-account-id* is the `catalog-id` of the client running the job and *grantor-account-id* is the owner of the resource. This example grants permission to all catalog resources in the grantor's account. To limit the scope of resources granted, you can provide specific ARNs for the catalog, database, table, and connection. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetConnection", "glue:GetDatabase", "glue:GetTable", "glue:GetPartition" ], "Principal": { "AWS": [ "arn:aws:iam::111122223333:root" ] }, "Resource": [ "arn:aws:glue:us-east-1:111122223333:*" ] } ] } ``` ------ **Note** If a table in the grantor's account points to an Amazon S3 location that is also in the grantor's account, the IAM role used to run an ETL job in the grantee's account must have permission to list and get objects from the grantor's account. Given that the client in Account A already has permission to create and run ETL jobs, the following are the basic steps to set up an ETL job for cross-account access: 1. Allow cross-account data access (skip this step if Amazon S3 cross-account access is already set up). 1. Update the Amazon S3 bucket policy in Account B to allow cross-account access from Account A. 1. Update the IAM policy in Account A to allow access to the bucket in Account B. 1. Allow cross-account Data Catalog access. 1. Create or update the resource policy attached to the Data Catalog in Account B to allow access from Account A. 1. Update the IAM policy in Account A to allow access to the Data Catalog in Account B. ## Cross-account CloudTrail logging When an AWS Glue extract, transform, and load (ETL) job accesses the underlying data of a Data Catalog table shared through AWS Lake Formation cross-account grants, there is additional AWS CloudTrail logging behavior. For purposes of this discussion, the AWS account that shared the table is the owner account, and the account that the table was shared with is the recipient account. When an ETL job in the recipient account accesses data in the table in the owner account, the data-access CloudTrail event that is added to the logs for the recipient account gets copied to the owner account's CloudTrail logs. This is so owner accounts can track data accesses by the various recipient accounts. By default, the CloudTrail events do not include a human-readable principal identifier (principal ARN). An administrator in the recipient account can opt in to include the principal ARN in the logs. For more information, see [Cross-account CloudTrail logging](https://docs.aws.amazon.com/lake-formation/latest/dg/cross-account-logging.html) in the *AWS Lake Formation Developer Guide*. **See Also** [Logging and monitoring in AWS Glue](logging-and-monitoring.md) ## Cross-account resource ownership and billing Cross-account ownership When a user in one AWS account (Account A) creates a new resource such as a database in a different account (Account B), that resource is then owned by Account B, the account where it was created. An administrator in Account B automatically gets full permissions to access the new resource, including reading, writing, and granting access permissions to a third account. The user in Account A can access the resource that they just created only if they have the appropriate permissions granted by Account B. Storage costs and other costs that are directly associated with the new resource are billed to Account B, the resource owner. The cost of requests from the user who created the resource are billed to the requester's account, Account A. For more information about AWS Glue billing and pricing, see [How AWS Pricing Works](https://d0.awsstatic.com/whitepapers/aws_pricing_overview.pdf). ## Cross-account access limitations Cross-account limitations AWS Glue cross-account access has the following limitations: + Cross-account access to AWS Glue is not allowed if you created databases and tables using Amazon Athena orAmazon Redshift Spectrum prior to a region's support for AWS Glue and the resource owner account has not migrated the Amazon Athena data catalog to AWS Glue. You can find the current migration status using the [GetCatalogImportStatus (get\$1catalog\$1import\$1status)](aws-glue-api-catalog-migration.md#aws-glue-api-catalog-migration-GetCatalogImportStatus). For more details on how to migrate an Athena catalog to AWS Glue, see [Upgrading to the AWS Glue Data Catalog step-by-step](https://docs.aws.amazon.com/athena/latest/ug/glue-upgrade.html) in the *Amazon Athena User Guide*. + Cross-account access is *only* supported for Data Catalog resources, including databases, tables, user-defined functions, and connections. + Cross-account access to the Data Catalog from Athena requires you to register the catalog as an Athena `DataCatalog` resource. For instructions, see [Registering an AWS Glue Data Catalog from another account](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue-cross-account.html) in the *Amazon Athena User Guide*. # Troubleshooting AWS Glue identity and access Troubleshooting Use the following information to help you diagnose and fix common issues that you might encounter when working with AWS Glue and IAM. **Topics** + [ ## I am not authorized to perform an action in AWS Glue ](#security_iam_troubleshoot-no-permissions) + [ ## I am not authorized to perform iam:PassRole ](#security_iam_troubleshoot-passrole) + [ ## I want to allow people outside of my AWS account to access my AWS Glue resources ](#security_iam_troubleshoot-cross-account-access) ## I am not authorized to perform an action in AWS Glue If you receive an error that you're not authorized to perform an action, your policies must be updated to allow you to perform the action. The following example error occurs when the `mateojackson` IAM user tries to use the console to view details about a fictional `my-example-widget` resource but doesn't have the fictional `glue:GetWidget` permissions. ``` User: arn:aws:iam::123456789012:user/mateojackson is not authorized to perform: glue:GetWidget on resource: my-example-widget ``` In this case, the policy for the `mateojackson` user must be updated to allow access to the `my-example-widget` resource by using the `glue:GetWidget` action. If you need help, contact your AWS administrator. Your administrator is the person who provided you with your sign-in credentials. ## I am not authorized to perform iam:PassRole If you receive an error that you're not authorized to perform the `iam:PassRole` action, your policies must be updated to allow you to pass a role to AWS Glue. Some AWS services allow you to pass an existing role to that service instead of creating a new service role or service-linked role. To do this, you must have permissions to pass the role to the service. The following example error occurs when an IAM user named `marymajor` tries to use the console to perform an action in AWS Glue. However, the action requires the service to have permissions that are granted by a service role. Mary does not have permissions to pass the role to the service. ``` User: arn:aws:iam::123456789012:user/marymajor is not authorized to perform: iam:PassRole ``` In this case, Mary's policies must be updated to allow her to perform the `iam:PassRole` action. If you need help, contact your AWS administrator. Your administrator is the person who provided you with your sign-in credentials. ## I want to allow people outside of my AWS account to access my AWS Glue resources You can create a role that users in other accounts or people outside of your organization can use to access your resources. You can specify who is trusted to assume the role. For services that support resource-based policies or access control lists (ACLs), you can use those policies to grant people access to your resources. To learn more, consult the following: + To learn whether AWS Glue supports these features, see [How AWS Glue works with IAM](security_iam_service-with-iam.md). + To learn how to provide access to your resources across AWS accounts that you own, see [Providing access to an IAM user in another AWS account that you own](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_aws-accounts.html) in the *IAM User Guide*. + To learn how to provide access to your resources to third-party AWS accounts, see [Providing access to AWS accounts owned by third parties](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) in the *IAM User Guide*. + To learn how to provide access through identity federation, see [Providing access to externally authenticated users (identity federation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_federated-users.html) in the *IAM User Guide*. + To learn the difference between using roles and resource-based policies for cross-account access, see [Cross account resource access in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html) in the *IAM User Guide*. # AWS Lake Formation access control models AWS Glue 5.0 and above supports two models for accessing data through AWS Lake Formation: **Topics** + [ # Using AWS Glue with AWS Lake Formation for Full Table Access ](security-access-control-fta.md) + [ # Using AWS Glue with AWS Lake Formation for fine-grained access control ](security-lf-enable.md) # Using AWS Glue with AWS Lake Formation for Full Table Access ## Introduction to Full Table Access AWS Glue 5.0 and above supports Full Table Access (FTA) control in Apache Spark based on your policies defined in AWS Lake Formation. This feature enables read and write operations from your AWS Glue Spark jobs on AWS Lake Formation registered tables when the job role has full table access. FTA is ideal for use cases that need to comply with security regulations at the table level and supports Spark capabilities including Resilient Distributed Datasets (RDDs), custom libraries, and User Defined Functions (UDFs) with AWS Lake Formation tables. When a AWS Glue Spark job is configured for Full Table Access (FTA), AWS Lake Formation credentials are used to read/write Amazon S3 data for AWS Lake Formation registered tables, while the job's runtime role credentials will be used to read/write tables not registered with AWS Lake Formation. This capability enables Data Manipulation Language (DML) operations including CREATE, ALTER, DELETE, UPDATE, and MERGE INTO statements on Apache Hive and Iceberg tables. **Note** Review your requirements and determine if Fine-Grained Access Control (FGAC) or Full Table Access (FTA) suits your needs. Only one AWS Lake Formation permission method can be enabled for a given AWS Glue job. A job cannot simultaneously run Full Table Access (FTA) and Fine-Grained Access Control (FGAC) at the same time. ## How Full-Table Access (FTA) works on AWS Glue AWS Lake Formation offers two approaches for data access control: Fine-Grained Access Control (FGAC) and Full Table Access (FTA). FGAC provides enhanced security through column, row, and cell-level filtering, ideal for scenarios requiring granular permissions. FTA is ideal for straightforward access control scenarios where you need table- level permissions. It simplifies implementation by eliminating the need to enable fine-grained access mode, improves performance and reduces cost by avoiding the system driver and system executors, and supports both read and write operations ( including CREATE, ALTER, DELETE, UPDATE, and MERGE INTO commands). In AWS Glue 4.0, AWS Lake Formation based data access worked through GlueContext class, the utility class provided by AWS Glue. In AWS Glue 5.0, AWS Lake Formation based data access is available through native Spark SQL, Spark DataFrames, and continues to be supported through GlueContext class. ## Implementing Full Table Access ### Step 1: Enable Full Table Access in AWS Lake Formation To use Full Table Access (FTA) mode, you need to allow third-party query engines to access data without the IAM session tag validation in AWS Lake Formation. To enable, follow the steps in [ Application integration for full table access ](https://docs.aws.amazon.com/lake-formation/latest/dg/full-table-credential-vending.html). ### Step 2: Setup IAM permissions for job runtime role For read or write access to underlying data, in addition to AWS Lake Formation permissions, a job runtime role needs the `lakeformation:GetDataAccess` IAM permission. With this permission, AWS Lake Formation grants the request for temporary credentials to access the data. The following is an example policy of how to provide IAM permissions to access a script in Amazon S3, uploading logs to Amazon S3, AWS Glue API permissions, and permission to access AWS Lake Formation. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "ScriptAccess", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/scripts/*" ] }, { "Sid": "LoggingAccess", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/logs/*" ] }, { "Sid": "GlueCatalogAccess", "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions", "glue:CreateTable", "glue:UpdateTable" ], "Resource": [ "arn:aws:glue:us-east-1:111122223333:catalog", "arn:aws:glue:us-east-1:111122223333:database/default", "arn:aws:glue:us-east-1:111122223333:table/default/*" ] }, { "Sid": "LakeFormationAccess", "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": "*" } ] } ``` ------ #### Step 2.1 Configure AWS Lake Formation permissions AWS Glue Spark jobs that read data from Amazon S3 require AWS Lake Formation SELECT permission. AWS Glue Spark jobs that write/delete data in Amazon S3 require AWS Lake Formation ALL permission. AWS Glue Spark jobs that interact with AWS Glue Data catalog require DESCRIBE, ALTER, DROP permission as appropriate. ### Step 3: Initialize a Spark session for Full Table Access using AWS Lake Formation To access tables registered with AWS Lake Formation, the following configurations need to be set during Spark initialization to configure Spark to use AWS Lake Formation credentials. To access tables registered with AWS Lake Formation, you need to explicitly configure your Spark session to use AWS Lake Formation credentials. Add the following configurations when initializing your Spark session: ``` from pyspark.sql import SparkSession # Initialize Spark session with Lake Formation configurations spark = SparkSession.builder \ .appName("Lake Formation Full Table Access") \ .config("spark.sql.catalog.glue_catalog", "org.apache.spark.sql.catalog.hive.GlueCatalog") \ .config("spark.sql.catalog.glue_catalog.glue.lakeformation-enabled", "true") \ .config("spark.sql.defaultCatalog", "glue_catalog") \ .getOrCreate() ``` Key configurations: + `spark.sql.catalog.glue_catalog`: Registers a catalog named "glue\$1catalog" that uses the GlueCatalog implementation + `spark.sql.catalog.glue_catalog.glue.lakeformation-enabled`: Explicitly enables AWS Lake Formation integration for this catalog + The catalog name ("glue\$1catalog" in this example) can be customized, but must be consistent in both configuration settings #### Hive ``` ‐‐conf spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.AWS Glue.accesscontrol.AWSLakeFormationCredentialResolver --conf spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true --conf spark.hadoop.fs.s3.folderObject.autoAction.disabled=true --conf spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true --conf spark.sql.catalog.createDirectoryAfterTable.enabled=true --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true ``` #### Iceberg ``` --conf spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.AWS Glue.accesscontrol.AWSLakeFormationCredentialResolver --conf spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true --conf spark.hadoop.fs.s3.folderObject.autoAction.disabled=true --conf spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true --conf spark.sql.catalog.createDirectoryAfterTable.enabled=true --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true --conf spark.sql.catalog..AWS Glue.lakeformation-enabled=true ``` + `spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.AWS Glue.accesscontrol.AWSLakeFormationCredentialResolver`: Configure EMR Filesystem (EMRFS) to use AWS Lake Formation S3 credentials for AWS Lake Formation registered tables. If the table is not registered, use the job's runtime role credentials. + `spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true` and `spark.hadoop.fs.s3.folderObject.autoAction.disabled=true`: Configure EMRFS to use content type header application/x-directory instead of \$1folder\$1 suffix when creating S3 folders. This is required when reading AWS Lake Formation tables, as AWS Lake Formation credentials do not allow reading table folders with \$1folder\$1 suffix. + `spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true`: Configure Spark to skip validating the table location's emptiness before creation. This is necessary for AWS Lake Formation registered tables, as AWS Lake Formation credentials to verify the empty location are available only after AWS Glue Data Catalog table creation. Without this configuration, the job's runtime role credentials will validate the empty table location. + `spark.sql.catalog.createDirectoryAfterTable.enabled=true`: Configure Spark to create the Amazon S3 folder after table creation in the Hive metastore. This is required for AWS Lake Formation registered tables, as AWS Lake Formation credentials to create the Amazon S3 folder are available only after AWS Glue Data Catalog table creation. + `spark.sql.catalog.dropDirectoryBeforeTable.enabled=true`: Configure Spark to drop the Amazon S3 folder before table deletion in the Hive metastore. This is necessary for AWS Lake Formation registered tables, as AWS Lake Formation credentials to drop the S3 folder are not available after table deletion from the AWS Glue Data Catalog. + `spark.sql.catalog..AWS Glue.lakeformation-enabled=true`: Configure Iceberg catalog to use AWS Lake Formation Amazon S3 credentials for AWS Lake Formation registered tables. If the table is not registered, use default environment credentials. ## Usage Patterns ### Using FTA with DataFrames For users familiar with Spark, DataFrames can be used with AWS Lake Formation Full Table Access. AWS Glue 5.0 adds native Spark support for Lake Formation Full Table Access, simplifying how you work with protected tables. This feature enables AWS Glue 5.0 AWS Glue Spark jobs to directly read and write data when full table access is granted, removing limitations that previously restricted certain Extract, Transform, and Load (ETL) operations. You can now leverage advanced Spark capabilities including Resilient Distributed Datasets (RDDs), custom libraries, and User Defined Functions (UDFs) with AWS Lake Formation tables. #### Native Spark FTA in AWS Glue 5.0 AWS Glue 5.0 supports full-table access (FTA) control in Apache Spark based on your policies defined in AWS Lake Formation. This level of control is ideal for use cases that need to comply with security regulations at the table level. #### Apache Iceberg Table Example ``` from pyspark.sql import SparkSession catalog_name = "spark_catalog" aws_region = "us-east-1" aws_account_id = "123456789012" warehouse_path = "s3://amzn-s3-demo-bucket/warehouse/" spark = SparkSession.builder \ .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ .config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkSessionCatalog") \ .config(f"spark.sql.catalog.{catalog_name}.warehouse", f"{warehouse_path}") \ .config(f"spark.sql.catalog.{catalog_name}.client.region",f"{aws_region}") \ .config(f"spark.sql.catalog.{catalog_name}.glue.account-id",f"{aws_account_id}") \ .config(f"spark.sql.catalog.{catalog_name}.glue.lakeformation-enabled","true") \ .config(f"spark.sql.catalog.dropDirectoryBeforeTable.enabled", "true") \ .config(f"spark.sql.catalog.{catalog_name}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \ .config(f"spark.sql.catalog.{catalog_name}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \ .config("spark.sql.defaultCatalog", catalog_name) \ # Add this line .getOrCreate() database_name = "your_database" table_name = "your_table" df = spark.sql(f"select * from {database_name}.{table_name}") df.show() ``` #### Required IAM Permissions Your AWS Glue job execution role must have: ``` { "Action": "lakeformation:GetDataAccess", "Resource": "*", "Effect": "Allow" } ``` Plus appropriate S3 access permissions for your data locations. #### Lake Formation Configuration Before using native Spark FTA in AWS Glue 5.0: 1. Allow third-party query engines to access data without IAM session tag validation in AWS Lake Formation 1. Grant appropriate table permissions to your AWS Glue job execution role through AWS Lake Formation console 1. Configure your Spark session with the required parameters shown in the example above ### Using FTA with DynamicFrames AWS Glue's native DynamicFrames can be used with AWS Lake Formation Full Table Access for optimized ETL operations. Full Table Access (FTA) provides a security model that grants permissions at the table level, allowing for faster data processing compared to Fine-Grained Access Control (FGAC) since it bypasses the overhead of row and column-level permission checks. This approach is useful when you need to process entire tables and table-level permissions meet your security requirements. In AWS Glue 4.0, DynamicFrames with FTA required specific GlueContext configuration. While existing AWS Glue 4.0 DynamicFrame code with FTA will continue to work in AWS Glue 5.0, the newer version also offers native Spark FTA support with greater flexibility. For new development, consider using the native Spark approach described in the DataFrames section, especially if you need additional capabilities such as Resilient Distributed Datasets (RDDs), custom libraries, and User Defined Functions (UDFs) with AWS Lake Formation tables. #### Required Permissions The IAM role executing your Glue job must have: + `lakeformation:GetDataAccess` permission + Appropriate Lake Formation table permissions granted through the Lake Formation console #### Example DynamicFrame Implementation in AWS Glue 5.0 ``` from awsglue.context import GlueContext from pyspark.context import SparkContext # Initialize Glue context sc = SparkContext() glueContext = GlueContext(sc) # Configure catalog for Iceberg tables catalog_name = "glue_catalog" aws_region = "us-east-1" aws_account_id = "123456789012" warehouse_path = "s3://amzn-s3-demo-bucket/warehouse/" spark = glueContext.spark_session spark.conf.set(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") spark.conf.set(f"spark.sql.catalog.{catalog_name}.warehouse", f"{warehouse_path}") spark.conf.set(f"spark.sql.catalog.{catalog_name}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") spark.conf.set(f"spark.sql.catalog.{catalog_name}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") spark.conf.set(f"spark.sql.catalog.{catalog_name}.glue.lakeformation-enabled","true") spark.conf.set(f"spark.sql.catalog.{catalog_name}.client.region",f"{aws_region}") spark.conf.set(f"spark.sql.catalog.{catalog_name}.glue.id", f"{aws_account_id}") # Read Lake Formation-protected table with DynamicFrame df = glueContext.create_data_frame.from_catalog( database="your_database", table_name="your_table" ) ``` ## Additional Configuration ### Configure full table access mode in AWS Glue Studio notebooks To access AWS Lake Formation registered tables from interactive Spark sessions in AWS Glue Studio notebooks, you must use compatibility permission mode. Use the `%%configure` magic command to set up your Spark configuration before starting your interactive session. This configuration must be the first command in your notebook, as it cannot be applied after the session has started. Choose the configuration based on your table type: #### For Hive tables ``` %%configure --conf spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.glue.accesscontrol.AWSLakeFormationCredentialResolver --conf spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true --conf spark.hadoop.fs.s3.folderObject.autoAction.disabled=true --conf spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true --conf spark.sql.catalog.createDirectoryAfterTable.enabled=true --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true ``` #### For Iceberg tables ``` %%configure --conf spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.glue.accesscontrol.AWSLakeFormationCredentialResolver --conf spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true --conf spark.hadoop.fs.s3.folderObject.autoAction.disabled=true --conf spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true --conf spark.sql.catalog.createDirectoryAfterTable.enabled=true --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true --conf spark.sql.catalog.glue_catalog.glue.lakeformation-enabled=true --conf spark.sql.catalog.glue_catalog.warehouse=s3://example-s3-bucket_DATA_LOCATION --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.glue.account-id=ACCOUNT_ID --conf spark.sql.catalog.glue_catalog.glue.region=REGION ``` Replace the placeholders: + S3\$1DATA\$1LOCATION: *s3://amzn-s3-demo-bucket* + REGION: *AWS Region (e.g., us-east-1)* + ACCOUNT\$1ID: *Your AWS Account ID* **Note** You must set these configurations before executing any Spark operations in your notebook. ### Supported Operations These operations will use AWS Lake Formation credentials to access the table data. **Note** On enabling AWS Lake Formation: For FTA: Enable the Spark configuration `spark.sql.catalog.{catalog_name}.glue.lakeformation-enabled` + CREATE TABLE + ALTER TABLE + INSERT INTO + INSERT OVERWRITE + SELECT + UPDATE + MERGE INTO + DELETE FROM + ANALYZE TABLE + REPAIR TABLE + DROP TABLE + Spark datasource queries + Spark datasource writes **Note** Operations not listed above will continue to use IAM permissions to access table data. ## Migrating from AWS Glue 4.0 to AWS Glue 5.0 FTA When migrating from AWS Glue 4.0 GlueContext FTA to AWS Glue 5.0 native Spark FTA: 1. Allow third-party query engines to access data without the IAM session tag validation in AWS Lake Formation. Follow [Step 1: Enable Full Table Access in AWS Lake Formation](#security-access-control-fta-step-1). 1. You do not need to change the job runtime role. However, verify that the AWS Glue job execution role has lakeformation:GetDataAccess IAM permission. 1. Modify spark session configurations in the script. Ensure the following spark configurations are present: ``` --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.warehouse=s3:///warehouse/ --conf spark.sql.catalog.spark_catalog.client.region= --conf spark.sql.catalog.spark_catalog.glue.account-id=ACCOUNT_ID --conf spark.sql.catalog.spark_catalog.glue.lakeformation-enabled=true --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true ``` 1. Update script such that GlueContext DataFrames are changed to native spark DataFrames. 1. Update your AWS Glue job to use AWS Glue 5.0 ## Considerations and Limitations + If a Hive table is created using a job that doesn't have full table access enabled, and no records are inserted, subsequent reads or writes from a job with full table access will fail. This is because AWS Glue Spark without full table access adds the \$1folder\$1 suffix to the table folder name. To resolve this, you can either: + Insert at least one row into the table from a job that does not have FTA enabled. + Configure the job that does not have FTA enabled to not use \$1folder\$1 suffix in folder name in S3. This can be achieved by setting Spark configuration `spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true`. + Create an Amazon S3 folder at the table location `s3://path/to/table/table_name` using the Amazon S3 console or Amazon S3 CLI. + Full Table Access works exclusively with EMR Filesystem (EMRFS). S3A filesystem is not compatible. + Full Table Access is supported for Hive and Iceberg tables. Support for Hudi and Delta tables has not yet been added. + Jobs referencing tables with AWS Lake Formation Fine-Grained Access Control (FGAC) rules or AWS Glue Data Catalog Views will fail. To query a table with an FGAC rules or a AWS Glue Data Catalog View, you need to use the FGAC mode. You can enable FGAC mode by following the steps outlined in the AWS documentation: Using AWS Glue with AWS Lake Formation for fine-grained access control. + Full table access does not support Spark Streaming. + Cannot be used simultaneously with FGAC. # Using AWS Glue with AWS Lake Formation for fine-grained access control Lake Formation for FGAC ## Overview With AWS Glue version 5.0 and higher, you can leverage AWS Lake Formation to apply fine-grained access controls on Data Catalog tables that are backed by S3. This capability lets you configure table, row, column, and cell level access controls for read queries within your AWS Glue for Apache Spark jobs. See the following sections to learn more about Lake Formation and how to use it with AWS Glue. `GlueContext`-based table-level access control with AWS Lake Formation permissions supported in Glue 4.0 or before is not supported in Glue 5.0. Use the new Spark native fine-grained access control (FGAC) in Glue 5.0. Note the following details: + If you need fine grained access control (FGAC) for row/column/cell access control, you will need to migrate from `GlueContext`/Glue DynamicFrame in Glue 4.0 and prior to Spark dataframe in Glue 5.0. For examples, see [Migrating from GlueContext/Glue DynamicFrame to Spark DataFrame](security-lf-migration-spark-dataframes.md) + If you need Full Table Access control (FTA), you can leverage FTA with DynamicFrames in AWS Glue 5.0. You can also migrate to native Spark approach for additional capabilities such as Resilient Distributed Datasets (RDDs), custom libraries, and User Defined Functions (UDFs) with AWS Lake Formation tables. For examples, see [ Migrating from AWS Glue 4.0 to AWS Glue 5.0](https://docs.aws.amazon.com/glue/latest/dg/migrating-version-50.html). + If you don't need FGAC, then no migration to Spark dataframe is necessary and `GlueContext` features like job bookmarks, push down predicates will continue to work. + Jobs with FGAC require a minimum of 4 workers: one user driver, one system driver, one system executor, and one standby user executor. Using AWS Glue with AWS Lake Formation incurs additional charges. ## How AWS Glue works with AWS Lake Formation How it works Using AWS Glue with Lake Formation lets you enforce a layer of permissions on each Spark job to apply Lake Formation permissions control when AWS Glue executes jobs. AWS Glue uses [ Spark resource profiles](https://spark.apache.org/docs/latest/api/java/org/apache/spark/resource/ResourceProfile.html) to create two profiles to effectively execute jobs. The user profile executes user-supplied code, while the system profile enforces Lake Formation policies. For more information, see [What is AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) and [Considerations and limitations](https://docs.aws.amazon.com/glue/latest/dg/security-lf-enable-considerations.html). The following is a high-level overview of how AWS Glue gets access to data protected by Lake Formation security policies. ![\[The diagram shows how fine-grained access control works with the AWS Glue StartJobRun API.\]](http://docs.aws.amazon.com/glue/latest/dg/images/glue-50-fgac-start-job-run-api-diagram.png) 1. A user calls the `StartJobRun` API on an AWS Lake Formation-enabled AWS Glue job. 1. AWS Glue sends the job to a user driver and runs the job in the user profile. The user driver runs a lean version of Spark that has no ability to launch tasks, request executors, access S3 or the Glue Catalog. It builds a job plan. 1. AWS Glue sets up a second driver called the system driver and runs it in the system profile (with a privileged identity). AWS Glue sets up an encrypted TLS channel between the two drivers for communication. The user driver uses the channel to send the job plans to the system driver. The system driver does not run user-submitted code. It runs full Spark and communicates with S3, and the Data Catalog for data access. It request executors and compiles the Job Plan into a sequence of execution stages. 1. AWS Glue then runs the stages on executors with the user driver or system driver. User code in any stage is run exclusively on user profile executors. 1. Stages that read data from Data Catalog tables protected by AWS Lake Formation or those that apply security filters are delegated to system executors. ## Minimum worker requirement Minimum workers A Lake Formation-enabled job in AWS Glue requires a minimum of 4 workers: one user driver, one system driver, one system executor, and one standby User Executor. This is up from the minimum of 2 workers required for standard AWS Glue jobs. A Lake Formation-enabled job in AWS Glue utilizes two Spark drivers—one for the system profile and another for the user profile. Similarly, the executors are also divided into two profiles: + System executors: handle tasks where Lake Formation data filters are applied. + User executors: are requested by the system driver as needed. As Spark jobs are lazy in nature, AWS Glue reserves 10% of the total workers (minimum of 1), after deducting the two drivers, for user executors. All Lake Formation-enabled jobs have auto-scaling enabled, meaning the user executors will only start when needed. For an example configuration, see [Considerations and limitations](https://docs.aws.amazon.com/glue/latest/dg/security-lf-enable-considerations.html). ## Job runtime role IAM permissions Enable runtime permissions Lake Formation permissions control access to AWS Glue Data Catalog resources, Amazon S3 locations, and the underlying data at those locations. IAM permissions control access to the Lake Formation and AWS Glue APIs and resources. Although you might have the Lake Formation permission to access a table in the Data Catalog (SELECT), your operation fails if you don’t have the IAM permission on the `glue:Get*` API operation. The following is an example policy of how to provide IAM permissions to access a script in S3, uploading logs to S3, AWS Glue API permissions, and permission to access Lake Formation. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "ScriptAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::*.amzn-s3-demo-bucket/scripts", "arn:aws:s3:::*.amzn-s3-demo-bucket/*" ] }, { "Sid": "LoggingAccess", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/logs/*" ] }, { "Sid": "GlueCatalogAccess", "Effect": "Allow", "Action": [ "glue:Get*", "glue:Create*", "glue:Update*" ], "Resource": [ "*" ] }, { "Sid": "LakeFormationAccess", "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] } ``` ------ ## Setting up Lake Formation permissions for job runtime role Set up runtime permissions First, register the location of your Hive table with Lake Formation. Then create permissions for your job runtime role on your desired table. For more details about Lake Formation, see [ What is AWS Lake Formation?](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) in the *AWS Lake Formation Developer Guide*. After you set up the Lake Formation permissions, you can submit Spark jobs on AWS Glue. ## Submitting a job run After you finish setting up the Lake Formation grants, you can submit Spark jobs on AWS Glue. To run Iceberg jobs, you must provide the following Spark configurations. To configure through Glue job parameters, put the following parameter: + Key: ``` --conf ``` + Value: ``` spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.warehouse= --conf spark.sql.catalog.spark_catalog.glue.account-id= --conf spark.sql.catalog.spark_catalog.client.region= --conf spark.sql.catalog.spark_catalog.glue.endpoint=https://glue..amazonaws.com ``` ## Using an Interactive Session After you finish setting up the AWS Lake Formation grants, you can use Interactive Sessions on AWS Glue. You must provide the following Spark configurations via the `%%configure` magic prior to executing code. ``` %%configure { "--enable-lakeformation-fine-grained-access": "true", "--conf": "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.warehouse= --conf spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog.client.region= --conf spark.sql.catalog.spark_catalog.glue.account-id= --conf spark.sql.catalog.spark_catalog.glue.endpoint=https://glue..amazonaws.com" } ``` ## FGAC for AWS Glue 5.0 Notebook or interactive sessions To enable Fine-Grained Access Control (FGAC) in AWS Glue you must specify the Spark confs required for Lake Formation as part of the %%configure magic before you create first cell. Specifying it later using the calls `SparkSession.builder().conf("").get()` or `SparkSession.builder().conf("").create()` will not be enough. This is a change from the AWS Glue 4.0 behavior. ## Open-table format support Supported operations AWS Glue version 5.0 or later includes support for fine-grained access control based on Lake Formation. AWS Glue supports Hive and Iceberg table types. The following table describes all of the supported operations. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/security-lf-enable.html) # Migrating from GlueContext/Glue DynamicFrame to Spark DataFrame The following are Python and Scala examples of migrating `GlueContext`/Glue `DynamicFrame` in Glue 4.0 to Spark `DataFrame` in Glue 5.0. **Python** Before: ``` escaped_table_name= '``.``' additional_options = { "query": f'select * from {escaped_table_name} WHERE column1 = 1 AND column7 = 7' } # DynamicFrame example dataset = glueContext.create_data_frame_from_catalog( database="", table_name=escaped_table_name, additional_options=additional_options) ``` After: ``` table_identifier= '``.``.``"' #catalogname is optional # DataFrame example dataset = spark.sql(f'select * from {table_identifier} WHERE column1 = 1 AND column7 = 7') ``` **Scala** Before: ``` val escapedTableName = "``.``" val additionalOptions = JsonOptions(Map( "query" -> s"select * from $escapedTableName WHERE column1 = 1 AND column7 = 7" ) ) # DynamicFrame example val datasource0 = glueContext.getCatalogSource( database="", tableName=escapedTableName, additionalOptions=additionalOptions).getDataFrame() ``` After: ``` val tableIdentifier = "``.``.``" //catalogname is optional # DataFrame example val datasource0 = spark.sql(s"select * from $tableIdentifier WHERE column1 = 1 AND column7 = 7") ``` # Considerations and limitations Considerations Consider the following considerations and limitations when you use Lake Formation with AWS Glue. AWS Glue with Lake Formation is available in all supported Regions except AWS GovCloud (US-East) and AWS GovCloud (US-West). + AWS Glue supports fine-grained access control via Lake Formation only for Apache Hive and Apache Iceberg tables. Apache Hive formats include Parquet, ORC, and CSV. + You can only use Lake Formation with Spark jobs. + AWS Glue with Lake Formation only supports a single Spark session throughout a job. + When Lake Formation is enabled,AWS Glue requires a greater number of workers because it requires one system driver, system executors, one user driver, and optionally user executors (required when your job has UDFs or `spark.createDataFrame`). + AWS Glue with Lake Formation only supports cross-account table queries shared through resource links. The resource-link needs to be named identically to the source account's resource. + To enable fine-grained access control for AWS Glue jobs, pass the `--enable-lakeformation-fine-grained-access` job parameter. + You can configure your AWS Glue jobs to work with the AWS Glue multi-catalog hierarchy. For information on the configuration parameters to use with the AWS Glue `StartJobRun` API, see [Working with AWS Glue multi-catalog hierarchy on EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/external-metastore-glue-multi.html). + The following aren't supported: + Resilient distributed datasets (RDD) + Spark streaming + Write with Lake Formation granted permissions + Access control for nested columns + AWS Glue blocks functionalities that might undermine the complete isolation of system driver, including the following: + UDTs, HiveUDFs, and any user-defined function that involves custom classes + Custom data sources + Supply of additional jars for Spark extension, connector, or metastore + `ANALYZE TABLE` command + To enforce access controls, `EXPLAIN PLAN` and DDL operations such as `DESCRIBE TABLE` don't expose restricted information. + AWS Glue restricts access to system driver Spark logs on Lake Formation-enabled applications. Since the system driver runs with more access, events and logs that the system driver generates can include sensitive information. To prevent unauthorized users or code from accessing this sensitive data, AWS Glue disabled access to system driver logs. For troubleshooting, contact AWS support. + If you registered a table location with Lake Formation, the data access path goes through the Lake Formation stored credentials regardless of the IAM permission for the AWS Glue job runtime role. If you misconfigure the role registered with table location, jobs submitted that use the role with S3 IAM permission to the table location will fail. + Writing to a Lake Formation table uses IAM permission rather than Lake Formation granted permissions. If your job runtime role has the necessary S3 permissions, you can use it to run write operations. The following are considerations and limitations when using Apache Iceberg: + You can only use Apache Iceberg with session catalog and not arbitrarily named catalogs. + Iceberg tables that are registered in Lake Formation only support the metadata tables `history`, `metadata_log_entries`, `snapshots`, `files`, `manifests`, and `refs`. AWS Glue hides the columns that might have sensitive data, such as `partitions`, `path`, and `summaries`. This limitation doesn't apply to Iceberg tables that aren't registered in Lake Formation. + Tables that you don't register in Lake Formation support all Iceberg stored procedures. The `register_table` and `migrate` procedures aren't supported for any tables. + We recommend that you use Iceberg DataFrameWriterV2 instead of V1. ## Example worker allocation For a job configured with the following parameters: ``` --enable-lakeformation-fine-grained-access=true --number-of-workers=20 ``` The worker allocation would be: + One worker for the user driver. + One worker for the system driver. + 10% of the remaining 18 workers (that is, 2 workers) reserved for the user executors. + Up to 16 workers allocated for system executors. With auto-scaling enabled, the user executors can utilize any of the unallocated capacity from the system executors if needed. ## Controlling user executor allocation You can adjust the reservation percentage for user executors using the following configuration: ``` --conf spark.dynamicAllocation.maxExecutorsRatio= ``` This configuration allows fine-tuned control over how many user executors are reserved relative to the total available capacity. # Troubleshooting See the following sections for troubleshooting solutions. ## Logging AWS Glue uses Spark resources profiles to split job execution. AWS Glue uses the user profile to run the code you supplied, while the system profile enforces Lake Formation policies. You can access the logs for the tasks ran as the user profile. ## Live UI and Spark History Server The Live UI and the Spark History Server have all Spark events generated from the user profile and redacted events generated from the system driver. You can see all of the tasks from both the user and system drivers in the **Executors** tab. However, log links are available only for the user profile. Also, some information is redacted from Live UI, such as the number of output records. ## Job failed with insufficient Lake Formation permissions Make sure that your job runtime role has the permissions to run SELECT and DESCRIBE on the table that you are accessing. ## Job with RDD execution failed AWS Glue currently doesn't support resilient distributed dataset (RDD) operations on Lake Formation-enabled jobs. ## Unable to access data files in Amazon S3 Make sure you have registered the location of the data lake in Lake Formation. ## Security validation exception AWS Glue detected a security validation error. Contact AWS support for assistance. ## Sharing AWS Glue Data Catalog and tables across accounts You can share databases and tables across accounts and still use Lake Formation. For more information, see [Cross-account data sharing in Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/cross-account-permissions.html) and [How do I share AWS Glue Data Catalog and tables cross-account using ?](https://repost.aws/knowledge-center/glue-lake-formation-cross-account). The following table describes summary of how to choose between Fine-grained access control (FGAC) and Full table access (FTA) for your workload. | Feature | Fine-grained access control (FGAC) | Full table access (FTA) | | --- |--- |--- | | Access Level | Column/row level | Full table | | Use Case | Queries and ETL with limited permissions | ETL | | Performance Impact | Requires system/user space transitions for access control evaluation, adding latency | Optimized performance | # Using Amazon S3 Access Grants with AWS Glue With Glue version 5.0, Amazon S3 Access Grants provide a scalable access control solution that you can use to augment access to your Amazon S3 data from AWS Glue. If you have a complex or large permission configuration for your S3 data, you can use S3 Access Grants to scale S3 data permissions for users and roles. Use S3 Access Grants to augment access to Amazon S3 data beyond the permissions that are granted by the runtime role or the IAM roles that are attached to the identities with access to your AWS Glue job. For more information, see [Managing access with S3 Access Grants](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants.html) in the *Amazon S3 User Guide*. ## How AWS Glue works with S3 Access Grants AWS Glue versions 5.0 and higher provide a native integration with S3 Access Grants. You can enable S3 Access Grants on AWS Glue and run Spark jobs. When a Spark job makes a request for S3 data, Amazon S3 provides temporary credentials that are scoped to the specific bucket, prefix, or object. The following is a high-level overview of how AWS Glue gets access to data that S3 Access Grants manages access to. ![\[The diagram shows a high-level overview of how AWS Glue gets access to data that S3 Access Grants manages access to.\]](http://docs.aws.amazon.com/glue/latest/dg/images/glue50-s3-access-grants.png) 1. A user submits an AWS Glue Spark job that uses data stored in Amazon S3. 1. AWS Glue makes a request for S3 Access Grants to vend temporary credentials for the user that give access to the bucket, prefix, or object. 1. AWS Glue returns temporary credentials in the form of an AWS Security Token Service (STS) token for the user. The token is scoped to access the S3 bucket, prefix, or object. 1. AWS Glue uses the STS token to retrieve data from S3. 1. AWS Glue receives the data from S3 and returns the results to the user. ## S3 Access Grants considerations with AWS Glue Take note of the following behaviors and limitations when you use S3 Access Grants with AWS Glue. **Feature support** + S3 Access Grants is supported with AWS Glue versions 5.0 and higher. + Spark is the only supported job type when you use S3 Access Grants with AWS Glue. + Delta Lake and Hudi are the only supported open-table formats when you use S3 Access Grants with AWS Glue. + The following capabilities are not supported for use with S3 Access Grants: + Apache Iceberg tables + AWS CLI requests to Amazon S3 that use IAM roles + S3 access through the open-source S3A protocol **Behavioral considerations** + AWS Glue provides a credentials cache to ensure that a user doesn't need to make repeated requests for the same credentials within a Spark job. Therefore, AWS Glue always requests the default-level privilege when it requests credentials. For more information, see [Request access to S3 data](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants-credentials.html) in the *Amazon S3 User Guide*. ## Set up S3 Access Grants with AWS Glue **Prerequisites** The caller or admin has created an S3 Access Grants instance. **Set up AWS Glue policies and job configuration** To set up S3 Access Grants with AWS Glue you must configure trust and IAM policies, and pass the configuration through job parameters. 1. Configure the following minimal trust and IAM policies on the role used for grants (the AWS Glue role that runs sessions or jobs). Trust policy: ``` { "Sid": "Stmt1234567891011", "Action": [ "sts:AssumeRole", "sts:SetSourceIdentity", "sts:SetContext" ], "Effect": "Allow", "Principal": { "Service": "access-grants.s3.amazonaws.com" }, "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012", "aws:SourceArn": "arn:aws:s3::123456789012:access-grants/default" } } } ``` IAM policy: ``` { "Sid": "S3Grants", "Effect": "Allow", "Action": [ "s3:GetDataAccess", "s3:GetAccessGrantsInstanceForPrefix" ], "Resource": "arn:aws:s3::123456789012:access-grants/default" }, { "Sid": "BucketLevelReadPermissions", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "123456789012" }, "ArnEquals": { "s3:AccessGrantsInstanceArn": [ "arn:aws:s3::123456789012:access-grants/default" ] } } }, { "Sid": "ObjectLevelReadPermissions", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl", "s3:GetObjectVersionAcl", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "123456789012" }, "ArnEquals": { "s3:AccessGrantsInstanceArn": [ "arn:aws:s3::123456789012:access-grants/default" ] } } } ``` 1. In your AWS Glue job, pass the following Spark configuration either through AWS Glue job parameters or `SparkConf`. ``` --conf spark.hadoop.fs.s3.s3AccessGrants.enabled=true \ --conf spark.hadoop.fs.s3.s3AccessGrants.fallbackToIAM=false ``` # Trusted Identity Propagation with AWS Glue ETL Trusted Identity Propagation With IAM Identity Center, you can connect to identity providers (IdPs) and centrally manage access for users and groups across AWS analytics services. You can integrate identity providers such as Okta, Ping, and Microsoft Entra ID (formerly Azure Active Directory) with IAM Identity Center for users in your organization to access data using a single-sign on experience. IAM Identity Center also supports connecting additional third-party identity providers. With AWS Glue 5.0 and higher, you can propagate user-identities from IAM Identity Center to AWS Glue interactive sessions. AWS Glue Interactive Sessions will further propagate supplied identity to downstream services such as Amazon S3 Access Grants, AWS Lake Formation, and Amazon Redshift, enabling secure data access via user identity in these downstream services. ## Overview [Identity Center](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) is the recommended approach for workforce authentication and authorization on AWS for organizations of any size and type. With Identity Center, you can create and manage user identities in AWS, or connect your existing identity source, including Microsoft Active Directory, Okta, Ping Identity, JumpCloud, Google Workspace, and Microsoft Entra ID (formerly Azure AD). [ Trusted identity propagation](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-overview.html) is an IAM Identity Center feature that administrators of connected AWS services can use to grant and audit access to service data. Access to this data is based on user attributes such as group associations. Setting up trusted identity propagation requires collaboration between the administrators of connected AWS services and the IAM Identity Center administrators. ## Features and benefits The AWS Glue interactive sessions integration with IAM Identity Center [ Trusted identity propagation ](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-overview.html) provides the following benefits: + The ability to enforce table-level authorization and fine grained access control with Identity Center identities on Lake Formation managed AWS Glue data catalog tables. + The ability to enforce authorization with Identity Center identities on Amazon Redshift clusters. + Enables end to end tracking of user actions for auditing. + The ability to enforce Amazon S3 prefix-level authorization with Identity Center identities on Amazon S3 Access Grants-managed Amazon S3 prefixes. ## Use cases **Interactive Data Exploration and Analysis** Data engineers use their corporate identities to seamlessly access and analyze data across multiple AWS accounts. Through SageMaker Studio, they launch interactive Spark sessions via AWS Glue ETL, connecting to various data sources including Amazon S3 and the AWS Glue Data Catalog. As engineers explore datasets, Spark enforces fine-grained access controls defined in Lake Formation based on their identities, ensuring they can only view authorized data. All queries and data transformations are logged with the user's identity, creating a clear audit trail. This streamlined approach enables rapid prototyping of new analytics products while maintaining strict data governance across client environments. **Data Preparation and Feature Engineering** Data scientists from multiple research teams collaborate on complex projects using a unified data platform. They log into SageMaker Studio with their corporate credentials, immediately accessing a vast, shared data lake that spans multiple AWS accounts. As they begin feature engineering for new machine learning models, Spark sessions launched through AWS Glue ETL enforce Lake Formation's column and row-level security policies based on their propagated identities. Scientists can efficiently prepare data and engineer features using familiar tools, while compliance teams have assurance that every data interaction is automatically tracked and audited. This secure, collaborative environment accelerates research pipelines while maintaining the strict data protection standards required in regulated industries. ## How it works ![\[Architecture diagram showing AWS Glue Interactive Sessions workflow. A user logs into client-facing applications (SageMaker Unified Studio, or custom applications) through IAM Identity Center. The user's identity is propagated to AWS Glue Interactive Sessions, which connects to access control services including IAM Identity Center, AWS Lake Formation, AWS Glue Data Catalog, and Amazon S3 Access Grant, before finally accessing S3 Storage.\]](http://docs.aws.amazon.com/glue/latest/dg/images/GlueISSMAI.png) A user logs into client-facing applications (SageMaker AI, or custom applications) using their corporate identity through IAM Identity Center. This identity is then propagated through the entire data access pipeline. The authenticated user launches AWS AWS Glue Interactive Sessions, which serve as the compute engine for data processing. These sessions maintain the user's identity context throughout the workflow. AWS Lake Formation and the AWS Glue Data Catalog work together to enforce fine-grained access controls. Lake Formation applies security policies based on the user's propagated identity, while Amazon S3 Access Grant provides additional permission layers, ensuring users can only access data they're authorized to view. Finally, the system connects to Amazon S3 Storage where the actual data resides. All access is governed by the combined security policies, maintaining data governance while enabling interactive data exploration and analysis. This architecture enables secure, identity-based data access across multiple AWS services while maintaining a seamless user experience for data scientists and engineers working with large datasets. ## Integrations ### AWS managed development environment The following AWS managed client-facing applications support trusted identity propagation with AWS Glue interactive sessions: + [Sagemaker Unified Studio](https://aws.amazon.com/sagemaker/unified-studio/) + [Amazon SageMaker AI](https://aws.amazon.com/sagemaker-ai/) **Sagemaker Unified Studio** To use trusted identity propagation with Sagemaker Unified Studio: 1. Set up Sagemaker Unified Studio project with trusted identity propagation enabled as the client-facing development environment. 1. Set up [ Lake Formation](https://docs.aws.amazon.com/en_us/singlesignon/latest/userguide/tip-tutorial-lf.html) to enable fine-grained access control for AWS Glue tables based on the user or group in IAM Identity Center. 1. [Set up Amazon S3 Access Grants ](https://docs.aws.amazon.com/en_us/singlesignon/latest/userguide/tip-tutorial-s3.html) to enable temporary access to the underlying data locations in Amazon S3. 1. Open Sagemaker Unified Studio JupyterLab IDE space and select AWS Glue as compute for notebook execution. ### Customer managed self-hosted Notebook environment To enable trusted identity propagation for users of custom-developed applications, see [ Access AWS services programmatically using trusted identity propagation ](https://aws.amazon.com/blogs/security/access-aws-services-programmatically-using-trusted-identity-propagation/) in the AWS Security Blog. # Getting started with trusted identity propagation in AWS Glue ETL This section helps you configure AWS Glue application with interactive sessions to integrate with IAM Identity Center and enable [Trusted identity propagation](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-overview.html). ## Prerequisites + An Identity Center instance in the AWS region where you want to create Trusted identity propagation enabled AWS Glue interactive sessions. An Identity Center instance can only exist in a single region for an AWS account. For more information, see [ Enable IAM Identity Center ](https://docs.aws.amazon.com/singlesignon/latest/userguide/get-started-enable-identity-center.html) and [ provision the users and groups from your source of identities into IAM Identity Center ](https://docs.aws.amazon.com/singlesignon/latest/userguide/tutorials.html). + Enable Trusted identity propagation for downstream services such as Lake Formation or Amazon S3 Access Grants or Amazon Redshift cluster with which interactive workload interacts to access data. ## Permissions needed to connect AWS Glue ETL with IAM Identity Center **Create an IAM role** The role that creates IAM Identity Center connection requires permissions to create and modify application configuration in AWS Glue and IAM Identity Center as in the following inline policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:CreateGlueIdentityCenterConfiguration", "sso:CreateApplication", "sso:PutApplicationAssignmentConfiguration", "sso:PutApplicationAuthenticationMethod", "sso:PutApplicationGrant", "sso:PutApplicationAccessScope", "sso:ListInstances" ], "Resource": [ "*" ] } ] } ``` ------ The following inline policies contain specific permissions required to view, update, and delete properties of AWS Glue integration with IAM Identity Center. Use the following inline policy to allow an IAM role to view a AWS Glue integration with IAM Identity Center. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetGlueIdentityCenterConfiguration" ], "Resource": [ "*" ] } ] } ``` ------ Use the following inline policy to allow an IAM role to update AWS Glue integration with IAM Identity Center. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:UpdateGlueIdentityCenterConfiguration", "sso:PutApplicationAccessScope", "sso:DeleteApplicationAccessScope" ], "Resource": [ "*" ] } ] } ``` ------ Use the following inline policy to allow an IAM role to delete a AWS Glue integration with IAM Identity Center. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:DeleteGlueIdentityCenterConfiguration", "sso:DeleteApplication" ], "Resource": [ "*" ] } ] } ``` ------ ### Permissions description + `glue:CreateGlueIdentityCenterConfiguration` – Grants permission to create the AWS Glue IdC configuration. + `glue:GetGlueIdentityCenterConfiguration` – Grants permission to get an existing IdC configuration. + `glue:DeleteGlueIdentityCenterConfiguration` – Grants permission to delete an existing AWS Glue IdC configuration. + `glue:UpdateGlueIdentityCenterConfiguration` – Grants permission to update an existing AWS Glue IdC configuration. + `sso:CreateApplication` – Grants permission to create a AWS Glue managed IAM Identity Center application. + `sso:DescribeApplication` - Grants permission to describe a AWS Glue managed IAM Identity Center application. + `sso:DeleteApplication` – Grants permission to delete a AWS Glue managed IAM Identity Center application. + `sso:UpdateApplication` – Grants permission to update a AWS Glue managed IAM Identity Center application. + `sso:PutApplicationGrant` – Grants permission to apply token-exchange, introspectToken, refreshToken and RevokeToken grants on IdC Application. + `sso:PutApplicationAuthenticationMethod` – Grants permission to put authenticationMethod on AWS Glue managed IdC Application that allows AWS Glue service principal to interact with IdC Application. + `sso:PutApplicationAccessScope` – Grants permission to add or update the list of authorized down stream service scopes on the AWS Glue managed IdC application. + `sso:DeleteApplicationAccessScope` - Grants permission to delete downstream scopes if a scope is removed for the AWS Glue managed IdC application. + `sso:PutApplicationAssignmentConfiguration` – Grants permission to set "User-assignment-not-required" setting on IdC Application. + `sso:ListInstances` – Grants permission to list instances and validate the IdC InstanceArn that you specify in identity-center-configuration parameter. ## Connecting AWS Glue with IAM Identity Center When AWS Glue is connected to IAM Identity Center, it creates a singleton managed IdC application per account. The following example shows how you can connect AWS Glue with IAM Identity Center: ``` aws glue create-glue-identity-center-configuration \ --instance-arn arn:aws:sso:::instance/ssoins-123456789 \ --scopes '["s3:access_grants:read_write", "redshift:connect","lakeformation:query"]' ``` To update the scopes of the managed application (usually done to propagate to more downstream services), you can use: ``` aws glue update-glue-identity-center-configuration \ --scopes '["s3:access_grants:read_write", "redshift:connect","lakeformation:query"]' ``` Scopes parameter is optional and all scopes will be added if not provided. The supported values are `s3:access_grants:read_write`, `redshift:connect` and `lakeformation:query`. To get the details of the configuration, you can use: ``` aws glue get-glue-identity-center-configuration ``` You can delete the connection between AWS Glue and IAM Identity Center by using the following command: ``` aws glue delete-glue-identity-center-configuration ``` **Note** AWS Glue creates a service managed Identity Center Application in your account that service leverages for identity validations and identity propagation to downstream services. AWS Glue created managed Identity Center Application is shared across all trusted-identity-propagation sessions in your account. **Warning:** Do not manually modify settings on the managed Identity Center Application. Any changes could affect all trusted-identity-propagation enabled AWS Glue interactive sessions in your account. ## Creating a AWS Glue Interactive Session with Trusted Identity Propagation Enabled After you connect AWS Glue with IAM Identity Center, you can use [ identity-enhanced role credentials](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-identity-enhanced-iam-role-sessions.html) to create a AWS Glue interactive session. You need not pass additional parameters when creating a 5.0 AWS Glue session. Since AWS Glue is connected with IAM identity center, if AWS Glue detects identity-enhanced-role-credentials, it will automatically propagate the identity information to downstream services which are called as part of your statements. However, the runtime role for the session needs to have the `sts:SetContext` permission as depicted below. **Runtime Role permissions to propagate identity** As AWS Glue sessions leverage [ Identity-enhanced credentials ](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-identity-enhanced-iam-role-sessions.html) to propagate identity to downstream AWS services, its runtime role's trust-policy need to have addition permission `sts:SetContext` to allow identity propagation to downstream services (Amazon S3 access-grant, Lake Formation, Amazon Redshift). To learn more about how to create a runtime role, see [Setting up a runtime role](https://docs.aws.amazon.com/glue/latest/dg/create-service-role.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com" }, "Action": [ "sts:AssumeRole", "sts:SetContext" ] } ] } ``` ------ Additionally, Runtime role would need permissions for downstream AWS services which job-run would invoke to fetch data using user identity. Please refer to the following links to configure Amazon S3 Access Grants and Lake Formation: + [Using Lake Formation with AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/security-lake-formation-fgac.html) + [Using Amazon S3 Access Grants with AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/security-s3-access-grants.html) # Considerations and limitations for AWS Glue ETL Trusted Identity Propagation integration **Important** By default sessions are not private which means one IdC user can access another IdC user's session. You can use [tagOnCreate](https://docs.aws.amazon.com/glue/latest/dg/glue-is-security.html#glue-is-tagoncreate) to make your sessions private. For example, the session can be tagged with an owner tag and the value of it as IDC User Id and then on the policy, you can use a global condition key like [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-identity-store-user-id](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-identity-store-user-id) to validate against the owner tag in the client principal/runtime role policy for all session API operations to ensure that one IdC User isn't able to access another IdC user's session. Consider the following points when you use IAM Identity Center Trusted Identity Propagation with AWS Glue Application: + Trusted Identity Propagation through Identity Center is supported on AWS Glue 5.0 and higher, and only with AWS Glue interactive sessions. + AWS Glue data catalog is covered under Lake Formation identity center integration. + Trusted Identity Propagation is limited to interactive sessions in AWS Glue, excluding other data processing entities like jobs, triggers, workflows, and ML tasks. All AWS Glue APIs, however, record user identities in AWS CloudTrail for auditing. + AWS Glue currently supports integration with IAM Identity Center exclusively through API and CLI interfaces, not via the console. + Once an application is enabled on AWS Glue side, make sure to create 5.0 sessions with IdC Credentials but don't create a 4.0 session with IdC credentials. + Trusted Identity Propagation with AWS Glue is supported in the following AWS Regions: + af-south-1 – Africa (Cape Town) + ap-east-1 – Asia Pacific (Hong Kong) + ap-northeast-1 – Asia Pacific (Tokyo) + ap-northeast-2 – Asia Pacific (Seoul) + ap-northeast-3 – Asia Pacific (Osaka) + ap-south-1 – Asia Pacific (Mumbai) + ap-southeast-1 – Asia Pacific (Singapore) + ap-southeast-2 – Asia Pacific (Sydney) + ap-southeast-3 – Asia Pacific (Jakarta) + ca-central-1 – Canada (Central) + eu-central-1 – Europe (Frankfurt) + eu-north-1 – Europe (Stockholm) + eu-south-1 – Europe (Milan) + eu-west-1 – Europe (Ireland) + eu-west-2 – Europe (London) + eu-west-3 – Europe (Paris) + me-south-1 – Middle East (Bahrain) + sa-east-1 – South America (São Paulo) + us-east-1 – US East (N. Virginia) + us-east-2 – US East (Ohio) + us-west-1 – US West (N. California) + us-west-2 – US West (Oregon) # User background sessions for AWS Glue ETL User background sessions User background sessions enable long-running analytics and machine learning workloads to continue even after the user has logged off from their notebook interface. This capability is implemented through AWS Glue's trusted identity propagation feature. The following page explains the configuration options and behaviors for user background sessions. **Note** User background sessions apply to AWS Glue interactive sessions initiated through notebook interfaces like SageMaker Unified Studio. Enabling or disabling this feature affects only new interactive sessions; existing active sessions are not impacted. ## Configure user background sessions User background sessions must be enabled at two levels for proper functionality: 1. IAM Identity Center instance level (configured by IdC administrators) 1. AWS Glue Identity Center configuration level (configured by AWS Glue administrators) ### Enable user background sessions for AWS Glue To enable user background sessions for AWS Glue, you must set the `userBackgroundSessionsEnabled` parameter to `true` in the Identity Center configuration when creating or updating the configuration. Prerequisites + Your IAM role that is used to create/update the AWS Glue Identity Center configuration must have the `sso:PutApplicationSessionConfiguration` permission. This permission allows AWS Glue to enable user background sessions at the AWS Glue-managed IdC application level. + Your AWS Glue interactive sessions must use AWS Glue version 5.0 or later and must be Trusted Identity Propagation enabled. To enable user background sessions using the AWS CLI: ``` aws glue create-glue-identity-center-configuration \ --instance-arn "arn:aws:sso:::instance/ssoins-1234567890abcdef" \ --user-background-sessions-enabled ``` To update an existing configuration: ``` aws glue update-glue-identity-center-configuration \ --user-background-sessions-enabled ``` #### Configuration matrix The effective user background session configuration depends on both the AWS Glue configuration setting and the IAM Identity Center instance-level settings: | IAM Identity Center userBackgroundSession Enabled? | AWS Glue userBackgroundSessionsEnabled | Behavior | | --- | --- | --- | | Yes | TRUE | User background sessions enabled | | Yes | FALSE | Session expires with user logout | | No | TRUE | Session creation fails with Exception | | No | FALSE | Session expires with user logout | ## Default user background session duration By default, all user background sessions have a duration limit of 7 days in IAM Identity Center. Administrators can modify this duration in the IAM Identity Center console. This setting applies at the IAM Identity Center instance level, affecting all supported IAM Identity Center applications within that instance. + Duration can be set to any value from 15 minutes up to 90 days + This setting is configured in the IAM Identity Center console under Settings → Authentication → Configure (Non-Interactive Jobs section) **Note** AWS Glue interactive sessions have a separate idle timeout limit of 48 hours by default. Sessions will terminate when either the AWS Glue session idle timeout or the user background session duration is reached, whichever comes first. ## Impact of disabling user background sessions When user background sessions are disabled at the AWS Glue configuration level: + **Existing interactive sessions:** Continue to run without interruption if they were started with user background sessions enabled. These sessions will continue using their existing background session tokens until they terminate naturally or are explicitly stopped. + **New interactive sessions:** Will use the standard trusted identity propagation flow and will terminate when the user logs out or their interactive session expires (such as when closing a SageMaker Unified Studio JupyterLab notebook). ### Changing user background sessions duration When the duration setting for user background sessions is modified in IAM Identity Center: + **Existing interactive sessions:** Continue to run with the same background session duration with which they were started + **New interactive sessions:** Will use the new session duration for background sessions ## Runtime considerations ### Session termination conditions When using user background sessions, a AWS Glue interactive session will continue running until one of the following occurs: + The user background session expires (based on IdC configuration, up to 90 days) + The user background session is manually revoked by an administrator + The AWS Glue interactive session reaches its idle timeout (default: 48 hours after the last executed statement) + The user explicitly stops or restarts the notebook kernel ### Data persistence When using user background sessions: + Users cannot reconnect to their notebook interface to view results once they have logged out + Configure your Spark statements to write results to persistent storage (such as Amazon S3) before execution completes ### Cost implications + Jobs will continue to run to completion even after users end their SageMaker Unified Studio JupyterLab session and will incur charges for the entire duration of the completed run + Monitor your active background sessions to avoid unnecessary costs from forgotten or abandoned sessions ### Feature availability User background sessions for AWS Glue are available for: + AWS Glue interactive sessions only (AWS Glue jobs and streaming jobs are not supported) + AWS Glue version 5.0 and later + Trusted Identity Propagation enabled configurations only # Logging and monitoring in AWS Glue Logging and monitoring You can automate the running of your ETL (extract, transform, and load) jobs. AWS Glue provides metrics for crawlers and jobs that you can monitor. After you set up the AWS Glue Data Catalog with the required metadata, AWS Glue provides statistics about the health of your environment. You can automate the invocation of crawlers and jobs with a time-based schedule based on cron. You can also trigger jobs when an event-based trigger fires. AWS Glue is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or AWS service in AWS Glue. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon Simple Storage Service (Amazon S3) bucket, Amazon CloudWatch Logs, and Amazon CloudWatch Events. Every event or log entry contains information about who generated the request. With AWS Glue version 5.1, audit context for AWS Glue and AWS Lake Formation API calls will be available in the AWS CloudTrail logs. Use Amazon CloudWatch Events to automate your AWS services and respond automatically to system events such as application availability issues or resource changes. Events from AWS services are delivered to CloudWatch Events in near-real time. You can write simple rules to indicate which events are of interest and what automated actions to take when an event matches a rule. **See also** [Automating AWS Glue with EventBridge](automating-awsglue-with-cloudwatch-events.md) [Cross-account CloudTrail logging](cross-account-access.md#cross-account-ct-logs) An important facet of security in the cloud is logging. You must configure logging in a way that does not capture secrets and confidential material while capturing information necessary to debug and secure your cloud infastructure. Make sure to familizarize yourself with what is being logged. # Compliance validation for AWS Glue Compliance validation To learn whether an AWS service is within the scope of specific compliance programs, see [AWS services in Scope by Compliance Program](https://aws.amazon.com/compliance/services-in-scope/) and choose the compliance program that you are interested in. For general information, see [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/). You can download third-party audit reports using AWS Artifact. For more information, see [Downloading Reports in AWS Artifact](https://docs.aws.amazon.com/artifact/latest/ug/downloading-documents.html). Your compliance responsibility when using AWS services is determined by the sensitivity of your data, your company's compliance objectives, and applicable laws and regulations. For more information about your compliance responsibility when using AWS services, see [AWS Security Documentation](https://docs.aws.amazon.com/security/). # Resilience in AWS Glue Resilience The AWS global infrastructure is built around AWS Regions and Availability Zones. AWS Regions provide multiple physically separated and isolated Availability Zones, which are connected with low-latency, high-throughput, and highly redundant networking. With Availability Zones, you can design and operate applications and databases that automatically fail over between Availability Zones without interruption. Availability Zones are more highly available, fault tolerant, and scalable than traditional single or multiple data center infrastructures. For more information about AWS Regions and Availability Zones, see [AWS Global Infrastructure](https://aws.amazon.com/about-aws/global-infrastructure/). For more information about AWS Glue job resiliency, see [Error: Failover behavior between VPCs in AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/glue-troubleshooting-errors.html#vpc-failover-behavior-error-10). # Infrastructure security in AWS Glue Infrastructure security As a managed service, AWS Glue is protected by the AWS global network security procedures that are described in the [Amazon Web Services: Overview of Security Processes](https://d0.awsstatic.com/whitepapers/Security/AWS_Security_Whitepaper.pdf) whitepaper. You use AWS published API calls to access AWS Glue through the network. Clients must support Transport Layer Security (TLS) 1.0 or later. We recommend TLS 1.2 or later. Clients must also support cipher suites with perfect forward secrecy (PFS) such as Ephemeral Diffie-Hellman (DHE) or Elliptic Curve Ephemeral Diffie-Hellman (ECDHE). Most modern systems such as Java 7 and later support these modes. Additionally, requests must be signed by using an access key ID and a secret access key that is associated with an IAM principal. Or you can use the [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html) (AWS STS) to generate temporary security credentials to sign requests. **Topics** + [ # Configuring interface VPC endpoints (AWS PrivateLink) for AWS Glue (AWS PrivateLink) ](vpc-interface-endpoints.md) + [ # Configuring shared Amazon VPCs ](shared-vpc.md) # Configuring interface VPC endpoints (AWS PrivateLink) for AWS Glue (AWS PrivateLink) Configuring interface VPC endpoints (AWS PrivateLink) for AWS Glue You can establish a private connection between your VPC and AWS Glue by creating an *interface VPC endpoint*. Interface endpoints are powered by [AWS PrivateLink](https://aws.amazon.com/privatelink), a technology that enables you to privately access AWS Glue APIs without an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Instances in your VPC don't need public IP addresses to communicate with AWS Glue APIs. Traffic between your VPC and AWS Glue does not leave the Amazon network. Each interface endpoint is represented by one or more [Elastic Network Interfaces](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) in your subnets. For more information, see [Interface VPC endpoints (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html) in the *Amazon VPC User Guide*. ## Considerations for AWS Glue VPC endpoints Before you set up an interface VPC endpoint for AWS Glue, ensure that you review [Interface endpoint properties and limitations](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#vpce-interface-limitations) in the *Amazon VPC User Guide*. AWS Glue supports making calls to all of its API actions from your VPC. ## Creating an interface VPC endpoint for AWS Glue You can create a VPC endpoint for the AWS Glue service using either the Amazon VPC console or the AWS Command Line Interface (AWS CLI). For more information, see [Creating an interface endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) in the *Amazon VPC User Guide*. Create a VPC endpoint for AWS Glue using the following service name: + com.amazonaws.*region*.glue If you enable private DNS for the endpoint, you can make API requests to AWS Glue using its default DNS name for the Region, for example, `glue.us-east-1.amazonaws.com`. For more information, see [Accessing a service through an interface endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#access-service-though-endpoint) in the *Amazon VPC User Guide*. ## Creating a VPC endpoint policy for AWS Glue You can attach an endpoint policy to your VPC endpoint that controls access to AWS Glue. The policy specifies the following information: + The principal that can perform actions. + The actions that can be performed. + The resources on which actions can be performed. For more information, see [Controlling access to services with VPC endpoints](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-access.html) in the *Amazon VPC User Guide*. **Example: VPC endpoint policy for AWS Glue to allow job creation and update** The following is an example of an endpoint policy for AWS Glue. When attached to an endpoint, this policy grants access to the listed AWS Glue actions for all principals on all resources. ``` { "Statement": [ { "Sid": "RestrictPassRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::123456789012:role/GlueServiceRole*", "Condition": { "StringEquals": { "iam:PassedToService": "glue.amazonaws.com" } } } ] } ``` **Example: VPC endpoint policy to allow read-only Data Catalog access** The following is an example of an endpoint policy for AWS Glue. When attached to an endpoint, this policy grants access to the listed AWS Glue actions for all principals on all resources. ``` { "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:GetTableVersion", "glue:GetTableVersions", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:SearchTables" ], "Resource": "*" } ] } ``` # Configuring shared Amazon VPCs AWS Glue supports shared virtual private clouds (VPCs) in Amazon Virtual Private Cloud. Amazon VPC sharing allows multiple AWS accounts to create their application resources, such as Amazon EC2 instances and Amazon Relational Database Service (Amazon RDS) databases, into shared, centrally-managed Amazon VPCs. In this model, the account that owns the VPC (owner) shares one or more subnets with other accounts (participants) that belong to the same organization from AWS Organizations. After a subnet is shared, the participants can view, create, modify, and delete their application resources in the subnets that are shared with them. In AWS Glue, to create a connection with a shared subnet, you must create a security group within your account and attach the security group to the shared subnet. For more information, see these topics: + [Working with Shared VPCs](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-sharing.html) in the *Amazon VPC User Guide* + [What Is AWS Organizations? ](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) in the *AWS Organizations User Guide*