# Connecting to SendGrid
SendGrid is a customer communication platform for transactional and marketing emails.
+ SendGrid connector helps in creating and managing contact lists and creating email marketing campaigns.
+ SendGrid allows online businesses, non-profits, and other online entities to create and send marketing emails to large audiences and monitor engagement with those emails.
**Topics**
+ [AWS Glue support for SendGrid](sendgrid-support.md)
+ [Policies containing the API operations for creating and using connections](sendgrid-configuring-iam-permissions.md)
+ [Configuring SendGrid](sendgrid-configuring.md)
+ [Configuring SendGrid connections](sendgrid-configuring-connections.md)
+ [Reading from SendGrid entities](sendgrid-reading-from-entities.md)
+ [SendGrid connection options](sendgrid-connection-options.md)
+ [SendGrid limitations](sendgrid-limitations.md)
# AWS Glue support for SendGrid
AWS Glue supports SendGrid as follows:
**Supported as a source?**
Yes. You can use AWS Glue ETL jobs to query data from SendGrid.
**Supported as a target?**
No.
**Supported SendGrid API versions**
The following SendGrid API versions are supported:
+ v3
# Policies containing the API operations for creating and using connections
The following sample policy describes the required AWS IAM permissions for creating and using connections. If you are creating a new role, create a policy that contains the following:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"glue:ListConnectionTypes",
"glue:DescribeConnectionType",
"glue:RefreshOAuth2Tokens",
"glue:ListEntities",
"glue:DescribeEntity"
],
"Resource": "*"
}
]
}
```
------
If you don't want to use the above method, alternatively use the following managed IAM policies:
+ [AWSGlueServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints.
+ [AWSGlueConsoleFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console.
# Configuring SendGrid
Before you can use AWS Glue to transfer data from SendGrid, you must meet these requirements:
## Minimum requirements
The following are minimum requirements:
+ You have a SendGrid account with an API key.
+ Your SendGrid account has API access with a valid license.
If you meet these requirements, you’re ready to connect AWS Glue to your SendGrid account. For typical connections, you don't need do anything else in SendGrid.
# Configuring SendGrid connections
SendGrid supports custom authentication.
For public SendGrid documentation on generating the required API keys for custom authentication, see [Authentication](https://www.twilio.com/docs/sendgrid/api-reference/how-to-use-the-sendgrid-v3-api/authentication).
To configure a SendGrid connection:
1. In AWS Secrets Manager, create a secret with the following details:
1. For the customer managed connected app, the Secret should contain the connected app Consumer Secret with `api_key` as key.
1. Note: you must create a secret for your connections in AWS Glue.
1. In AWS Glue Glue Studio, create a connection under **Data Connections** by following the steps below:
1. When selecting a **Connection type**, select SendGrid.
1. Provide the `INSTANCE_URL` of the SendGrid instance you want to connect to.
1. Select the AWS IAM role which AWS Glue can assume and has permissions for following actions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "*"
}
]
}
```
------
1. Select the `secretName` which you want to use for this connection in AWS Glue to put the tokens.
1. Select the network options if you want to use your network.
1. Grant the IAM role associated with your AWS Glue job permission to read `secretName`.
# Reading from SendGrid entities
**Prerequisite**
A SendGrid object you would like to read from. You will need the object name such as `lists`, `singlesends` or `segments`.
**Supported entities for source**:
| Entity | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partitioning |
| --- | --- | --- | --- | --- | --- |
| Lists | No | Yes | No | Yes | No |
| Single Sends | Yes | Yes | No | Yes | No |
| Marketing Campaign Stats-Automations | Yes | Yes | No | Yes | No |
| Marketing Campaign Stats-Single Sends | Yes | Yes | No | Yes | No |
| Segments | Yes | No | No | Yes | No |
| Contacts | Yes | No | No | Yes | No |
| Category | No | No | No | Yes | No |
| Stats | Yes | No | No | Yes | No |
| Unsubscribe Groups | Yes | No | No | Yes | No |
**Example**:
```
sendgrid_read = glueContext.create_dynamic_frame.from_options(
connection_type="sendgrid",
connection_options={
"connectionName": "connectionName",
"ENTITY_NAME": "lists",
"API_VERSION": "v3",
"INSTANCE_URL": "instanceUrl"
}
```
**SendGrid entity and field details**:
Entities with static metadata:
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/sendgrid-reading-from-entities.html)
**Note**
Struct and List data types are converted to String data type, and DateTime data type is converted to Timestamp in the response of the connectors.
## Partitioning queries
SendGrid doesn't support filter-based partitioning or record-based partitioning.
# SendGrid connection options
The following are connection options for SendGrid:
+ `ENTITY_NAME`(String) - (Required) Used for Read. The name of your object in SendGrid.
+ `API_VERSION`(String) - (Required) Used for Read. SendGrid Rest API version you want to use.
+ `INSTANCE_URL`(String) - (Required) Used for Read. A valid SendGrid instance URL.
+ `SELECTED_FIELDS`(List) - Default: empty(SELECT \$1). Used for Read. Columns you want to select for the object.
+ `FILTER_PREDICATE`(String) - Default: empty. Used for Read. It should be in the Spark SQL format.
+ `QUERY`(String) - Default: empty. Used for Read. Full Spark SQL query.
# SendGrid limitations
The following are limitations or notes for SendGrid:
+ Incremental pull is only supported by the Stats entity on the `start_date` field and by the Contact entity on the `event_timestamp` field.
+ Pagination is only supported by the Marketing Campaign Stats (Automations), Marketing Campaign Stats (Single Sends), Single Sends, and Lists entities.
+ For the Stats entity, `start_date` is a mandatory filter parameter.
+ An API key with Restricted Access can’t support read access for the Email API and Stats entities. Use an API key with Full Access. For more information, see [API Overview](https://www.twilio.com/docs/sendgrid/api-reference/api-keys/create-api-keys#api-overview).