Reading from Salesforce Marketing Cloud entities - AWS Glue
Partitioning queries

Reading from Salesforce Marketing Cloud entities

Prerequisite

A Salesforce Marketing Cloud object you would like to read from. You will need the object name such as Activity or Campaigns. The following table shows the supported entities.

Supported entities for source:

Entity Interface Can be filtered Supports limit Supports Order by Supports SELECT * Supports partitioning
Event Notification Callback REST No No No Yes No
Seed-List REST No Yes No Yes No
Setup REST Yes Yes No Yes No
Domain Verification REST Yes Yes Yes Yes No
Objects Nested Tags REST Yes No No Yes No
Contact REST No Yes No Yes No
Event Notification Subscription REST No No No Yes No
Messaging REST No Yes No Yes No
Activity SOAP No No No Yes Yes
Bounce Event SOAP No No No Yes Yes
Click Event SOAP No No No Yes Yes
Content Area SOAP No No No Yes Yes
Data Extension SOAP No Yes No Yes Yes
Email SOAP No Yes No Yes Yes
Forwarded Email Event SOAP No Yes No Yes Yes
Forward Email OptInEvent SOAP No Yes No Yes Yes
Link SOAP No Yes No Yes Yes
Link Send SOAP No Yes No Yes Yes
List SOAP No Yes No Yes Yes
List Subscriber SOAP No Yes No Yes Yes
Not Sent Event SOAP No Yes No Yes Yes
Open Event SOAP No Yes No Yes Yes
Send SOAP No Yes No Yes Yes
Sent Event SOAP No Yes No Yes Yes
Subscriber SOAP No Yes No Yes Yes
Survey Event SOAP No Yes No Yes Yes
Unsub Event SOAP No Yes No Yes Yes
Audit Events REST No Yes Yes Yes No
Campaigns REST No Yes Yes Yes No
Interactions REST No Yes Yes Yes No
Content Assets REST No Yes Yes Yes No

Example for REST:

salesforcemarketingcloud_read = glueContext.create_dynamic_frame.from_options(
    connection_type="salesforcemarketingcloud",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "Campaigns",
        "API_VERSION": "v1",
        "INSTANCE_URL": "https://**********************.rest.marketingcloudapis.com"
    }
)

Example for SOAP:

salesforcemarketingcloud_read = glueContext.create_dynamic_frame.from_options(
    connection_type="salesforcemarketingcloud",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "Activity",
        "API_VERSION": "v1",
        "INSTANCE_URL": "https://**********************.soap.marketingcloudapis.com"
    }
)

Salesforce Marketing Cloud entity and field details:

The following tables describe the Salesforce Marketing Cloud entities. There are REST entities with static metadata and SOAP entities with dynamic metadata.

REST entities with static metadata:

Entity Field Data type Supported operators
Event Notification Callback callbackId String
callbackName String
url String
maxBatchSize Integer
status String
statusReason String
Seed-List id String
name String
description String
activeSeedCount Integer
Setup customerKey String
name String
description String
locationType String '='
awsFileTransferLocation Struct
Domain verification enterpriseId Integer
status String '='
domainType String '='
memberId Integer
emailSendTime DateTime
domain String
isSendable Boolean
Objects Nested Tags id Integer
modifiedDate DateTime
tags List
name String
description String
parentId Integer
Contact values List
Event Notification Subscription subscriptionName String
callbackId String
callbackName String
eventCategoryTypes List
filters List
url String
maxBatchSize Integer
subscriptionId String
status String
statusReason String
Messaging deliveryTime DateTime
id String
messageId String
status String
to Struct
Interactions status String '='
id String
key String
name String
lastPublishedDate DateTime
description String
version Integer
workflowApiVersion Integer
createdDate DateTime
modifiedDate DateTime
goals Struct
stats Struct
entryMode String
defaults Struct
executionMode Struct
definitionId String
Content Assets id Integer
customerKey String
objectId String
contentType String
assetType Struct
name String
description String
owner Struct
createdDate DateTime
createdBy Struct
modifiedDate DateTime
modifiedBy Struct
thumbnail Struct
category Struct
meta Struct
views Struct
availableViews Struct
data Struct
legacyData Struct
modelVersion Integer
Version Integer
Locked Boolean
FileProperties Struct
Tags List
Content String
Design String
SuperContent String
CustomFields Struct
Blocks Struct
MinBlocks Integer
MaxBlocks Integer
Channels Struct
AllowedBlocks List
Slots Struct
BusinessUnitAvailability Struct
sharingProperties Struct
sharingProperties.sharedWith Struct
sharingProperties.sharingType String
Template Struct
File String
GenerateFrom String
Audit Events id Integer
createdDate DateTime
memberId Integer
enterpriseId Integer
employee Struct
objectType Struct
operation Struct
object Struct
transactionId String
Campaigns id Integer
createdDate DateTime
modifiedDate DateTime
name String
description String
campaignCode String
color String
favorite Boolean

SOAP entities with dynamic metadata:

Entity Data type Supported operators
Activity String LIKE,!=,=
Struct
Integer !=,=,>=,<=,<,>
Double !=,=,>=,<=,<,>
Boolean !=,=
DateTime >=,<=,<,>,=,BETWEEN
Bounce Event Integer !=,=,>=,<=,<,>
DateTime >=,<=,<,>,=,BETWEEN
String LIKE,!=,=
Struct
Click Event Integer !=,=,>=,<=,<,>
DateTime >=,<=,<,>,=,BETWEEN
String LIKE,!=,=
Struct
Content Area Struct
String LIKE,!=,=
Integer !=,=,>=,<=,<,>
DateTime >=,<=,<,>,=,BETWEEN
Boolean !=,=
Data Extension DateTime >=,<=,<,>,=,BETWEEN
String LIKE,!=,=
Email Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Boolean !=,=
Struct
Forwarded Email Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Forwarded Email OptInEvent Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Link Integer !=,=,>=,<=,<,>
Link Send Integer !=,=,>=,<=,<,>
String LIKE,!=,=
Double !=,=,>=,<=,<,>
List Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
List Subscriber Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Not Sent Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Open Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Send Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Boolean !=,=
Struct
Sent Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Subscriber Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Survey Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Struct
Unsub Event Integer !=,=,>=,<=,<,>
String LIKE,!=,=
DateTime >=,<=,<,>,=,BETWEEN
Boolean !=,=
Struct

Partitioning queries

In Salesforce Marketing Cloud, the Integer and DateTime datatype fields support field-based partitioning.

You can provide the additional Spark options PARTITION_FIELD, LOWER_BOUND, UPPER_BOUND, and NUM_PARTITIONS if you want to utilize concurrency in Spark. With these parameters, the original query would be split into NUM_PARTITIONS number of sub-queries that can be executed by Spark tasks concurrently.

  • PARTITION_FIELD: the name of the field to be used to partition the query.

  • LOWER_BOUND: an inclusive lower bound value of the chosen partition field.

    For the timestamp field, we accept the Spark timestamp format used in Spark SQL queries.

    Examples of valid value:

    “2024-05-07T02:03:00.00Z"

  • UPPER_BOUND: an exclusive upper bound value of the chosen partition field.

  • NUM_PARTITIONS: the number of partitions.

Example:

salesforcemarketingcloud_read = glueContext.create_dynamic_frame.from_options(
    connection_type="salesforcemarketingcloud",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "ListSubscriber",
        "API_VERSION": "v1",
        "PARTITION_FIELD": "CreatedDate",
        "LOWER_BOUND": "2023-09-07T02:03:00.000Z",
        "UPPER_BOUND": "2024-05-07T02:03:00.000Z",
        "NUM_PARTITIONS": "10"
    }
)