Reading from Twilio entities - AWS Glue
Partitioning queries

Reading from Twilio entities

Prerequisite

A Twilio object you would like to read from. You will need the object name such as SMS-Message or SMS-CountryPricing.

Supported entities for source:

Entity Interface Can be filtered Supports limit Supports Order by Supports Select * Supports partitioning
SMS-Message REST Yes Yes No Yes Yes
SMS-CountryPricing REST No No No Yes No
Voice-Call REST Yes Yes No Yes No
Voice-Application REST Yes Yes No Yes No
Voice-OutgoingCallerID REST Yes Yes No Yes No
Voice-Queue REST Yes Yes No Yes No
Conversations-Conversation REST Yes Yes No Yes No
Conversations-User REST No Yes No Yes No
Conversations-Role REST No Yes No Yes No
Conversations-Configuration REST No No No Yes No
Conversations-AddressConfiguration REST Yes Yes No Yes No
Conversations-WebhookConfiguration REST No No No Yes No
Conversations-ParticipantConversation REST No No No Yes No
Conversations-Credential REST No Yes No Yes No
Conversations-ConversationService REST No Yes No Yes No

Example:

twilio_read = glueContext.create_dynamic_frame.from_options(
    connection_type="twilio",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "sms-message",
        "API_VERSION": "2010-04-01",
        "Edge_Location": "sydney.us1"
    }

Twilio entity and field details:

Entity Field Data type Supported operators
SMS-Message account_sid String N/A
api_version String N/A
body String N/A
date_created Datetime N/A
date_sent Datetime >=, <=, =
date_updated Datetime N/A
direction String N/A
error_code Integer N/A
error_message String N/A
from Integer =
messaging_service_sid String N/A
num_media String N/A
num_segments String N/A
price String N/A
price_unit Struct N/A
sid Integer N/A
status String N/A
subresource_uris Map N/A
to Integer =
uri Datetime N/A
SMS-CountryPricing country String N/A
iso_country String N/A
url String N/A
outbound_sms_prices List N/A
inbound_sms_prices List N/A
price_unit String N/A
Voice-Call account_sid String N/A
annotation String N/A
answered_by String N/A
api_version String N/A
caller_name String N/A
date_created Datetime N/A
date_updated Datetime N/A
direction String N/A
duration String N/A
end_time Datetime >=, <=, =
forwarded_from String N/A
from String =
from_formatted String N/A
group_sid String N/A
parent_call_sid String N/A
phone_number_sid String N/A
price String N/A
price_unit String N/A
sid String N/A
start_time Datetime >=, <=, =
status String =
subresource_uris String N/A
to String =
to_formatted String N/A
trunk_sid String N/A
uri String N/A
queue_time String N/A
Voice-Application account_sid String N/A
api_version String N/A
date_created Datetime N/A
date_updated Datetime N/A
friendly_name String =
message_status_callback String N/A
sid String N/A
sms_fallback_method String N/A
sms_fallback_url String N/A
sms_method String N/A
sms_status_callback String N/A
sms_url String N/A
status_callback String N/A
status_callback_method String N/A
uri String N/A
voice_caller_id_lookup Boolean N/A
voice_fallback_method String N/A
voice_fallback_url String N/A
voice_method String N/A
voice_url String N/A
public_application_connect_enabled Boolean N/A
Voice-OutgoingCallerID sid String N/A
date_created Datetime N/A
date_updated Datetime N/A
account_sid String N/A
friendly_name String =
phone_number String =
uri String N/A
Voice-Queue date_created Datetime N/A
date_updated Datetime N/A
current_size Integer N/A
friendly_name String N/A
uri String N/A
account_sid String N/A
average_wait_time Integer N/A
sid String N/A
max_size Integer N/A
Conversations-Conversation account_sid String N/A
chat_service_sid String N/A
messaging_service_sid String N/A
sid String N/A
friendly_name String N/A
unique_name String N/A
attributes String N/A
state String =
date_created Datetime N/A
date_updated Datetime N/A
timers Struct N/A
url String N/A
links Struct N/A
bindings Struct N/A
start_date Datetime =
end_date Datetime =
Timers.DateInactive String N/A
Timers.DateClosed String N/A
Conversations-User sid String N/A
account_sid String N/A
chat_service_sid String N/A
role_sid String N/A
identity String N/A
friendly_name String N/A
attributes String N/A
is_online Boolean N/A
is_notifiable Boolean N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
links Struct N/A
Conversations-Role sid String N/A
account_sid String N/A
chat_service_sid String N/A
friendly_name String N/A
type String N/A
permissions String N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
Conversations-Configuration account_sid Long N/A
default_chat_service_sid String N/A
default_messaging_service_sid String N/A
default_inactive_timer String N/A
default_closed_timer String N/A
url String N/A
links Map N/A
Conversations-AddressConfiguration sid String N/A
account_sid String N/A
type String N/A
address String N/A
friendly_name String N/A
auto_creation Struct N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
address_country String N/A
AutoCreation.Enabled Boolean N/A
AutoCreation.Type String N/A
AutoCreation.ConversationServiceSid String N/A
AutoCreation.WebhookUrl String N/A
AutoCreation.WebhookMethod String N/A
AutoCreation.WebhookFilters List N/A
AutoCreation.StudioFlowSid String N/A
AutoCreation.StudioRetryCount Integer N/A
Conversations-WebhookConfiguration account_sid String N/A
method String N/A
filters List N/A
pre_webhook_url String N/A
post_webhook_url String N/A
target String N/A
url String N/A
Converations-ParticipantConversation account_sid String N/A
chat_service_sid String N/A
participant_sid String N/A
participant_user_sid String N/A
participant_identity String N/A
participant_messaging_binding Struct N/A
Conversation_sid String N/A
conversation_unique_name String N/A
conversation_friendly_name String N/A
conversation_attributes String N/A
conversation_date_created Datetime N/A
conversation_date_updated Datetime N/A
conversation_created_by String N/A
conversation_state String N/A
conversation_timers Struct N/A
links Map N/A
address String =
identity String =
Conversation-Credentials sid String N/A
account_sid String N/A
friendly_name String N/A
type String N/A
sandbox String N/A
date_created Datetime N/A
dated_updated Datetime N/A
url String N/A
certificate String N/A
private_key String N/A
api_key String N/A
secret String N/A
Conversations-ConversationService sid String N/A
account_sid String N/A
friendly_name String N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
links Map N/A

Partitioning queries

Fields supporting partitioning:

In Twilio, the DateTime datatype fields support field-based partitioning.

You can provide the additional Spark options PARTITION_FIELD, LOWER_BOUND, UPPER_BOUND, and NUM_PARTITIONS if you want to utilize concurrency in Spark. With these parameters, the original query would be split into NUM_PARTITIONS number of sub-queries that can be executed by Spark tasks concurrently.

  • PARTITION_FIELD: the name of the field to be used to partition the query.

  • LOWER_BOUND: an inclusive lower bound value of the chosen partition field.

    For the Datetime field, we accept the Spark timestamp format used in Spark SQL queries.

    Examples of valid value:

    "2024-05-01T20:55:02.000Z"

  • UPPER_BOUND: an exclusive upper bound value of the chosen partition field.

  • NUM_PARTITIONS: the number of partitions.

Example:

twilio_read = glueContext.create_dynamic_frame.from_options(
    connection_type="twilio",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "sms-message",
        "API_VERSION": "2010-04-01",
        "PARTITION_FIELD": "date_sent"
        "LOWER_BOUND": "2024-05-01T20:55:02.000Z"
        "UPPER_BOUND": "2024-06-01T20:55:02.000Z"
        "NUM_PARTITIONS": "10"
    }