Reading from Pipedrive entities - AWS Glue
Partitioning queries

Reading from Pipedrive entities

Prerequisites

  • A Pipedrive Object you would like to read from. Refer the supported entities table below to check the available entities.

Supported entities

Entity Can be Filtered Supports Limit Supports Order By Supports Select * Supports Partitioning
Activities Yes Yes No Yes Yes
Activity Type No No No Yes No
Call Logs No No No Yes No
Currencies Yes Yes No Yes No
Deals Yes Yes Yes Yes Yes
Leads Yes Yes Yes Yes No
Lead Sources No Yes No Yes No
Lead Labels No No No No No
Notes Yes Yes Yes Yes Yes
Organization Yes Yes No Yes Yes
Permission Sets Yes No No Yes No
Persons Yes Yes Yes Yes Yes
Pipelines No Yes No Yes No
Products Yes Yes No Yes Yes
Roles No Yes No Yes No
Stages Yes Yes No Yes No
Users No No No Yes No

Example 

pipedrive_read= glueContext.create_dynamic_frame.from_options(
    connection_type="PIPEDRIVE",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "activites",
        "API_VERSION": "v1"
    }

Pipedrive entity and field details

Entities list:

Entity Data Type Supported Operators
Activities, Deals, Notes, Organization, Persons and Products. Date '='
Integer '='
String '='
Boolean '='

Partitioning queries

In Pipedrive, only one field (due_date) from Activities entity supports field-based partitioning. It is a Date field.

Additional spark options PARTITION_FIELD, LOWER_BOUND, UPPER_BOUND, NUM_PARTITIONS can be provided if you want to utilize concurrency in Spark. With these parameters, the original query would be split into NUM_PARTITIONS number of sub-queries that can be executed by spark tasks concurrently.

  • PARTITION_FIELD: the name of the field to be used to partition query.

  • LOWER_BOUND: an inclusive lower bound value of the chosen partition field.

    For date, we accept the Spark date format used in Spark SQL queries. Example of valid values: "2024-02-06".

  • UPPER_BOUND: an exclusive upper bound value of the chosen partition field.

  • NUM_PARTITIONS: number of partitions.

Example 

pipedrive_read = glueContext.create_dynamic_frame.from_options(
    connection_type="PIPEDRIVE",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "activites",
        "API_VERSION": "v1",
        "PARTITION_FIELD": "due_date"
        "LOWER_BOUND": "2023-09-07T02:03:00.000Z"
        "UPPER_BOUND": "2024-05-07T02:03:00.000Z"
        "NUM_PARTITIONS": "10"
    }