Reading from WooCommerce entities - AWS Glue
Partitioning queries

Reading from WooCommerce entities

Prerequisite

A WooCommerce object you would like to read from. You will need the object name such as coupon, order, product, etc.

Supported entities for source:

Entity Can be filtered Supports limit Supports Order by Supports Select * Supports partitioning
Coupon Yes Yes Yes Yes Yes
Coupon Total No No No Yes No
Customers Total No No No Yes No
Order Yes Yes Yes Yes Yes
Orders Total No No No Yes No
Payment Gateway No No No Yes No
Product Yes Yes Yes Yes Yes
Product attribute Yes Yes Yes Yes Yes
Product category Yes Yes Yes Yes Yes
Product review Yes Yes Yes Yes Yes
Product shipping class Yes Yes Yes Yes Yes
Product tag Yes Yes Yes Yes Yes
Product variation Yes Yes Yes Yes Yes
Products Total No No No Yes No
Report (List) No No No Yes No
Reviews Total No No No Yes No
Sales Report Yes No No Yes No
Shipping Method No No No Yes No
Shipping Zone No No No Yes No
Shipping Zone Location No No No Yes No
Shipping Zone Method No No No Yes No
Tax Rate Yes Yes Yes Yes Yes
Tax Class No No No Yes No
Top Sellers Report Yes No No Yes No

Example:

woocommerce_read = glueContext.create_dynamic_frame.from_options(
    connection_type="glue.spark.woocommerce",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "coupon",
        "API_VERSION": "v3",
        "INSTANCE_URL": "instanceUrl"
    }

WooCommerce entity and field details:

Entity Field Data type Supported operators
coupon id Integer N/A
code String EQUAL_TO
amount String N/A
status String N/A
date_created DateTime N/A
date_created_gmt DateTime N/A
date_modified DateTime N/A
date_modified_gmt DateTime N/A
discount_type String N/A
description String N/A
date_expires String N/A
date_expires_gmt String N/A
usage_count Integer N/A
individual_use Boolean N/A
product_ids List N/A
excluded_product_ids List N/A
usage_limit Integer N/A
usage_limit_per_user Integer N/A
limit_usage_to_x_items Integer N/A
free_shipping Boolean N/A
product_categories List N/A
excluded_product_categories List N/A
exclude_sale_items Boolean N/A
minimum_amount String N/A
maximum_amount String N/A
email_restrictions List N/A
used_by List N/A
meta_data List N/A
context String EQUAL_TO
search String EQUAL_TO
after DateTime EQUAL_TO
before DateTime EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
modified_after DateTime EQUAL_TO
modified_before DateTime EQUAL_TO
dates_are_gmt Boolean EQUAL_TO
coupon-total slug String N/A
name String N/A
total Integer N/A
customer-total slug String N/A
name String N/A
total Integer N/A
order id Integer N/A
parent_id Integer N/A
number String N/A
order_key String N/A
created_via String N/A
status String N/A
currency String N/A
version String N/A
date_created DateTime N/A
date_modified DateTime N/A
discount_total String N/A
discount_tax String N/A
shipping_total String N/A
shipping_tax String N/A
cart_tax String N/A
total String N/A
total_tax String N/A
prices_include_tax Boolean N/A
customer_id Integer N/A
customer_ip_address String N/A
customer_user_agent String N/A
customer_note String N/A
billing Struct N/A
shipping Struct N/A
payment_method String N/A
payment_method_title String N/A
transaction_id String N/A
date_paid DateTime N/A
date_completed DateTime N/A
cart_hash String N/A
meta_data List N/A
line_items List N/A
tax_lines List N/A
shipping_lines List N/A
fee_lines List N/A
coupon_lines List N/A
refunds List N/A
payment_url String N/A
is_editable Boolean N/A
needs_payment Boolean N/A
needs_processing Boolean N/A
date_created_gmt DateTime N/A
date_modified_gmt DateTime N/A
date_completed_gmt DateTime N/A
date_paid_gmt DateTime N/A
currency_symbol String N/A
set_paid Boolean N/A
context String EQUAL_TO
search String EQUAL_TO
after DateTime EQUAL_TO
before DateTime EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
customer Integer EQUAL_TO
product Integer EQUAL_TO
dp Integer EQUAL_TO
modified_before DateTime EQUAL_TO
modified_after DateTime EQUAL_TO
dates_are_gmt Boolean EQUAL_TO
order-total slug String N/A
name String N/A
total Integer N/A
payment-gateway title String N/A
description String N/A
order String N/A
enabled Boolean N/A
method_title String N/A
method_description String N/A
method_supports List N/A
settings String N/A
needs_setup Boolean N/A
post_install_scripts List N/A
settings_url String N/A
connection_url String N/A
setup_help_text String N/A
required_settings_keys List N/A
product id Integer N/A
name String N/A
type String EQUAL_TO
permalink String N/A
date_created DateTime N/A
date_created_gmt DateTime N/A
date_modified DateTime N/A
date_modified_gmt DateTime N/A
catalog_visibility String N/A
description String N/A
short_description String N/A
price String N/A
regular_price String N/A
sale_price String N/A
date_on_sale_from DateTime N/A
date_on_sale_from_gmt DateTime N/A
date_on_sale_to DateTime N/A
date_on_sale_to_gmt DateTime N/A
price_html String N/A
purchasable Boolean N/A
total_sales Integer N/A
virtual Boolean N/A
downloadable Boolean N/A
downloads List N/A
download_limit Integer N/A
download_expiry Integer N/A
external_url String N/A
button_text String N/A
tax_status String N/A
manage_stock Boolean N/A
stock_quantity Integer N/A
backorders String N/A
backorders_allowed Boolean N/A
backordered Boolean N/A
sold_individually Boolean N/A
weight String N/A
dimensions Struct N/A
shipping_required Boolean N/A
shipping_taxable Boolean N/A
shipping_class_id Integer N/A
reviews_allowed Boolean N/A
average_rating String N/A
rating_count Integer N/A
related_ids List N/A
upsell_ids List N/A
cross_sell_ids List N/A
parent_id Integer N/A
purchase_note String N/A
categories List N/A
tags List N/A
images List N/A
attributes List N/A
default_attributes List N/A
variations List N/A
grouped_products List N/A
menu_order Integer N/A
meta_data List N/A
low_stock_amount Integer N/A
jetpack_publicize_connections List N/A
jetpack-related-posts List N/A
jetpack_likes_enabled Boolean N/A
jetpack_sharing_enabled Boolean N/A
context String EQUAL_TO
search String EQUAL_TO
after DateTime EQUAL_TO
before DateTime EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
slug String EQUAL_TO
status String EQUAL_TO
sku String EQUAL_TO
featured Boolean EQUAL_TO
tag String EQUAL_TO
shipping_class String EQUAL_TO
tax_class String EQUAL_TO
on_sale Boolean EQUAL_TO
stock_status String EQUAL_TO
has_options Boolean N/A
modified_after DateTime EQUAL_TO
modified_before DateTime EQUAL_TO
dates_are_gmt Boolean EQUAL_TO
category String EQUAL_TO
attribute String EQUAL_TO
min_price String EQUAL_TO
max_price String EQUAL_TO
product-attribute id Integer N/A
name String N/A
slug String N/A
type String N/A
order_by String N/A
has_archives Boolean N/A
context String EQUAL_TO
product-attribute-term id Integer N/A
name String N/A
slug String N/A
description String N/A
menu_order Integer N/A
count Integer N/A
context String EQUAL_TO
search String EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
hide_empty Boolean EQUAL_TO
parent Integer EQUAL_TO
product Integer EQUAL_TO
product-category id Integer N/A
name String N/A
slug String EQUAL_TO
description String N/A
display String N/A
image Struct N/A
menu_order Integer N/A
count Integer N/A
context String EQUAL_TO
search String EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
hide_empty Boolean EQUAL_TO
parent Integer EQUAL_TO
product Integer EQUAL_TO
product-review id Integer N/A
date_created DateTime N/A
date_created_gmt DateTime N/A
product_id Integer N/A
product_name String N/A
product_permalink String N/A
review String N/A
rating Integer N/A
verified Boolean N/A
reviewer String N/A
reviewer_email String N/A
reviewer_avatar_urls Struct N/A
context String EQUAL_TO
search String EQUAL_TO
after DateTime EQUAL_TO
before DateTime EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
status String EQUAL_TO
product-shipping-class id Integer N/A
name String N/A
slug String EQUAL_TO
description String N/A
count Integer N/A
context String EQUAL_TO
search String EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
hide_empty String EQUAL_TO
product Integer EQUAL_TO
product-tag id Integer N/A
name String N/A
slug String EQUAL_TO
description String N/A
count Integer N/A
context String EQUAL_TO
search String EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
hide_empty Boolean EQUAL_TO
product Integer EQUAL_TO
product-total slug String N/A
name String N/A
total Integer N/A
product-variation id Integer N/A
date_created DateTime N/A
date_created_gmt DateTime N/A
date_modified DateTime N/A
date_modified_gmt DateTime N/A
description String N/A
permalink String N/A
price String N/A
regular_price String N/A
sale_price String N/A
date_on_sale_from DateTime N/A
date_on_sale_from_gmt DateTime N/A
date_on_sale_to DateTime N/A
date_on_sale_to_gmt DateTime N/A
purchasable Boolean N/A
virtual Boolean N/A
downloadable Boolean N/A
downloads List N/A
download_limit Integer N/A
download_expiry Integer N/A
tax_status String N/A
manage_stock Boolean N/A
stock_quantity Integer N/A
backorders String N/A
backorders_allowed Boolean N/A
backordered Boolean N/A
low_stock_amount Integer N/A
weight String N/A
dimensions Struct N/A
shipping_class String N/A
shipping_class_id Integer N/A
image Struct N/A
attributes List N/A
menu_order Integer N/A
meta_data List N/A
context String EQUAL_TO
search String EQUAL_TO
after DateTime EQUAL_TO
before DateTime EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
slug String EQUAL_TO
status String EQUAL_TO
sku String EQUAL_TO
tax_class String EQUAL_TO
on_sale Boolean EQUAL_TO
min_price String EQUAL_TO
max_price String EQUAL_TO
stock_status String EQUAL_TO
report slug String N/A
description String N/A
review-total slug String N/A
name String N/A
total Integer N/A
sales-report total_sales String N/A
net_sales String N/A
average_sales String N/A
total_orders Integer N/A
total_items Integer N/A
total_tax String N/A
total_shipping String N/A
total_refunds Integer N/A
total_discount String N/A
totals_grouped_by String N/A
totals Struct N/A
total_customers Integer N/A
context String EQUAL_TO
period String EQUAL_TO
date_min Date EQUAL_TO
date_max Date EQUAL_TO
shipping-method id String N/A
title String N/A
description String N/A
shipping-zone id Integer EQUAL_TO
name String N/A
order Integer N/A
shipping-zone-location code String N/A
type String N/A
shipping-zone-method instance_id Integer N/A
id Integer EQUAL_TO
title String N/A
order Integer N/A
enabled Boolean N/A
method_id String N/A
method_title String N/A
method_description String N/A
settings Struct N/A
tax-class slug String N/A
name String N/A
tax-rate id Integer N/A
country String N/A
state String N/A
postcode String N/A
city String N/A
postcodes List N/A
cities List N/A
rate String N/A
name String N/A
priority Integer N/A
compound Boolean N/A
shipping Boolean N/A
context String EQUAL_TO
order String EQUAL_TO
orderby String EQUAL_TO
class String EQUAL_TO
top-seller-report name String N/A
product_id Integer N/A
quantity Integer N/A
context String EQUAL_TO
period String EQUAL_TO
date_min Date EQUAL_TO
date_max Date EQUAL_TO
Note

Struct and List data types are converted to String data type, and DateTime data type is converted to Timestamp in the response of the connectors.

Partitioning queries

Record-based partitioning:

You can provide the additional Spark option NUM_PARTITIONS if you want to utilize concurrency in Spark. With these parameters, the original query would be split into NUM_PARTITIONS number of sub-queries that can be executed by Spark tasks concurrently.

In record-based partitioning, the total number of records present is queried from the WooCommerce API, and divided by a NUM_PARTITIONS number provided. The resulting number of records are then concurrently fetched by each sub-query.

  • NUM_PARTITIONS: the number of partitions.

The following entities support record-based partitioning:

  • coupon

  • order

  • product

  • product-attribute

  • product-attribute-term

  • product-category

  • product-review

  • product-shipping-class

  • product-tag

  • product-variation

  • tax-rate

Example:

woocommerce_read = glueContext.create_dynamic_frame.from_options(
    connection_type="glue.spark.woocommerce",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "coupon",
        "API_VERSION": "v3",
        "INSTANCE_URL": "instanceUrl"
        "NUM_PARTITIONS": "10"
    }

Record-based partitioning:

The original query is splitinto NUM_PARTITIONS number of sub-queries that can be executed by Spark tasks concurrently:

  • NUM_PARTITIONS: the number of partitions.

Example:

WooCommerce_read = glueContext.create_dynamic_frame.from_options(
    connection_type="WooCommerce",
    connection_options={
        "connectionName": "connectionName",
        "REALMID": "1234567890123456789",
        "ENTITY_NAME": "Bill",
        "API_VERSION": "v3",
        "NUM_PARTITIONS": "10"
    }