# Transform data with custom visual transforms Custom visual transforms allow you to create transforms and make them available for use in AWS Glue Studio jobs. Custom visual transforms enable ETL developers, who may not be familiar with coding, to search and use a growing library of transforms using the AWS Glue Studio interface. You can create a custom visual transform, then upload it to Amazon S3 to make available for use through the visual editor in AWS Glue Studio to work with these jobs. **Topics** + [Getting started with custom visual transforms](custom-visual-transform-getting-started.md) + [Step 1. Create a JSON config file](custom-visual-transform-json-config-file.md) + [Step 2. Implement the transform logic](custom-visual-transform-implementation.md) + [Step 3. Validate and troubleshoot custom visual transforms in AWS Glue Studio](custom-visual-transform-validation.md) + [Step 4. Update custom visual transforms as needed](custom-visual-transform-updating-transforms.md) + [Step 5. Use custom visual transforms in AWS Glue Studio](custom-visual-transform-create-gs.md) + [Usage examples](custom-visual-transform-example-json.md) + [Examples of custom visual scripts](custom-visual-transform-example-scripts.md) + [Video](#custom-visual-transform-video) # Getting started with custom visual transforms To create a custom visual transform, you go through the following steps. + Step 1. Create a JSON config file + Step 2. Implement the transform logic + Step 3. Validate the custom visual transform + Step 4. Update the custom visual transform as needed + Step 5. Use the custom visual transform in AWS Glue Studio Get started by setting up the Amazon S3 bucket and continue to **Step 1. Create a JSON config file.** ## Prerequisites Customer-supplied transforms reside within a customer AWS account. That account owns the transforms and therefore has all permissions to view (search and use), edit, or delete them. In order to use a custom transform in AWS Glue Studio, you will need to create and upload two files to the Amazon S3 assets bucket in that AWS account: + **Python file** – contains the transform function + **JSON file** – describes the transform. This is also known as the config file that is required to define the transform. In order to pair the files together, use the same name for both. For example: + myTransform.json + myTransform.py Optionally, you can give your custom visual transform a custom icon by providing a **SVG file** containing the icon. In order to pair the files together, use the same name for the icon: + myTransform.svg AWS Glue Studio will automatically match them using their respective file names. File names cannot be the same for any existing module. ## Recommended convention for transform file name AWS Glue Studio will import your file as module (for example, `import myTransform`) in your job script. Therefore, your file name must follow the same naming rules set for python variable names (identifiers). Specifically, they must start with either a letter or an underscore and then be composed entirely of letters, digits, and/or underscores. **Note** Ensure your transform file name is not conflicting with existing loaded python modules (for example, `sys, array, copy` etc.) to avoid unexpected runtime issues. ## Setting up the Amazon S3 bucket Transforms you create are stored in Amazon S3 and is owned by your AWS account. You create new custom visual transforms by simply uploading files (json and py) to the Amazon S3 assets folder where all job scripts are currently stored (for example, `s3://aws-glue-assets--/transforms`). If using a custom icon, upload it as well. By default, AWS Glue Studio will read all .json files from the /transforms folder in the same S3 bucket. # Step 1. Create a JSON config file A JSON config file is required to define and describe your custom visual transform. The schema for the config file is as follows. ## JSON file structure **Fields** + `name: string` – (required) the transform system name used to identify transforms. Follow the same naming rules set for python variable names (identifiers). Specifically, they must start with either a letter or an underscore and then be composed entirely of letters, digits, and/or underscores. + `displayName: string` – (optional) the name of the transform displayed in the AWS Glue Studio visual job editor. If no `displayName` is specified, the `name` is used as the name of the transform in AWS Glue Studio. + `description: string` – (optional) the transform description is displayed in AWS Glue Studio and is searchable. + `functionName: string` – (required) the Python function name is used to identify the function to call in the Python script. + `path: string` – (optional) the full Amazon S3 path to the Python source file. If not specified, AWS Glue uses file name matching to pair the .json and .py files together. For example, the name of the JSON file, `myTransform.json`, will be paired to the Python file, `myTransform.py`, on the same Amazon S3 location. + `parameters: Array of TransformParameter object` – (optional) the list of parameters to be displayed when you configure them in the AWS Glue Studio visual editor. **TransformParameter fields** + `name: string` – (required) the parameter name that will be passed to the python function as a named argument in the job script. Follow the same naming rules set for python variable names (identifiers). Specifically, they must start with either a letter or an underscore and then be composed entirely of letters, digits, and/or underscores. + `displayName: string` – (optional) the name of the transform displayed in the AWS Glue Studio visual job editor. If no `displayName` is specified, the `name` is used as the name of the transform in AWS Glue Studio. + `type: string` – (required) the parameter type accepting common Python data types. Valid values: 'str' \$1 'int' \$1 'float' \$1 'list' \$1 'bool'. + `isOptional: boolean` – (optional) determines whether the parameter is optional. By default all parameters are required. + `description: string` — (optional) description is displayed in AWS Glue Studio to help the user configure the transform parameter. + `validationType: string` – (optional) defines the way this parameter is validated. Currently, it only supports regular expressions. By default, the validation type is set to `RegularExpression`. + `validationRule: string` – (optional) regular expression used to validate form input before submit when `validationType` is set to `RegularExpression`. Regular expression syntax must be compatible with [ RegExp Ecmascript specifications](https://tc39.es/ecma262/multipage/text-processing.html#sec-regexp-regular-expression-objects). + `validationMessage: string` – (optional) the message to display when validation fails. + `listOptions: An array of TransformParameterListOption object` OR a `string` or the string value ‘column’ – (optional) options to display in Select or Multiselect UI control. Accepting a list of comma separated value or a strongly type JSON object of type `TransformParameterListOption`. It can also dynamically populate the list of columns from the parent node schema by specifying the string value “column”. + `listType: string` – (optional) Define options types for type = 'list'. Valid values: 'str' \$1 'int' \$1 'float' \$1 'list' \$1 'bool'. Parameter type accepting common python data types. **TransformParameterListOption fields** + `value: string | int | float | bool` – (required) option value. + `label: string` – (optional) option label displayed in the select dropdown. ## Transform parameters in AWS Glue Studio By default, parameters are required unless mark as `isOptional` in the .json file. In AWS Glue Studio, parameters are displayed in the **Transform** tab. The example shows user-defined parameters such as Email Address, Phone Number, Your age, Your gender and Your origin country. ![\[The screenshot shows a custom visual transform selected and the Transform tab with user-defined parameters.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-parameters.png) You can enforce some validations in AWS Glue Studio using regular expressions in the json file by specifying the `validationRule` parameter and specifying a validation message in `validationMessage`. ``` "validationRule": "^\\(?(\\d{3})\\)?[- ]?(\\d{3})[- ]?(\\d{4})$", "validationMessage": "Please enter a valid US number" ``` **Note** Since validation occurs in the browser, your regular expression syntax must be compatible with [ RegExp Ecmascript specifications](https://tc39.es/ecma262/multipage/text-processing.html#sec-regexp-regular-expression-objects). Python syntax is not supported for these regular expressions. Adding validation will prevent the user from saving the job with incorrect user input. AWS Glue Studio displays the validation message as displayed in the example: ![\[The screenshot shows a custom visual transform parameter with a validation error message: Please enter a valid email address.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-validation-message.png) Parameters are displayed in AWS Glue Studio based on the parameter configuration. + When `type` is any of the following: `str`, `int` or `float`, a text input field is displayed. For example, the screenshot shows input fields for 'Email Address' and 'Your age' parameters. ![\[The screenshot shows a custom visual transform parameter with text input field.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-email-address.png) ![\[The screenshot shows a custom visual transform parameter with text input field.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-your-age.png) + When `type` is `bool`, a checkbox is displayed. ![\[The screenshot shows a custom visual transform parameter with text input field.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-bool.png) + When `type` is `str` and `listOptions` is provided, a single select list is displayed. ![\[The screenshot shows a custom visual transform parameter with a single select list drop-down.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-single-list.png) + When `type` is `list` and `listOptions` and `listType` are provided, a multi-select list is displayed. ![\[The screenshot shows a custom visual transform parameter with a list drop-down.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-list-options.png) ### Displaying a column selector as parameter If the configuration requires the user to choose a column from the schema, you can display a column selector so the user isn't required to type the column name. By setting the `listOptions` field to '“column”, AWS Glue Studio dynamically displays a column selector based on the parent node output schema. AWS Glue Studio can display either a single or multiple column selector. The following example uses the schema: ![\[The screenshot shows a sample output schema.\]](http://docs.aws.amazon.com/glue/latest/dg/images/custom-visual-transform-example-schema.png) **To define your Custom Visual Transform parameter to display a single column:** 1. In your JSON file, for the `parameters` object, set the `listOptions` value to "column". This allows a user to choose a column from a pick list in AWS Glue Studio. ![\[The screenshot shows a sample JSON file with the listOptions parameter set to "column" and the resulting user interface in In AWS Glue Studio.\]](http://docs.aws.amazon.com/glue/latest/dg/images/custom-visual-transform-example-listoptions-column.png) 1. You can also allow multiple columns selection by defining the parameter as: + `listOptions: "column"` + `type: "list"` ![\[The screenshot shows a sample JSON file with the listOptions parameter set to "column" and the type set to "list", and resulting user interface in AWS Glue Studio.\]](http://docs.aws.amazon.com/glue/latest/dg/images/custom-visual-transform-example-listoptions-column-type-list.png) # Step 2. Implement the transform logic **Note** Custom visual transforms only support Python scripts. Scala is not supported. To add the code that implements the function defined by the .json config file, it is recommended to place the Python file in the same location as the .json file, with the same name but with the “.py” extension. AWS Glue Studio automatically pairs the .json and .py files so that you don’t need to specify the path of the Python file in the config file. In the Python file, add the declared function, with the named parameters configured and register it to be used in `DynamicFrame`. The following is an example of a Python file: ``` from awsglue import DynamicFrame # self refers to the DynamicFrame to transform, # the parameter names must match the ones defined in the config # if it's optional, need to provide a default value def myTransform(self, email, phone, age=None, gender="", country="", promotion=False): resulting_dynf = # do some transformation on self return resulting_dynf DynamicFrame.myTransform = myTransform ``` It is recommended to use an AWS Glue notebook for the quickest way to develop and test the python code. See [Getting started with notebooks in AWS Glue Studio](https://docs.aws.amazon.com/glue/latest/ug/notebook-getting-started.html). To illustrate how to implement the transform logic, the custom visual transform in the example below is a transform to filter incoming data to keep only the data related to a specific US state. The .json file contains the parameter for `functionName` as `custom_filter_state` and two arguments ("state" and "colName" with type "str"). The example config .json file is: ``` { "name": "custom_filter_state", "displayName": "Filter State", "description": "A simple example to filter the data to keep only the state indicated.", "functionName": "custom_filter_state", "parameters": [ { "name": "colName", "displayName": "Column name", "type": "str", "description": "Name of the column in the data that holds the state postal code" }, { "name": "state", "displayName": "State postal code", "type": "str", "description": "The postal code of the state whole rows to keep" } ] } ``` **To implement the companion script in Python** 1. Start a AWS Glue notebook and run the initial cell provided for the session to be started. Running the initial cell creates the basic components required. 1. Create a function that performs the filtering as describe in the example and register it on `DynamicFrame`. Copy the code below and paste into a cell in the AWS Glue notebook. ``` from awsglue import DynamicFrame def custom_filter_state(self, colName, state): return self.filter(lambda row: row[colName] == state) DynamicFrame.custom_filter_state = custom_filter_state ``` 1. Create or load sample data to test the code in the same cell or a new cell. If you add the sample data in a new cell, don't forget to run the cell. For example: ``` # A few of rows of sample data to test data_sample = [ {"state": "CA", "count": 4}, {"state": "NY", "count": 2}, {"state": "WA", "count": 3} ] df1 = glueContext.sparkSession.sparkContext.parallelize(data_sample).toDF() dynf1 = DynamicFrame.fromDF(df1, glueContext, None) ``` 1. Test to validate the “custom\$1filter\$1state” with different arguments: ![\[The screenshot shows a cell in a AWS Glue notebook with the arguments passed to the dynamicFrame.show function.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-notebook-test-python.png) 1. After running several tests, save the code with the .py extension and name the .py file with a name that mirrors the .json file name. The .py and .json files should be in the same transform folder. Copy the following code and paste it to a file and rename it with a .py file extension. ``` from awsglue import DynamicFrame def custom_filter_state(self, colName, state): return self.filter(lambda row: row[colName] == state) DynamicFrame.custom_filter_state = custom_filter_state ``` 1. In AWS Glue Studio, open a visual job and add the transform to the job by selecting it from the list of available **Transforms**. To reuse this transform in a Python script code, add the Amazon S3 path to the .py file in the job under “Referenced files path” and in the script, import the name of the python file (without the extension) by adding it to the top of the file. For example: `import` # Step 3. Validate and troubleshoot custom visual transforms in AWS Glue Studio AWS Glue Studio validates the JSON config file before custom visual transforms are loaded into AWS Glue Studio. Validation includes: + Presence of required fields + JSON format validation + Incorrect or invalid parameters + Presence of both the .py and .json files in the same Amazon S3 path + Matching filenames for the .py and .json If validation succeeds, the transform is listed in the list of available **Actions** in the visual editor. If a custom icon has been provided, it should be visible beside the **Action**. If validation fails, AWS Glue Studio does not load the custom visual transform. # Step 4. Update custom visual transforms as needed Once created and used, the transform script can be updated as long as the transform follows the corresponding json definition: + The name used when assigning to DynamicFrame much match the json `functionName`. + The function arguments must be defined in the json file as described in [Step 1. Create a JSON config file](custom-visual-transform-json-config-file.md). + The Amazon S3 path of the Python file cannot change, since the jobs depend directly on it. **Note** If any updates need to be made, ensure the script and the .json file are consistently updated and any visual jobs are correctly saved again with the new transform. If visual jobs are not saved after the updates were made, the updates will not be applied and validated. If the Python script file is renamed or not placed next to the .json file, then you need to specify the full path in the .json file. **Custom icon** If you determine the default icon for your **Action** does not visually distinguish it as part of your workflows, you can provide a custom icon, as described in [Getting started with custom visual transforms](custom-visual-transform-getting-started.md). You can update the icon by updating the corresponding SVG hosted in Amazon S3. For best results, design your image to be viewed at 32x32px following guidelines from the Cloudscape Design System. For more information about Cloudscape guidelines, see [The Cloudscape documentation](https://cloudscape.design/foundation/visual-foundation/iconography/#custom-icons) # Step 5. Use custom visual transforms in AWS Glue Studio To use a custom visual transform in AWS Glue Studio, you upload the config and source files, then select the transform from the **Action** menu. Any parameters that need values or input are available to you in the **Transform** tab. 1. Upload the two files (Python source file and JSON config file) to the Amazon S3 assets folder where the job scripts are stored. By default, AWS Glue pulls all JSON files from the **/transforms** folder within the same Amazon S3 bucket. 1. From the **Action** menu, choose the custom visual transform. It is named with the transform `displayName` or name that you specified in the .json config file. 1. Enter values for any parameters that were configured in the config file. ![\[The screenshot shows a custom visual transform with parameters for the user to complete in the Transform tab.\]](http://docs.aws.amazon.com/glue/latest/dg/images/dynamic-transform-parameters.png) # Usage examples The following is an example of all possible parameters in a .json config file. ``` { "name": "MyTransform", "displayName": "My Transform", "description": "This transform description will be displayed in UI", "functionName": "myTransform", "parameters": [ { "name": "email", "displayName": "Email Address", "type": "str", "description": "Enter your work email address below", "validationType": "RegularExpression", "validationRule": "^\\w+([\\.-]?\\w+)*@\\w+([\\.-]?\\w+)*(\\.\\w{2,3})+$", "validationMessage": "Please enter a valid email address" }, { "name": "phone", "displayName": "Phone Number", "type": "str", "description": "Enter your mobile phone number below", "validationRule": "^\\(?(\\d{3})\\)?[- ]?(\\d{3})[- ]?(\\d{4})$", "validationMessage": "Please enter a valid US number" }, { "name": "age", "displayName": "Your age", "type": "int", "isOptional": true }, { "name": "gender", "displayName": "Your gender", "type": "str", "listOptions": [ {"label": "Male", "value": "male"}, {"label": "Female", "value": "female"}, {"label": "Other", "value": "other"} ], "isOptional": true }, { "name": "country", "displayName": "Your origin country ?", "type": "list", "listOptions": "Afghanistan,Albania,Algeria,American Samoa,Andorra,Angola,Anguilla,Antarctica,Antigua and Barbuda,Argentina,Armenia,Aruba,Australia,Austria,Azerbaijan,Bahamas,Bahrain,Bangladesh,Barbados,Belarus,Belgium,Belize,Benin,Bermuda,Bhutan,Bolivia,Bosnia and Herzegovina,Botswana,Bouvet Island,Brazil,British Indian Ocean Territory,Brunei Darussalam,Bulgaria,Burkina Faso,Burundi,Cambodia,Cameroon,Canada,Cape Verde,Cayman Islands,Central African Republic,Chad,Chile,China,Christmas Island,Cocos (Keeling Islands),Colombia,Comoros,Congo,Cook Islands,Costa Rica,Cote D'Ivoire (Ivory Coast),Croatia (Hrvatska,Cuba,Cyprus,Czech Republic,Denmark,Djibouti,Dominica,Dominican Republic,East Timor,Ecuador,Egypt,El Salvador,Equatorial Guinea,Eritrea,Estonia,Ethiopia,Falkland Islands (Malvinas),Faroe Islands,Fiji,Finland,France,France,Metropolitan,French Guiana,French Polynesia,French Southern Territories,Gabon,Gambia,Georgia,Germany,Ghana,Gibraltar,Greece,Greenland,Grenada,Guadeloupe,Guam,Guatemala,Guinea,Guinea-Bissau,Guyana,Haiti,Heard and McDonald Islands,Honduras,Hong Kong,Hungary,Iceland,India,Indonesia,Iran,Iraq,Ireland,Israel,Italy,Jamaica,Japan,Jordan,Kazakhstan,Kenya,Kiribati,Korea (North),Korea (South),Kuwait,Kyrgyzstan,Laos,Latvia,Lebanon,Lesotho,Liberia,Libya,Liechtenstein,Lithuania,Luxembourg,Macau,Macedonia,Madagascar,Malawi,Malaysia,Maldives,Mali,Malta,Marshall Islands,Martinique,Mauritania,Mauritius,Mayotte,Mexico,Micronesia,Moldova,Monaco,Mongolia,Montserrat,Morocco,Mozambique,Myanmar,Namibia,Nauru,Nepal,Netherlands,Netherlands Antilles,New Caledonia,New Zealand,Nicaragua,Niger,Nigeria,Niue,Norfolk Island,Northern Mariana Islands,Norway,Oman,Pakistan,Palau,Panama,Papua New Guinea,Paraguay,Peru,Philippines,Pitcairn,Poland,Portugal,Puerto Rico,Qatar,Reunion,Romania,Russian Federation,Rwanda,Saint Kitts and Nevis,Saint Lucia,Saint Vincent and The Grenadines,Samoa,San Marino,Sao Tome and Principe,Saudi Arabia,Senegal,Seychelles,Sierra Leone,Singapore,Slovak Republic,Slovenia,Solomon Islands,Somalia,South Africa,S. Georgia and S. Sandwich Isls.,Spain,Sri Lanka,St. Helena,St. Pierre and Miquelon,Sudan,Suriname,Svalbard and Jan Mayen Islands,Swaziland,Sweden,Switzerland,Syria,Tajikistan,Tanzania,Thailand,Togo,Tokelau,Tonga,Trinidad and Tobago,Tunisia,Turkey,Turkmenistan,Turks and Caicos Islands,Tuvalu,Uganda,Ukraine,United Arab Emirates,United Kingdom (Britain / UK),United States of America (USA),US Minor Outlying Islands,Uruguay,Uzbekistan,Vanuatu,Vatican City State (Holy See),Venezuela,Viet Nam,Virgin Islands (British),Virgin Islands (US),Wallis and Futuna Islands,Western Sahara,Yemen,Yugoslavia,Zaire,Zambia,Zimbabwe", "description": "What country were you born in?", "listType": "str", "isOptional": true }, { "name": "promotion", "displayName": "Do you want to receive promotional newsletter from us?", "type": "bool", "isOptional": true } ] } ``` # Examples of custom visual scripts The following examples perform equivalent transformations. However, the second example (SparkSQL) is the cleanest and most efficient, followed by the Pandas UDF and finally the low level mapping in the first example. The following example is a complete example of a simple transformation to add up two columns: ``` from awsglue import DynamicFrame # You can have other auxiliary variables, functions or classes on this file, it won't affect the runtime def record_sum(rec, col1, col2, resultCol): rec[resultCol] = rec[col1] + rec[col2] return rec # The number and name of arguments must match the definition on json config file # (expect self which is the current DynamicFrame to transform # If an argument is optional, you need to define a default value here # (resultCol in this example is an optional argument) def custom_add_columns(self, col1, col2, resultCol="result"): # The mapping will alter the columns order, which could be important fields = [field.name for field in self.schema()] if resultCol not in fields: # If it's a new column put it at the end fields.append(resultCol) return self.map(lambda record: record_sum(record, col1, col2, resultCol)).select_fields(paths=fields) # The name we assign on DynamicFrame must match the configured "functionName" DynamicFrame.custom_add_columns = custom_add_columns ``` The following example is an equivalent transform leveraging the SparkSQL API. ``` from awsglue import DynamicFrame # The number and name of arguments must match the definition on json config file # (expect self which is the current DynamicFrame to transform # If an argument is optional, you need to define a default value here # (resultCol in this example is an optional argument) def custom_add_columns(self, col1, col2, resultCol="result"): df = self.toDF() return DynamicFrame.fromDF( df.withColumn(resultCol, df[col1] + df[col2]) # This is the conversion logic , self.glue_ctx, self.name) # The name we assign on DynamicFrame must match the configured "functionName" DynamicFrame.custom_add_columns = custom_add_columns ``` The following example uses the same transformations but using a pandas UDF, which is more efficient that using a plain UDF. For more information about writing pandas UDFs see: [Apache Spark SQL documentation](https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.pandas_udf.html). ``` from awsglue import DynamicFrame import pandas as pd from pyspark.sql.functions import pandas_udf # The number and name of arguments must match the definition on json config file # (expect self which is the current DynamicFrame to transform # If an argument is optional, you need to define a default value here # (resultCol in this example is an optional argument) def custom_add_columns(self, col1, col2, resultCol="result"): @pandas_udf("integer") # We need to declare the type of the result column def add_columns(value1: pd.Series, value2: pd.Series) → pd.Series: return value1 + value2 df = self.toDF() return DynamicFrame.fromDF( df.withColumn(resultCol, add_columns(col1, col2)) # This is the conversion logic , self.glue_ctx, self.name) # The name we assign on DynamicFrame must match the configured "functionName" DynamicFrame.custom_add_columns = custom_add_columns ``` ## Video The following video provides an introduction to visual custom transforms and demonstrates how to use them. [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/xFpAhANcVcg/0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/xFpAhANcVcg)