# Data format options for inputs and outputs in AWS Glue for Spark Data format options These pages offer information about feature support and configuration parameters for data formats supported by AWS Glue for Spark. See the following for a description of the usage and applicablity of this information. ## Feature support across data formats in AWS Glue Feature support Each data format may support different AWS Glue features. The following common features may or may not be supported based on your format type. Refer to the documentation for your data format to understand how to leverage our features to meet your requirements. | | | | --- |--- | | Read | AWS Glue can recognize and interpret this data format without additional resources, such as connectors. | | Write | AWS Glue can write data in this format without additional resources. You can include third-party libraries in your job and use standard Apache Spark functions to write data, as you would in other Spark environments. For more information about including libraries, see [Using Python libraries with AWS Glue](aws-glue-programming-python-libraries.md). | | Streaming read | AWS Glue can recognize and interpret this data format from an Apache Kafka, Amazon Managed Streaming for Apache Kafka or Amazon Kinesis message stream. We expect streams to present data in a consistent format, so they are read in as DataFrames. | | Group small files | AWS Glue can group files together to batch work sent to each node when performing AWS Glue transforms. This can significantly improve performance for workloads involving large amounts of small files. For more information, see [Reading input files in larger groups](grouping-input-files.md). | | Job bookmarks | AWS Glue can track the progress of transforms performing the same work on the same dataset across job runs with job bookmarks. This can improve performance for workloads involving datasets where work only needs to be done on new data since the last job run. For more information, see [Tracking processed data using job bookmarks](monitor-continuations.md). | ## Parameters used to interact with data formats in AWS Glue Data format parameters Certain AWS Glue connection types support multiple `format` types, requiring you to specify information about your data format with a `format_options` object when using methods like `GlueContext.write_dynamic_frame.from_options`. + `s3` – For more information, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can also view the documentation for the methods facilitating this connection type: [create\$1dynamic\$1frame\$1from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) and [write\$1dynamic\$1frame\$1from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) in Python and the corresponding Scala methods [def getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) and [def getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat). + `kinesis` – For more information, see Connection types and options for ETL in AWS Glue: [Kinesis connection parameters](aws-glue-programming-etl-connect-kinesis-home.md#aws-glue-programming-etl-connect-kinesis). You can also view the documentation for the method facilitating this connection type: [create\$1data\$1frame\$1from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create-dataframe-from-options) and the corresponding Scala method [def createDataFrameFromOptions](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-createDataFrameFromOptions). + `kafka` – For more information, see Connection types and options for ETL in AWS Glue: [Kafka connection parameters](aws-glue-programming-etl-connect-kafka-home.md#aws-glue-programming-etl-connect-kafka). You can also view the documentation for the method facilitating this connection type: [create\$1data\$1frame\$1from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create-dataframe-from-options) and the corresponding Scala method [def createDataFrameFromOptions](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-createDataFrameFromOptions). Some connection types do not require `format_options`. For example, in normal use, a JDBC connection to a relational database retrieves data in a consistent, tabular data format. Therefore, reading from a JDBC connection would not require `format_options`. Some methods to read and write data in glue do not require `format_options`. For example, using `GlueContext.create_dynamic_frame.from_catalog` with AWS Glue crawlers. Crawlers determine the shape of your data. When using crawlers, a AWS Glue classifier will examine your data to make smart decisions about how to represent your data format. It will then store a representation of your data in the AWS Glue Data Catalog, which can be used within a AWS Glue ETL script to retrieve your data with the `GlueContext.create_dynamic_frame.from_catalog` method. Crawlers remove the need to manually specify information about your data format. For jobs that access AWS Lake Formation governed tables, AWS Glue supports reading and writing all formats supported by Lake Formation governed tables. For the current list of supported formats for AWS Lake Formation governed tables, see [Notes and Restrictions for Governed Tables](https://docs.aws.amazon.com/lake-formation/latest/dg/governed-table-restrictions.html) in the *AWS Lake Formation Developer Guide*. **Note** For writing Apache Parquet, AWS Glue ETL only supports writing to a governed table by specifying an option for a custom Parquet writer type optimized for Dynamic Frames. When writing to a governed table with the `parquet` format, you should add the key `useGlueParquetWriter` with a value of `true` in the table parameters. **Topics** + [ ## Feature support across data formats in AWS Glue ](#aws-glue-programming-etl-format-features) + [ ## Parameters used to interact with data formats in AWS Glue ](#aws-glue-programming-etl-format-parameters) + [ # Using the CSV format in AWS Glue ](aws-glue-programming-etl-format-csv-home.md) + [ # Using the Parquet format in AWS Glue ](aws-glue-programming-etl-format-parquet-home.md) + [ # Using the XML format in AWS Glue ](aws-glue-programming-etl-format-xml-home.md) + [ # Using the Avro format in AWS Glue ](aws-glue-programming-etl-format-avro-home.md) + [ # Using the grokLog format in AWS Glue ](aws-glue-programming-etl-format-grokLog-home.md) + [ # Using the Ion format in AWS Glue ](aws-glue-programming-etl-format-ion-home.md) + [ # Using the JSON format in AWS Glue ](aws-glue-programming-etl-format-json-home.md) + [ # Using the ORC format in AWS Glue ](aws-glue-programming-etl-format-orc-home.md) + [ # Using data lake frameworks with AWS Glue ETL jobs ](aws-glue-programming-etl-datalake-native-frameworks.md) + [ ## Shared configuration reference ](#aws-glue-programming-etl-format-shared-reference) # Using the CSV format in AWS Glue CSV AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the CSV data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the comma-separated value (CSV) format. This format is a minimal, row-based data format. CSVs often don't strictly conform to a standard, but you can refer to [RFC 4180](https://tools.ietf.org/html/rfc4180) and [RFC 7111](https://tools.ietf.org/html/rfc7111) for more information. You can use AWS Glue to read CSVs from Amazon S3 and from streaming sources as well as write CSVs to Amazon S3. You can read and write `bzip` and `gzip` archives containing CSV files from S3. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue features support the CSV format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Supported | Supported | Supported | Supported | ## Example: Read CSV files or folders from S3 Example: Read CSV **Prerequisites:** You will need the S3 paths (`s3path`) to the CSV files or folders that you want to read. **Configuration:** In your function options, specify `format="csv"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can configure how the reader interacts with S3 in `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how the reader interprets CSV files in your `format_options`. For details, see [CSV Configuration Reference](#aws-glue-programming-etl-format-csv-reference). The following AWS Glue ETL script shows the process of reading CSV files or folders from S3. We provide a custom CSV reader with performance optimizations for common workflows through the `optimizePerformance` configuration key. To determine if this reader is right for your workload, see [Optimize read performance with vectorized SIMD CSV reader](#aws-glue-programming-etl-format-simd-csv-reader). ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` # Example: Read CSV from S3 # For show, we handle a CSV with a header row. Set the withHeader option. # Consider whether optimizePerformance is right for your workflow. from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) spark = glueContext.spark_session dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="csv", format_options={ "withHeader": True, # "optimizePerformance": True, }, ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame = spark.read\ .format("csv")\ .option("header", "true")\ .load("s3://s3path") ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` // Example: Read CSV from S3 // For show, we handle a CSV with a header row. Set the withHeader option. // Consider whether optimizePerformance is right for your workflow. import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( formatOptions=JsonOptions("""{"withHeader": true}"""), connectionType="s3", format="csv", options=JsonOptions("""{"paths": ["s3://s3path"], "recurse": true}""") ).getDynamicFrame() } } ``` You can also use DataFrames in a script (`org.apache.spark.sql.DataFrame`). ``` val dataFrame = spark.read .option("header","true") .format("csv") .load("s3://s3path“) ``` ------ ## Example: Write CSV files and folders to S3 Example: Write CSV **Prerequisites:** You will need an initialized DataFrame (`dataFrame`) or a DynamicFrame (`dynamicFrame`). You will also need your expected S3 output path, `s3path`. **Configuration:** In your function options, specify `format="csv"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can configure how the writer interacts with S3 in `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how your operation writes the contents of your files in `format_options`. For details, see [CSV Configuration Reference](#aws-glue-programming-etl-format-csv-reference). The following AWS Glue ETL script shows the process of writing CSV files and folders to S3. ------ #### [ Python ] For this example, use the [write\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) method. ``` # Example: Write CSV to S3 # For show, customize how we write string type values. Set quoteChar to -1 so our values are not quoted. from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) glueContext.write_dynamic_frame.from_options( frame=dynamicFrame, connection_type="s3", connection_options={"path": "s3://s3path"}, format="csv", format_options={ "quoteChar": -1, }, ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame.write\ .format("csv")\ .option("quote", None)\ .mode("append")\ .save("s3://s3path") ``` ------ #### [ Scala ] For this example, use the [getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat) method. ``` // Example: Write CSV to S3 // For show, customize how we write string type values. Set quoteChar to -1 so our values are not quoted. import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getSinkWithFormat( connectionType="s3", options=JsonOptions("""{"path": "s3://s3path"}"""), format="csv" ).writeDynamicFrame(dynamicFrame) } } ``` You can also use DataFrames in a script (`org.apache.spark.sql.DataFrame`). ``` dataFrame.write .format("csv") .option("quote", null) .mode("Append") .save("s3://s3path") ``` ------ ## CSV configuration reference CSV reference You can use the following `format_options` wherever AWS Glue libraries specify `format="csv"`: + `separator` –Specifies the delimiter character. The default is a comma, but any other character can be specified. + **Type:** Text, **Default:** `","` + `escaper` – Specifies a character to use for escaping. This option is used only when reading CSV files, not writing. If enabled, the character that immediately follows is used as-is, except for a small set of well-known escapes (`\n`, `\r`, `\t`, and `\0`). + **Type:** Text, **Default:** none + `quoteChar` – Specifies the character to use for quoting. The default is a double quote. Set this to `-1` to turn off quoting entirely. + **Type:** Text, **Default:** `'"'` + `multiLine` – Specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to `True` if any record spans multiple lines. Enabling `multiLine` might decrease performance because it requires more cautious file-splitting while parsing. + **Type:** Boolean, **Default:** `false` + `withHeader` – Specifies whether to treat the first line as a header. This option can be used in the `DynamicFrameReader` class. + **Type:** Boolean, **Default:** `false` + `writeHeader` – Specifies whether to write the header to output. This option can be used in the `DynamicFrameWriter` class. + **Type:** Boolean, **Default:** `true` + `skipFirst` – Specifies whether to skip the first data line. + **Type:** Boolean, **Default:** `false` + `optimizePerformance` – Specifies whether to use the advanced SIMD CSV reader along with Apache Arrow–based columnar memory formats. Only available in AWS Glue 3.0\$1. + **Type:** Boolean, **Default:** `false` + `strictCheckForQuoting` – When writing CSVs, Glue may add quotes to values it interprets as strings. This is done to prevent ambiguity in what is written out. To save time when deciding what to write, Glue may quote in certain situations where quotes are not necessary. Enabling a strict check will perform a more intensive computation and will only quote when strictly necessary. Only available in AWS Glue 3.0\$1. + **Type:** Boolean, **Default:** `false` ## Optimize read performance with vectorized SIMD CSV reader Using optimized CSV reader AWS Glue version 3.0 adds an optimized CSV reader that can significantly speed up overall job performance compared to row-based CSV readers. The optimized reader: + Uses CPU SIMD instructions to read from disk + Immediately writes records to memory in a columnar format (Apache Arrow) + Divides the records into batches This saves processing time when records would be batched or converted to a columnar format later on. Some examples are when changing schemas or retrieving data by column. To use the optimized reader, set `"optimizePerformance"` to `true` in the `format_options` or table property. ``` glueContext.create_dynamic_frame.from_options( frame = datasource1, connection_type = "s3", connection_options = {"paths": ["s3://s3path"]}, format = "csv", format_options={ "optimizePerformance": True, "separator": "," }, transformation_ctx = "datasink2") ``` **Limitations for the vectorized CSV reader** Note the following limitations of the vectorized CSV reader: + It doesn't support the `multiLine` and `escaper` format options. It uses the default `escaper` of double quote char `'"'`. When these options are set, AWS Glue automatically falls back to using the row-based CSV reader. + It doesn't support creating a DynamicFrame with [ChoiceType](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-types.html#aws-glue-api-crawler-pyspark-extensions-types-awsglue-choicetype). + It doesn't support creating a DynamicFrame with [error records](https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-apis-glue-dynamicframe-class.html#glue-etl-scala-apis-glue-dynamicframe-class-defs-errorsAsDynamicFrame). + It doesn't support reading CSV files with multibyte characters such as Japanese or Chinese characters. # Using the Parquet format in AWS Glue Parquet AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the Parquet data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the Parquet format. This format is a performance-oriented, column-based data format. For an introduction to the format by the standard authority see, [Apache Parquet Documentation Overview](https://parquet.apache.org/docs/overview/). You can use AWS Glue to read Parquet files from Amazon S3 and from streaming sources as well as write Parquet files to Amazon S3. You can read and write `bzip` and `gzip` archives containing Parquet files from S3. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue features support the Parquet format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Supported | Supported | Unsupported | Supported\$1 | \$1 Supported in AWS Glue version 1.0\$1 ## Example: Read Parquet files or folders from S3 Example: Read Parquet ** Prerequisites:** You will need the S3 paths (`s3path`) to the Parquet files or folders that you want to read. **Configuration:** In your function options, specify `format="parquet"`. In your `connection_options`, use the `paths` key to specify your `s3path`. You can configure how the reader interacts with S3 in the `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how the reader interprets Parquet files in your `format_options`. For details, see [Parquet Configuration Reference](#aws-glue-programming-etl-format-parquet-reference). The following AWS Glue ETL script shows the process of reading Parquet files or folders from S3: ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` # Example: Read Parquet from S3 from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) spark = glueContext.spark_session dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type = "s3", connection_options = {"paths": ["s3://s3path/"]}, format = "parquet" ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame = spark.read.parquet("s3://s3path/") ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) method. ``` // Example: Read Parquet from S3 import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( connectionType="s3", format="parquet", options=JsonOptions("""{"paths": ["s3://s3path"]}""") ).getDynamicFrame() } } ``` You can also use DataFrames in a script (`org.apache.spark.sql.DataFrame`). ``` spark.read.parquet("s3://s3path/") ``` ------ ## Example: Write Parquet files and folders to S3 Example: Write Parquet ** Prerequisites:** You will need an initialized DataFrame (`dataFrame`) or DynamicFrame (`dynamicFrame`). You will also need your expected S3 output path, `s3path`. **Configuration:** In your function options, specify `format="parquet"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can further alter how the writer interacts with S3 in the `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how your operation writes the contents of your files in `format_options`. For details, see [Parquet Configuration Reference](#aws-glue-programming-etl-format-parquet-reference). The following AWS Glue ETL script shows the process of writing Parquet files and folders to S3. We provide a custom Parquet writer with performance optimizations for DynamicFrames, through the `useGlueParquetWriter` configuration key. To determine if this writer is right for your workload, see [Glue Parquet Writer](#aws-glue-programming-etl-format-glue-parquet-writer). ------ #### [ Python ] For this example, use the [write\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) method. ``` # Example: Write Parquet to S3 # Consider whether useGlueParquetWriter is right for your workflow. from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) glueContext.write_dynamic_frame.from_options( frame=dynamicFrame, connection_type="s3", format="parquet", connection_options={ "path": "s3://s3path", }, format_options={ # "useGlueParquetWriter": True, }, ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` df.write.parquet("s3://s3path/") ``` ------ #### [ Scala ] For this example, use the [getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat) method. ``` // Example: Write Parquet to S3 // Consider whether useGlueParquetWriter is right for your workflow. import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getSinkWithFormat( connectionType="s3", options=JsonOptions("""{"path": "s3://s3path"}"""), format="parquet" ).writeDynamicFrame(dynamicFrame) } } ``` You can also use DataFrames in a script (`org.apache.spark.sql.DataFrame`). ``` df.write.parquet("s3://s3path/") ``` ------ ## Parquet configuration reference Parquet reference You can use the following `format_options` wherever AWS Glue libraries specify `format="parquet"`: + `useGlueParquetWriter` – Specifies the use of a custom Parquet writer that has performance optimizations for DynamicFrame workflows. For usage details, see [Glue Parquet Writer](#aws-glue-programming-etl-format-glue-parquet-writer). + **Type:** Boolean, **Default:**`false` + `compression` – Specifies the compression codec used. Values are fully compatible with `org.apache.parquet.hadoop.metadata.CompressionCodecName`. + **Type:** Enumerated Text, **Default:** `"snappy"` + Values: `"uncompressed"`, `"snappy"`, `"gzip"`, and `"lzo"` + `blockSize` – Specifies the size in bytes of a row group being buffered in memory. You use this for tuning performance. Size should divide exactly into a number of megabytes. + **Type:** Numerical, **Default:**`134217728` + The default value is equal to 128 MB. + `pageSize` – Specifies the size in bytes of a page. You use this for tuning performance. A page is the smallest unit that must be read fully to access a single record. + **Type:** Numerical, **Default:**`1048576` + The default value is equal to 1 MB. **Note** Additionally, any options that are accepted by the underlying SparkSQL code can be passed to this format by way of the `connection_options` map parameter. For example, you can set a Spark configuration such as [mergeSchema](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#schema-merging) for the AWS Glue Spark reader to merge the schema for all files. ## Optimize write performance with AWS Glue Parquet writer Write with AWS Glue Parquet writer **Note** The AWS Glue Parquet writer has historically been accessed through the `glueparquet` format type. This access pattern is no longer advocated. Instead, use the `parquet` type with `useGlueParquetWriter` enabled. The AWS Glue Parquet writer has performance enhancements that allow faster Parquet file writes. The traditional writer computes a schema before writing. The Parquet format doesn't store the schema in a quickly retrievable fashion, so this might take some time. With the AWS Glue Parquet writer, a pre-computed schema isn't required. The writer computes and modifies the schema dynamically, as data comes in. Note the following limitations when you specify `useGlueParquetWriter`: + The writer supports only schema evolution (such as adding or removing columns), but not changing column types, such as with `ResolveChoice`. + The writer doesn't support writing empty DataFrames—for example, to write a schema-only file. When integrating with the AWS Glue Data Catalog by setting `enableUpdateCatalog=True`, attempting to write an empty DataFrame will not update the Data Catalog. This will result in creating a table in the Data Catalog without a schema. If your transform doesn't require these limitations, turning on the AWS Glue Parquet writer should increase performance. # Using the XML format in AWS Glue XML AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the XML data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the XML format. This format represents highly configurable, rigidly defined data structures that aren't row or column based. XML is highly standardized. For an introduction to the format by the standard authority, see [XML Essentials](https://www.w3.org/standards/xml/core). You can use AWS Glue to read XML files from Amazon S3, as well as `bzip` and `gzip` archives containing XML files. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue features support the XML format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Unsupported | Unsupported | Supported | Supported | ## Example: Read XML from S3 Example: Read XML The XML reader takes an XML tag name. It examines elements with that tag within its input to infer a schema and populates a DynamicFrame with corresponding values. The AWS Glue XML functionality behaves similarly to the [XML Data Source for Apache Spark](https://github.com/databricks/spark-xml). You might be able to gain insight around basic behavior by comparing this reader to that project's documentation. ** Prerequisites:** You will need the S3 paths (`s3path`) to the XML files or folders that you want to read, and some information about your XML file. You will also need the tag for the XML element you want to read, `xmlTag`. **Configuration:** In your function options, specify `format="xml"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can further configure how the reader interacts with S3 in the `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). In your `format_options`, use the `rowTag` key to specify `xmlTag`. You can further configure how the reader interprets XML files in your `format_options`. For details, see [XML Configuration Reference](#aws-glue-programming-etl-format-xml-reference). The following AWS Glue ETL script shows the process of reading XML files or folders from S3. ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` # Example: Read XML from S3 # Set the rowTag option to configure the reader. from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="xml", format_options={"rowTag": "xmlTag"}, ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame = spark.read\ .format("xml")\ .option("rowTag", "xmlTag")\ .load("s3://s3path") ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` // Example: Read XML from S3 // Set the rowTag option to configure the reader. import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.GlueContext import org.apache.spark.sql.SparkSession val glueContext = new GlueContext(SparkContext.getOrCreate()) val sparkSession: SparkSession = glueContext.getSparkSession object GlueApp { def main(sysArgs: Array[String]): Unit = { val dynamicFrame = glueContext.getSourceWithFormat( formatOptions=JsonOptions("""{"rowTag": "xmlTag"}"""), connectionType="s3", format="xml", options=JsonOptions("""{"paths": ["s3://s3path"], "recurse": true}""") ).getDynamicFrame() } ``` You can also use DataFrames in a script (`org.apache.spark.sql.DataFrame`). ``` val dataFrame = spark.read .option("rowTag", "xmlTag") .format("xml") .load("s3://s3path“) ``` ------ ## XML configuration reference XML reference You can use the following `format_options` wherever AWS Glue libraries specify `format="xml"`: + `rowTag` – Specifies the XML tag in the file to treat as a row. Row tags cannot be self-closing. + **Type:** Text, **Required** + `encoding` – Specifies the character encoding. It can be the name or alias of a [Charset](https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html) supported by our runtime environment. We don't make specific guarantees around encoding support, but major encodings should work. + **Type:** Text, **Default:** `"UTF-8"` + `excludeAttribute` – Specifies whether you want to exclude attributes in elements or not. + **Type:** Boolean, **Default:** `false` + `treatEmptyValuesAsNulls` – Specifies whether to treat white space as a null value. + **Type:** Boolean, **Default:** `false` + `attributePrefix` – A prefix for attributes to differentiate them from child element text. This prefix is used for field names. + **Type:** Text, **Default:** `"_"` + `valueTag` – The tag used for a value when there are attributes in the element that have no child. + **Type:** Text, **Default:** `"_VALUE"` + `ignoreSurroundingSpaces` – Specifies whether the white space that surrounds values should be ignored. + **Type:** Boolean, **Default:** `false` + `withSchema` – Contains the expected schema, in situations where you want to override the inferred schema. If you don't use this option, AWS Glue infers the schema from the XML data. + **Type:** Text, **Default:** Not applicable + The value should be a JSON object that represents a `StructType`. ## Manually specify the XML schema Specify XML schema **Manual XML schema example** This is an example of using the `withSchema` format option to specify the schema for XML data. ``` from awsglue.gluetypes import * schema = StructType([ Field("id", IntegerType()), Field("name", StringType()), Field("nested", StructType([ Field("x", IntegerType()), Field("y", StringType()), Field("z", ChoiceType([IntegerType(), StringType()])) ])) ]) datasource0 = create_dynamic_frame_from_options( connection_type, connection_options={"paths": ["s3://xml_bucket/someprefix"]}, format="xml", format_options={"withSchema": json.dumps(schema.jsonValue())}, transformation_ctx = "" ) ``` # Using the Avro format in AWS Glue Avro AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the Avro data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the Avro format. This format is a performance-oriented, row-based data format. For an introduction to the format by the standard authority see, [Apache Avro 1.8.2 Documentation](https://avro.apache.org/docs/1.8.2/). You can use AWS Glue to read Avro files from Amazon S3 and from streaming sources as well as write Avro files to Amazon S3. You can read and write `bzip2` and `gzip` archives containing Avro files from S3. Additionally, you can write `deflate`, `snappy`, and `xz` archives containing Avro files. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue operations support the Avro format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Supported | Supported\$1 | Unsupported | Supported | \$1Supported with restrictions. For more information, see [Notes and restrictions for Avro streaming sources](add-job-streaming.md#streaming-avro-notes). ## Example: Read Avro files or folders from S3 Example: Read Avro ** Prerequisites:** You will need the S3 paths (`s3path`) to the Avro files or folders that you want to read. **Configuration:** In your function options, specify `format="avro"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can configure how the reader interacts with S3 in the `connection_options`. For details, see Data format options for ETL inputs and outputs in AWS Glue: [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how the reader interprets Avro files in your `format_options`. For details, see [Avro Configuration Reference](#aws-glue-programming-etl-format-avro-reference). The following AWS Glue ETL script shows the process of reading Avro files or folders from S3: ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="avro" ) ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.GlueContext import org.apache.spark.sql.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( connectionType="s3", format="avro", options=JsonOptions("""{"paths": ["s3://s3path"]}""") ).getDynamicFrame() } ``` ------ ## Example: Write Avro files and folders to S3 Example: Avro write ** Prerequisites:** You will need an initialized DataFrame (`dataFrame`) or DynamicFrame (`dynamicFrame`). You will also need your expected S3 output path, `s3path`. **Configuration:** In your function options, specify `format="avro"`. In your `connection_options`, use the `paths` key to specify your `s3path`. You can further alter how the writer interacts with S3 in the `connection_options`. For details, see Data format options for ETL inputs and outputs in AWS Glue: [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can alter how the writer interprets Avro files in your `format_options`. For details, see [Avro Configuration Reference](#aws-glue-programming-etl-format-avro-reference). The following AWS Glue ETL script shows the process of writing Avro files or folders to S3. ------ #### [ Python ] For this example, use the [write\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) method. ``` from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) glueContext.write_dynamic_frame.from_options( frame=dynamicFrame, connection_type="s3", format="avro", connection_options={ "path": "s3://s3path" } ) ``` ------ #### [ Scala ] For this example, use the [getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat) method. ``` import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getSinkWithFormat( connectionType="s3", options=JsonOptions("""{"path": "s3://s3path"}"""), format="avro" ).writeDynamicFrame(dynamicFrame) } } ``` ------ ## Avro configuration reference Avro reference You can use the following `format_options` values wherever AWS Glue libraries specify `format="avro"`: + `version` — Specifies the version of Apache Avro reader/writer format to support. The default is "1.7". You can specify `format_options={"version": “1.8”}` to enable Avro logical type reading and writing. For more information, see the [Apache Avro 1.7.7 Specification](https://avro.apache.org/docs/1.7.7/spec.html) and [Apache Avro 1.8.2 Specification](https://avro.apache.org/docs/1.8.2/spec.html). The Apache Avro 1.8 connector supports the following logical type conversions: For the reader: this table shows the conversion between Avro data type (logical type and Avro primitive type) and AWS Glue `DynamicFrame` data type for Avro reader 1.7 and 1.8. | Avro Data Type: Logical Type | Avro Data Type: Avro Primitive Type | GlueDynamicFrame Data Type:Avro Reader 1.7 | GlueDynamicFrame Data Type: Avro Reader 1.8 | | --- | --- | --- | --- | | Decimal | bytes | BINARY | Decimal | | Decimal | fixed | BINARY | Decimal | | Date | int | INT | Date | | Time (millisecond) | int | INT | INT | | Time (microsecond) | long | LONG | LONG | | Timestamp (millisecond) | long | LONG | Timestamp | | Timestamp (microsecond) | long | LONG | LONG | | Duration (not a logical type) | fixed of 12 | BINARY | BINARY | For the writer: this table shows the conversion between AWS Glue `DynamicFrame` data type and Avro data type for Avro writer 1.7 and 1.8. | AWS Glue `DynamicFrame` Data Type | Avro Data Type:Avro Writer 1.7 | Avro Data Type:Avro Writer 1.8 | | --- | --- | --- | | Decimal | String | decimal | | Date | String | date | | Timestamp | String | timestamp-micros | ## Avro Spark DataFrame support Avro DataFrame support In order to use Avro from the Spark DataFrame API, you need to install the Spark Avro plugin for the corresponding Spark version. The version of Spark available in your job is determined by your AWS Glue version. For more information about Spark versions, see [AWS Glue versions](release-notes.md). This plugin is maintained by Apache, we do not make specific guarantees of support. In AWS Glue 2.0 - use version 2.4.3 of the Spark Avro plugin. You can find this JAR on Maven Central, see [org.apache.spark:spark-avro\$12.12:2.4.3](https://search.maven.org/artifact/org.apache.spark/spark-avro_2.12/3.1.1/jar). In AWS Glue 3.0 - use version 3.1.1 of the Spark Avro plugin. You can find this JAR on Maven Central, see [org.apache.spark:spark-avro\$12.12:3.1.1](https://search.maven.org/artifact/org.apache.spark/spark-avro_2.12/3.1.1/jar). To include extra JARs in a AWS Glue ETL job, use the `--extra-jars` job parameter. For more information about job parameters, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). You can also configure this parameter in the AWS Management Console. # Using the grokLog format in AWS Glue grokLog AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in a loosely structured plaintext format, this document introduces you available features for using your data in AWS Glue through Grok patterns. AWS Glue supports using Grok patterns. Grok patterns are similar to regular expression capture groups. They recognize patterns of character sequences in a plaintext file and give them a type and purpose. In AWS Glue, their primary purpose is to read logs. For an introduction to the Grok by the authors, see [Logstash Reference: Grok filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html). | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Not Applicable | Supported | Supported | Unsupported | ## grokLog configuration reference grokLog You can use the following `format_options` values with `format="grokLog"`: + `logFormat` — Specifies the Grok pattern that matches the log's format. + `customPatterns` — Specifies additional Grok patterns used here. + `MISSING` — Specifies the signal to use in identifying missing values. The default is `'-'`. + `LineCount` — Specifies the number of lines in each log record. The default is `'1'`, and currently only single-line records are supported. + `StrictMode` — A Boolean value that specifies whether strict mode is turned on. In strict mode, the reader doesn't do automatic type conversion or recovery. The default value is `"false"`. # Using the Ion format in AWS Glue Ion AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the Ion data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the Ion format. This format represents data structures (that aren't row or column based) in interchangeable binary and plaintext representations. For an introduction to the format by the authors, see [Amazon Ion](https://amzn.github.io/ion-docs/). (For more information, see the [Amazon Ion Specification](https://amzn.github.io/ion-docs/spec.html).) You can use AWS Glue to read Ion files from Amazon S3. You can read `bzip` and `gzip` archives containing Ion files from S3. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue operations support the Ion format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Unsupported | Unsupported | Supported | Unsupported | ## Example: Read Ion files and folders from S3 Example: Read Ion ** Prerequisites:** You will need the S3 paths (`s3path`) to the Ion files or folders that you want to read. **Configuration:** In your function options, specify `format="json"`. In your `connection_options`, use the `paths` key to specify your `s3path`. You can configure how the reader interacts with S3 in the `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). The following AWS Glue ETL script shows the process of reading Ion files or folders from S3: ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` # Example: Read ION from S3 from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="ion" ) ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` // Example: Read ION from S3 import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.GlueContext import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( connectionType="s3", format="ion", options=JsonOptions("""{"paths": ["s3://s3path"], "recurse": true}""") ).getDynamicFrame() } } ``` ------ ## Ion configuration reference Ion reference There are no `format_options` values for `format="ion"`. # Using the JSON format in AWS Glue JSON AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the JSON data format, this document introduces you to available features for using your data in AWS Glue. AWS Glue supports using the JSON format. This format represents data structures with consistent shape but flexible contents, that aren't row or column based. JSON is defined by parallel standards issued by several authorities, one of which is ECMA-404. For an introduction to the format by a commonly referenced source, see [Introducing JSON](https://www.json.org/). You can use AWS Glue to read JSON files from Amazon S3, as well as `bzip` and `gzip` compressed JSON files. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Supported | Supported | Supported | Supported | ## Example: Read JSON files or folders from S3 Example: Read JSON ** Prerequisites:** You will need the S3 paths (`s3path`) to the JSON files or folders you would like to read. **Configuration:** In your function options, specify `format="json"`. In your `connection_options`, use the `paths` key to specify your `s3path`. You can further alter how your read operation will traverse s3 in the connection options, consult [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) for details. You can configure how the reader interprets JSON files in your `format_options`. For details, see [JSON Configuration Reference](#aws-glue-programming-etl-format-json-reference). The following AWS Glue ETL script shows the process of reading JSON files or folders from S3: ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` # Example: Read JSON from S3 # For show, we handle a nested JSON file that we can limit with the JsonPath parameter # For show, we also handle a JSON where a single entry spans multiple lines # Consider whether optimizePerformance is right for your workflow. from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) spark = glueContext.spark_session dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="json", format_options={ "jsonPath": "$.id", "multiline": True, # "optimizePerformance": True, -> not compatible with jsonPath, multiline } ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame = spark.read\ .option("multiline", "true")\ .json("s3://s3path") ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` // Example: Read JSON from S3 // For show, we handle a nested JSON file that we can limit with the JsonPath parameter // For show, we also handle a JSON where a single entry spans multiple lines // Consider whether optimizePerformance is right for your workflow. import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( formatOptions=JsonOptions("""{"jsonPath": "$.id", "multiline": true, "optimizePerformance":false}"""), connectionType="s3", format="json", options=JsonOptions("""{"paths": ["s3://s3path"], "recurse": true}""") ).getDynamicFrame() } } ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` val dataFrame = spark.read .option("multiline", "true") .json("s3://s3path") ``` ------ ## Example: Write JSON files and folders to S3 Example: Write JSON ** Prerequisites:**You will need an initialized DataFrame (`dataFrame`) or DynamicFrame (`dynamicFrame`). You will also need your expected S3 output path, `s3path`. **Configuration:** In your function options, specify `format="json"`. In your `connection_options`, use the `paths` key to specify `s3path`. You can further alter how the writer interacts with S3 in the `connection_options`. For details, see Data format options for ETL inputs and outputs in AWS Glue : [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). You can configure how the writer interprets JSON files in your `format_options`. For details, see [JSON Configuration Reference](#aws-glue-programming-etl-format-json-reference). The following AWS Glue ETL script shows the process of writing JSON files or folders from S3: ------ #### [ Python ] For this example, use the [write\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) method. ``` # Example: Write JSON to S3 from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) glueContext.write_dynamic_frame.from_options( frame=dynamicFrame, connection_type="s3", connection_options={"path": "s3://s3path"}, format="json" ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` df.write.json("s3://s3path/") ``` ------ #### [ Scala ] For this example, use the [getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat) method. ``` // Example: Write JSON to S3 import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getSinkWithFormat( connectionType="s3", options=JsonOptions("""{"path": "s3://s3path"}"""), format="json" ).writeDynamicFrame(dynamicFrame) } } ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` df.write.json("s3://s3path") ``` ------ ## Json configuration reference json reference You can use the following `format_options` values with `format="json"`: + `jsonPath` — A [JsonPath](https://github.com/json-path/JsonPath) expression that identifies an object to be read into records. This is particularly useful when a file contains records nested inside an outer array. For example, the following JsonPath expression targets the `id` field of a JSON object. ``` format="json", format_options={"jsonPath": "$.id"} ``` + `multiline` — A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to `"true"` if any record spans multiple lines. The default value is `"false"`, which allows for more aggressive file-splitting during parsing. + `optimizePerformance` — A Boolean value that specifies whether to use the advanced SIMD JSON reader along with Apache Arrow based columnar memory formats. Only available in AWS Glue 3.0. Not compatible with `multiline` or `jsonPath`. Providing either of those options will instruct AWS Glue to fall back to the standard reader. + `withSchema` — A String value that specifies a table schema in the format described in [Manually specify the XML schema](aws-glue-programming-etl-format-xml-home.md#aws-glue-programming-etl-format-xml-withschema). Only used with `optimizePerformance` when reading from non-Catalog connections. ## Using vectorized SIMD JSON reader with Apache Arrow columnar format Using optimized JSON reader AWS Glue version 3.0 adds a vectorized reader for JSON data. It performs 2x faster under certain conditions, compared to the standard reader. This reader comes with certain limitations users should be aware of before use, documented in this section. To use the optimized reader, set `"optimizePerformance"` to True in the `format_options` or table property. You will also need to provide `withSchema` unless reading from the catalog. `withSchema` expects an input as described in the [Manually specify the XML schema](aws-glue-programming-etl-format-xml-home.md#aws-glue-programming-etl-format-xml-withschema) ``` // Read from S3 data source glueContext.create_dynamic_frame.from_options( connection_type = "s3", connection_options = {"paths": ["s3://s3path"]}, format = "json", format_options={ "optimizePerformance": True, "withSchema": SchemaString }) // Read from catalog table glueContext.create_dynamic_frame.from_catalog( database = database, table_name = table, additional_options = { // The vectorized reader for JSON can read your schema from a catalog table property. "optimizePerformance": True, }) ``` For more information about the building a *SchemaString* in the AWS Glue library, see [PySpark extension types](aws-glue-api-crawler-pyspark-extensions-types.md). **Limitations for the vectorized CSV reader** Note the following limitations: + JSON elements with nested objects or array values are not supported. If provided, AWS Glue will fall back to the standard reader. + A schema must be provided, either from the Catalog or with `withSchema`. + Not compatible with `multiline` or `jsonPath`. Providing either of those options will instruct AWS Glue to fall back to the standard reader. + Providing input records that do not match the input schema will cause the reader to fail. + [Error records](https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-apis-glue-dynamicframe-class.html#glue-etl-scala-apis-glue-dynamicframe-class-defs-errorsAsDynamicFrame) will not be created. + JSON files with multi-byte characters (such as Japanese or Chinese characters) are not supported. # Using the ORC format in AWS Glue ORC AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats. If your data is stored or transported in the ORC data format, this document introduces you available features for using your data in AWS Glue. AWS Glue supports using the ORC format. This format is a performance-oriented, column-based data format. For an introduction to the format by the standard authority see, [Apache Orc](https://orc.apache.org/docs/). You can use AWS Glue to read ORC files from Amazon S3 and from streaming sources as well as write ORC files to Amazon S3. You can read and write `bzip` and `gzip` archives containing ORC files from S3. You configure compression behavior on the [S3 connection parameters](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3) instead of in the configuration discussed on this page. The following table shows which common AWS Glue operations support the ORC format option. | Read | Write | Streaming read | Group small files | Job bookmarks | | --- | --- | --- | --- | --- | | Supported | Supported | Supported | Unsupported | Supported\$1 | \$1Supported in AWS Glue version 1.0\$1 ## Example: Read ORC files or folders from S3 Example: Read ORC ** Prerequisites:** You will need the S3 paths (`s3path`) to the ORC files or folders that you want to read. **Configuration:** In your function options, specify `format="orc"`. In your `connection_options`, use the `paths` key to specify your `s3path`. You can configure how the reader interacts with S3 in the `connection_options`. For details, see Connection types and options for ETL in AWS Glue: [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). The following AWS Glue ETL script shows the process of reading ORC files or folders from S3: ------ #### [ Python ] For this example, use the [create\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create_dynamic_frame_from_options) method. ``` from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) dynamicFrame = glueContext.create_dynamic_frame.from_options( connection_type="s3", connection_options={"paths": ["s3://s3path"]}, format="orc" ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` dataFrame = spark.read\ .orc("s3://s3path") ``` ------ #### [ Scala ] For this example, use the [getSourceWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSourceWithFormat) operation. ``` import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.GlueContext import org.apache.spark.sql.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dynamicFrame = glueContext.getSourceWithFormat( connectionType="s3", format="orc", options=JsonOptions("""{"paths": ["s3://s3path"]}""") ).getDynamicFrame() } } ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` val dataFrame = spark.read .orc("s3://s3path") ``` ------ ## Example: Write ORC files and folders to S3 Example: Write ORC ** Prerequisites:** You will need an initialized DataFrame (`dataFrame`) or DynamicFrame (`dynamicFrame`). You will also need your expected S3 output path, `s3path`. **Configuration:** In your function options, specify `format="orc"`. In your connection options, use the `paths` key to specify `s3path`. You can further alter how the writer interacts with S3 in the `connection_options`. For details, see Data format options for ETL inputs and outputs in AWS Glue: [Amazon S3 connection option reference](aws-glue-programming-etl-connect-s3-home.md#aws-glue-programming-etl-connect-s3). The following code example shows the process: ------ #### [ Python ] For this example, use the [write\$1dynamic\$1frame.from\$1options](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_dynamic_frame_from_options) method. ``` from pyspark.context import SparkContext from awsglue.context import GlueContext sc = SparkContext.getOrCreate() glueContext = GlueContext(sc) glueContext.write_dynamic_frame.from_options( frame=dynamicFrame, connection_type="s3", format="orc", connection_options={ "path": "s3://s3path" } ) ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` df.write.orc("s3://s3path/") ``` ------ #### [ Scala ] For this example, use the [getSinkWithFormat](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getSinkWithFormat) method. ``` import com.amazonaws.services.glue.util.JsonOptions import com.amazonaws.services.glue.{DynamicFrame, GlueContext} import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getSinkWithFormat( connectionType="s3", options=JsonOptions("""{"path": "s3://s3path"}"""), format="orc" ).writeDynamicFrame(dynamicFrame) } } ``` You can also use DataFrames in a script (`pyspark.sql.DataFrame`). ``` df.write.orc("s3://s3path/") ``` ------ ## ORC configuration reference ORC reference There are no `format_options` values for `format="orc"`. However, any options that are accepted by the underlying SparkSQL code can be passed to it by way of the `connection_options` map parameter. # Using data lake frameworks with AWS Glue ETL jobs Data lake frameworks Open-source data lake frameworks simplify incremental data processing for files that you store in data lakes built on Amazon S3. AWS Glue 3.0 and later supports the following open-source data lake frameworks: + Apache Hudi + Linux Foundation Delta Lake + Apache Iceberg We provide native support for these frameworks so that you can read and write data that you store in Amazon S3 in a transactionally consistent manner. There's no need to install a separate connector or complete extra configuration steps in order to use these frameworks in AWS Glue ETL jobs. When you manage datasets through the AWS Glue Data Catalog, you can use AWS Glue methods to read and write data lake tables with Spark DataFrames. You can also read and write Amazon S3 data using the Spark DataFrame API. In this video, you can learn about the basics of how Apache Hudi, Apache Iceberg, and Delta Lake work. You'll see how to insert, update, and delete data in your data lake and how each of these frameworks works. [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/fryfx0Zg7KA/0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/fryfx0Zg7KA) **Topics** + [ # Limitations ](aws-glue-programming-etl-datalake-native-frameworks-limitations.md) + [ # Using the Hudi framework in AWS Glue ](aws-glue-programming-etl-format-hudi.md) + [ # Using the Delta Lake framework in AWS Glue ](aws-glue-programming-etl-format-delta-lake.md) + [ # Using the Iceberg framework in AWS Glue ](aws-glue-programming-etl-format-iceberg.md) # Limitations Consider the following limitations before you use data lake frameworks with AWS Glue. + The following AWS Glue `GlueContext` methods for DynamicFrame don't support reading and writing data lake framework tables. Use the `GlueContext` methods for DataFrame or Spark DataFrame API instead. + `create_dynamic_frame.from_catalog` + `write_dynamic_frame.from_catalog` + `getDynamicFrame` + `writeDynamicFrame` + The following `GlueContext` methods for DataFrame are supported with Lake Formation permission control: + `create_data_frame.from_catalog` + `write_data_frame.from_catalog` + `getDataFrame` + `writeDataFrame` + [Grouping small files](grouping-input-files.md) is not supported. + [Job bookmarks](monitor-continuations.md) are not supported. + Apache Hudi 0.10.1 for AWS Glue 3.0 doesn't support Hudi Merge on Read (MoR) tables. + `ALTER TABLE … RENAME TO` is not available for Apache Iceberg 0.13.1 for AWS Glue 3.0. ## Limitations for data lake format tables managed by Lake Formation permissions The data lake formats are integrated with AWS Glue ETL via Lake Formation permissions. Creating a DynamicFrame using `create_dynamic_frame` is not supported. For more information, see the following examples: + [Example: Read and write Iceberg table with Lake Formation permission control](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html#aws-glue-programming-etl-format-iceberg-read-write-lake-formation-tables) + [Example: Read and write Hudi table with Lake Formation permission control](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-hudi.html#aws-glue-programming-etl-format-hudi-read-write-lake-formation-tables) + [Example: Read and write Delta Lake table with Lake Formation permission control](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html#aws-glue-programming-etl-format-delta-lake-read-write-lake-formation-tables) **Note** The integration with AWS Glue ETL via Lake Formation permissions for Apache Hudi, Apache Iceberg, and Delta Lake is supported only in AWS Glue version 4.0. Apache Iceberg has the best integration with AWS Glue ETL via Lake Formation permissions. It supports almost all operations and includes SQL support. Hudi supports most basic operations with the exception of administrative operations. This is because these options generally are done via writing of dataframes and specified via `additional_options`. You need to use AWS Glue APIs to create DataFrames for your operations as SparkSQL is not supported. Delta Lake only supports the reading and appending and overwriting of table data. Delta Lake requires the use of their own libraries to be able to perform various tasks such as updates. The following features are not available for Iceberg tables managed by Lake Formation permissions. + Compaction using AWS Glue ETL + Spark SQL support via AWS Glue ETL The following are limitations of Hudi tables managed by Lake Formation permissions: + Removal of orphaned files The following are limitations of Delta Lake tables managed by Lake Formation permissions: + All features other than inserting and reading from Delta Lake tables. # Using the Hudi framework in AWS Glue Hudi AWS Glue 3.0 and later supports Apache Hudi framework for data lakes. Hudi is an open-source data lake storage framework that simplifies incremental data processing and data pipeline development. This topic covers available features for using your data in AWS Glue when you transport or store your data in a Hudi table. To learn more about Hudi, see the official [Apache Hudi documentation](https://hudi.apache.org/docs/overview/). You can use AWS Glue to perform read and write operations on Hudi tables in Amazon S3, or work with Hudi tables using the AWS Glue Data Catalog. Additional operations including insert, update, and all of the [Apache Spark operations](https://hudi.apache.org/docs/quick-start-guide/) are also supported. **Note** The Apache Hudi 0.15.0 implementation in AWS Glue 5.0 internally reverts [HUDI-7001](https://github.com/apache/hudi/pull/9936). It does not exhibit the regression related to Complex Key gen when the record key consists of a single field. However this behavior differs from OSS Apache Hudi 0.15.0. Apache Hudi 0.10.1 for AWS Glue 3.0 doesn't support Hudi Merge on Read (MoR) tables. The following table lists the Hudi version that is included in each AWS Glue version. **** | AWS Glue version | Supported Hudi version | | --- | --- | | 5.1 | 1.0.2 | | 5.0 | 0.15.0 | | 4.0 | 0.12.1 | | 3.0 | 0.10.1 | To learn more about the data lake frameworks that AWS Glue supports, see [Using data lake frameworks with AWS Glue ETL jobs](aws-glue-programming-etl-datalake-native-frameworks.md). ## Enabling Hudi Enabling Hudi To enable Hudi for AWS Glue, complete the following tasks: + Specify `hudi` as a value for the `--datalake-formats` job parameter. For more information, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). + Create a key named `--conf` for your AWS Glue job, and set it to the following value. Alternatively, you can set the following configuration using `SparkConf` in your script. These settings help Apache Spark correctly handle Hudi tables. ``` spark.serializer=org.apache.spark.serializer.KryoSerializer ``` + Lake Formation permission support for Hudi is enabled by default for AWS Glue 4.0. No additional configuration is needed for reading/writing to Lake Formation-registered Hudi tables. To read a registered Hudi table, the AWS Glue job IAM role must have the SELECT permission. To write to a registered Hudi table, the AWS Glue job IAM role must have the SUPER permission. To learn more about managing Lake Formation permissions, see [Granting and revoking permissions on Data Catalog resources](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-catalog-permissions.html). **Using a different Hudi version** To use a version of Hudi that AWS Glue doesn't support, specify your own Hudi JAR files using the `--extra-jars` job parameter. Do not include `hudi` as a value for the `--datalake-formats` job parameter. If you use AWS Glue 5.0 or above, you must set `--user-jars-first true` job parameter. ## Example: Write a Hudi table to Amazon S3 and register it in the AWS Glue Data Catalog Example: Write Hudi This example script demonstrates how to write a Hudi table to Amazon S3 and register the table to the AWS Glue Data Catalog. The example uses the Hudi [Hive Sync tool](https://hudi.apache.org/docs/syncing_metastore/) to register the table. **Note** This example requires you to set the `--enable-glue-datacatalog` job parameter in order to use the AWS Glue Data Catalog as an Apache Spark Hive metastore. To learn more, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). ------ #### [ Python ] ``` # Example: Create a Hudi table from a DataFrame # and register the table to Glue Data Catalog additional_options={ "hoodie.table.name": "", "hoodie.database.name": "", "hoodie.datasource.write.storage.type": "COPY_ON_WRITE", "hoodie.datasource.write.operation": "upsert", "hoodie.datasource.write.recordkey.field": "", "hoodie.datasource.write.precombine.field": "", "hoodie.datasource.write.partitionpath.field": "", "hoodie.datasource.write.hive_style_partitioning": "true", "hoodie.datasource.hive_sync.enable": "true", "hoodie.datasource.hive_sync.database": "", "hoodie.datasource.hive_sync.table": "", "hoodie.datasource.hive_sync.partition_fields": "", "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.use_jdbc": "false", "hoodie.datasource.hive_sync.mode": "hms", "path": "s3://" } dataFrame.write.format("hudi") \ .options(**additional_options) \ .mode("overwrite") \ .save() ``` ------ #### [ Scala ] ``` // Example: Example: Create a Hudi table from a DataFrame // and register the table to Glue Data Catalog val additionalOptions = Map( "hoodie.table.name" -> "", "hoodie.database.name" -> "", "hoodie.datasource.write.storage.type" -> "COPY_ON_WRITE", "hoodie.datasource.write.operation" -> "upsert", "hoodie.datasource.write.recordkey.field" -> "", "hoodie.datasource.write.precombine.field" -> "", "hoodie.datasource.write.partitionpath.field" -> "", "hoodie.datasource.write.hive_style_partitioning" -> "true", "hoodie.datasource.hive_sync.enable" -> "true", "hoodie.datasource.hive_sync.database" -> "", "hoodie.datasource.hive_sync.table" -> "", "hoodie.datasource.hive_sync.partition_fields" -> "", "hoodie.datasource.hive_sync.partition_extractor_class" -> "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.use_jdbc" -> "false", "hoodie.datasource.hive_sync.mode" -> "hms", "path" -> "s3://") dataFrame.write.format("hudi") .options(additionalOptions) .mode("append") .save() ``` ------ ## Example: Read a Hudi table from Amazon S3 using the AWS Glue Data Catalog Example: Read Hudi This example reads the Hudi table that you created in the [Example: Write a Hudi table to Amazon S3 and register it in the AWS Glue Data Catalog](#aws-glue-programming-etl-format-hudi-write) from Amazon S3. **Note** This example requires you to set the `--enable-glue-datacatalog` job parameter in order to use the AWS Glue Data Catalog as an Apache Spark Hive metastore. To learn more, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). ------ #### [ Python ] For this example, use the `GlueContext.create\$1data\$1frame.from\$1catalog()` method. ``` # Example: Read a Hudi table from Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) dataFrame = glueContext.create_data_frame.from_catalog( database = "", table_name = "" ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSource](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSource) method. ``` // Example: Read a Hudi table from Glue Data Catalog import com.amazonaws.services.glue.GlueContext import org.apache.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val dataFrame = glueContext.getCatalogSource( database = "", tableName = "" ).getDataFrame() } } ``` ------ ## Example: Update and insert a `DataFrame` into a Hudi table in Amazon S3 Example: Update and insert into a Hudi table This example uses the AWS Glue Data Catalog to insert a DataFrame into the Hudi table that you created in [Example: Write a Hudi table to Amazon S3 and register it in the AWS Glue Data Catalog](#aws-glue-programming-etl-format-hudi-write). **Note** This example requires you to set the `--enable-glue-datacatalog` job parameter in order to use the AWS Glue Data Catalog as an Apache Spark Hive metastore. To learn more, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). ------ #### [ Python ] For this example, use the `GlueContext.write\$1data\$1frame.from\$1catalog()` method. ``` # Example: Upsert a Hudi table from Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) glueContext.write_data_frame.from_catalog( frame = dataFrame, database = "", table_name = "", additional_options={ "hoodie.table.name": "", "hoodie.database.name": "", "hoodie.datasource.write.storage.type": "COPY_ON_WRITE", "hoodie.datasource.write.operation": "upsert", "hoodie.datasource.write.recordkey.field": "", "hoodie.datasource.write.precombine.field": "", "hoodie.datasource.write.partitionpath.field": "", "hoodie.datasource.write.hive_style_partitioning": "true", "hoodie.datasource.hive_sync.enable": "true", "hoodie.datasource.hive_sync.database": "", "hoodie.datasource.hive_sync.table": "", "hoodie.datasource.hive_sync.partition_fields": "", "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.use_jdbc": "false", "hoodie.datasource.hive_sync.mode": "hms" } ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSink](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSink) method. ``` // Example: Upsert a Hudi table from Glue Data Catalog import com.amazonaws.services.glue.GlueContext import com.amazonaws.services.glue.util.JsonOptions import org.apacke.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getCatalogSink("", "", additionalOptions = JsonOptions(Map( "hoodie.table.name" -> "", "hoodie.database.name" -> "", "hoodie.datasource.write.storage.type" -> "COPY_ON_WRITE", "hoodie.datasource.write.operation" -> "upsert", "hoodie.datasource.write.recordkey.field" -> "", "hoodie.datasource.write.precombine.field" -> "", "hoodie.datasource.write.partitionpath.field" -> "", "hoodie.datasource.write.hive_style_partitioning" -> "true", "hoodie.datasource.hive_sync.enable" -> "true", "hoodie.datasource.hive_sync.database" -> "", "hoodie.datasource.hive_sync.table" -> "", "hoodie.datasource.hive_sync.partition_fields" -> "", "hoodie.datasource.hive_sync.partition_extractor_class" -> "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.use_jdbc" -> "false", "hoodie.datasource.hive_sync.mode" -> "hms" ))) .writeDataFrame(dataFrame, glueContext) } } ``` ------ ## Example: Read a Hudi table from Amazon S3 using Spark Example: Read a Hudi table using Spark This example reads a Hudi table from Amazon S3 using the Spark DataFrame API. ------ #### [ Python ] ``` # Example: Read a Hudi table from S3 using a Spark DataFrame dataFrame = spark.read.format("hudi").load("s3://") ``` ------ #### [ Scala ] ``` // Example: Read a Hudi table from S3 using a Spark DataFrame val dataFrame = spark.read.format("hudi").load("s3://") ``` ------ ## Example: Write a Hudi table to Amazon S3 using Spark Example: Write a Hudi table using Spark This example writes a Hudi table to Amazon S3 using Spark. ------ #### [ Python ] ``` # Example: Write a Hudi table to S3 using a Spark DataFrame dataFrame.write.format("hudi") \ .options(**additional_options) \ .mode("overwrite") \ .save("s3://) ``` ------ #### [ Scala ] ``` // Example: Write a Hudi table to S3 using a Spark DataFrame dataFrame.write.format("hudi") .options(additionalOptions) .mode("overwrite") .save("s3://") ``` ------ ## Example: Read and write Hudi table with Lake Formation permission control Example: Read and write Hudi table with Lake Formation permission control This example reads and writes a Hudi table with Lake Formation permission control. 1. Create a Hudi table and register it in Lake Formation. 1. To enable Lake Formation permission control, you’ll first need to register the table Amazon S3 path on Lake Formation. For more information, see [Registering an Amazon S3 location](https://docs.aws.amazon.com/lake-formation/latest/dg/register-location.html). You can register it either from the Lake Formation console or by using the AWS CLI: ``` aws lakeformation register-resource --resource-arn arn:aws:s3:::/ --use-service-linked-role --region ``` Once you register an Amazon S3 location, any AWS Glue table pointing to the location (or any of its child locations) will return the value for the `IsRegisteredWithLakeFormation` parameter as true in the `GetTable` call. 1. Create a Hudi table that points to the registered Amazon S3 path through the Spark dataframe API: ``` hudi_options = { 'hoodie.table.name': table_name, 'hoodie.database.name': database_name, 'hoodie.datasource.write.storage.type': 'COPY_ON_WRITE', 'hoodie.datasource.write.recordkey.field': 'product_id', 'hoodie.datasource.write.table.name': table_name, 'hoodie.datasource.write.operation': 'upsert', 'hoodie.datasource.write.precombine.field': 'updated_at', 'hoodie.datasource.write.hive_style_partitioning': 'true', 'hoodie.upsert.shuffle.parallelism': 2, 'hoodie.insert.shuffle.parallelism': 2, 'path': , 'hoodie.datasource.hive_sync.enable': 'true', 'hoodie.datasource.hive_sync.database': database_name, 'hoodie.datasource.hive_sync.table': table_name, 'hoodie.datasource.hive_sync.use_jdbc': 'false', 'hoodie.datasource.hive_sync.mode': 'hms' } df_products.write.format("hudi") \ .options(**hudi_options) \ .mode("overwrite") \ .save() ``` 1. Grant Lake Formation permission to the AWS Glue job IAM role. You can either grant permissions from the Lake Formation console, or using the AWS CLI. For more information, see [Granting table permissions using the Lake Formation console and the named resource method](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-table-permissions.html) 1. Read the Hudi table registered in Lake Formation. The code is same as reading a non-registered Hudi table. Note that the AWS Glue job IAM role needs to have the SELECT permission for the read to succeed. ``` val dataFrame = glueContext.getCatalogSource( database = "", tableName = "" ).getDataFrame() ``` 1. Write to a Hudi table registered in Lake Formation. The code is same as writing to a non-registered Hudi table. Note that the AWS Glue job IAM role needs to have the SUPER permission for the write to succeed. ``` glueContext.getCatalogSink("", "", additionalOptions = JsonOptions(Map( "hoodie.table.name" -> "", "hoodie.database.name" -> "", "hoodie.datasource.write.storage.type" -> "COPY_ON_WRITE", "hoodie.datasource.write.operation" -> "", "hoodie.datasource.write.recordkey.field" -> "", "hoodie.datasource.write.precombine.field" -> "", "hoodie.datasource.write.partitionpath.field" -> "", "hoodie.datasource.write.hive_style_partitioning" -> "true", "hoodie.datasource.hive_sync.enable" -> "true", "hoodie.datasource.hive_sync.database" -> "", "hoodie.datasource.hive_sync.table" -> "", "hoodie.datasource.hive_sync.partition_fields" -> "", "hoodie.datasource.hive_sync.partition_extractor_class" -> "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.use_jdbc" -> "false", "hoodie.datasource.hive_sync.mode" -> "hms" ))) .writeDataFrame(dataFrame, glueContext) ``` # Using the Delta Lake framework in AWS Glue Delta Lake AWS Glue 3.0 and later supports the Linux Foundation Delta Lake framework. Delta Lake is an open-source data lake storage framework that helps you perform ACID transactions, scale metadata handling, and unify streaming and batch data processing. This topic covers available features for using your data in AWS Glue when you transport or store your data in a Delta Lake table. To learn more about Delta Lake, see the official [Delta Lake documentation](https://docs.delta.io/latest/delta-intro.html). You can use AWS Glue to perform read and write operations on Delta Lake tables in Amazon S3, or work with Delta Lake tables using the AWS Glue Data Catalog. Additional operations such as insert, update, and [Table batch reads and writes](https://docs.delta.io/0.7.0/api/python/index.html) are also supported. When you use Delta Lake tables, you also have the option to use methods from the Delta Lake Python library such as `DeltaTable.forPath`. For more information about the Delta Lake Python library, see Delta Lake's Python documentation. The following table lists the version of Delta Lake included in each AWS Glue version. **** | AWS Glue version | Supported Delta Lake version | | --- | --- | | 5.1 | 3.3.2 | | 5.0 | 3.3.0 | | 4.0 | 2.1.0 | | 3.0 | 1.0.0 | To learn more about the data lake frameworks that AWS Glue supports, see [Using data lake frameworks with AWS Glue ETL jobs](aws-glue-programming-etl-datalake-native-frameworks.md). ## Enabling Delta Lake for AWS Glue Enabling Delta Lake To enable Delta Lake for AWS Glue, complete the following tasks: + Specify `delta` as a value for the `--datalake-formats` job parameter. For more information, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). + Create a key named `--conf` for your AWS Glue job, and set it to the following value. Alternatively, you can set the following configuration using `SparkConf` in your script. These settings help Apache Spark correctly handle Delta Lake tables. ``` spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore ``` + Lake Formation permission support for Delta tables is enabled by default for AWS Glue 4.0. No additional configuration is needed for reading/writing to Lake Formation-registered Delta tables. To read a registered Delta table, the AWS Glue job IAM role must have the SELECT permission. To write to a registered Delta table, the AWS Glue job IAM role must have the SUPER permission. To learn more about managing Lake Formation permissions, see [Granting and revoking permissions on Data Catalog resources](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-catalog-permissions.html). **Using a different Delta Lake version** To use a version of Delta lake that AWS Glue doesn't support, specify your own Delta Lake JAR files using the `--extra-jars` job parameter. Do not include `delta` as a value for the `--datalake-formats` job parameter. If you use AWS Glue 5.0 or above, you must set `--user-jars-first true` job parameter. To use the Delta Lake Python library in this case, you must specify the library JAR files using the `--extra-py-files` job parameter. The Python library comes packaged in the Delta Lake JAR files. ## Example: Write a Delta Lake table to Amazon S3 and register it to the AWS Glue Data Catalog Example: Write Delta Lake The following AWS Glue ETL script demonstrates how to write a Delta Lake table to Amazon S3 and register the table to the AWS Glue Data Catalog. ------ #### [ Python ] ``` # Example: Create a Delta Lake table from a DataFrame # and register the table to Glue Data Catalog additional_options = { "path": "s3://" } dataFrame.write \ .format("delta") \ .options(**additional_options) \ .mode("append") \ .partitionBy("") \ .saveAsTable(".") ``` ------ #### [ Scala ] ``` // Example: Example: Create a Delta Lake table from a DataFrame // and register the table to Glue Data Catalog val additional_options = Map( "path" -> "s3://" ) dataFrame.write.format("delta") .options(additional_options) .mode("append") .partitionBy("") .saveAsTable(".") ``` ------ ## Example: Read a Delta Lake table from Amazon S3 using the AWS Glue Data Catalog Example: Read Delta Lake The following AWS Glue ETL script reads the Delta Lake table that you created in [Example: Write a Delta Lake table to Amazon S3 and register it to the AWS Glue Data Catalog](#aws-glue-programming-etl-format-delta-lake-write). ------ #### [ Python ] For this example, use the [create\$1data\$1frame.from\$1catalog](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-create-dataframe-from-catalog) method. ``` # Example: Read a Delta Lake table from Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) df = glueContext.create_data_frame.from_catalog( database="", table_name="", additional_options=additional_options ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSource](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSource) method. ``` // Example: Read a Delta Lake table from Glue Data Catalog import com.amazonaws.services.glue.GlueContext import org.apacke.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val df = glueContext.getCatalogSource("", "", additionalOptions = additionalOptions) .getDataFrame() } } ``` ------ ## Example: Insert a `DataFrame` into a Delta Lake table in Amazon S3 using the AWS Glue Data Catalog Example: Insert into a Delta Lake table This example inserts data into the Delta Lake table that you created in [Example: Write a Delta Lake table to Amazon S3 and register it to the AWS Glue Data Catalog](#aws-glue-programming-etl-format-delta-lake-write). **Note** This example requires you to set the `--enable-glue-datacatalog` job parameter in order to use the AWS Glue Data Catalog as an Apache Spark Hive metastore. To learn more, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). ------ #### [ Python ] For this example, use the [write\$1data\$1frame.from\$1catalog](aws-glue-api-crawler-pyspark-extensions-glue-context.md#aws-glue-api-crawler-pyspark-extensions-glue-context-write_data_frame_from_catalog) method. ``` # Example: Insert into a Delta Lake table in S3 using Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) glueContext.write_data_frame.from_catalog( frame=dataFrame, database="", table_name="", additional_options=additional_options ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSink](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSink) method. ``` // Example: Insert into a Delta Lake table in S3 using Glue Data Catalog import com.amazonaws.services.glue.GlueContext import org.apacke.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getCatalogSink("", "", additionalOptions = additionalOptions) .writeDataFrame(dataFrame, glueContext) } } ``` ------ ## Example: Read a Delta Lake table from Amazon S3 using the Spark API Example: Read a Delta Lake table using Spark This example reads a Delta Lake table from Amazon S3 using the Spark API. ------ #### [ Python ] ``` # Example: Read a Delta Lake table from S3 using a Spark DataFrame dataFrame = spark.read.format("delta").load("s3://") ``` ------ #### [ Scala ] ``` // Example: Read a Delta Lake table from S3 using a Spark DataFrame val dataFrame = spark.read.format("delta").load("s3://") ``` ------ ## Example: Write a Delta Lake table to Amazon S3 using Spark Example: Write a Delta Lake table using Spark This example writes a Delta Lake table to Amazon S3 using Spark. ------ #### [ Python ] ``` # Example: Write a Delta Lake table to S3 using a Spark DataFrame dataFrame.write.format("delta") \ .options(**additional_options) \ .mode("overwrite") \ .partitionBy("") .save("s3://") ``` ------ #### [ Scala ] ``` // Example: Write a Delta Lake table to S3 using a Spark DataFrame dataFrame.write.format("delta") .options(additionalOptions) .mode("overwrite") .partitionBy("") .save("s3://") ``` ------ ## Example: Read and write Delta Lake table with Lake Formation permission control Example: Read and write Delta Lake table with Lake Formation permission control This example reads and writes a Delta Lake table with Lake Formation permission control. 1. Create a Delta table and register it in Lake Formation 1. To enable Lake Formation permission control, you’ll first need to register the table Amazon S3 path on Lake Formation. For more information, see [Registering an Amazon S3 location](https://docs.aws.amazon.com/lake-formation/latest/dg/register-location.html). You can register it either from the Lake Formation console or by using the AWS CLI: ``` aws lakeformation register-resource --resource-arn arn:aws:s3:::/ --use-service-linked-role --region ``` Once you register an Amazon S3 location, any AWS Glue table pointing to the location (or any of its child locations) will return the value for the `IsRegisteredWithLakeFormation` parameter as true in the `GetTable` call. 1. Create a Delta table that points to the registered Amazon S3 path through Spark: **Note** The following are Python examples. ``` dataFrame.write \ .format("delta") \ .mode("overwrite") \ .partitionBy("") \ .save("s3://") ``` After the data has been written to Amazon S3, use the AWS Glue crawler to create a new Delta catalog table. For more information, see [Introducing native Delta Lake table support with AWS Glue crawlers](https://aws.amazon.com/blogs/big-data/introducing-native-delta-lake-table-support-with-aws-glue-crawlers/). You can also create the table manually through the AWS Glue `CreateTable` API. 1. Grant Lake Formation permission to the AWS Glue job IAM role. You can either grant permissions from the Lake Formation console, or using the AWS CLI. For more information, see [Granting table permissions using the Lake Formation console and the named resource method](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-table-permissions.html) 1. Read the Delta table registered in Lake Formation. The code is same as reading a non-registered Delta table. Note that the AWS Glue job IAM role needs to have the SELECT permission for the read to succeed. ``` # Example: Read a Delta Lake table from Glue Data Catalog df = glueContext.create_data_frame.from_catalog( database="", table_name="", additional_options=additional_options ) ``` 1. Write to a Delta table registered in Lake Formation. The code is same as writing to a non-registered Delta table. Note that the AWS Glue job IAM role needs to have the SUPER permission for the write to succeed. By default AWS Glue uses `Append` as saveMode. You can change it by setting the saveMode option in `additional_options`. For information about saveMode support in Delta tables, see [Write to a table](https://docs.delta.io/latest/delta-batch.html#write-to-a-table). ``` glueContext.write_data_frame.from_catalog( frame=dataFrame, database="", table_name="", additional_options=additional_options ) ``` # Using the Iceberg framework in AWS Glue Iceberg AWS Glue 3.0 and later supports the Apache Iceberg framework for data lakes. Iceberg provides a high-performance table format that works just like a SQL table. This topic covers available features for using your data in AWS Glue when you transport or store your data in an Iceberg table. To learn more about Iceberg, see the official [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/). You can use AWS Glue to perform read and write operations on Iceberg tables in Amazon S3, or work with Iceberg tables using the AWS Glue Data Catalog. Additional operations including insert and all [Spark Queries](https://iceberg.apache.org/docs/latest/spark-queries/) [Spark Writes](https://iceberg.apache.org/docs/latest/spark-writes/) are also supported. Update is not supported for Iceberg tables. **Note** `ALTER TABLE … RENAME TO` is not available for Apache Iceberg 0.13.1 for AWS Glue 3.0. The following table lists the version of Iceberg included in each AWS Glue version. **** | AWS Glue version | Supported Iceberg version | | --- | --- | | 5.1 | 1.10.0 | | 5.0 | 1.7.1 | | 4.0 | 1.0.0 | | 3.0 | 0.13.1 | To learn more about the data lake frameworks that AWS Glue supports, see [Using data lake frameworks with AWS Glue ETL jobs](aws-glue-programming-etl-datalake-native-frameworks.md). ## Enabling the Iceberg framework Enabling Iceberg To enable Iceberg for AWS Glue, complete the following tasks: + Specify `iceberg` as a value for the `--datalake-formats` job parameter. For more information, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). + Create a key named `--conf` for your AWS Glue job, and set it to the following value. Alternatively, you can set the following configuration using `SparkConf` in your script. These settings help Apache Spark correctly handle Iceberg tables. ``` spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.warehouse=s3:/// --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO ``` If you are reading or writing to Iceberg tables that are registered with Lake Formation, follow the guidance in [Using AWS Glue with AWS Lake Formation for fine-grained access control](security-lf-enable.md) in AWS Glue 5.0 and later. In AWS Glue 4.0, add the following configuration to enable Lake Formation support. ``` --conf spark.sql.catalog.glue_catalog.glue.lakeformation-enabled=true --conf spark.sql.catalog.glue_catalog.glue.id= ``` If you use AWS Glue 3.0 with Iceberg 0.13.1, you must set the following additional configurations to use Amazon DynamoDB lock manager to ensure atomic transaction. AWS Glue 4.0 or later uses optimistic locking by default. For more information, see [Iceberg AWS Integrations](https://iceberg.apache.org/docs/latest/aws/#dynamodb-lock-manager) in the official Apache Iceberg documentation. ``` --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManager --conf spark.sql.catalog.glue_catalog.lock.table= ``` **Using a different Iceberg version** To use a version of Iceberg that AWS Glue doesn't support, specify your own Iceberg JAR files using the `--extra-jars` job parameter. Do not include `iceberg` as a value for the `--datalake-formats` parameter. If you use AWS Glue 5.0 or above, you must set `--user-jars-first true` job parameter. **Enabling encryption for Iceberg tables** **Note** Iceberg tables have their own mechanisms to enable server-side encryption. You should enable this configuration in addition to AWS Glue's security configuration. To enable server-side encryption on Iceberg tables, review the guidance from the [Iceberg documentation](https://iceberg.apache.org/docs/latest/aws/#s3-server-side-encryption). **Add Spark configuration for Iceberg cross region** To add additional spark configuration for Iceberg cross-region table access with the AWS Glue Data Catalog and AWS Lake Formation, follow the steps below: 1. Create a [Multi-region access point](https://docs.aws.amazon.com/AmazonS3/latest/userguide/multi-region-access-point-create-examples.html). 1. Set the following Spark properties: ``` ----- --conf spark.sql.catalog.my_catalog.s3.use-arn-region-enabled=true \ --conf spark.sql.catalog.{CATALOG}.s3.access-points.bucket1", "arn:aws:s3:::accesspoint/.mrap \ --conf spark.sql.catalog.{CATALOG}.s3.access-points.bucket2", "arn:aws:s3:::accesspoint/.mrap ----- ``` ## Example: Write an Iceberg table to Amazon S3 and register it to the AWS Glue Data Catalog Example: Write Iceberg This example script demonstrates how to write an Iceberg table to Amazon S3. The example uses [Iceberg AWS Integrations](https://iceberg.apache.org/docs/latest/aws/) to register the table to the AWS Glue Data Catalog. ------ #### [ Python ] ``` # Example: Create an Iceberg table from a DataFrame # and register the table to Glue Data Catalog dataFrame.createOrReplaceTempView("tmp_") query = f""" CREATE TABLE glue_catalog.. USING iceberg TBLPROPERTIES ("format-version"="2") AS SELECT * FROM tmp_ """ spark.sql(query) ``` ------ #### [ Scala ] ``` // Example: Example: Create an Iceberg table from a DataFrame // and register the table to Glue Data Catalog dataFrame.createOrReplaceTempView("tmp_") val query = """CREATE TABLE glue_catalog.. USING iceberg TBLPROPERTIES ("format-version"="2") AS SELECT * FROM tmp_ """ spark.sql(query) ``` ------ Alternatively, you can write an Iceberg table to Amazon S3 and the Data Catalog using Spark methods. Prerequisites: You will need to provision a catalog for the Iceberg library to use. When using the AWS Glue Data Catalog, AWS Glue makes this straightforward. The AWS Glue Data Catalog is pre-configured for use by the Spark libraries as `glue_catalog`. Data Catalog tables are identified by a *databaseName* and a *tableName*. For more information about the AWS Glue Data Catalog, see [Data discovery and cataloging in AWS Glue](catalog-and-crawler.md). If you are not using the AWS Glue Data Catalog, you will need to provision a catalog through the Spark APIs. For more information, see [Spark Configuration](https://iceberg.apache.org/docs/latest/spark-configuration/) in the Iceberg documentation. This example writes an Iceberg table to Amazon S3 and the Data Catalog using Spark. ------ #### [ Python ] ``` # Example: Write an Iceberg table to S3 on the Glue Data Catalog # Create (equivalent to CREATE TABLE AS SELECT) dataFrame.writeTo("glue_catalog.databaseName.tableName") \ .tableProperty("format-version", "2") \ .create() # Append (equivalent to INSERT INTO) dataFrame.writeTo("glue_catalog.databaseName.tableName") \ .tableProperty("format-version", "2") \ .append() ``` ------ #### [ Scala ] ``` // Example: Write an Iceberg table to S3 on the Glue Data Catalog // Create (equivalent to CREATE TABLE AS SELECT) dataFrame.writeTo("glue_catalog.databaseName.tableName") .tableProperty("format-version", "2") .create() // Append (equivalent to INSERT INTO) dataFrame.writeTo("glue_catalog.databaseName.tableName") .tableProperty("format-version", "2") .append() ``` ------ ## Example: Read an Iceberg table from Amazon S3 using the AWS Glue Data Catalog Example: Read Iceberg This example reads the Iceberg table that you created in [Example: Write an Iceberg table to Amazon S3 and register it to the AWS Glue Data Catalog](#aws-glue-programming-etl-format-iceberg-write). ------ #### [ Python ] For this example, use the `GlueContext.create\$1data\$1frame.from\$1catalog()` method. ``` # Example: Read an Iceberg table from Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) df = glueContext.create_data_frame.from_catalog( database="", table_name="", additional_options=additional_options ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSource](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSource) method. ``` // Example: Read an Iceberg table from Glue Data Catalog import com.amazonaws.services.glue.GlueContext import org.apacke.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) val df = glueContext.getCatalogSource("", "", additionalOptions = additionalOptions) .getDataFrame() } } ``` ------ ## Example: Insert a `DataFrame` into an Iceberg table in Amazon S3 using the AWS Glue Data Catalog Example: Insert into an Iceberg table This example inserts data into the Iceberg table that you created in [Example: Write an Iceberg table to Amazon S3 and register it to the AWS Glue Data Catalog](#aws-glue-programming-etl-format-iceberg-write). **Note** This example requires you to set the `--enable-glue-datacatalog` job parameter in order to use the AWS Glue Data Catalog as an Apache Spark Hive metastore. To learn more, see [Using job parameters in AWS Glue jobs](aws-glue-programming-etl-glue-arguments.md). ------ #### [ Python ] For this example, use the `GlueContext.write\$1data\$1frame.from\$1catalog()` method. ``` # Example: Insert into an Iceberg table from Glue Data Catalog from awsglue.context import GlueContext from pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) glueContext.write_data_frame.from_catalog( frame=dataFrame, database="", table_name="", additional_options=additional_options ) ``` ------ #### [ Scala ] For this example, use the [getCatalogSink](glue-etl-scala-apis-glue-gluecontext.md#glue-etl-scala-apis-glue-gluecontext-defs-getCatalogSink) method. ``` // Example: Insert into an Iceberg table from Glue Data Catalog import com.amazonaws.services.glue.GlueContext import org.apacke.spark.SparkContext object GlueApp { def main(sysArgs: Array[String]): Unit = { val spark: SparkContext = new SparkContext() val glueContext: GlueContext = new GlueContext(spark) glueContext.getCatalogSink("", "", additionalOptions = additionalOptions) .writeDataFrame(dataFrame, glueContext) } } ``` ------ ## Example: Read an Iceberg table from Amazon S3 using Spark Example: Read an Iceberg table using Spark Prerequisites: You will need to provision a catalog for the Iceberg library to use. When using the AWS Glue Data Catalog, AWS Glue makes this straightforward. The AWS Glue Data Catalog is pre-configured for use by the Spark libraries as `glue_catalog`. Data Catalog tables are identified by a *databaseName* and a *tableName*. For more information about the AWS Glue Data Catalog, see [Data discovery and cataloging in AWS Glue](catalog-and-crawler.md). If you are not using the AWS Glue Data Catalog, you will need to provision a catalog through the Spark APIs. For more information, see [Spark Configuration](https://iceberg.apache.org/docs/latest/spark-configuration/) in the Iceberg documentation. This example reads an Iceberg table in Amazon S3 from the Data Catalog using Spark. ------ #### [ Python ] ``` # Example: Read an Iceberg table on S3 as a DataFrame from the Glue Data Catalog dataFrame = spark.read.format("iceberg").load("glue_catalog.databaseName.tableName") ``` ------ #### [ Scala ] ``` // Example: Read an Iceberg table on S3 as a DataFrame from the Glue Data Catalog val dataFrame = spark.read.format("iceberg").load("glue_catalog.databaseName.tableName") ``` ------ ## Example: Read and write Iceberg table with Lake Formation permission control Example: Read and write Iceberg table with Lake Formation permission control This example reads and writes an Iceberg table with Lake Formation permission control. **Note** This example works only in AWS Glue 4.0. In AWS Glue 5.0 and later, follow the guidance in [Using AWS Glue with AWS Lake Formation for fine-grained access control](security-lf-enable.md). 1. Create an Iceberg table and register it in Lake Formation: 1. To enable Lake Formation permission control, you’ll first need to register the table Amazon S3 path on Lake Formation. For more information, see [Registering an Amazon S3 location](https://docs.aws.amazon.com/lake-formation/latest/dg/register-location.html). You can register it either from the Lake Formation console or by using the AWS CLI: ``` aws lakeformation register-resource --resource-arn arn:aws:s3:::/ --use-service-linked-role --region ``` Once you register an Amazon S3 location, any AWS Glue table pointing to the location (or any of its child locations) will return the value for the `IsRegisteredWithLakeFormation` parameter as true in the `GetTable` call. 1. Create an Iceberg table that points to the registered path through Spark SQL: **Note** The following are Python examples. ``` dataFrame.createOrReplaceTempView("tmp_") query = f""" CREATE TABLE glue_catalog.. USING iceberg AS SELECT * FROM tmp_ """ spark.sql(query) ``` You can also create the table manually through AWS Glue `CreateTable` API. For more information, see [Creating Apache Iceberg tables](https://docs.aws.amazon.com/lake-formation/latest/dg/creating-iceberg-tables.html). **Note** The `UpdateTable` API does not currently support Iceberg table format as an input to the operation. 1. Grant Lake Formation permission to the job IAM role. You can either grant permissions from the Lake Formation console, or using the AWS CLI. For more information, see: https://docs.aws.amazon.com/lake-formation/latest/dg/granting-table-permissions.html 1. Read an Iceberg table registered with Lake Formation. The code is same as reading a non-registered Iceberg table. Note that your AWS Glue job IAM role needs to have the SELECT permission for the read to succeed. ``` # Example: Read an Iceberg table from the AWS Glue Data Catalog from awsglue.context import GlueContextfrom pyspark.context import SparkContext sc = SparkContext() glueContext = GlueContext(sc) df = glueContext.create_data_frame.from_catalog( database="", table_name="", additional_options=additional_options ) ``` 1. Write to an Iceberg table registered with Lake Formation. The code is same as writing to a non-registered Iceberg table. Note that your AWS Glue job IAM role needs to have the SUPER permission for the write to succeed. ``` glueContext.write_data_frame.from_catalog( frame=dataFrame, database="", table_name="", additional_options=additional_options ) ``` ## Shared configuration reference Shared configuration reference You can use the following `format_options` values with any format type. + `attachFilename` — A string in the appropriate format to be used as a column name. If you provide this option, the name of the source file for the record will be appended to the record. The parameter value will be used as the column name. + `attachTimestamp` — A string in the appropriate format to be used as a column name. If you provide this option, the modification time of the source file for the record will be appended to the record. The parameter value will be used as the column name.