View a markdown version of this page

Using the Federal Aviation Administration SWIM Data Lake - Using the Federal Aviation Administration SWIM Data Lake
FAA SWIM data lake diagramFurther readingDiagram history

Using the Federal Aviation Administration SWIM Data Lake

Publication date: March 10, 2021 (Diagram history)

The Federal Aviation Administration (FAA) System Wide Information Management (SWIM) Data Lake provides real-time delivery of aeronautical flight and weather information. This helps airline companies optimize their operations.

This architecture processes SWIM data through a series of AWS Lambda functions and Amazon Kinesis Data Streams. It transforms, enriches, and analyzes flight data. You can then visualize the results with Amazon Quick Sight or consume them through custom applications.

FAA SWIM data lake diagram

Architecture for FAA SWIM data lake using Amazon Kinesis, AWS Lambda, and AWS Glue.

The following steps describe the architecture:

  1. The SWIM Consumer App consumes SWIM data. It sends the data to an Kinesis Data Stream.

  2. Lambda converts the XML data to JSON format.

  3. Lambda simplifies the JSON data for downstream processing.

  4. Lambda filters customer flight data. It writes one copy to Kinesis Data Stream and a second to the subsequent Lambda function.

  5. Lambda enriches customer flight messages with operational or external datasets.

  6. Lambda writes enriched flight data to Amazon Aurora.

  7. Lambda writes enriched flight data to Kinesis Data Stream.

  8. AWS Glue extract, transform, and load (ETL) jobs transform enriched flight data from JSON to Parquet with appropriate partitions. The jobs register Parquet files as external tables in AWS Glue Data Catalog.

  9. Services such as Athena, Amazon Redshift, and Amazon Aurora analyze the transformed and enriched flight data.

  10. Services such as Quick, custom applications, and APIs consume the analyzed data.

  11. AWS Glue Data Catalog stores databases and tables. These represent raw, transformed, enriched, and analyzed datasets in Amazon S3, Amazon Aurora, and Amazon Redshift.

Further reading

For additional information, see the following resources:

Diagram history

To receive updates about this reference architecture diagram, subscribe to the RSS feed.

ChangeDescriptionDate

Initial publication

Reference architecture diagram first published.

March 10, 2021

RSS subscription requirement

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.