View a markdown version of this page

Expose Your Data Lake as a GraphQL API - Expose Your Data Lake as a GraphQL API
Expose Your Data Lake as a GraphQL APIFurther readingDiagram history

Expose Your Data Lake as a GraphQL API

Publication date: April 25, 2022 (Diagram history)

This architecture shows how to build a data lake using Lake Formation and share it as a self-documented GraphQL API. You can automatically build the API with a reactive CI/CD pipeline, and allow authorized external clients to query your data lake using GraphQL syntax over HTTPS.

Expose Your Data Lake as a GraphQL API

Architecture diagram showing how to expose a data lake as a GraphQL API using Amazon AppSync, Athena, Lake Formation, and AWS Glue.

The following steps describe the architecture:

  1. Set up a data lake using Lake Formation and Amazon Simple Storage Service.

  2. Crawl the data in the Amazon S3 buckets with AWS Glue to build a data catalog.

  3. Monitor any changes in the AWS Glue Data Catalog with Amazon CloudWatch, and trigger an AWS Lambda function on every catalog change to update the schema of the external API.

  4. In the Lambda function, query the latest data catalog and security settings, automatically generate source code to build and deploy a new version of the external API's schema and resolvers, and commit the code to AWS CodeCommit.

  5. Set up CodeCommit to automatically start a new CI/CD pipeline on AWS CodePipeline after any code commit.

  6. Use CodePipeline to build a new version of the API with AWS CodeBuild using the newly generated source code, and use AWS CodeDeploy to deploy both the new Amazon AppSync GraphQL API and the new Lambda query resolvers.

  7. Allow your GraphQL clients to securely invoke the external API over HTTPS with your chosen authentication mechanism. Activate the AppSync server-side caching.

  8. Use the AppSync Lambda query resolvers to translate the GraphQL query into SQL queries supported by Athena.

  9. Run Athena jobs synchronously with the athena-express library, appropriately timing out and limiting the result set.

Further reading

For additional information, refer to the following resources:

Diagram history

To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.

ChangeDescriptionDate

Initial publication

Reference architecture diagram first published.

April 25, 2022

Note

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.