Architecture overview

This chapter provides a high-level overview of the Automotive Data Platform v0.2 architecture — its foundation topology, data product catalog, governance layer, and two-stage rollout model. For deploy topology detail, see Platform foundation. For the full data product catalog, see Data products.

Foundation topology

The platform-foundation/ CDK app is the single entry point for all ADP infrastructure. It provisions five per-stage stacks plus one account-singular bootstrap stack. Stack names follow the adp-{stage}-foundation- prefix pattern; resources within each stack follow adp_{stage}_ naming for Glue databases and adp-{stage}-* for IAM Identity Center groups.

Per-stage stacks

The five per-stage stacks deploy in dependency order (1–5) and are gated behind a STAGE-required Makefile entry point. Both staging and prod stages are supported; staging and prod deploy independently into the same AWS account in us-east-1.

Stage names and stack-name prefixes are strictly controlled. The Makefile fails closed if STAGE is omitted. Resource naming is a deterministic function of {stage} — no timestamps, no random suffixes — to support idempotent re-deploys and cross-stage comparison.

Two-stage rollout

ADP supports two stages: staging and prod. Each stage is a fully independent deploy with its own S3 lake bucket, DataZone domain, Glue databases, and governance stack. The staging stack names follow adp-staging-foundation- ; production follows adp-prod-foundation-. Resources are isolated by name prefix and by Lake Formation permission scope — no cross-stage data access.

This prefix-based isolation means that make deploy STAGE=staging and make deploy STAGE=prod can safely co-exist in the same AWS account.

DataZone V2 domain and projects

The adp-{stage}-foundation-datazone-projects stack creates one Amazon DataZone V2 domain with 10 projects:

The DataZone domain name follows adp-{stage}-foundation-domain. All 9 data products are governed within this domain; consumers (CVX agents, SageMaker Studio notebooks, BI tools) subscribe via DataZone to access product assets under Lake Formation-enforced permissions.

For the complete data product catalog — technical names, domains, partitions, and per-product subscription patterns — see Data products.

Cross-cutting governance layer

ADP’s governance layer is deployed as part of the governance stack rather than as an independent guidance. It covers four dimensions:

Lake Formation tag-based access control

The governance stack registers the S3 lake bucket with AWS Lake Formation and applies tag-based access control (LF-TBAC) across all 10 Glue databases. Data owners are granted access via the adp-{stage}-data-owners IAM Identity Center group; consumers via adp-{stage}-data-consumers; platform administrators via adp-{stage}-platform-admins. Fine-grained column-level permissions are applied on PII-bearing tables.

Amazon Macie classification

The adp-shared-bootstrap stack enables an Amazon Macie session at account level (once per account, not per stage). Macie automated discovery jobs classify PII-bearing prefixes on the lake S3 bucket. Macie findings surface in Amazon Security Hub.

Note: Macie has a 30-day cool-down period after disabling. If you see "Macie is already enabled" during bootstrap, the session is already active — no action required. See the deployment runbook at docs/DEPLOYMENT.md for details.

CloudTrail data events

The governance stack creates an AWS CloudTrail trail configured for data-event logging on the lake S3 bucket. Every GetObject, PutObject, and DeleteObject API call is recorded, providing an immutable audit log of all data access within the lake.

IAM Identity Center groups

The governance stack provisions three IAM Identity Center (IDC) groups per stage:

IDC groups are regional and live in us-east-1. This is the primary reason ADP is single-region (us-east-1) in v0.2 — the IDC instance cannot be split across regions. EU Data Act / GDPR cross-region split is out of scope for foundation v1.

Platform layers

The platform-foundation model organizes the data platform into four integrated layers:

DataZone catalog layer

Amazon DataZone V2 provides the data catalog, lineage tracking, and self-service subscription surface. Producers publish data products to the domain; consumers discover, subscribe, and access products via DataZone’s UI or API. The subscription model is auto-granted within the domain — no manual approval step is required for intra-domain subscriptions during smoke-test validation.

See Data products for the full catalog of 9 published data products.

Lake layer

Amazon S3 provides the storage substrate for all data products. Eight of the 9 products are stored as Apache Iceberg tables (ACID transactions, schema evolution, time-travel queries). One product (vehicle_knowledge_base) uses direct S3 storage backed by a Bedrock Knowledge Base with an Amazon OpenSearch Serverless index.

AWS Glue provides the catalog metadata layer (10 databases, one per product plus one shared dimensions database) and the compute layer (Glue 5.1 Spark jobs for data generation and transformation). Amazon Athena Engine V3 provides the serverless SQL query layer for analyst and notebook consumers.

Governance layer

The cross-cutting governance layer (Lake Formation, Macie, CloudTrail, IDC groups) is described in the previous section. This layer ensures that every data access event is logged, PII is classified, and fine-grained permissions are enforced at Glue database / table / column level.

Consumers layer

Consumers access data products via Amazon DataZone subscriptions. The primary consumer patterns documented in this release are:

AWS services used

This section lists the primary AWS services deployed by the foundation. For the complete service footprint, see the stack definitions under platform-foundation/stacks/.

Data lake and storage

Catalog and governance

Knowledge and AI

Analytics and machine learning

Networking

Well-Architected Framework alignment

Operational excellence

Security

Reliability

Performance efficiency

Cost optimization

Sustainability

Next steps

For detailed deployment procedures — including make bootstrap, make deploy STAGE=, make seed STAGE=, and make smoke-test STAGE= — see Platform foundation.

For the full data product catalog including technical names, partition schemes, and DataZone subscription patterns, see Data products.

For cross-product Athena query patterns and CVX agent integration, see docs/cvx-integration-contract.md.