Understanding CDC records

Aurora DSQL CDC delivers each change as a JSON record. The record uses an envelope structure with operation type, before and after row images, and source metadata.

How records map to Amazon Kinesis

Aurora DSQL writes each CDC record as a single Kinesis record. The Kinesis record's Data field contains the JSON payload. Aurora DSQL uses a randomized Kinesis partition key to distribute CDC records evenly across shards. To read all changes, consume all shards on the Kinesis data stream. If a record exceeds the Kinesis record size limit, Aurora DSQL splits it across multiple Kinesis records. For details, see Handling oversized records.

Primary key in the payload

For tables with a primary key, the primary key column values appear in the payload:

For example, consider a table with a composite primary key:

CREATE TABLE order_items (
    order_id INT,
    item_id INT,
    quantity INT,
    price NUMERIC,
    PRIMARY KEY (order_id, item_id)
);

A delete on this table produces a payload where "before": {"order_id": 1001, "item_id": 42}.

Record payload

The payload uses the following JSON envelope format.

INSERT example

The following example shows a CDC record for an INSERT operation:

{
    "type": "full",
    "op": "c",
    "before": null,
    "after": {"order_id": 1001, "item_id": 42, "quantity": 5, "price": "29.99"},
    "source": {
        "version": "1.0",
        "ts_ms": 1705318200000,
        "ts_ns": 1705318200000000000,
        "txId": "ffthunp5stx6ffs2vyfqoatmfu",
        "schema": "public",
        "table": "order_items",
        "db": "postgres",
        "cluster": "kmabugltfmjdaj2siqr2qbxgju"
    },
    "ts_ms": 1705318200125,
    "ts_ns": 1705318200125483291
}

UPDATE example

The following example shows a CDC record for an UPDATE operation:

{
    "type": "full",
    "op": "u",
    "before": null,
    "after": {"order_id": 1001, "item_id": 42, "quantity": 10, "price": "29.99"},
    "source": {
        "version": "1.0",
        "ts_ms": 1705318300000,
        "ts_ns": 1705318300000000000,
        "txId": "qvtiesgmd55cvlfukm3dfuotji",
        "schema": "public",
        "table": "order_items",
        "db": "postgres",
        "cluster": "kmabugltfmjdaj2siqr2qbxgju"
    },
    "ts_ms": 1705318300125,
    "ts_ns": 1705318300125483291
}

DELETE example

For a DELETE on a table with a primary key, the before field contains the primary key values:

{
    "type": "full",
    "op": "d",
    "before": {"order_id": 1001, "item_id": 42},
    "after": null,
    "source": {
        "version": "1.0",
        "ts_ms": 1705318400000,
        "ts_ns": 1705318400000000000,
        "txId": "xyzabc123def456ghi789jklmno",
        "schema": "public",
        "table": "order_items",
        "db": "postgres",
        "cluster": "kmabugltfmjdaj2siqr2qbxgju"
    },
    "ts_ms": 1705318400125,
    "ts_ns": 1705318400125483291
}

Payload fields

Format details

The following details describe how Aurora DSQL CDC formats records. Design your app to handle these behaviors.

Write-set compaction

Aurora DSQL compacts the write set of each committed transaction before publishing CDC records. Aurora DSQL compares each row's state before the transaction to its state after, and emits at most one record per row reflecting the net effect.

Compaction rules

Aurora DSQL determines the op code based on the row's net state change:

Handling oversized records

When a CDC record's serialized JSON exceeds 9 MiB, Aurora DSQL splits the before and/or after images, delivering multiple Kinesis records. Each record contains a top-level type field that indicates its structure: full for a complete record, chunked for a main record that references fragments, and fragment for an individual piece of a chunked image. The op, source, ts_ms, and ts_ns fields on a chunked main record behave the same as on a full record. Records that fit in a single Kinesis record have type set to full and don't require any extra handling.

A chunk_id is stable across retries. If Aurora DSQL redelivers a fragment, it carries the same chunk_id as the original delivery, so your app can continue buffering under the same identifier without handling partial sets from previous attempts.

Main record

A chunked main record replaces the top-level before or after field for the split image with a chunked object that describes how to reassemble it. Each entry under chunked has a chunk_id (the identifier that links fragments to this record), total_fragments (the number of fragments that make up that image), and crc32c (a CRC32C checksum, as a decimal string, over the reassembled image text). If one image is inline and the other is chunked, the inline image still appears at the top level as either a value or null.

{
    "type": "chunked",
    "op": "c",
    "before": null,
    "after": null,
    "source": {
        "version": "1.0",
        "ts_ms": 1705318200000,
        "ts_ns": 1705318200000000000,
        "txId": "ffthunp5stx6ffs2vyfqoatmfu",
        "schema": "public",
        "table": "order_items",
        "db": "postgres",
        "cluster": "cluster-id"
    },
    "chunked": {
        "after": {
            "chunk_id": "chunk-id",
            "total_fragments": 3,
            "crc32c": "2073618257"
        }
    },
    "ts_ms": 1705318200125,
    "ts_ns": 1705318200125483291
}

Fragment record

Each fragment is its own Kinesis record with type set to fragment and three fields: chunk_id matches the value in the corresponding chunked.before.chunk_id or chunked.after.chunk_id on the main record, index is the zero-based position of the fragment within the image, and data is a segment of the image's JSON text split on UTF-8 character boundaries (each fragment's data value is a valid UTF-8 string on its own). Because Aurora DSQL CDC uses UNORDERED mode and randomized partition keys, fragments and the main record can arrive on different shards and in any order. To read all fragments, consume all shards on the Kinesis data stream. For more information about delivery ordering, see Ordering.

{
    "type": "fragment",
    "chunk_id": "chunk-id",
    "index": 0,
    "data": "partial-JSON-text"
}

To reassemble an oversized image, buffer each record with type fragment by its chunk_id. When you receive a main record with type chunked, wait until you have total_fragments fragments for each chunk_id referenced under chunked.before or chunked.after, sort the fragments by index ascending, and concatenate the data strings. The concatenated result is the original before or after object as JSON text—parse it to access the column values. To verify delivery integrity, compute CRC32C over the concatenated string and compare the result to chunked.before.crc32c or chunked.after.crc32c.

Data type serialization

The following tables describe how Aurora DSQL serializes each PostgreSQL data type in CDC records.

Integer types

Values of bigint beyond ±2^53 might lose precision in JavaScript environments. Use BigInt or arbitrary-precision libraries in those cases.

Floating-point types

Boolean

Character types

Binary

Date and time types

Other types

NULL values

For any data type, NULL column values are represented as JSON null.

Schema evolution in CDC records

When you modify a table's schema—for example, by adding, dropping, or renaming a column—CDC records reflect the change starting from the transaction that committed the DDL change. Records from transactions committed before the DDL change use the previous schema. For example:

Track schema changes in your downstream consumer by inspecting the column names present in each record's after and before fields. The source.version field in each record identifies the CDC envelope format.