For similar capabilities to Amazon Timestream for LiveAnalytics, consider Amazon Timestream for InfluxDB. It offers simplified data ingestion and single-digit millisecond query response times for real-time analytics. Learn more [here](https://docs.aws.amazon.com//timestream/latest/developerguide/timestream-for-influxdb.html).
# Customer-defined partition keys
Amazon Timestream for LiveAnalytics customer-defined partition keys is a feature in Timestream for LiveAnalytics that enables customers to define their own partition keys for their tables. Partitioning is a technique used to distribute data across multiple physical storage units, allowing for faster and more efficient data retrieval. With customer-defined partition keys, customers can create a partitioning schema that better aligns with their query patterns and use cases.
With Timestream for LiveAnalytics customer-defined partition keys, customers can choose one dimension names as a partition key for their tables. This allows for more flexibility in defining the partitioning schema for their data. By selecting the right partition key, customers can optimize their data model, improving their query performance, and reduce query latency.
**Topics**
+ [
# Using customer-defined partition keys
](customer-defined-partition-keys-using.md)
+ [
# Getting started with customer-defined partition keys
](customer-defined-partition-keys-getting-started.md)
+ [
# Checking partitioning schema configuration
](customer-defined-partition-keys-checking-configuration.md)
+ [
# Updating partitioning schema configuration
](customer-defined-partition-keys-updating-configuration.md)
+ [
# Advantages of customer-defined partition keys
](customer-defined-partition-keys-advantages.md)
+ [
# Limitations of customer-defined partition keys
](customer-defined-partition-keys-limitations.md)
+ [
# Customer-defined partition keys and low cardinality dimensions
](customer-defined-partition-keys-low-cardinality-dimensions.md)
+ [
# Creating partition keys for existing tables
](customer-defined-partition-keys-creating.md)
+ [
# Timestream for LiveAnalytics schema validation with custom composite partition keys
](customer-defined-partition-keys-schema-validation.md)
# Using customer-defined partition keys
If you have a well-defined query pattern with high cardinality dimensions and require low query latency, a Timestream for LiveAnalytics customer-defined partition key can be a useful tool to enhance your data model. For instance, if you are a retail company tracking customer interactions on your website, the main access patterns would likely be by customer ID and timestamp. By defining customer ID as the partition key, your data can be distributed evenly, allowing for reduced latency, ultimately improving the user experience.
Another example is in the healthcare industry, where wearable devices collect sensor data to track patients' vital signs. The main access pattern would be by Device ID and timestamp, with high cardinality on both dimensions. By defining Device ID as the partition key, can optimize your query execution and ensure a sustained long term query performance.
In summary, Timestream for LiveAnalytics customer-defined partition keys are most useful when you have a clear query pattern, high cardinality dimensions, and need low latency for your queries. By defining a partition key that aligns with your query pattern, you can optimize your query execution and ensure a sustained long term performance query performance.
# Getting started with customer-defined partition keys
From the console, choose **Tables** and create a new table. You can also use an SDK to access the `CreateTable` action to create new tables that can include a customer-defined partition key.
## Create a table with a dimension type partition key
You can use the following code snippets to create a table with a dimension type partition key.
------
#### [ Java ]
```
public void createTableWithDimensionTypePartitionKeyExample() {
System.out.println("Creating table");
CreateTableRequest createTableRequest = new CreateTableRequest();
createTableRequest.setDatabaseName(DATABASE_NAME);
createTableRequest.setTableName(TABLE_NAME);
final RetentionProperties retentionProperties = new RetentionProperties()
.withMemoryStoreRetentionPeriodInHours(HT_TTL_HOURS)
.withMagneticStoreRetentionPeriodInDays(CT_TTL_DAYS);
createTableRequest.setRetentionProperties(retentionProperties);
// Can specify enforcement level with OPTIONAL or REQUIRED
final List partitionKeyWithDimensionAndOptionalEnforcement = Collections.singletonList(new PartitionKey()
.withName(COMPOSITE_PARTITION_KEY_DIM_NAME)
.withType(PartitionKeyType.DIMENSION)
.withEnforcementInRecord(PartitionKeyEnforcementLevel.OPTIONAL));
Schema schema = new Schema();
schema.setCompositePartitionKey(partitionKeyWithDimensionAndOptionalEnforcement);
createTableRequest.setSchema(schema);
try {
writeClient.createTable(createTableRequest);
System.out.println("Table [" + TABLE_NAME + "] successfully created.");
} catch (ConflictException e) {
System.out.println("Table [" + TABLE_NAME + "] exists on database [" + DATABASE_NAME + "] . Skipping database creation");
}
}
```
------
#### [ Java v2 ]
```
public void createTableWithDimensionTypePartitionKeyExample() {
System.out.println("Creating table");
final RetentionProperties retentionProperties = RetentionProperties.builder()
.memoryStoreRetentionPeriodInHours(HT_TTL_HOURS)
.magneticStoreRetentionPeriodInDays(CT_TTL_DAYS)
.build();
// Can specify enforcement level with OPTIONAL or REQUIRED
final List partitionKeyWithDimensionAndOptionalEnforcement = Collections.singletonList(PartitionKey
.builder()
.name(COMPOSITE_PARTITION_KEY_DIM_NAME)
.type(PartitionKeyType.DIMENSION)
.enforcementInRecord(PartitionKeyEnforcementLevel.OPTIONAL)
.build());
final Schema schema = Schema.builder()
.compositePartitionKey(partitionKeyWithDimensionAndOptionalEnforcement).build();
final CreateTableRequest createTableRequest = CreateTableRequest.builder()
.databaseName(DATABASE_NAME)
.tableName(TABLE_NAME)
.retentionProperties(retentionProperties)
.schema(schema)
.build();
try {
writeClient.createTable(createTableRequest);
System.out.println("Table [" + TABLE_NAME + "] successfully created.");
} catch (ConflictException e) {
System.out.println("Table [" + TABLE_NAME + "] exists on database [" + DATABASE_NAME + "] . Skipping database creation");
}
}
```
------
#### [ Go v1 ]
```
func createTableWithDimensionTypePartitionKeyExample(){
// Can specify enforcement level with OPTIONAL or REQUIRED
partitionKeyWithDimensionAndOptionalEnforcement := []*timestreamwrite.PartitionKey{
{
Name: aws.String(CompositePartitionKeyDimName),
EnforcementInRecord: aws.String("OPTIONAL"),
Type: aws.String("DIMENSION"),
},
}
createTableInput := ×treamwrite.CreateTableInput{
DatabaseName: aws.String(*databaseName),
TableName: aws.String(*tableName),
// Enable MagneticStoreWrite for Table
MagneticStoreWriteProperties: ×treamwrite.MagneticStoreWriteProperties{
EnableMagneticStoreWrites: aws.Bool(true),
// Persist MagneticStoreWrite rejected records in S3
MagneticStoreRejectedDataLocation: ×treamwrite.MagneticStoreRejectedDataLocation{
S3Configuration: ×treamwrite.S3Configuration{
BucketName: aws.String("timestream-sample-bucket"),
ObjectKeyPrefix: aws.String("TimeStreamCustomerSampleGo"),
EncryptionOption: aws.String("SSE_S3"),
},
},
},
Schema: ×treamwrite.Schema{
CompositePartitionKey: partitionKeyWithDimensionAndOptionalEnforcement,
}
}
_, err := writeSvc.CreateTable(createTableInput)
}
```
------
#### [ Go v2 ]
```
func (timestreamBuilder TimestreamBuilder) CreateTableWithDimensionTypePartitionKeyExample() error {
partitionKeyWithDimensionAndOptionalEnforcement := []types.PartitionKey{
{
Name: aws.String(CompositePartitionKeyDimName),
EnforcementInRecord: types.PartitionKeyEnforcementLevelOptional,
Type: types.PartitionKeyTypeDimension,
},
}
_, err := timestreamBuilder.WriteSvc.CreateTable(context.TODO(), ×treamwrite.CreateTableInput{
DatabaseName: aws.String(databaseName),
TableName: aws.String(tableName),
MagneticStoreWriteProperties: &types.MagneticStoreWriteProperties{
EnableMagneticStoreWrites: aws.Bool(true),
// Persist MagneticStoreWrite rejected records in S3
MagneticStoreRejectedDataLocation: &types.MagneticStoreRejectedDataLocation{
S3Configuration: &types.S3Configuration{
BucketName: aws.String(s3BucketName),
EncryptionOption: "SSE_S3",
},
},
},
Schema: &types.Schema{
CompositePartitionKey: partitionKeyWithDimensionAndOptionalEnforcement,
},
})
if err != nil {
fmt.Println("Error:")
fmt.Println(err)
} else {
fmt.Println("Create table is successful")
}
return err
}
```
------
#### [ Python ]
```
def create_table_with_measure_name_type_partition_key(self):
print("Creating table")
retention_properties = {
'MemoryStoreRetentionPeriodInHours': HT_TTL_HOURS,
'MagneticStoreRetentionPeriodInDays': CT_TTL_DAYS
}
partitionKey_with_measure_name = [
{'Type': 'MEASURE'}
]
schema = {
'CompositePartitionKey': partitionKey_with_measure_name
}
try:
self.client.create_table(DatabaseName=DATABASE_NAME, TableName=TABLE_NAME,
RetentionProperties=retention_properties, Schema=schema)
print("Table [%s] successfully created." % TABLE_NAME)
except self.client.exceptions.ConflictException:
print("Table [%s] exists on database [%s]. Skipping table creation" % (
TABLE_NAME, DATABASE_NAME))
except Exception as err:
print("Create table failed:", err)
```
------
# Checking partitioning schema configuration
You can check how a table configuration for partitioning schema in a couple of ways. From the console, choose **Databases** and choose the table to check. You can also use an SDK to access the `DescribeTable` action.
## Describe a table with a partition key
You can use the following code snippets to describe a table with a partition key.
------
#### [ Java ]
```
public void describeTable() {
System.out.println("Describing table");
final DescribeTableRequest describeTableRequest = new DescribeTableRequest();
describeTableRequest.setDatabaseName(DATABASE_NAME);
describeTableRequest.setTableName(TABLE_NAME);
try {
DescribeTableResult result = amazonTimestreamWrite.describeTable(describeTableRequest);
String tableId = result.getTable().getArn();
System.out.println("Table " + TABLE_NAME + " has id " + tableId);
// If table is created with composite partition key, it can be described with
// System.out.println(result.getTable().getSchema().getCompositePartitionKey());
} catch (final Exception e) {
System.out.println("Table " + TABLE_NAME + " doesn't exist = " + e);
throw e;
}
}
```
The following is an example output.
1. Table has dimension type partition key
```
[{Type: DIMENSION,Name: hostId,EnforcementInRecord: OPTIONAL}]
```
1. Table has measure name type partition key
```
[{Type: MEASURE,}]
```
1. Getting composite partition key from a table created without specifying composite partition key
```
[{Type: MEASURE,}]
```
------
#### [ Java v2 ]
```
public void describeTable() {
System.out.println("Describing table");
final DescribeTableRequest describeTableRequest = DescribeTableRequest.builder()
.databaseName(DATABASE_NAME).tableName(TABLE_NAME).build();
try {
DescribeTableResponse response = writeClient.describeTable(describeTableRequest);
String tableId = response.table().arn();
System.out.println("Table " + TABLE_NAME + " has id " + tableId);
// If table is created with composite partition key, it can be described with
// System.out.println(response.table().schema().compositePartitionKey());
} catch (final Exception e) {
System.out.println("Table " + TABLE_NAME + " doesn't exist = " + e);
throw e;
}
}
```
The following is an example output.
1. Table has dimension type partition key
```
[PartitionKey(Type=DIMENSION, Name=hostId, EnforcementInRecord=OPTIONAL)]
```
1. Table has measure name type partition key
```
[PartitionKey(Type=MEASURE)]
```
1. Getting composite partition key from a table created without specifying composite partition key will return
```
[PartitionKey(Type=MEASURE)]
```
------
#### [ Go v1 ]
```
Go
```
The following is an example output.
```
{
Table: {
Arn: "arn:aws:timestream:us-west-2:533139590831:database/devops/table/host_metrics_dim_pk_1",
CreationTime: 2023-05-31 01:52:00.511 +0000 UTC,
DatabaseName: "devops",
LastUpdatedTime: 2023-05-31 01:52:00.511 +0000 UTC,
MagneticStoreWriteProperties: {
EnableMagneticStoreWrites: true,
MagneticStoreRejectedDataLocation: {
S3Configuration: {
BucketName: "timestream-sample-bucket-west",
EncryptionOption: "SSE_S3",
ObjectKeyPrefix: "TimeStreamCustomerSampleGo"
}
}
},
RetentionProperties: {
MagneticStoreRetentionPeriodInDays: 73000,
MemoryStoreRetentionPeriodInHours: 6
},
Schema: {
CompositePartitionKey: [{
EnforcementInRecord: "OPTIONAL",
Name: "hostId",
Type: "DIMENSION"
}]
},
TableName: "host_metrics_dim_pk_1",
TableStatus: "ACTIVE"
}
}
```
------
#### [ Go v2 ]
```
func (timestreamBuilder TimestreamBuilder) DescribeTable() (*timestreamwrite.DescribeTableOutput, error) {
describeTableInput := ×treamwrite.DescribeTableInput{
DatabaseName: aws.String(databaseName),
TableName: aws.String(tableName),
}
describeTableOutput, err := timestreamBuilder.WriteSvc.DescribeTable(context.TODO(), describeTableInput)
if err != nil {
fmt.Printf("Failed to describe table with Error: %s", err.Error())
} else {
fmt.Printf("Describe table is successful : %s\n", JsonMarshalIgnoreError(*describeTableOutput))
// If table is created with composite partition key, it will be included in the output
}
return describeTableOutput, err
}
```
The following is an example output.
```
{
"Table": {
"Arn":"arn:aws:timestream:us-east-1:351861611069:database/cdpk-wr-db/table/host_metrics_dim_pk",
"CreationTime":"2023-05-31T22:36:10.66Z",
"DatabaseName":"cdpk-wr-db",
"LastUpdatedTime":"2023-05-31T22:36:10.66Z",
"MagneticStoreWriteProperties":{
"EnableMagneticStoreWrites":true,
"MagneticStoreRejectedDataLocation":{
"S3Configuration":{
"BucketName":"error-configuration-sample-s3-bucket-cq8my",
"EncryptionOption":"SSE_S3",
"KmsKeyId":null,"ObjectKeyPrefix":null
}
}
},
"RetentionProperties":{
"MagneticStoreRetentionPeriodInDays":73000,
"MemoryStoreRetentionPeriodInHours":6
},
"Schema":{
"CompositePartitionKey":[{
"Type":"DIMENSION",
"EnforcementInRecord":"OPTIONAL",
"Name":"hostId"
}]
},
"TableName":"host_metrics_dim_pk",
"TableStatus":"ACTIVE"
},
"ResultMetadata":{}
}
```
------
#### [ Python ]
```
def describe_table(self):
print('Describing table')
try:
result = self.client.describe_table(DatabaseName=DATABASE_NAME, TableName=TABLE_NAME)
print("Table [%s] has id [%s]" % (TABLE_NAME, result['Table']['Arn']))
# If table is created with composite partition key, it can be described with
# print(result['Table']['Schema'])
except self.client.exceptions.ResourceNotFoundException:
print("Table doesn't exist")
except Exception as err:
print("Describe table failed:", err)
```
The following is an example output.
1. Table has dimension type partition key
```
[{'CompositePartitionKey': [{'Type': 'DIMENSION', 'Name': 'hostId', 'EnforcementInRecord': 'OPTIONAL'}]}]
```
1. Table has measure name type partition key
```
[{'CompositePartitionKey': [{'Type': 'MEASURE'}]}]
```
1. Getting composite partition key from a table created without specifying composite partition key
```
[{'CompositePartitionKey': [{'Type': 'MEASURE'}]}]
```
------
# Updating partitioning schema configuration
You can update table configuration for partitioning schema with an SDK with access the `UpdateTable` action.
## Update a table with a partition key
You can use the following code snippets to update a table with a partition key.
------
#### [ Java ]
```
public void updateTableCompositePartitionKeyEnforcement() {
System.out.println("Updating table");
UpdateTableRequest updateTableRequest = new UpdateTableRequest();
updateTableRequest.setDatabaseName(DATABASE_NAME);
updateTableRequest.setTableName(TABLE_NAME);
// Can update enforcement level for dimension type partition key with OPTIONAL or REQUIRED enforcement
final List partitionKeyWithDimensionAndRequiredEnforcement = Collections.singletonList(new PartitionKey()
.withName(COMPOSITE_PARTITION_KEY_DIM_NAME)
.withType(PartitionKeyType.DIMENSION)
.withEnforcementInRecord(PartitionKeyEnforcementLevel.REQUIRED));
Schema schema = new Schema();
schema.setCompositePartitionKey(partitionKeyWithDimensionAndRequiredEnforcement);
updateTableRequest.withSchema(schema);
writeClient.updateTable(updateTableRequest);
System.out.println("Table updated");
```
------
#### [ Java v2 ]
```
public void updateTableCompositePartitionKeyEnforcement() {
System.out.println("Updating table");
// Can update enforcement level for dimension type partition key with OPTIONAL or REQUIRED enforcement
final List partitionKeyWithDimensionAndRequiredEnforcement = Collections.singletonList(PartitionKey
.builder()
.name(COMPOSITE_PARTITION_KEY_DIM_NAME)
.type(PartitionKeyType.DIMENSION)
.enforcementInRecord(PartitionKeyEnforcementLevel.REQUIRED)
.build());
final Schema schema = Schema.builder()
.compositePartitionKey(partitionKeyWithDimensionAndRequiredEnforcement).build();
final UpdateTableRequest updateTableRequest = UpdateTableRequest.builder()
.databaseName(DATABASE_NAME).tableName(TABLE_NAME).schema(schema).build();
writeClient.updateTable(updateTableRequest);
System.out.println("Table updated");
```
------
#### [ Go v1 ]
```
// Update table partition key enforcement attribute
updateTableInput := ×treamwrite.UpdateTableInput{
DatabaseName: aws.String(*databaseName),
TableName: aws.String(*tableName),
// Can update enforcement level for dimension type partition key with OPTIONAL or REQUIRED enforcement
Schema: ×treamwrite.Schema{
CompositePartitionKey: []*timestreamwrite.PartitionKey{
{
Name: aws.String(CompositePartitionKeyDimName),
EnforcementInRecord: aws.String("REQUIRED"),
Type: aws.String("DIMENSION"),
},
}},
}
updateTableOutput, err := writeSvc.UpdateTable(updateTableInput)
if err != nil {
fmt.Println("Error:")
fmt.Println(err)
} else {
fmt.Println("Update table is successful, below is the output:")
fmt.Println(updateTableOutput)
}
```
------
#### [ Go v2 ]
```
// Update table partition key enforcement attribute
updateTableInput := ×treamwrite.UpdateTableInput{
DatabaseName: aws.String(*databaseName),
TableName: aws.String(*tableName),
// Can update enforcement level for dimension type partition key with OPTIONAL or REQUIRED enforcement
Schema: &types.Schema{
CompositePartitionKey: []types.PartitionKey{
{
Name: aws.String(CompositePartitionKeyDimName),
EnforcementInRecord: types.PartitionKeyEnforcementLevelRequired,
Type: types.PartitionKeyTypeDimension,
},
}},
}
updateTableOutput, err := timestreamBuilder.WriteSvc.UpdateTable(context.TODO(), updateTableInput)
if err != nil {
fmt.Println("Error:")
fmt.Println(err)
} else {
fmt.Println("Update table is successful, below is the output:")
fmt.Println(updateTableOutput)
}
```
------
#### [ Python ]
```
def update_table(self):
print('Updating table')
try:
# Can update enforcement level for dimension type partition key with OPTIONAL or REQUIRED enforcement
partition_key_with_dimension_and_required_enforcement = [
{
'Type': 'DIMENSION',
'Name': COMPOSITE_PARTITION_KEY_DIM_NAME,
'EnforcementInRecord': 'REQUIRED'
}
]
schema = {
'CompositePartitionKey': partition_key_with_dimension_and_required_enforcement
}
self.client.update_table(DatabaseName=DATABASE_NAME, TableName=TABLE_NAME,
Schema=schema)
print('Table updated.')
except Exception as err:
print('Update table failed:', err)
```
------
# Advantages of customer-defined partition keys
**Enhanced query performance: **Customer-defined partition keys enable you to optimize your query execution and improve overall query performance. By defining partition keys that align with your query patterns, you can minimize data scanning and optimize data pruning, resulting in lower query latency.
**Better long term performance predictability: **Customer-defined partition keys allow customers to distribute data evenly across partitions, improving the efficiency of data management. This will ensure that your query performance remains stable as your data stored scales over time.
# Limitations of customer-defined partition keys
As a Timestream for LiveAnalytics user, it's important to keep in mind the limitations around a customer partition key. Firstly, it requires a good understanding of your workload and query patterns. This means that you should have a clear idea of which dimensions are most frequently use as main filtering conditions in queries and have high cardinality to make the most effective use of partition keys.
Secondly, partition keys need to be defined at the time of table creation and cannot be added to existing tables. This means that you should carefully consider your partitioning strategy before creating a table to ensure that it aligns with your business needs.
Lastly, it's important to note that once the table has been created, you cannot change the partition key afterwards. This means that you should thoroughly test and evaluate your partitioning strategy before committing to it. With these limitations in mind, Timestream's customer-defined partition key can greatly improve query performance and long term satisfaction.
# Customer-defined partition keys and low cardinality dimensions
If you decide to use a partition key with very low cardinality, such as a specific region or state, it is important to note that the data for for other entities such as `customerID`, `ProductCategory`, and others, could end up spread across too many partitions sometimes with little or no data present. This can lead to inefficient query execution and decreased performance.
To avoid this, we recommend you choose dimensions that are not only part of your key filtering condition but have higher cardinality. This will help ensure that the data is evenly distributed across the partitions and improve query performance.
# Creating partition keys for existing tables
If you already have tables in Timestream for LiveAnalytics and want to use customer-defined partition keys, you will need to migrate your data into a new table with the desired partitioning schema definition. This can be done using export to S3 and batch load together, which involves exporting the data from the existing table to S3, modifying the data to include the partition key (if necessary) and adding headers to your CSV files, and then importing the data into a new table with the desired partitioning schema defined. Keep in mind that this method can be time consuming and costly, especially for large tables.
Alternatively, you can use scheduled queries to migrate your data to a new table with the desired partitioning schema. This method involves creating a scheduled query that reads from the existing table and writes to the new table. The scheduled query can be set up to run on a regular basis until all the data has been migrated. Keep in mind that you will be charged for reading and writing the data during the migration process.
# Timestream for LiveAnalytics schema validation with custom composite partition keys
Schema validation in Timestream for LiveAnalytics helps ensure that data ingested into the database complies with the specified schema, minimizing ingestion errors and improving data quality. In particular, schema validation is especially useful when adopting customer-defined partition key with the goal of optimizing your query performance.
## What is Timestream for LiveAnalytics schema validation with customer-defined partition keys?
Timestream for LiveAnalytics schema validation is a feature that validates data being ingested into a Timestream for LiveAnalytics table based on a predefined schema. This schema defines the data model, including partition key, data types, and constraints for the records being inserted.
When using a customer-defined partition key, schema validation becomes even more crucial. Partition keys allow you to specify a partition key, which determines how your data is stored in Timestream for LiveAnalytics. By validating the incoming data against the schema with a custom partition key, you can enforce data consistency, detect errors early, and improve the overall quality of the data stored in Timestream for LiveAnalytics.
## How to Use Timestream for LiveAnalytics schema validation with custom composite partition keys
To use Timestream for LiveAnalytics schema validation with custom composite partition keys, follow these steps:
**Think about what your query patterns will look like: **To properly choose and define the schema for your Timestream for LiveAnalytics table you should start with your query requirements.
**Specify custom composite partition keys: **When creating the table, specify a custom partition key. This key determines the attribute that will be used to partition the table data. You can choose between dimension keys and measure keys for partitioning. A dimension key partitions data based on a dimension name, while a measure key partitions data based on the measure name.
**Set enforcement levels: **To ensure proper data partitioning and the benefits that come with it, Amazon Timestream for LiveAnalytics allows you to set enforcement levels for each partition key in your schema. The enforcement level determines whether the partition key dimension is required or optional when ingesting records. You can choose between two options: `REQUIRED`, which means the partition key must be present in the ingested record, and `OPTIONAL`, which means the partition key doesn't have to be present. It is recommended that you use the `REQUIRED` enforcement level when using a customer-defined partition to ensure that your data is properly partitioned and you get the full benefits of this feature. Additionally, you can change the enforcement level configuration at any time after the schema creation to adjust to your data ingestion requirements.
**Ingest data: **When ingesting data into the Timestream for LiveAnalytics table, the schema validation process will check the records against the defined schema with custom composite partition keys. If the records do not adhere to the schema, Timestream for LiveAnalytics will return a validation error.
**Handle validation errors:** In case of validation errors, Timestream for LiveAnalytics will return a `ValidationException` or a `RejectedRecordsException`, depending on the type of error. Make sure to handle these exceptions in your application and take appropriate action, such as fixing the incorrect records and retrying the ingestion.
**Update enforcement levels: **If necessary, you can update the enforcement level of partition keys after table creation using the `UpdateTable` action. However, it's important to note that some aspects of the partition key configuration, such as the name, and type, cannot be changed after table creation. If you change the enforcement level from `REQUIRED` to `OPTIONAL`, all records will be accepted regardless of the presence of the attribute selected as the customer-defined partition key. Conversely, if you change the enforcement level from `OPTIONAL` to `REQUIRED`, you may start seeing 4xx write errors for records that don't meet this condition. Therefore, it's essential to choose the appropriate enforcement level for your use case when creating your table, based on your data's partitioning requirements.
## When to use Timestream for LiveAnalytics schema validation with custom composite partition keys
Timestream for LiveAnalytics schema validation with custom composite partition keys should be used in scenarios where data consistency, quality, and optimized partitioning are crucial. By enforcing a schema during data ingestion, you can prevent errors and inconsistencies that might lead to incorrect analysis or loss of valuable insights.
## Interaction with batch load jobs
When setting up a batch load job to import data into a table with a customer-defined partition key, there are a few scenarios that could affect the process:
1. If the enforcement level is set to `OPTIONAL`, an alert will be displayed on the console during the creation flow if the partition key is not mapped during job configuration. This alert will not appear when using the API or CLI.
1. If the enforcement level is set to `REQUIRED`, the job creation will be rejected unless the partition key is mapped to a source data column.
1. If the enforcement level is changed to `REQUIRED` after the job is created, the job will continue to execute, but any records that do not have the proper mapping for the partition key will be rejected with a 4xx error.
## Interaction with scheduled query
When setting up a scheduled query job for calculating and storing aggregates, rollups, and other forms of preprocessed data into a table with a customer-defined partition key, there are a few scenarios that could affect the process:
1. If the enforcement level is set to `OPTIONAL`, an alert will be displayed if the partition key is not mapped during job configuration. This alert will not appear when using the API or CLI.
1. If the enforcement level is set to `REQUIRED`, the job creation will be rejected unless the partition key is mapped to a source data column.
1. If the enforcement level is changed to `REQUIRED` after the job is created and the scheduled query results does not contain the partition key dimension, all the next iterations of the job will fail.