After careful consideration, we decided to end support for Amazon FinSpace, effective October 7, 2026. Amazon FinSpace will no longer accept new customers beginning October 7, 2025. As an existing customer with an Amazon FinSpace environment created before October 7, 2025, you can continue to use the service as normal. After October 7, 2026, you will no longer be able to use Amazon FinSpace. For more information, see [Amazon FinSpace end of support](https://docs.aws.amazon.com/finspace/latest/userguide/amazon-finspace-end-of-support.html). # Managed kdb Insights databases Managed kdb databases A Managed kdb Insights database acts as a highly available and scalable repository to store your kdb data files so that they can be used with one or more historical database (HDB) clusters in FinSpace kdb. Data in a database may consist of either kdb objects, kdb splayed tables, or kdb partitioned tables. These represent different types of kdb table structures and each must follow a prescribed file and path layout. You can learn more about each of these structures [here](https://code.kx.com/q/database/). Data is loaded into a database by defining a changeset, which lets you import a file or set of files into a database. The files in the kdb database are placed into logical paths called the *Database paths*. Creating a database does not automatically load any data. You must add data to the kdb database through changesets. # Managing kdb databases The following sections provide a detailed overview of the operations that you can perform by using a Managed kdb database. ## Creating a kdb database **To create a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. Choose **Create database**. 1. On the **Create database** page, enter a unique name for the database. 1. (Optional) Enter a description for your database. 1. (Optional) Add a new tag to assign it to your kdb database. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your database. 1. Choose **Create database**. The environment details page opens and the table under **Databases** lists the newly created database. You can choose the database name from the list to view its details in database details page. ## Managing data in a kdb database The Managed kdb Insights database allows you to add, update, or delete a set of files. When you create a database, there is no data loaded in it. You must add data to the database through changesets. A changeset represents a versioned set of changes that are applied to a database. ### Creating an Amazon S3 bucket policy Before you can ingest data into your database, you must have a valid Amazon S3 bucket IAM policy in place to allow FinSpace to access the data you will ingest into it. The following is an example of such a policy. In the following example, replace each **user input placeholder** with your own values. Replace **555555555555** with the AWS account where you created your Managed kdb Insights environment. **Example — Sample Amazon S3 bucket policy** **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": [ "s3:GetObject", "s3:GetObjectTagging" ], "Resource": "arn:aws:s3:::managed-kdb-data/*", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" } } }, { "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-data", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" } } } ] } ``` ### Creating a new changeset You can add, update, and delete data in a database by creating a new changeset. You can either use the console or the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset) API to create a changeset. To add a data to your database, create a changeset by providing the changeset type as `PUT`, database path, and S3 URI path. To update data in a database, you need to create another changeset with the same database path you chose while adding the data. To delete data in a database, create a new changeset with changeset type as `DELETE`. **Note** You should only add data in the correct kdb file format that follows a valid kdb path structure. Other file formats and structures are not supported when accessed from a FinSpace Managed kdb cluster. You can learn more about valid kdb path structures [here](https://code.kx.com/q/database/). **To create a changeset from the console** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. The table under this tab displays a list of databases. 1. Choose a database name to view its details. 1. On the database details page, choose the **Changesets** tab. 1. Choose **Create changeset**. 1. On the **Create changeset** page, select one of the following types of changeset. + *PUT* – Adds or updates files in a database. + *DELETE* – Deletes files in a database. This option is not available when creating the changeset for the first time. 1. For **Database path**, specify a path within the database directory where you want to add data. If the data already exists at this path, it will be updated. 1. For **S3 URI ** provide the source path of the file to add data. 1. Choose **Create changeset**. The database details page opens where you can see the status of the changeset in the changeset table. You can choose the changeset ID to view details of a changeset. ## Updating a kdb database **To update the metadata of a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose the one that you want to update. The database details page opens. 1. On the database details page, choose **Edit**. 1. Edit the database description. 1. Choose **Update database**. ## Viewing kdb database details **To view and get details of a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. The table under this tab displays a list of databases. 1. Choose a database name to view its details. The database details page opens where you can view details about the database. You can also add and view changesets and tags associated with this database. ## Deleting a kdb database **Note** This action is irreversible. Deleting a kdb database will delete all of its contents. **To delete a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose the one that you want to delete. The database details page opens. 1. On the database details page, choose **Delete**. 1. On the confirmation dialog box, enter *confirm*. 1. Choose **Delete**. # Dataviews for querying data Dataviews allow you to place portions of your Managed kdb Insights object store database onto disk for faster read-only access from your kdb clusters. To your kdb process, the dataview looks like a kdb segmented database, with data placed across one or more disk mounts (volumes) and the object store. This lets you place frequently-queried data on a fast-access disk for more performant access while keeping the rest of the data in the object store layer for less frequent access. With dataviews, the golden copy of your database’s data still remains in the object store format. The data stored on disk for faster access is a copy. ![\[A diagram that shows how dataviews work.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-dataviews-intro-diagram.png) Dataviews can be accessed from HDB and General purpose (GP) type clusters for read only access. The data within a dataview is accessible from the cluster as a kdb [segmented database](https://code.kx.com/q/database/segment/) that is automatically configured when you associate the dataview with the cluster. A segment is a mount point that can contain a portion of a database. Different segments could contain different data partitions, tables, or even columns. A kdb *par.txt * file that FinSpace automatically creates when you mount a database defines the segments. The segments of this segmented database can reside on different kdb Insights disk volumes. A segment of your database can be any portion of it. For example, consider a database with contents as the following date-partitioned layout. ``` /sym /2023.10.01/trades/price /2023.10.01/trades/time /2023.10.01/trades/quality /2023.10.01/trades/price /2023.10.02/trades/time /2023.10.02/trades/quality /2023.10.02/trades/price /2023.10.03/trades/time /2023.10.03/trades/quality /2023.10.03/trades/price /2023.10.04/trades/time /2023.10.04/trades/quality /2023.10.05/trades/price /2023.10.05/trades/time /2023.10.05/trades/quality /2023.10.05/trades/price /2023.10.01/trades/.d /2023.10.02/trades/.d /2023.10.03/trades/.d /2023.10.04/trades/.d /2023.10.05/trades/.d ``` In this example, `trades` is a table and `time`, `quantity`, and `price` are columns. You can store the most recent day of data on a high throughput volume, two days prior to that on 250 MB/s/TiB volume, with the rest accessible as a segment from the object store layer. The following table shows the data and segments. | Database contents | Segments | | --- | --- | | /2023.10.05/trades/time /2023.10.05/trades/quality /2023.10.05/trades/price | **Segment**: **Dataview Segment 1** **Stored On**: Managed kdb Insights Volume 1 [High throughput – 1000 MB/s/TiB] | | /2023.10.04/trades/time /2023.10.04/trades/quality /2023.10.04/trades/price /2023.10.03/trades/time /2023.10.03/trades/quality /2023.10.03/trades/price | **Segment**: **Dataview Segment 2** **Stored On**: Managed kdb Insights Volume 2 [Medium Throughput – 250 MB/s/TiB] | | /2023.10.02/trades/time /2023.10.02/trades/quality /2023.10.02/trades/price /2023.10.01/trades/time /2023.10.01/trades/quality /2023.10.01/trades/price | **Segment**: **Dataview Default Segment** **Stored On**: Object store | This gives you control to place copies of portions of your database on the appropriate type of disk for access, if you require higher performance access than what is available with the default object store storage. In addition, having the ability to explicitly place data on different volumes when creating a dataview, the contents directly under the root (/) path of the database, such as `/sym` in this example, are always copied to the cluster’s local storage for fast access. ## Auto-updating vs static dataviews When you create a dataview, you can specify from one of the following types of dataview. + **Auto-updating** –An auto-update dataview contains the most recent version of the data in the database. Its contents are automatically updated as new data is added to the database. + **Static** – For a static dataview, the data within the view is not updated automatically as new data is added to the database. When creating a static dataview, you specify a database version identifier that is the changeset ID. The dataview will contain contents of the database as of that changeset ID. To refresh the contents of a static dataview, you need to update it. If you do not provide a changeset ID when updating a dataview, system picks the latest one by default. ## Dataview versions When you create a dataview, it is assigned an initial version. Each update, whether automatic or manual, creates a new version of a dataview. A dataview version becomes active when it is mountable. A dataview version is released when it is not attached to any clusters and when it's no longer the latest active version. ## Data placement For each volume, you specify a list of paths for the data that you want to place on the volume. This can be done by using the db paths. Your paths can include the wildcard characters — asterisk (\$1) and question mark (?). Here are a few examples of how you can use db paths for segment configuration. + To specify a particular partition – `/2020.01.02/*` or `/2020.01.02*` + To specify all partitions for Jan 2020– `/2020.01.*` or `/2020.01*` + To specify all partitions for 1st of each month in 2020 – `/2020.??.01` or `/2020.*.01` + To specify all partitions – `/*` or `*` ## Data cardinality You can create multiple dataviews for a single database. For example, you may wish to create one dataview based on an older version of the database for historical analysis, at the same time you may want an auto updating dataview for applications to query more recent data in your database. You can also use multiple dataviews with the same data in each, as a way to spread query load from a large number of clusters querying the data. You can create two different dataviews on the same changeset version. ## Consideration + Dataviews are only available for clusters running on a scaling group. They are not supported on dedicated clusters. + The paths placed on different volumes cannot overlap. For example, you could not place a path of `/2023.10.31/*` on one volume of a dataview and `/2023.10*` on another volume of the same dataview because the paths overlap. This constraint is because each volume is a different segment in the `par.txt` file on the database and contents of a segment can’t overlap. # Managing kdb dataviews The following sections provide a detailed overview of the operations that you can perform by using a Managed kdb dataview. ## Creating a kdb dataview **To create a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. On database details page, choose the **Dataviews** tab. 1. Choose **Create dataview**. 1. On the **Create dataview** page, enter a unique name for the dataview. 1. (Optional) Enter a description for your dataview. 1. Choose the availability zone that you want to associate with the dataview. Currently, you can only choose single availability zone. 1. Choose a how you want to update data in the dataview from one of the following options. + **Auto-update** – Adds the most recent version of the data in a database. The dataview is automatically updated as new data is added to the database. + **Static** – Add data based on the changeset ID that you specify. The dataview is not automatically updated as new data is added to the database. To refresh the contents of a static dataview, you need to update it and specify a new changeset ID. When you choose this, the **Read Write** option enables. 1. If you choose **Static**, specify the **Changeset ID** to indicate which version of data you want. 1. If you choose **Static**, you get the option to make dataviews writable. Select **True** if you want to make the dataview writable to perform database maintenance operations. By default, this value is set to **False**. For more information, read [this](finspace-managed-kdb-databases-dbmaint.md) section. 1. (Optional) For **segment configuration**, specify the database path of the data that you want to place on each selected volume. You can also enable **On demand caching** on the selected database path when a particular file or a column of a database is accessed. **Note** Each [segment](finspace-managed-kdb-dataviews.md#segment) must have a unique database path for each volume. Every data view has a default segment associated with it. The default segment is S3/object store segment. All database paths not explicitly specified to be on a volume are accessible from the cluster through this default segment. The **Segment configuration** is required if **Read Write** is **True**. You can only add one segment for a writeable dataview. The **Database path** is disabled and defaults to **\$1\$1** when **Read Write** is **True** as you cannot have partial writeable dataviews on cache. 1. Choose **Create dataview**. The database details page opens and the table under **Dataviews** lists the newly created database along with its status. You can choose the dataview name from the list to view its details. ## Viewing kdb dataview details **To view and get details of a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose a database name. The database details page opens. 1. On the database details page, choose the **Dataviews** tab that shows a list of dataviews along with its status, availability zones where they were created, and their creation time. 1. From the list of dataviews, choose a name to view its details. The dataviews details page opens where you can view the following details. + **Dataview details** section – Displays the metadata of the dataview that you created. + **Configuration** tab – Displays the details about the dataview update mode and ID, and the availability zones ID. + **Active versions** tab – Displays a list of active versions of the dataview. Each update of the dataview creates a new version, including changeset details and the cache configurations. Each version triggers a workflow to cache database based on the cache configuration. A dataview version becomes active once the workflow finishes. The dataview version is deactivated under the following conditions + It's not the latest active version. + No cluster is currently mounting this version. You can choose the **Version ID** to see details of each active version. + **Clusters** tab – Displays a list of clusters that mounts the dataview. ## Updating a kdb dataview **To update a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose a database name. The database details page opens. 1. On the database details page, choose the **Dataviews** tab. 1. From the list of dataviews, choose a name and then choose **Edit**. 1. On the edit page, you can update the description for the dataview. If the dataview is **Static**, you can also update the **Changeset ID**. 1. Choose **Save changes**. ## Deleting a kdb dataview Before deleting a dataview, make sure that it is not in use by any cluster. You can check this from the **Clusters** tab in the dataview details page. **Note** This action is irreversible. Deleting a kdb dataview will delete all of its metadata. **To delete a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose the one whose dataview you want to delete. The database details page opens. 1. On the database details page, choose the **Dataviews** tab. 1. From the list of dataviews, choose a name and then choose **Delete**. 1. On the confirmation dialog box, enter **confirm** to provide a written consent to delete the resource permanently. 1. Choose **Delete**. # Database maintenance Amazon FinSpace Managed kdb allows you to perform schema changes to your database like adding a new column, updating a column type, and renaming columns, etc. You can perform the database maintenance operations by creating a general purpose cluster with a writable dataview. A writable dataview allows you to make updates to your kdb database locally on a cluster. To avoid caching the whole kdb database on a cluster, you can enable on-demand caching for your dataview segments. The dataview will only load the filesystem metadata of your database files for the segments with on-demand caching and loads the actual file content as they are accessed by a database maintenance operation. You can implement a database maintenance script and run it as an initialization script. An initialization script can run for multiple hours without being interrupted, which is required for long-running database maintenance tasks. When database maintenance script is running, monitor the cluster logs for progress and any errors. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes by using the `commit_kx_database` q API. The API creates a changeset and returns the changeset id, which you can use to monitor the changeset status through either the FinSpace API or console. You can also automate verification and commit steps in your database maintenance script itself. For more information, see the following sections. **Topics** + [ # Setting up for database maintenance ](dbmaint-writable-database-dataviews.md) + [ # Performing database maintenance ](dbmaint-long-running-dbmaint.md) # Setting up for database maintenance Setting up for database maintenance To perform the database maintenance operations, you need a writeable shallow copy of a database . A writable shallow copy of a kdb database only loads the metadata of your database files to make them visible on the file system and loads the actual file content as they are accessed. To optimise time and memory utilization, it is recommended not to load file content initially, as not all files may be necessary for a database maintenance operation. For instance, in the case of renaming a table, no files are read or updated directly. To create a writeable shallow copy of database, you can create dataviews with read write property set as true and enable on-demand caching in the configuration. A dataview performs minimal loading of files on the file system as needed by a database maintenance operation when on-demand caching is enabled. Reading an existing database file for the first time is slower as compared to accessing the files that have been previously read or newly written. This is because files are loaded onto the file system as they are accessed in case of on-demand caching. ## Creating writeable dataviews by using console Before you proceed, complete the following prerequisites: + Create a kdb environment. For more information, see [Creating a kdb environment](using-kdb-environment.md#create-kdb-environment). + Create a kdb database. For more information, see [Creating a kdb database](using-kdb-db.md#create-kdb-db). + Create a new changeset. For more information, see [Creating a new changeset](using-kdb-db.md#kdb-db-changesets). + Create a kdb volume. Make sure this volume is not used by any other resource. For more information, see [Creating a Managed kdb volume](create-volumes.md). **To create a writeable dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose a database name. 1. On database details page, choose the **Dataviews** tab. 1. Choose **Create dataview**. 1. On the **Create dataview** page, enter a unique name for the dataview. 1. (Optional) Enter a description for your dataview. 1. Choose the availability zone that you want to associate with the dataview. Currently, you can only choose single availability zone. 1. Under **Changeset update settings**, do the following. 1. Choose **Static** mode of update. **Note** The **Read Write** option is only available for **Static** update mode as you cannot perform automatic updates on a writeable dataview. 1. Select the **Changeset ID** for the changeset you created to indicate which version of data you want. 1. Choose **Read Write** as **True** to make this dataview as writeable. You cannot change this later. 1. Add **Segment configuration**. **Note** The **Segment configuration** is required if **Read Write** is **True**. You can only add one segment for a writeable dataview. The **Database path** is disabled and defaults to **\$1\$1** when **Read Write** is **True** as you cannot have partial writeable dataviews on cache. 1. Choose a volume for caching. Use an exclusive volume for writable dataviews, it should not be in use by any other dataviews. 1. For **On demand caching**, choose **True** to enable on demand caching on the selected database path when a particular file or a column of a database is accessed. When you enable on demand caching, files will only be copied to the dataview when they are accessed by code for reading or writing. When you disable on demand caching, everything is cached. The default value is **False**. 1. Choose **Create dataview**. The database details page opens and the table under **Dataviews** lists the newly created database along with its status. ## Creating writeable dataviews by using FinSpace API operations Before you proceed, complete the following prerequisites: + Create a kdb environment by using the [CreateKxEnvironment](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxEnvironment.html) API operation. + Create a kdb database by using the [CreateKxDatabase](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxDatabase.html) API operation. + Create a new changeset by using the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset.html) API operation. + Create a kdb volume by using the [CreateKxVolume](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxVolume.html) API operation. Make sure this volume is unique for this dataview and is not used by any other resource. To create a dataview with writable shallow copy of a database, create a dataview with the volume that has writable segments by using the [CreateKxDataview](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxDataview.html) API operation. You can make dataview as writeable by setting the `readWrite` parameter as true. You can only use this parameter for a static update mode. The `onDemand` parameter allows you to enable or disable on-demand caching on the selected dbPaths. Sample `CreateKxDataview` API request ``` { "autoUpdate": false, "availabilityZoneId": "use1-az1", "clientToken": "65117136-4421-4371-0f1a-ce012823126", "changesetId": "latest_changesetId", "readWrite": true, "segmentConfigurations": [{ "volumeName": "test_vol", "dbPaths": [ "/*" ], "onDemand": true } ], "azMode": "SINGLE", "dataviewName": "test_dv" } ``` Following are some of the considerations for the above request. + The `autoUpdate` must be `false` for if `readWrite` is `true` on the dataviews. + You need exclusive volume for creating a writable dataview. The volume mentioned in the `segmentConfiguration` should not be used by any other dataview. + The `dbPath` must be set as "/\$1" for writable dataview. + Only a single `segmentConfiguration` is allowed when `readWrite` is true. The `dbPaths` on the segment should be set as "\$1\$1" . + A dataview with `readWrite` set as `true` is not allowed to be updated. + You cannot update the `readWrite` property later. + A dataview can only have a single segment if `onDemand` is `true` on a segment. # Performing database maintenance After you create a writeable dataview, you create a scaling group general purpose cluster to run a long-running database maintenance script. For this, you use the cluster `initializationScript` attribute. The database maintenance script could run for multiple hours without being terminated. When database maintenance script is running, monitor the cluster logs for progress and any errors from the database maintenance script. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes to the underlying kdb database by using the `commit_kx_database` q API. You can also automate these steps in your database maintenance script itself. ## Steps to perform database maintenance using a scaling group general purpose cluster 1. Create a general purpose cluster in the scaling group with the previously created data view and provide database maintenance script using `initializationScript` in the [CreateKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) API operation. After you create the cluster, wait till the status changes to `Running`. During this time, you can monitor the logs from the cluster for progress and any errors from the database maintenance script. 1. Call the [GetKxConnectionString](https://docs.aws.amazon.com/finspace/latest/management-api/API_GetKxConnectionString.html) API to get a `signedConnectionString` for the cluster. 1. Connect to the cluster and verify the kdb database state by running q commands. 1. Call the `commit_kx_database` q API with the database name to apply the changes to the source kdb database. 1. Call the [GetKxChangset](https://docs.aws.amazon.com/finspace/latest/management-api/API_GetKxChangeset.html) API operation to check the status of the commit database changeset. After the kdb database is successfully updated, you can load the updated kdb database on an existing HDB cluster by calling the [UpdateKxClusterDatabases](https://docs.aws.amazon.com/finspace/latest/management-api/API_UpdateKxClusterDatabases.html) API operation or on a new HDB cluster by calling the [CreateKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) API operation. ## Steps to perform database maintenance using `dbmaint.q` This is section shows how you can perform database maintenance on a partitioned database by using a `dbmaint.q` script. The following example explains how you can load the `dbmaint.q` script on a general purpose cluster that runs on a scaling group, add a new column to a table, and finally commit the database to create a changeset. 1. Load the [https://github.com/KxSystems/kdb/blob/master/utils/dbmaint.q](https://github.com/KxSystems/kdb/blob/master/utils/dbmaint.q) script by running the following command. This script contains utility functions for maintenance of partitioned database tables in kdb\$1. ``` q) \l /opt/kx/app/code/dbmaint/dbmaint.q ``` 1. Load a database. ``` q) \l /opt/kx/app/db/welcomedb ``` 1. Inspect the table schema in your database. ``` q) meta example c | t f a ------| ----- date | d sym | s p time | p number| j ``` 1. Change to the database parent directory. ``` q) \cd /opt/kx/app/db ``` 1. Add a new column using the `addcol` function from the `dbmaint.q` script. ``` addcol[`:welcomedb;`example;`price;0h]; ``` 1. Inspect the updated table schema with the newly added column. ``` q)con "meta example" c | t f a ------| ----- date | d sym | s p time | p number| j price | h ``` 1. Commit the database changes by calling the `.aws.commit_kx_changeset` q API. The API creates a changeset and returns the id, which you can use to monitor the changeset status through the FinSpace API or console. ``` q) .aws.commit_kx_database["welcomedb"] id | "UscXQcZ2htijCQlr1xNaIA" status| "PENDING" ``` **Note** The recommended way to perform a long-running database maintenance is to implement a database maintenance script and execute it as cluster initialization script. An initialization script can run for multiple hours without being interrupted which is required for long-running database maintenance tasks. When database maintenance script is running, monitor the cluster logs for progress and any errors. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes to the underlying kdb database by using the commit\$1kx\$1database q API. You can also automate verification and commit steps in your database maintenance script itself.