After careful consideration, we decided to end support for Amazon FinSpace, effective October 7, 2026. Amazon FinSpace will no longer accept new customers beginning October 7, 2025. As an existing customer with an Amazon FinSpace environment created before October 7, 2025, you can continue to use the service as normal. After October 7, 2026, you will no longer be able to use Amazon FinSpace. For more information, see [Amazon FinSpace end of support](https://docs.aws.amazon.com/finspace/latest/userguide/amazon-finspace-end-of-support.html). # Managed kdb Insights Managed kdb Insights Amazon FinSpace with Managed kdb Insights provides a fully managed service for the latest version of kdb’s analytics engine. Kdb Insights is the leading time-series analytics engine used by capital markets customers to power their business-critical analytics workloads, such as liquidity insights, pricing, transaction cost analysis, and back-testing. With Amazon FinSpace Managed kdb Insights, you can deploy your existing real-time stream processing and high-performance kdb code in the AWS cloud to power time-sensitive analytics workloads. This is fundamental to running investment and trading businesses. Using Managed kdb Insights’ clusters, you can quickly set up a managed data processing and analytics hub. With a few clicks in the Managed kdb Insights application, you can migrate your existing kdb datasets to FinSpace. By configuring Managed kdb Insights to auto scale the kdb clusters up and down, you can meet the availability and runtime performance needs of each of your kdb workloads. You can configure your Managed kdb Insights clusters to automatically deploy across multiple Availability Zones and Regions to ensure that your analytics environment is available during the most critical business hours. As a result, you no longer require teams of specialists to monitor the infrastructure. This is because Managed kdb Insights continuously monitors underlying server health and capacity, and automatically replaces servers when they fail and patches servers in need of updates. In addition, it simplifies the work required to set up and deploy new clusters so the kdb administrators can focus more on business needs. Managed kdb Insights also supports running the same customer-developed kdb scripts that they run today on premises, and provides the same familiar kdb interfaces. **How it works** ![\[A diagram that shows how Managed kdb Insights works.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/fs-managed-kdb-how-it-works.png) **The diagram describes key components of Managed kdb Insights:** + You can manage Managed kdb Insights resources by using the AWS Management Console or the SDK/CLI. + Data is stored in durable object store back databases. + Compute clusters running kdb software access data in the database. + Data can be cached from the database on high performance disk cache for fast access by the cluster. + Developers and quantitative analysts (quants) can access clusters via kdb IPC connections. + Access can be controlled through IAM. + Activity is logged to CloudTrail and CloudWatch. ** **Topics** ** + [ # Permissions required for Managed kdb ](permissions-managed-kdb.md) + [ # Managed kdb Insights environments ](finspace-managed-kdb-environment.md) + [ # Managed kdb Insights databases ](finspace-managed-kdb-db.md) + [ # Managed kdb scaling groups ](finspace-managed-kdb-scaling-groups.md) + [ # Managed kdb volumes ](finspace-managed-kdb-volumes.md) + [ # Managed kdb Insights clusters ](finspace-managed-kdb-clusters.md) + [ # Logging and monitoring ](kdb-cluster-logging-monitoring.md) # Permissions required for Managed kdb You must have certain IAM permissions to use Managed kdb. In addition to the [finspace:\$1permissions](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonfinspace.html), you might need additional permissions to use the resources in your AWS account. FinSpace uses these permissions on your behalf to configure resources in your account where it needs to function. Add these permissions by using the IAM policies to IAM roles that you use to interact with Managed kdb. The following table shows a list of permissions and what they are needed for. **** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/finspace/latest/userguide/permissions-managed-kdb.html) ## Permissions FinSpace needs to resources in your account You will need to grant permission to FinSpace to access certain resources in your account. To do this, follow steps in the following sections. ### Granting permission to your AWS KMS key to encrypt data and code stored in Managed kdb You must grant the FinSpace service access by using the AWS KMS key policy to create Managed kdb changesets and load code onto a cluster. The following is an example of such a policy. In the following example, replace each **user input placeholder** with your own values. #### Sample AWS KMS key policy ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Sid": "FinSpace Permissions", "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:DescribeKey", "kms:Encrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*" ], "Resource": "arn:aws:kms:us-east-1:555555555555:key/f935d84c-d365-4753-875Y-1c014ab4f61Z", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" } } } ] } ``` ------ ### Granting permission to your Amazon S3 code bucket to load code onto your Managed kdb cluster To load code onto your cluster you must first grant the FinSpace service access to the Amazon S3 bucket that stores the code you want to load. The following is an example of the policy that you can use to grant access to code location. #### Example policy to grant access to the code location In the following example, replace each **user input placeholder** with your own values. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::managed-kdb-code/*", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" }, "ArnLike": { "aws:SourceArn": "arn:aws:finspace:us-east-1:555555555555:kxEnvironment//*" } } }, { "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-code", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" }, "ArnLike": { "aws:SourceArn": "arn:aws:finspace:us-east-1:555555555555:kxEnvironment//*" } } } ] } ``` ------ After you grant the FinSpace service access to the S3 bucket , you must ensure that the IAM role that you use when you [create a cluster](create-kdb-clusters.md) or when you [update the code on a cluster](update-cluster-code.md) has permission to access the files on the Amazon S3 bucket. The following is an example of the policy that you can use to grant access to the role. In the following example, replace each **user input placeholder** with your own values. #### Example policy for granting calling role access to the code location ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::managed-kdb-code/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-code" } ] } ``` ------ When you set permissions on the role, you can control which Amazon S3 locations a user can access. You can also set **Deny** policies on this role to prevent access to resources. For example, you can use the **Deny** policy to prevent access to resources in another account. ### Granting permission to your Amazon S3 data staging bucket to ingest data into Managed kdb To ingest data from Amazon S3 into your database through a changeset, you must first grant FinSpace access to the S3 bucket that stores the data you want to import as Managed kdb changesets. The following is an example of such a policy. In the following example, replace each **user input placeholder** with your own values. #### Example policy to grant the FinSpace service principal access to the code location In the following example, replace each **user input placeholder** with your own values. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::managed-kdb-data/*", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" }, "ArnLike": { "aws:SourceArn": "arn:aws:finspace:us-east-1:555555555555:kxEnvironment//*" } } }, { "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-data", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" }, "ArnLike": { "aws:SourceArn": "arn:aws:finspace:us-east-1:555555555555:kxEnvironment//*" } } } ] } ``` ------ After you grant FinSpace access to the Amazon S3 bucket, you must ensure that the IAM role you use when you [create a changeset](using-kdb-db.md#kdb-db-changesets) has permission to access the files on the Amazon S3 bucket. The following is an example of such a policy. In the following example, replace each **user input placeholder** with your own values. #### Example policy to grant role access to the changeset location ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::managed-kdb-data/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-data" } ] } ``` ------ When you set permissions on the role, you can control which Amazon S3 locations a user can access. You can also set **Deny** policies on this role to prevent access to resources. For example, you can use the **Deny** policy to prevent access to resources in another account. # Managed kdb Insights environments Managed kdb environments The Managed kdb Insights environment provides a logical container where you can launch and run clusters, and store data from kdb that can be used by the clusters. All resources in the Managed kdb environment run in AWS managed accounts and not in the customer account. The Managed kdb environment dedicated account is not shared with the existing FinSpace dataset browser environment. # Managing kdb environments The following sections provide a detailed overview of the operations that you can perform by using a Managed kdb Insights environment. ## Creating a kdb environment **Note** You can only create one kdb environment per Region per AWS account. **To create a kdb environment** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. On the getting started page, choose **Create kdb environment**. 1. On **Create kdb environment** page, enter the environment name and description. 1. Choose a symmetric encryption KMS key to encrypt data in your kdb environment. If a KMS key is not available in the Region where you want to create your FinSpace environment, create a new key. For more information, see [Creating keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) in the *AWS Key Management Service Developer Guide*. 1. (Optional) Add a new tag to assign it to your kdb environment. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your environment. 1. Choose **Create kdb environment**. The environment creation process begins and the environment details page opens. The environment creation process takes few minutes to finish in the background. You can view the status of environment creation under the kdb environment configuration section. After the environment is successfully created, you can add network configuration, databases, and clusters to the environment. ## Updating a kdb environment **To update a kdb environment** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Edit**. 1. Edit the environment details. **Note** You can only edit the **Name** and **Description** . 1. Choose **Update kdb environment**. You can view the updated details on the environment details page. ## Viewing kdb environment details **To view and get details of a kdb environment** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. The environment details page opens where you can view details about the environment, add or view network configuration, create new databases, and add clusters. ## Deleting a kdb environment **Note** This action is irreversible. Deleting a kdb environment will delete all resources (users, clusters, and databases) and their metadata in the account. After you initiate a deletion request, the billing for resources in an environment will stop immediately. **To delete a kdb environment** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Delete**. 1. On the confirmation dialog box, enter *confirm*. 1. Choose **Delete**. # Managing environment network settings For each Managed kdb Insights environment, you can configure a network connection to allow the Managed kdb clusters running in your environment infrastructure account to access resources in your internal network. You can create a connection by connecting your infrastructure account to an existing transit gateway in your organization. After you add a network, you can also specify details for the DNS servers that your Managed kdb clusters will use to resolve resources outside of your Managed kdb environment. After your Managed kdb environment is connected to your network, you can optionally configure your network to allow outbound traffic from your environment to the internet. This connectivity is managed by your network infrastructure. Managed kdb doesn't support direct internet access (inbound or outbound). ## Prerequisites Before you proceed, complete the following prerequisites: + Make sure that a kdb environment has been created. For more information, see [Creating a kdb environment](using-kdb-environment.md#create-kdb-environment). + Make sure that a transit gateway has been created in AWS Transit Gateway. For more information, see [Create the transit gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html#step-create-tgw) in the *AWS Transit Gateway User Guide*. + Make sure that you have a */26 (64)* IP address range from the *100.64.0.0/10* range that you can allocate to the subnets that connect to your transit gateway. ## Creating a network connection You can configure a network connection to allow the Managed kdb clusters running in your environment infrastructure account to access resources in your internal network. Optionally, you can also define how you manage the outbound traffic from kdb network to your internal network. You do this by configuring the attachment network access control lists (ACLs). A network ACL allows or denies specific outbound traffic at the subnet level. You can use the default network ACL for your VPC. Alternatively, to add an additional layer of security to your VPC, you can create a custom network ACL for your VPC with rules that are similar to the rules for your security groups. For more information, see the [Network ACL rules](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#nacl-rules) in the *Amazon VPC User Guide*. **Note** You can only configure one network connection per Managed kdb environment. You cannot delete a network connection. To remove the existing network and the network ACL attachments, delete the Managed kdb environment. **To create a network connection** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, under **Network** tab, choose **Add network configuration**. 1. On **Add network configuration** page, enter a transit gateway ID and the CIDR range that will be used for the subnets connecting to your internal network. For more information, see the [*Amazon VPC Transit Gateways User Guide*.](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html#create-tgw) **Note** When you add a transit gateway without creating a network ACL, all outbound traffic is allowed by default. 1. (Optional) Add rules to define how you want to manage the outbound traffic from kdb network to your internal network. Choose **Add new rule** to allow or deny outbound traffic for each port range and destination. **Note** When you create a network ACL rule, by default all the other traffic are denied. We process the ACL rules according to the rule numbers, in ascending order. 1. Choose **Save**. The connection creation process begins and the environment details page opens from where you can check the status under the **Network** tab. ## Editing a network **Note** You can't edit the transit gateway ID and CIDR routable space for your network. You only edit the network ACL configurations for your network. **To edit a network connection** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, under **Network** tab, choose **Edit network**. 1. On **Edit network** page, add or modify the network ACL rules as required. 1. Choose **Save changes**. The updates are available on the environment details page. ## Adding DNS details You can set the DNS resolver that the Managed kdb Insights compute nodes will use for resolving IP addresses. This is useful if you want to connect from your Managed kdb compute clusters to resources like on-premises kdb ticker plants or other resources. We recommend you add DNS details only after you have successfully configured a network in your Managed kdb environment. **Note** You can only add one DNS server and IP address per Managed kdb environment. **To add DNS details** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. Under **DNS details**, choose **Add details**. 1. On **Add DNS details** page, enter the DNS server name and IP address that the clusters running in the Managed kdb environment will use. 1. Choose **Add DNS details**. The **environment details** page opens and the DNS details are added in the **DNS details** section, from where you can edit the DNS details. # Tutorial: Configuring and validating outbound network connectivity through transit gateway Tutorial: Configuring and validating outbound network connectivity Amazon FinSpace Managed kdb environment allows you to connect to kdb or q processes in your account through transit gateway, without going over the internet. This section demonstrates how to setup outbound network connectivity from FinSpace Managed kdb environment to your virtual private cloud (VPC) and validate connectivity from an RDB cluster to a q process on an Amazon EC2 instance in your network. **Topics** + [ ## Prerequisites ](#prereq-kdb-tgw) + [ ## Setup diagram ](#tgw-setup-diag) + [ # Step 1: Configuring a network connection to create FinSpace VPC transit gateway attachment ](step1-config-ntw.md) + [ # Step 2: Adding DNS details to your network connection ](step2-dns-details.md) + [ # Step 3: Setting up a transit gateway VPC attachment from your VPC ](step3-setup-tgw-attachment.md) + [ # Step 4: Configuring routes in your VPC route tables ](step4-config-routing-tgw.md) + [ # Step 5: Configuring security group inbound rules ](step5-config-inbound-rule.md) + [ # Step 6: Validating network connectivity ](step6-validate-ntw.md) + [ # Step 7: Validating connection using the DNS server configuration ](step7-validate-connection-dns-server.md) ## Prerequisites Before you proceed, complete the following prerequisites: + Create a kdb environment. For more information, see [Creating a kdb environment](using-kdb-environment.md#create-kdb-environment). **Note** Note down the `Availability Zone Ids` after creating a kdb environment. You will need them when you create an attachment from your VPC to a transit gateway. + Make sure that you create a transit gateway in AWS Transit Gateway. For more information, see [Creating the transit gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html#step-create-tgw) in the *Amazon VPC Transit Gateways User Guide*. **Note** When creating the transit gateway, you only need to specify the name and description. For the rest of the fields, choose the default values. For example, for DNS-Support, VPN ECMP support, Default route table association, and Default route table propagation options should be selected by default. + Make sure you are familiar with the process of [Creating a kdb environment](using-kdb-environment.md#create-kdb-environment), [Creating a kdb user](finspace-managed-kdb-users.md#create-kdb-user), and [Creating a Managed kdb Insights cluster](create-kdb-clusters.md). ## Setup diagram This diagram shows a high level of configuration steps that are further described in the following sections. ![\[A diagram that shows steps to set up transit gateway.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/tgw-setup-diag.png) # Step 1: Configuring a network connection to create FinSpace VPC transit gateway attachment Step 1: Configuring a network connection **To create a network connection** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, under **Network** tab, choose **Add network configuration**. 1. On **Add network configuration** page, enter a transit gateway ID and the CIDR range that will be used for the subnets connecting to your internal network. For more information, see the [*Amazon VPC Transit Gateways User Guide*.](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html#create-tgw) **Note** When you add a transit gateway without creating a network ACL, all outbound traffic is allowed by default. 1. (Optional) Add rules to define how you want to manage the outbound traffic from kdb network to your internal network. Choose **Add new rule** to allow or deny outbound traffic for each port range and destination. **Note** When you create a network ACL rule, by default all the other traffic are denied. We process the ACL rules according to the rule numbers, in ascending order. 1. Choose **Save**. The connection creation process begins and the environment details page opens from where you can check the status under the **Network** tab. **Note** When you configure a network connection, make sure that you have a /26 (64) IP address range from the *100.64.0.0/10* range. The CIDR range should not be used in your network or any other environments that are connected by this TGW. A few valid examples of this CIDR range are *100.64.0.0/26*, *100.64.1.0/26*, *100.64.2.0/26*, *100.64.3.0/26*. We will pick *100.64.0.0/26* for this tutorial. This step creates a transit gateway VPC attachment to connect FinSpace environment to the transit gateway. After you configure a network, check the **Network** tab for details of your network. # Step 2: Adding DNS details to your network connection Step 2: Adding DNS details The **Network** tab on the Kdb environments details page allows you to add custom DNS server name and IP address. This is used when you have a custom DNS server that you want to query for internal host names. The DNS server IP is used for DNS resolution of queries. **To add DNS details** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. Under **DNS details**, choose **Add details**. 1. On **Add DNS details** page, enter *example.com* as the DNS server name and *173.31.0.2* as the DNS server IP. This means that any DNS queries for *example.com* from the FinSpace clusters will return the DNS resolver at *172.31.0.2* in the your VPC. **Note** The IP *172.31.0.2* is the second IP address in the default VPC CIDR and corresponds to the IP of the DNS Resolver for an Amazon VPC. Any DNS queries for *example.com* from the FinSpace clusters will return the DNS resolver at *172.31.0.2* in your custom VPC. 1. Choose **Add DNS details**. The **environment details** page opens and the DNS details are added in the **DNS details** section, from where you can edit the DNS details. # Step 3: Setting up a transit gateway VPC attachment from your VPC Step 3: Setting up a transit gateway VPC attachment **Note** It may take a few minutes for [Step 1](step1-config-ntw.md) and [Step 2](step2-dns-details.md) to complete. Wait till these steps are successful before proceeding. In the previous step you created a network connectivity from FinSpace environment to your transit gateway but FinSpace cannot reach into your network unless you create a VPC attachment from your VPC to Transit Gateway and set up routing and rules for the traffic to flow into your network. In this step, you create a transit gateway attachment and validate that it is associated in the transit gateway associations. **To create a transit gateway VPC attachment from your VPC** 1. Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). 1. On the navigation pane, choose **Transit Gateway Attachments**. 1. Choose **Create transit gateway attachment**. 1. For **Transit gateway ID**, choose the transit gateway for the attachment that you created in [step 1](step1-config-ntw.md) of this tutorial. 1. For **Attachment type**, choose **VPC**. 1. For **VPC ID**, choose the [default VPC](https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html) to attach to the transit gateway. This VPC must have at least one subnet associated with it. **Note** There is a default VPC for every AWS account. The default VPC ID is the value of the VPC ID column of the VPC table. To view your default VPC: Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). In the navigation pane, choose **Your VPCs**. In the **Default VPC** column, look for a value of **Yes**. Take note of the ID of the default VPC. 1. For **Subnet IDs**, choose 3 subnets from the availability zones where the environment is created. To check the availability zones ID mapping for your AWS account, go to the AWS Resource Access Manager in your account. Navigate to the product console, find the AZ ID at the bottom right of the page. **To validate the TGW associations** 1. Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). 1. In the navigation pane, choose **Transit Gateway ID** for transit gateway that you created earlier. 1. Under **Details**, choose **Association route table ID**. The **Association** tab shows the two VPC attachments, one from FinSpace infrastructure VPC and the other from your VPC. # Step 4: Configuring routes in your VPC route tables Step 4: Configuring routes With a VPC, you must create routes to send traffic to the transit gateway. The following steps show how you can update your default VPC route tables to have an entry for traffic to return to FinSpace VPC. **To configure route tables** 1. Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). 1. On the navigation pane, choose **Route Tables**. 1. Choose the route table for the default VPC ID. 1. Choose **Edit routes**. 1. On **Edit routes** page, choose **Add route** and enter *100.64.0.0/26* as the **Destination**. This value is the same as the CIDR range that you added while creating the network connectivity in [Step 1: Configuring a network connection to create FinSpace VPC transit gateway attachment](step1-config-ntw.md). 1. For **Target** choose **Transit Gateway** and select your transit gateway ID. 1. Choose **Save changes**. # Step 5: Configuring security group inbound rules Step 5: Configuring security group inbound rules After you set up routing, you need to add inbound rule for the default security group to allow inbound traffic. The default security group comes with your AWS account. For more information, see [Default security groups](https://docs.aws.amazon.com/vpc/latest/userguide/default-security-group.html) in the *Amazon VPC User Guide*. A security group acts as a firewall that controls the traffic allowed to and from the resources in your VPC. You can choose the ports and protocols to allow for inbound traffic or outbound traffic. For each security group, you add separate sets of rules for inbound traffic and outbound traffic. For more information, see [Security group rules](https://docs.aws.amazon.com/vpc/latest/userguide/security-group-rules.html) in the *Amazon VPC User Guide*. As an example, add an entry to allow TCP traffic for port *5005* to connect to a q process in your account running on port *5005*. This makes port *5005* of any host launched with the default security group to be reachable. **To create an inbound rule** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. On the navigation pane, choose **Security Groups**. 1. Under the **Inbound rules** tab, choose **Edit inbound rules**. 1. On **Inbound rules** page, choose **Add rules**. 1. For **Type**, choose *Custom TCP*. 1. For **Port range** enter *5005*. As another example, you can also allow all traffic from FinSpace to all ports. To allow all ports by default, follow the above steps of creating an inbound rule. In step 5, for **Type**, choose *All TCP*. **Note** If you need to restrict outbound traffic to specific ports and destination, add [network ACL](step1-config-ntw.md#nacl) while creating a network connection to deny outbound traffic from FinSpace for each port range and destination. When you create an Amazon EC2 instance, you need to specify the default security group for these inbound rules to apply. See next section for an example of how an Amazon EC2 instance is created with this security group. If you have hosts with different port rules you can create a security group for each host. When you launch an EC2 instance, use the security group with the port rules for your host. # Step 6: Validating network connectivity Step 6: Validating network connectivity After you’ve successfully created an outbound network connectivity between FinSpace VPC and your VPC using transit gateway, you can validate the network configuration. To do this, run a test to connect to a customer EC2 instance q process from an RDB cluster in the FinSpace environment. The following procedure shows how to connect to an RDB cluster and then connect to a q/kdb process running on EC2 instance in the your VPC account. In this step, you will create two EC2 instances: + **customerEc2Instance** – This is a q process to which the RDB would connect to. + **clientEc2Instance** – This is a q client to connect to the RDB cluster. ## Create an RDB Cluster Create an RDB cluster with a single-AZ mode by following the steps in [this](create-kdb-clusters.md) tutorial. ## Create an EC2 instance Use the following command to create an EC2 instance with a name *customerEc2Instance* instance to which an RDB would connect to. ``` echo '{"Version": "2012-10-17", "Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}' > policy.json aws iam create-role --role-name ssmrole --assume-role-policy-document file://policy.json aws iam attach-role-policy --role-name ssmrole --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess aws iam attach-role-policy --role-name ssmrole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore aws iam attach-role-policy --role-name ssmrole --policy-arn arn:aws:iam::aws:policy/AmazonSSMPatchAssociation aws iam create-instance-profile --instance-profile-name "SSMRole" aws iam add-role-to-instance-profile --instance-profile-name SSMRole --role-name ssmrole aws ec2 run-instances \ --count 1 \ --instance-type t2.micro \ --security-group-ids \ --subnet-id \ --iam-instance-profile Name=SSMRole \ --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=CustomerEc2Instance}]" \ --image-id $(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 --region us-east-2 | jq ".Parameters[0].Value" -r) \ --metadata-options "HttpEndpoint=enabled,HttpTokens=required" ``` ## Start a q process and listen on port 5005 1. Connect to the *CustomerEc2Instance* instance. For more information, see [this](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/connecting_to_windows_instance.html#session-manager) section. 1. Install the q client. For more information on installation, see [Installing kdb\$1](https://code.kx.com/q/learn/install/). 1. Launch a q process and run the following command to listen on port *5005*. ``` q) \p 5005 ``` ## Create another EC2 instance Create another instance with a name *clientEc2Instance*, which you can use to connect to the RDB cluster. The EC2 instance should use the same security group and subnet that you chose for the cluster. ``` aws ec2 run-instances \ --count 1 \ --instance-type t2.micro \ --security-group-ids \ --subnet-id \ --iam-instance-profile Name=SSMRole \ --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=Bastion}]" \ --image-id $(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 --region us-east-1 | jq ".Parameters[0].Value" -r) \ --metadata-options "HttpEndpoint=enabled,HttpTokens=required" ``` ## Test the connection Test the connection from q process on EC2 instance to the RDB cluster. Create an RDB cluster with a single-AZ mode by following the steps in [this](create-kdb-clusters.md) tutorial. 1. Connect to the *clientEc2Instance* by following the steps in [this](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/connecting_to_windows_instance.html#session-manager) section. 1. Install the q client. For more information on installation, see [Installing kdb\$1](https://code.kx.com/q/learn/install/). 1. Start a q process and connect to the RDB cluster on port *5005* by using the following example command. ``` q)cs_rdb1:   q)cs_rdb1: ssr[cs_rdb1;"\n";""]  q)conn: hopen cs_rdb1  q)conn hopen(": q)cs_rdb1: ssr[cs_rdb1;"\n";""] q)conn: hopen cs_rdb1 q)conn hopen(": q)cs_rdb1: ssr[cs_rdb1;"\n";""] q)conn: hopen cs_rdb1 q)conn hopen(":example.com:5005"; 10) ``` The connection test using the DNS name should work successfully. # Managed kdb Insights databases Managed kdb databases A Managed kdb Insights database acts as a highly available and scalable repository to store your kdb data files so that they can be used with one or more historical database (HDB) clusters in FinSpace kdb. Data in a database may consist of either kdb objects, kdb splayed tables, or kdb partitioned tables. These represent different types of kdb table structures and each must follow a prescribed file and path layout. You can learn more about each of these structures [here](https://code.kx.com/q/database/). Data is loaded into a database by defining a changeset, which lets you import a file or set of files into a database. The files in the kdb database are placed into logical paths called the *Database paths*. Creating a database does not automatically load any data. You must add data to the kdb database through changesets. # Managing kdb databases The following sections provide a detailed overview of the operations that you can perform by using a Managed kdb database. ## Creating a kdb database **To create a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. Choose **Create database**. 1. On the **Create database** page, enter a unique name for the database. 1. (Optional) Enter a description for your database. 1. (Optional) Add a new tag to assign it to your kdb database. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your database. 1. Choose **Create database**. The environment details page opens and the table under **Databases** lists the newly created database. You can choose the database name from the list to view its details in database details page. ## Managing data in a kdb database The Managed kdb Insights database allows you to add, update, or delete a set of files. When you create a database, there is no data loaded in it. You must add data to the database through changesets. A changeset represents a versioned set of changes that are applied to a database. ### Creating an Amazon S3 bucket policy Before you can ingest data into your database, you must have a valid Amazon S3 bucket IAM policy in place to allow FinSpace to access the data you will ingest into it. The following is an example of such a policy. In the following example, replace each **user input placeholder** with your own values. Replace **555555555555** with the AWS account where you created your Managed kdb Insights environment. **Example — Sample Amazon S3 bucket policy** **** ``` { "Version":"2012-10-17", "Id": "FinSpaceServiceAccess", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": [ "s3:GetObject", "s3:GetObjectTagging" ], "Resource": "arn:aws:s3:::managed-kdb-data/*", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" } } }, { "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com" }, "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::managed-kdb-data", "Condition": { "StringEquals": { "aws:SourceAccount": "555555555555" } } } ] } ``` ### Creating a new changeset You can add, update, and delete data in a database by creating a new changeset. You can either use the console or the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset) API to create a changeset. To add a data to your database, create a changeset by providing the changeset type as `PUT`, database path, and S3 URI path. To update data in a database, you need to create another changeset with the same database path you chose while adding the data. To delete data in a database, create a new changeset with changeset type as `DELETE`. **Note** You should only add data in the correct kdb file format that follows a valid kdb path structure. Other file formats and structures are not supported when accessed from a FinSpace Managed kdb cluster. You can learn more about valid kdb path structures [here](https://code.kx.com/q/database/). **To create a changeset from the console** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. The table under this tab displays a list of databases. 1. Choose a database name to view its details. 1. On the database details page, choose the **Changesets** tab. 1. Choose **Create changeset**. 1. On the **Create changeset** page, select one of the following types of changeset. + *PUT* – Adds or updates files in a database. + *DELETE* – Deletes files in a database. This option is not available when creating the changeset for the first time. 1. For **Database path**, specify a path within the database directory where you want to add data. If the data already exists at this path, it will be updated. 1. For **S3 URI ** provide the source path of the file to add data. 1. Choose **Create changeset**. The database details page opens where you can see the status of the changeset in the changeset table. You can choose the changeset ID to view details of a changeset. ## Updating a kdb database **To update the metadata of a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose the one that you want to update. The database details page opens. 1. On the database details page, choose **Edit**. 1. Edit the database description. 1. Choose **Update database**. ## Viewing kdb database details **To view and get details of a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. The table under this tab displays a list of databases. 1. Choose a database name to view its details. The database details page opens where you can view details about the database. You can also add and view changesets and tags associated with this database. ## Deleting a kdb database **Note** This action is irreversible. Deleting a kdb database will delete all of its contents. **To delete a kdb database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose the one that you want to delete. The database details page opens. 1. On the database details page, choose **Delete**. 1. On the confirmation dialog box, enter *confirm*. 1. Choose **Delete**. # Dataviews for querying data Dataviews allow you to place portions of your Managed kdb Insights object store database onto disk for faster read-only access from your kdb clusters. To your kdb process, the dataview looks like a kdb segmented database, with data placed across one or more disk mounts (volumes) and the object store. This lets you place frequently-queried data on a fast-access disk for more performant access while keeping the rest of the data in the object store layer for less frequent access. With dataviews, the golden copy of your database’s data still remains in the object store format. The data stored on disk for faster access is a copy. ![\[A diagram that shows how dataviews work.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-dataviews-intro-diagram.png) Dataviews can be accessed from HDB and General purpose (GP) type clusters for read only access. The data within a dataview is accessible from the cluster as a kdb [segmented database](https://code.kx.com/q/database/segment/) that is automatically configured when you associate the dataview with the cluster. A segment is a mount point that can contain a portion of a database. Different segments could contain different data partitions, tables, or even columns. A kdb *par.txt * file that FinSpace automatically creates when you mount a database defines the segments. The segments of this segmented database can reside on different kdb Insights disk volumes. A segment of your database can be any portion of it. For example, consider a database with contents as the following date-partitioned layout. ``` /sym /2023.10.01/trades/price /2023.10.01/trades/time /2023.10.01/trades/quality /2023.10.01/trades/price /2023.10.02/trades/time /2023.10.02/trades/quality /2023.10.02/trades/price /2023.10.03/trades/time /2023.10.03/trades/quality /2023.10.03/trades/price /2023.10.04/trades/time /2023.10.04/trades/quality /2023.10.05/trades/price /2023.10.05/trades/time /2023.10.05/trades/quality /2023.10.05/trades/price /2023.10.01/trades/.d /2023.10.02/trades/.d /2023.10.03/trades/.d /2023.10.04/trades/.d /2023.10.05/trades/.d ``` In this example, `trades` is a table and `time`, `quantity`, and `price` are columns. You can store the most recent day of data on a high throughput volume, two days prior to that on 250 MB/s/TiB volume, with the rest accessible as a segment from the object store layer. The following table shows the data and segments. | Database contents | Segments | | --- | --- | | /2023.10.05/trades/time /2023.10.05/trades/quality /2023.10.05/trades/price | **Segment**: **Dataview Segment 1** **Stored On**: Managed kdb Insights Volume 1 [High throughput – 1000 MB/s/TiB] | | /2023.10.04/trades/time /2023.10.04/trades/quality /2023.10.04/trades/price /2023.10.03/trades/time /2023.10.03/trades/quality /2023.10.03/trades/price | **Segment**: **Dataview Segment 2** **Stored On**: Managed kdb Insights Volume 2 [Medium Throughput – 250 MB/s/TiB] | | /2023.10.02/trades/time /2023.10.02/trades/quality /2023.10.02/trades/price /2023.10.01/trades/time /2023.10.01/trades/quality /2023.10.01/trades/price | **Segment**: **Dataview Default Segment** **Stored On**: Object store | This gives you control to place copies of portions of your database on the appropriate type of disk for access, if you require higher performance access than what is available with the default object store storage. In addition, having the ability to explicitly place data on different volumes when creating a dataview, the contents directly under the root (/) path of the database, such as `/sym` in this example, are always copied to the cluster’s local storage for fast access. ## Auto-updating vs static dataviews When you create a dataview, you can specify from one of the following types of dataview. + **Auto-updating** –An auto-update dataview contains the most recent version of the data in the database. Its contents are automatically updated as new data is added to the database. + **Static** – For a static dataview, the data within the view is not updated automatically as new data is added to the database. When creating a static dataview, you specify a database version identifier that is the changeset ID. The dataview will contain contents of the database as of that changeset ID. To refresh the contents of a static dataview, you need to update it. If you do not provide a changeset ID when updating a dataview, system picks the latest one by default. ## Dataview versions When you create a dataview, it is assigned an initial version. Each update, whether automatic or manual, creates a new version of a dataview. A dataview version becomes active when it is mountable. A dataview version is released when it is not attached to any clusters and when it's no longer the latest active version. ## Data placement For each volume, you specify a list of paths for the data that you want to place on the volume. This can be done by using the db paths. Your paths can include the wildcard characters — asterisk (\$1) and question mark (?). Here are a few examples of how you can use db paths for segment configuration. + To specify a particular partition – `/2020.01.02/*` or `/2020.01.02*` + To specify all partitions for Jan 2020– `/2020.01.*` or `/2020.01*` + To specify all partitions for 1st of each month in 2020 – `/2020.??.01` or `/2020.*.01` + To specify all partitions – `/*` or `*` ## Data cardinality You can create multiple dataviews for a single database. For example, you may wish to create one dataview based on an older version of the database for historical analysis, at the same time you may want an auto updating dataview for applications to query more recent data in your database. You can also use multiple dataviews with the same data in each, as a way to spread query load from a large number of clusters querying the data. You can create two different dataviews on the same changeset version. ## Consideration + Dataviews are only available for clusters running on a scaling group. They are not supported on dedicated clusters. + The paths placed on different volumes cannot overlap. For example, you could not place a path of `/2023.10.31/*` on one volume of a dataview and `/2023.10*` on another volume of the same dataview because the paths overlap. This constraint is because each volume is a different segment in the `par.txt` file on the database and contents of a segment can’t overlap. # Managing kdb dataviews The following sections provide a detailed overview of the operations that you can perform by using a Managed kdb dataview. ## Creating a kdb dataview **To create a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. On database details page, choose the **Dataviews** tab. 1. Choose **Create dataview**. 1. On the **Create dataview** page, enter a unique name for the dataview. 1. (Optional) Enter a description for your dataview. 1. Choose the availability zone that you want to associate with the dataview. Currently, you can only choose single availability zone. 1. Choose a how you want to update data in the dataview from one of the following options. + **Auto-update** – Adds the most recent version of the data in a database. The dataview is automatically updated as new data is added to the database. + **Static** – Add data based on the changeset ID that you specify. The dataview is not automatically updated as new data is added to the database. To refresh the contents of a static dataview, you need to update it and specify a new changeset ID. When you choose this, the **Read Write** option enables. 1. If you choose **Static**, specify the **Changeset ID** to indicate which version of data you want. 1. If you choose **Static**, you get the option to make dataviews writable. Select **True** if you want to make the dataview writable to perform database maintenance operations. By default, this value is set to **False**. For more information, read [this](finspace-managed-kdb-databases-dbmaint.md) section. 1. (Optional) For **segment configuration**, specify the database path of the data that you want to place on each selected volume. You can also enable **On demand caching** on the selected database path when a particular file or a column of a database is accessed. **Note** Each [segment](finspace-managed-kdb-dataviews.md#segment) must have a unique database path for each volume. Every data view has a default segment associated with it. The default segment is S3/object store segment. All database paths not explicitly specified to be on a volume are accessible from the cluster through this default segment. The **Segment configuration** is required if **Read Write** is **True**. You can only add one segment for a writeable dataview. The **Database path** is disabled and defaults to **\$1\$1** when **Read Write** is **True** as you cannot have partial writeable dataviews on cache. 1. Choose **Create dataview**. The database details page opens and the table under **Dataviews** lists the newly created database along with its status. You can choose the dataview name from the list to view its details. ## Viewing kdb dataview details **To view and get details of a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose a database name. The database details page opens. 1. On the database details page, choose the **Dataviews** tab that shows a list of dataviews along with its status, availability zones where they were created, and their creation time. 1. From the list of dataviews, choose a name to view its details. The dataviews details page opens where you can view the following details. + **Dataview details** section – Displays the metadata of the dataview that you created. + **Configuration** tab – Displays the details about the dataview update mode and ID, and the availability zones ID. + **Active versions** tab – Displays a list of active versions of the dataview. Each update of the dataview creates a new version, including changeset details and the cache configurations. Each version triggers a workflow to cache database based on the cache configuration. A dataview version becomes active once the workflow finishes. The dataview version is deactivated under the following conditions + It's not the latest active version. + No cluster is currently mounting this version. You can choose the **Version ID** to see details of each active version. + **Clusters** tab – Displays a list of clusters that mounts the dataview. ## Updating a kdb dataview **To update a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Databases** tab. 1. From the list of databases, choose a database name. The database details page opens. 1. On the database details page, choose the **Dataviews** tab. 1. From the list of dataviews, choose a name and then choose **Edit**. 1. On the edit page, you can update the description for the dataview. If the dataview is **Static**, you can also update the **Changeset ID**. 1. Choose **Save changes**. ## Deleting a kdb dataview Before deleting a dataview, make sure that it is not in use by any cluster. You can check this from the **Clusters** tab in the dataview details page. **Note** This action is irreversible. Deleting a kdb dataview will delete all of its metadata. **To delete a kdb dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose the one whose dataview you want to delete. The database details page opens. 1. On the database details page, choose the **Dataviews** tab. 1. From the list of dataviews, choose a name and then choose **Delete**. 1. On the confirmation dialog box, enter **confirm** to provide a written consent to delete the resource permanently. 1. Choose **Delete**. # Database maintenance Amazon FinSpace Managed kdb allows you to perform schema changes to your database like adding a new column, updating a column type, and renaming columns, etc. You can perform the database maintenance operations by creating a general purpose cluster with a writable dataview. A writable dataview allows you to make updates to your kdb database locally on a cluster. To avoid caching the whole kdb database on a cluster, you can enable on-demand caching for your dataview segments. The dataview will only load the filesystem metadata of your database files for the segments with on-demand caching and loads the actual file content as they are accessed by a database maintenance operation. You can implement a database maintenance script and run it as an initialization script. An initialization script can run for multiple hours without being interrupted, which is required for long-running database maintenance tasks. When database maintenance script is running, monitor the cluster logs for progress and any errors. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes by using the `commit_kx_database` q API. The API creates a changeset and returns the changeset id, which you can use to monitor the changeset status through either the FinSpace API or console. You can also automate verification and commit steps in your database maintenance script itself. For more information, see the following sections. **Topics** + [ # Setting up for database maintenance ](dbmaint-writable-database-dataviews.md) + [ # Performing database maintenance ](dbmaint-long-running-dbmaint.md) # Setting up for database maintenance Setting up for database maintenance To perform the database maintenance operations, you need a writeable shallow copy of a database . A writable shallow copy of a kdb database only loads the metadata of your database files to make them visible on the file system and loads the actual file content as they are accessed. To optimise time and memory utilization, it is recommended not to load file content initially, as not all files may be necessary for a database maintenance operation. For instance, in the case of renaming a table, no files are read or updated directly. To create a writeable shallow copy of database, you can create dataviews with read write property set as true and enable on-demand caching in the configuration. A dataview performs minimal loading of files on the file system as needed by a database maintenance operation when on-demand caching is enabled. Reading an existing database file for the first time is slower as compared to accessing the files that have been previously read or newly written. This is because files are loaded onto the file system as they are accessed in case of on-demand caching. ## Creating writeable dataviews by using console Before you proceed, complete the following prerequisites: + Create a kdb environment. For more information, see [Creating a kdb environment](using-kdb-environment.md#create-kdb-environment). + Create a kdb database. For more information, see [Creating a kdb database](using-kdb-db.md#create-kdb-db). + Create a new changeset. For more information, see [Creating a new changeset](using-kdb-db.md#kdb-db-changesets). + Create a kdb volume. Make sure this volume is not used by any other resource. For more information, see [Creating a Managed kdb volume](create-volumes.md). **To create a writeable dataview** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Databases** tab. 1. From the list of databases, choose a database name. 1. On database details page, choose the **Dataviews** tab. 1. Choose **Create dataview**. 1. On the **Create dataview** page, enter a unique name for the dataview. 1. (Optional) Enter a description for your dataview. 1. Choose the availability zone that you want to associate with the dataview. Currently, you can only choose single availability zone. 1. Under **Changeset update settings**, do the following. 1. Choose **Static** mode of update. **Note** The **Read Write** option is only available for **Static** update mode as you cannot perform automatic updates on a writeable dataview. 1. Select the **Changeset ID** for the changeset you created to indicate which version of data you want. 1. Choose **Read Write** as **True** to make this dataview as writeable. You cannot change this later. 1. Add **Segment configuration**. **Note** The **Segment configuration** is required if **Read Write** is **True**. You can only add one segment for a writeable dataview. The **Database path** is disabled and defaults to **\$1\$1** when **Read Write** is **True** as you cannot have partial writeable dataviews on cache. 1. Choose a volume for caching. Use an exclusive volume for writable dataviews, it should not be in use by any other dataviews. 1. For **On demand caching**, choose **True** to enable on demand caching on the selected database path when a particular file or a column of a database is accessed. When you enable on demand caching, files will only be copied to the dataview when they are accessed by code for reading or writing. When you disable on demand caching, everything is cached. The default value is **False**. 1. Choose **Create dataview**. The database details page opens and the table under **Dataviews** lists the newly created database along with its status. ## Creating writeable dataviews by using FinSpace API operations Before you proceed, complete the following prerequisites: + Create a kdb environment by using the [CreateKxEnvironment](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxEnvironment.html) API operation. + Create a kdb database by using the [CreateKxDatabase](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxDatabase.html) API operation. + Create a new changeset by using the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset.html) API operation. + Create a kdb volume by using the [CreateKxVolume](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxVolume.html) API operation. Make sure this volume is unique for this dataview and is not used by any other resource. To create a dataview with writable shallow copy of a database, create a dataview with the volume that has writable segments by using the [CreateKxDataview](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxDataview.html) API operation. You can make dataview as writeable by setting the `readWrite` parameter as true. You can only use this parameter for a static update mode. The `onDemand` parameter allows you to enable or disable on-demand caching on the selected dbPaths. Sample `CreateKxDataview` API request ``` { "autoUpdate": false, "availabilityZoneId": "use1-az1", "clientToken": "65117136-4421-4371-0f1a-ce012823126", "changesetId": "latest_changesetId", "readWrite": true, "segmentConfigurations": [{ "volumeName": "test_vol", "dbPaths": [ "/*" ], "onDemand": true } ], "azMode": "SINGLE", "dataviewName": "test_dv" } ``` Following are some of the considerations for the above request. + The `autoUpdate` must be `false` for if `readWrite` is `true` on the dataviews. + You need exclusive volume for creating a writable dataview. The volume mentioned in the `segmentConfiguration` should not be used by any other dataview. + The `dbPath` must be set as "/\$1" for writable dataview. + Only a single `segmentConfiguration` is allowed when `readWrite` is true. The `dbPaths` on the segment should be set as "\$1\$1" . + A dataview with `readWrite` set as `true` is not allowed to be updated. + You cannot update the `readWrite` property later. + A dataview can only have a single segment if `onDemand` is `true` on a segment. # Performing database maintenance After you create a writeable dataview, you create a scaling group general purpose cluster to run a long-running database maintenance script. For this, you use the cluster `initializationScript` attribute. The database maintenance script could run for multiple hours without being terminated. When database maintenance script is running, monitor the cluster logs for progress and any errors from the database maintenance script. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes to the underlying kdb database by using the `commit_kx_database` q API. You can also automate these steps in your database maintenance script itself. ## Steps to perform database maintenance using a scaling group general purpose cluster 1. Create a general purpose cluster in the scaling group with the previously created data view and provide database maintenance script using `initializationScript` in the [CreateKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) API operation. After you create the cluster, wait till the status changes to `Running`. During this time, you can monitor the logs from the cluster for progress and any errors from the database maintenance script. 1. Call the [GetKxConnectionString](https://docs.aws.amazon.com/finspace/latest/management-api/API_GetKxConnectionString.html) API to get a `signedConnectionString` for the cluster. 1. Connect to the cluster and verify the kdb database state by running q commands. 1. Call the `commit_kx_database` q API with the database name to apply the changes to the source kdb database. 1. Call the [GetKxChangset](https://docs.aws.amazon.com/finspace/latest/management-api/API_GetKxChangeset.html) API operation to check the status of the commit database changeset. After the kdb database is successfully updated, you can load the updated kdb database on an existing HDB cluster by calling the [UpdateKxClusterDatabases](https://docs.aws.amazon.com/finspace/latest/management-api/API_UpdateKxClusterDatabases.html) API operation or on a new HDB cluster by calling the [CreateKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) API operation. ## Steps to perform database maintenance using `dbmaint.q` This is section shows how you can perform database maintenance on a partitioned database by using a `dbmaint.q` script. The following example explains how you can load the `dbmaint.q` script on a general purpose cluster that runs on a scaling group, add a new column to a table, and finally commit the database to create a changeset. 1. Load the [https://github.com/KxSystems/kdb/blob/master/utils/dbmaint.q](https://github.com/KxSystems/kdb/blob/master/utils/dbmaint.q) script by running the following command. This script contains utility functions for maintenance of partitioned database tables in kdb\$1. ``` q) \l /opt/kx/app/code/dbmaint/dbmaint.q ``` 1. Load a database. ``` q) \l /opt/kx/app/db/welcomedb ``` 1. Inspect the table schema in your database. ``` q) meta example c | t f a ------| ----- date | d sym | s p time | p number| j ``` 1. Change to the database parent directory. ``` q) \cd /opt/kx/app/db ``` 1. Add a new column using the `addcol` function from the `dbmaint.q` script. ``` addcol[`:welcomedb;`example;`price;0h]; ``` 1. Inspect the updated table schema with the newly added column. ``` q)con "meta example" c | t f a ------| ----- date | d sym | s p time | p number| j price | h ``` 1. Commit the database changes by calling the `.aws.commit_kx_changeset` q API. The API creates a changeset and returns the id, which you can use to monitor the changeset status through the FinSpace API or console. ``` q) .aws.commit_kx_database["welcomedb"] id | "UscXQcZ2htijCQlr1xNaIA" status| "PENDING" ``` **Note** The recommended way to perform a long-running database maintenance is to implement a database maintenance script and execute it as cluster initialization script. An initialization script can run for multiple hours without being interrupted which is required for long-running database maintenance tasks. When database maintenance script is running, monitor the cluster logs for progress and any errors. After the database maintenance script completes, connect to the cluster to verify the updated kdb database and commit changes to the underlying kdb database by using the commit\$1kx\$1database q API. You can also automate verification and commit steps in your database maintenance script itself. # Managed kdb scaling groups Managed kdb scaling groups Many kdb customers today use a deployment architecture consisting of multiple kdb processes running on a single compute host. When workloads are such that the resource of demands of the different processes don’t conflict, this approach can maximize use of computing resources (CPU, RAM, I/O) to achieve more efficient use of computing resources. Scaling groups allows you to take this same approach with Managed kdb Insights. Scaling group terminology + Scaling group – Shared compute you can run your kdb workloads (clusters) on. + Scaling group host – A single unit of compute in a scaling group. Scaling groups currently can only have a single host. + Cluster – A set of one or more identically configured kdb process (nodes). + Cluster node – A single kdb process, running within a cluster. With scaling groups, you can run multiple kdb workloads or clusters on shared compute (a host) that you provision. This allows you to maximize utilization of compute in your FinSpace Managed kdb Insights environment. You can run multiple clusters on a single scaling group host. Each cluster can have one or more nodes, each with a kdb process. ![\[A diagram that shows an example of four clusters running on a scaling group host.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-sg-fig1.png) The previous diagram is an example of four clusters running on a scaling group host. Cluster 1 has four nodes, Cluster 2 has two nodes, cluster 3 has three nodes and cluster 4 has one node. As memory requirements for an individual cluster vary throughout the day, each may consume different amounts of memory. By placing workloads that have memory needs that peak at different times throughout the day, you can place more workloads or clusters in a fixed set of compute than it is possible if you used FinSpace [dedicated](kdb-clusters-running-clusters-comparison.md) cluster option. For example, you may have multiple HDB workloads where memory requirement of any individual HDB will vary at different times of the day, but in total they will all remain within a certain known memory footprint. You can place all of these workloads onto a scaling group to share resources like CPU and memory as shown in the following diagram. ![\[A diagram that shows an example of four clusters running on a scaling group host.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-scalinggroup-fig2.png) ## High level workflow for running clusters on a scaling group Before running a cluster on a scaling group, you need to create the scaling group itself. Once you create the scaling group, you can launch one or more clusters on it. You can display clusters running on a scaling group by using the `ListKxClusters` API or from the **Clusters** tab in Amazon FinSpace console. When you delete a cluster running in a scaling group, the host and any other running clusters on the scaling group remain available. If there are no clusters running on a scaling group, you may delete it. ## Resource management with scaling groups When launching a cluster to run on a scaling group,the total available amount of memory on the scaling group host is limited. The following table describes the limits of each host. | Compute type | vCPUs | Memory available for kdb (GiB) | | --- | --- | --- | | kx.sg.large | 2 | 16 | | kx.sg.xlarge | 4 | 32 | | kx.sg.2xlarge | 8 | 64 | | kx.sg.4xlarge | 16 | 108 | | kx.sg.8xlarge | 32 | 216 | | kx.sg.16xlarge | 64 | 432 | | kx.sg.32xlarge | 128 | 864 | | kx.sg1.16xlarge | 64 | 1949 | | kx.sg1.24xlarge | 96 | 2948 | When launching a kdb cluster to run on a scaling group, you specify the minimum memory required for each kdb process in the cluster (node) as well as expected amount of memory. If there is insufficient memory on the scaling group host to meet this required value, the cluster will not start. You can also specify an expected value for the amount of memory the cluster will require. The scheduler will use this to avoid launching the cluster if the memory value is not sufficient. For clusters with more than one node or kdb processes, the amount of memory used is the sum of the kdb process memory that each node consumes. ## Considerations + Currently, a scaling group consists of a single scaling group host and clusters can only run on one scaling group at a time. If you need to run more clusters in your environment than can fit on a single scaling group host, you may run multiple and put different clusters from your set on to different scaling groups. + You cannot delete a scaling group until you delete all the clusters running on it. + [Savedown storage](kdb-cluster-types.md#kdb-cluster-savedown-storage) does not work with General purpose (GP) and RDB clusters running on scaling groups. Instead, you should use volumes for the temporary storage of your savedown data. + HDB and GP clusters, when they are run as a part of a scaling group, don't support high performance HDB disk cache. You may instead use dataviews if you need to place portions of your database on high performance disk. # Managing kdb scaling groups The following sections provide a detailed overview of the operations that you can perform by using Managed kdb scaling groups. ** **Topics** ** + [ # Creating a Managed kdb scaling group ](create-scaling-groups.md) + [ # Viewing a Managing kdb scaling group ](view-kdb-scaling-groups.md) + [ # Deleting a Managing kdb scaling group ](delete-kdb-scaling-groups.md) # Creating a Managed kdb scaling group **To create a scaling group** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Kdb scaling groups** tab. 1. Choose **Create kdb scaling group**. 1. On the **Create kdb scaling group** page, enter a unique name for the scaling group details. 1. Choose a **Host Type** based on the available throughput and size. 1. Choose the availability zone that you want to associate with the scaling group. Currently, you can choose only single availability zone. 1. (Optional) Add a new tag to assign it to your scaling group. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your user. 1. Choose **Create kdb scaling group**. The scaling group creation process starts and the kdb environment details page opens where you can see the status of creation under the **Kdb scaling groups** tab. # Viewing a Managing kdb scaling group **To view and get details of a Managing kdb scaling group** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Kdb scaling groups** tab. 1. From the list of scaling groups, choose a name to view its details. The Kdb scaling group details page opens where you can view the following details. + **Scaling group details** section – Displays the metadata of the scaling group that you view. + **Configuration** tab – Displays the availability zone for the scaling group. + **Monitoring** tab – Displays the dashboard of scaling group metrics. You can view activity logs for your scaling group here. + **Clusters** tab – Displays a list of clusters running on this scaling group. For information on how to create clusters, see [Creating a Managed kdb Insights cluster](create-kdb-clusters.md). + **Tags** tab – Displays a list of key-value pairs that are associated with the scaling group. If you did not provide tags during volume creation, choose **Manage tags** to add new tags. # Deleting a Managing kdb scaling group **Note** This action is irreversible. Deleting a scaling group will delete all of its data. **To delete a Managing kdb scaling group** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Kdb scaling groups** tab. 1. From the list of scaling groups, choose the one that you want to delete and choose **Delete**. Alternatively, you can choose the scaling group name and open the details page to delete it. 1. On the confirmation dialog box, enter **confirm** to provide a written consent to delete the resource permanently. 1. Choose **Delete**. # Managed kdb volumes Volumes are managed storage that resides in your Managed kdb Insights environment and can be associated with clusters for storage of data such as TickerPlant (TP) logs, Real-time Database (RDB) savedown files, and temporary storage on General purpose (GP) type clusters. Volumes can also be used by dataview to store copies of your database of disk for fast access when reading data from a database. You can also use volumes to share data between RDB and Intra-day DB (IDB). ## Volumes for temporary data storage You can use a Managed kdb volume data storage for your cluster. When creating a TP cluster, you must specify a volume that will hold the TP logs. For an RDB or GP cluster running on a scaling group, you can specify a volume to hold savedown or temporary files. Multiple clusters can share a single volume for simplicity as shown in Figure 1, or you can configure multiple volumes and associate them with specific clusters for workload isolation as shown in Figure 2. ![\[A diagram that shows how clusters can share volumes.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-storage-volumes.png) ## Volumes with dataviews You can use Managed kdb volumes when you create dataviews. Dataviews store a copy of the data in your database on one or more volumes for fast access. When creating a dataview you can specify one or more volumes to store a portion of your database for faster data access compared to querying the data from the default object store format of the data in the database. For more information about using volumes as part of a dataview, see [Creating a kdb dataview](managing-kdb-dataviews.md#create-kdb-dataview). ## Volume types Volumes are available in different price or performance characteristics based on your need. Currently FinSpace offers the following three types of volume with three throughput characteristics. + 1000 MB/s/TiB – + 250 MB/s/TiB + 12 MB/s/TiB ## Considerations + When you delete a cluster, the data remains on the volume. If you don’t want this delete data before deleting the cluster. + You can access data mounted on a volume from within a cluster from the path `/opt/kx/app/shared/$VOLUME_NAME/$CLUSTER_NAME`. # Managing kdb volumes The following sections provide a detailed overview of the operations that you can perform by using Managed kdb volumes. ** **Topics** ** + [ # Creating a Managed kdb volume ](create-volumes.md) + [ # Viewing a Managed kdb volume ](view-volumes.md) + [ # Updating a Managed kdb volume ](update-volumes.md) + [ # Deleting a Managed kdb volume ](delete-volumes.md) # Creating a Managed kdb volume **To create a Managed kdb volume** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Volumes** tab. 1. Choose **Create volume**. 1. On the **Create volume** page, enter the volume details and choose the **Volume type**. Currently, FinSpace only supports **NAS\$11** (network attached storage) volume type. 1. Choose the throughput from one of the following types. + **SSD\$11000** + **SSD\$1250** + **HDD\$112** 1. Enter the size for the network attached storage configuration. For storage type **SSD\$11000** and **SSD\$1250** you can select the minimum size as 1200 GB or increments of 2400 GB. For storage type **HDD\$112** you can select the minimum size as 6000 GB or increments of 6000 GB. 1. Choose the availability zone that you want to associate with the volume. 1. (Optional) Add a new tag to assign it to your Managed kdb volume. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your user. 1. Choose **Create volume**. The volume creation process starts and kdb environment details page opens where you can see the status of volume creation under the **Volumes** tab. # Viewing a Managed kdb volume **To view and get details of a Managed kdb volume** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Volumes** tab. The table under this tab displays a list of volumes created in the environment. 1. Choose a volume name to view its details. The volume details page opens where you can view the following details. + **Volume details** section – Displays the metadata of the volume that you created. + **Configuration** tab – Displays the details about the network attached storage and availability zones. + **Monitoring** tab – Displays the dashboard of volume metrics. You can view activity logs for your volume here. + **Clusters** tab – Displays a list of clusters attached to this volume. For information on how to create clusters, see [Creating a Managed kdb Insights cluster](create-kdb-clusters.md). + **Tags** tab – Displays a list of key-value pairs associated with the volume. If you did not provide tags during volume creation, choose **Manage tags** to add new tags. # Updating a Managed kdb volume You can only edit the description and size of a volume. When you update a volume, you can only increase the volume size but cannot reduce it. During the update process, the filesystem might be unavailable for a few minutes. You can retry any operations after the update is complete. **To update a Managed kdb volume** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the list of environments, choose a kdb environment. 1. On the environment details page, choose the **Volumes** tab. 1. From the list of clusters, choose the one that you want to edit. The volume details page opens. 1. Choose **Edit** and update the required details. 1. Choose **Save changes**. # Deleting a Managed kdb volume **Note** This action is irreversible. You cannot delete a volume if it's attached to any cluster or dataview. **To delete a Managed kdb volume** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Volumes** tab. 1. From the list of volumes, choose the volume that you want to delete and choose **Delete**. Alternatively, you can choose the volume name and open the volume details page to delete it. 1. On the confirmation dialog box, enter **confirm** to provide a written consent to delete the resource permanently. 1. Choose **Delete**. # Managed kdb Insights clusters Managed kdb clusters A FinSpace Managed kdb Insights cluster is a set of compute resources that run kdb processes in a FinSpace Managed kdb environment. By using FinSpace Managed kdb clusters, you can easily set up your own private managed data processing and analytics hub for capital markets data. This provides access to real-time and historical data along with high-performance analytics. ** **Topics** ** + [ # Running a clusters on scaling groups vs as a dedicated cluster ](kdb-clusters-running-clusters-comparison.md) + [ # Cluster types ](kdb-cluster-types.md) + [ # Managing kdb clusters ](managing-kdb-clusters.md) + [ # Using Managed kdb Insights clusters ](using-kdb-clusters.md) # Running a clusters on scaling groups vs as a dedicated cluster Scaling groups cluster vs dedicated cluster The original Amazon FinSpace Managed kdb cluster launch configuration is now referred to as a dedicated cluster. In a dedicated cluster, each node or kdb process in the cluster runs on its own dedicated compute host. ![\[A diagram that shows dedicated cluster.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-cluster-coparison-image1.png) This configuration provides strong workload isolation between clusters and nodes in a single cluster at the expense of needing a fixed amount of compute per node. In contrast, with a cluster on scaling group a single set of compute is shared by multiple workloads (clusters) running on shared compute, allowing you to share a fixed amount of compute. ![\[A diagram that shows shared compute.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-cluster-coparison-image2.png) **Considerations** + Currently, a kdb scaling group is limited to only one host residing in one Availability Zone. + The [HDB clusters](kdb-cluster-types.md#kdb-clusters-hdb) running on kdb scaling groups must use dataviews instead of cluster-specific [disk cache](kdb-cluster-types.md#kdb-cluster-cache-config) to store database data for high-performance read access. + RDB and General Purpose clusters running on scaling groups must use a [kdb volume](finspace-managed-kdb-volumes.md) for their savedown storage configuration. # Cluster types Amazon FinSpace supports a variety of kdb clusters that you can use for different uses cases such as to implement a standard kdb tick architecture. ## General purpose You can use a *general purpose *cluster if your kdb application doesn't require any specific features that are available on more specialized clusters—like the multi-node, Multi-AZ read-only query of an HDB cluster or the multi-node, Multi-AZ gateways. With a general purpose cluster, you can mount a kdb Insights database for read-only access, as well as storage ([savedown storage](#kdb-cluster-savedown-storage)) for writing. This ability to read a database and write contents from a single cluster makes general purpose clusters suitable for various maintenance tasks. For example, you can use a general purpose cluster for tasks that require the ability to read and write data, and for creating derived datasets from an HDB cluster, in support of use cases such as one-time analysis by quantitative analysts (quants). **Features of a general purpose cluster** The following are the features of a general purpose cluster. + The node count for this cluster type is fixed at 1. + It only supports Single-AZ mode. + It can mount a kdb Insights database for read-only access to data. + You can configure savedown storage at the time of creating the cluster. You can use this space for writing savedown files before loading into a FinSpace database, or as a writeable space of other temporary files. For dedicated clusters, the savedown storage becomes unavailable when the cluster node is deleted. + For clusters running on a scaling group, the savedown storage location will use a shared volume. This volume exists even after you delete the cluster and can be used by other clusters. You can remove the data on the volume before deleting the cluster or it remains available for use by other clusters. + It can update databases and cache with the [UpdateKxClusterDatabase](https://docs.aws.amazon.com/finspace/latest/management-api/API_UpdateKxClusterDatabases.html) operation. ## Tickerplant A tickerplant (TP) acts as a message bus that subscribes to data or gets data pushed to it by Feed Handlers and then publishes it to one or more consumers, typically a realtime database (RDB). It persists a copy of each message received to a durable log of messages that are called the TP Log, so that downstream subscribers can request a replay of messages if needed. The following diagram explains that you can configure a TP cluster to save logs to a volume in Managed kdb Insights, from where you can replay the logs from an RDB type cluster. ![\[A diagram that shows how ticker plant works.\]](http://docs.aws.amazon.com/finspace/latest/userguide/images/11-managed-kx/finspace-tp-diagram.png) **Features of a tickerplant cluster** Following are the features of a tickerplant type cluster:
 + It supports only single-node that is only one kdb process. + It shares storage with RDB clusters. + It does not support the Multi-AZ mode. If you need Multi-AZ redundancy, run two TP type clusters in parallel. ## Gateway In the vast majority of kdb\$1 systems, data is stored across several processes, which results in the need to access data across these processes. You do this by using a gateway to act as a single interface point that separates the end user from the configuration of underlying databases or services. With a gateway, you don't need to know where data is stored, and you don't need to make multiple requests to retrieve it. To support running your custom gateway logic, Managed kdb Insights provides a *gateway* cluster type. You can deploy your own routing logic using the initialization scripts and custom code. You can configure gateways to a multi-node, Multi-AZ deployment for resiliency. **Features of a gateway cluster** The following are the features of a gateway type cluster:
 + It provides support to run gateways with your custom allocation hosted inside of a Managed kdb environment. + It provides support for hosting code with custom allocation logic for allocating load across different kdb clusters or nodes. + It integrates with the discovery service to understand available clusters, monitor their health status, and provide an endpoint for the cluster. + It provides a network path from your custom code running on the gateway to the cluster supporting IPC connections. ## Real-time database (RDB) You can use a *real-time database* cluster to capture all the data from another kdb process, such as a ticker plant, and store it in memory for query or real-time processing. Because the data volume can eventually exceed the amount of available memory, kdb customers typically move the data from the RDB to a historical database (HDB) using a process called *savedown*. This process typically occurs at the end of a business day. You can create, list, and delete RDB clusters with single or multiple nodes through both console and FinSpace API operations. ### Savedown storage RDB clusters require local space for temporary storage of data during the savedown process. This temporary storage is used to hold data for the period between when a cluster has flushed it from memory and when it is successfully loaded into a kdb Insights database. To support this, RDB clusters have writeable disk that is used as storage space for savedown data. You can use the data saved down to the FinSpace database from and RDB by creating an HDB cluster that points to the database. **Considerations** The following are some considerations related to savedown storage: + You can configure savedown storage at the time of creating the cluster. You can use this space to write savedown files before loading into a FinSpace database, or as a writeable space for other temporary files. + For dedicated clusters, the savedown storage becomes unavailable when you delete a cluster node. + For clusters running on a scaling group, the savedown storage location will use a shared volume. This volume exists even after you delete the cluster and can be used by other clusters. You can remove the data on the volume before deleting the cluster or it remains available for use by other clusters. ## Historical database (HDB) A historical database holds data from a day before the current day. Each day, new records are added to the HDB at the end of day. To access data in Managed kdb databases from an HDB cluster, you must attach the databases you want to access as an option when launching the cluster. You can do this at the time of creating a cluster through the console, or by using the [create cluster API operation](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) in the *Amazon FinSpace Management API Reference*. The HDB cluster can access this data in a read-only mode. ### Cache configuration When you attach a database to a cluster for access, by default, the read operations are performed directly against the object store that the database data is stored in. Alternatively, you can also define a file cache, in which you can load data for faster performance. You do this by specifying cache configuration when you associate the database with the cluster. You can specify a certain amount of cache, and then separately specify the contents of the database that you want to cache. FinSpace supports the following cache types: + **CACHE\$11000** – This type allows a throughput of 1000 MB/s per unit storage (TiB). + **CACHE\$1250** – This type allows a throughput of 250 MB/s per unit storage (TiB). + **CACHE\$112** – This type allows a throughput of 12 MB/s per unit storage (TiB). **Considerations** The following are some considerations related to storage and billing: + Caching is only available on dedicated clusters. For clusters running on a scaling group, use [dataviews](finspace-managed-kdb-dataviews.md). + You can only configure initial cache size at the time of cluster creation. To run a cluster with a different sized cache, you need to terminate the cluster and launch a new one with smaller database cache size. + Billing for cache storage starts when storage is available for use by the cluster and stops when the cluster is terminated. ### Auto scaling With the HDB auto scaling feature, you can take away some nodes to save costs when the usage is low, and add more nodes to improve availability and performance when the usage is high. For auto scaling HDB clusters, you specify the CPU utilization targets for your scaling policy. You can auto scale an HDB cluster at the time of [cluster creation](create-kdb-clusters.md) in two ways. You can use the console or use the [createKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster.html) API operation, where you provide minimum and maximum node count, the metric policy, and a target utilization percentage. As a result, FinSpace scales in or scales out the clusters based on service utilization that's determined by CPU consumed by the kdb\$1 node. **Note** Auto scaling is only available for dedicated clusters and is not supported for clusters running on scaling groups. ## Summary of capabilities by cluster type | Capability | General purpose | Gateway | RDB | TP | HDB | | --- | --- | --- | --- | --- | --- | | Attaches a Managed kdb Insights database for read-only access | Yes | No | No | No | Yes | | Attaches writable local (savedown) storage to a node | Yes | No | Yes | No | No | | Number of nodes supported | Single | Multi | Multi | Single | Multi | | Supports AZ configurations (for dedicated clusters) | Single | Single or Multi | Single or Multi | Single | Single or Multi | # Managing kdb clusters The following sections provide a detailed overview of the operations that you can perform by using Managed kdb clusters. ** **Topics** ** + [ # Activating your Managed kdb Insights license ](kdb-licensing.md) + [ # Managed kdb Insights cluster software bundles ](kdb-software-bundles.md) + [ # Maintaining a Managed kdb Insights cluster ](maintaining-kdb-clusters.md) + [ # Creating a Managed kdb Insights cluster ](create-kdb-clusters.md) + [ # Viewing kdb cluster detail ](view-kdb-clusters.md) + [ # Updating code configurations on a running cluster ](update-cluster-code.md) + [ # Updating a kdb cluster database ](update-kdb-clusters-databases.md) + [ # Deleting a kdb cluster ](delete-kdb-clusters.md) # Activating your Managed kdb Insights license Activating your kdb license To run Managed kdb Insights clusters, you must first have an existing kdb Insights license from KX. That kdb Insights license needs to be activated for your Managed kdb Insights environment(s). You're responsible for working directly with KX (KX Systems, Inc., a subsidiary of FD Technologies plc) to obtain this. To activate an existing kdb Insights license for your Managed kdb Insights environment(s), do the following: + Contact your KX account manager or KX sales representative and provide them with the AWS account number for all the accounts where you want to use your Managed kdb Insights environment(s). + Once arranged with KX, the kdb Insights license will be automatically applied to your Managed kdb Insights environment(s). **Note** If you do not have an existing kdb Insights license, you can request a 30-day trial license from KX [here](https://kx.com/amazon-finspace-with-managed-kdb-insights/). KX will then activate a 30-day trial license for you. + You will receive an activation email and your 30 day trial license will be automatically applied to your Managed kdb Insights environment. **Note** If KX has already enabled a kdb license for use with Managed kdb Insights in your AWS account and the license has not expired, you can start using clusters in your environment as soon as it is created. You do not need to request a new license. # Managed kdb Insights cluster software bundles Cluster software bundles When you launch a cluster, you can choose the software versions that will run on your cluster. This allows you to test and use application versions that fit your compatibility requirements. You can specify the release version using the `Release Label`. Release labels are in the form `x.x`. The following table lists the software versions that each release label includes. Currently it only includes the kdb Insights core. | Managed kdb Insights release label | Kdb Insights core version | | --- | --- | | 1.0 | 4.0.3 | # Maintaining a Managed kdb Insights cluster Maintaining a kdb cluster involves updates to the cluster's underlying operating system or to the container hosting the Managed kdb Insights software. FinSpace manages and applies all such updates. Some maintenance may require FinSpace to take your Managed kdb cluster offline for a short time. This includes, installing or upgrading required operating system or database patches. This maintenance is automatically scheduled for patches that are related to security and instance reliability. The maintenance window determines when pending operations start, but it doesn't limit the total execution time of these operations. Maintenance operations that don't finish before the maintenance window ends can continue beyond the specified end time. ## Managed kdb Insights maintenance window Every Manged kdb environment has a weekly maintenance window during which system changes are applied. You can control when modifications and software patches occurs during a maintenance window. If a maintenance event is scheduled for a given week, it is initiated during the maintenance window. | AWS Region name | Time block | | --- | --- | | Canada (Central) | 15:00–16:30 UTC | | US West (N. California) | 18:00–19:30 UTC | | US West (Oregon) | 18:00–19:30 UTC | | US East (N. Virginia) | 15:00–16:30 UTC | | US East (Ohio) | 15:00–16:30 UTC | | Europe (Ireland) | 10:00–11:30 UTC | | Europe (London) | 09:00–10:30 UTC | | Europe (Frankfurt) | 08:00–09:30 UTC | | Asia Pacific (Singapore) | 02:00–03:30 UTC | | Asia Pacific (Sydney) | 23:00–12:30 UTC | | Asia Pacific (Tokyo) | 01:00–02:30 UTC | # Creating a Managed kdb Insights cluster Creating a cluster You can either use the console or the [CreateKxCluster](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxCluster) API to create a cluster. When you create a cluster from the console, you choose one of the following cluster types available in FinSpace – [General purpose](kdb-cluster-types.md#kdb-clusters-gp), [Tickerplant](kdb-cluster-types.md#kdb-clusters-tp), [HDB](kdb-cluster-types.md#kdb-clusters-hdb), [RDB](kdb-cluster-types.md#kdb-clusters-rdb), and [Gateway](kdb-cluster-types.md#kdb-clusters-gw). The create cluster workflow includes a step-wise wizard, where you will add various details based on the cluster type you choose. The fields on each page can differ based on various selections throughout the cluster creation process. ## Prerequisites Before you proceed, complete the following prerequisites: + If you want to run clusters on a scaling group, [create a scaling group](create-scaling-groups.md). + If you want to run a TP, GP, or RDB cluster on e scaling group [create a volume](create-volumes.md). + If you want to run an HDB type cluster on a scaling group, [create a dataview](managing-kdb-dataviews.md#create-kdb-dataview). ** **Topics** ** + [ ## Prerequisites ](#create-cluster-prereq) + [ # Opening the cluster wizard ](create-cluster-tab.md) + [ # Step 1: Add cluster details ](create-cluster-step1.md) + [ # Step 2: Add code ](create-cluster-step2.md) + [ # Step 3: Configure VPC settings ](create-cluster-step3.md) + [ # Step 4: Configure data and storage ](create-cluster-step4.md) + [ # Step 5: Review and create ](create-cluster-step5.md) # Opening the cluster wizard **To open the create cluster wizard** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. In the kdb environments table, choose the name of the environment. 1. On the environment details page, choose **Clusters** tab. 1. Choose **Create cluster**. A step-wise wizard to create a cluster opens. # Step 1: Add cluster details Specify details for each of the following sections on **Add cluster details** page. ## Cluster details 1. Choose from one of the following types of clusters that you want to add. + **(HDB) Historical Database** + **(RDB) Realtime Database** + **Gateway** + **General purpose** + **Tickerplant** For more information about cluster types, see [Managed kdb Insights clusters](finspace-managed-kdb-clusters.md). **Note** Currently, you can only create dedicated HDB clusters and all cluster types that are running on a scaling group directly from the console. To create other types of clusters, you need to first create a [Support case](https://aws.amazon.com/contact-us/), and then proceed with steps in this tutorial. The parameters that are displayed on the **Step 5: Configure data and storage** page will change based on the cluster type and running mode that you select in this step. 1. Add a unique name and a brief description for your cluster. 1. For `Release label`, choose the package version to run in the cluster. 1. (Optional) Choose the IAM role that defines a set of permissions associated with this cluster. This is an execution role that will be associated with the cluster. You can use this role to control access to other clusters in your Managed kdb environment. ## Cluster running mode 1. Choose if you want to add this cluster as a dedicated cluster or as a part of scaling groups. + **Run on kdb scaling group** – Allows you to share a single set of compute with multiple clusters. + **Run as a dedicated cluster** – Allows you to run each process on its own compute hose. 1. If you choose **Run as a dedicated cluster**, you also need to provide the Availability Zones where you want to create a cluster. 1. Choose **AZ mode** to specify the number of Availability Zones where you want to create a cluster. You can choose from one of the following options: + **Single** – Allows you to create a cluster in one Availability Zone that you select. If you choose this option, you must specify only one Availability Zone value and only one subnet in the next step. The subnet must reside in one of the three AZs that your kdb environment uses, and the Availability Zone must align with one of the three AZs. + **Multiple** – Allows you to create a cluster with nodes automatically allocated across all the Availability Zones that are used by your Managed kdb environment. This option provides resiliency for node or cache failures in a Single-AZ. If you choose this option, you must specify three subnets, one in each of the three AZs that your kdb environment uses. **Note** For the **General purpose** and **Tickerplant** type cluster, you can only choose Single-AZ. 1. Choose the Availability Zone IDs that include the subnets you want to add. ## Scaling group details **Note** This section is only available when you choose to add cluster as a part of scaling groups. Choose the name of the scaling group where you want to create this cluster. The drop down shows the metadata for each scaling group along with their names to help you decide which one to pick. If a scaling group is not available, choose **Create kdb scaling group** to add a new one. For more information, see [Creating a Managed kdb scaling group](create-scaling-groups.md). ## Node details In this section, you can choose the capacity configuration for your clusters. The fields in this section vary for dedicated and scaling group clusters. ------ #### [ Scaling group cluster ] You can decide the memory and CPU usage that will be shared with the instances for your scaling group clusters by providing the following information. 1. Under **Node details**, for **Node count**, enter the number of instances in a cluster. **Note** For a **General purpose** and **Tickerplant** type cluster, the node count is fixed at 1. 1. Enter the memory reservation and limits per node. Specifying the memory limit is optional. The memory limit should be equal to or greater than the memory reservation. 1. (Optional) Enter the number of vCPUs that you want to reserve for each node of this scaling group cluster. ------ #### [ Dedicated cluster ] For a dedicated cluster you can provide an initial node count and choose the capacity configuration from a pre-defined list of node types. For example, the node type `kx.s.large` allows you to use two vCPUs and 12 GiB of memory for your instance. 1. Under **Node details**, for **Initial node count**, enter the number of instances in a cluster. **Note** For a **General purpose** and **Tickerplant** type cluster, the node count is fixed at 1. 1. For **Node type**, choose the memory and storage capabilities for your cluster instance. You can choose from one of the following options: + `kx.s.large` – The node type with a configuration of 12 GiB memory and 2 vCPUs. + `kx.s.xlarge` – The node type with a configuration of 27 GiB memory and 4 vCPUs. + `kx.s.2xlarge` – The node type with a configuration of 54 GiB memory and 8 vCPUs. + `kx.s.4xlarge` – The node type with a configuration of 108 GiB memory and 16 vCPUs. + `kx.s.8xlarge` – The node type with a configuration of 216 GiB memory and 32 vCPUs. + `kx.s.16xlarge` – The node type with a configuration of 432 GiB memory and 64 vCPUs. + `kx.s.32xlarge` – The node type with a configuration of 864 GiB memory and 128 vCPUs. ------ ## Auto-scaling **Note** This section is only available when you add an HDB cluster type as a dedicated cluster. Specify details to scale in or scale out the based on service utilization. For more information, see [Auto scaling](kdb-cluster-types.md#kdb-cluster-hdb-autoscaling). 1. Enter a minimum node count. Valid numbers: 1–5. 1. Enter a maximum node count. Valid numbers: 1–5. 1. Choose the metrics to auto scale your cluster. Currently, FinSpace only supports CPU utilization. 1. Enter the cooldown time before initiating another scaling event. ## Tags 1. (Optional) Add a new tag to assign to your kdb cluster. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your cluster. 1. Choose **Next** for next step of the wizard. # Step 2: Add code You can load q code onto your kdb cluster so that you can run it when the cluster is running. Additionally, you can configure your cluster to automatically run a particular q command script on cluster startup. By default, q writes files uncompressed. You can pass command line arguments to set compression defaults `.z.d` at the time of creating a cluster from the console or through CLI, which can be updated later. **Note** This step is required for the **Gateway** and **Tickerplant** cluster type. On the **Add code** page, add the following details of your custom code that you want to use when analyzing the data in the cluster. 1. (Optional) Specify the **S3 URI** and the **Object version**. You can choose the *.zip* file that contains code that should be available on the cluster. 1. (Optional) For **Initialization script**, enter the relative path that contains a q program script that will run at the launch of a cluster. If you choose to load the database by using the initialization script, it will autoload on startup. If you add a changeset that has a missing sym file, the cluster creation fails. **Note** This step is optional. If you choose to enter the initialization script, you must also provide the S3 URI. 1. (Optional) Enter key-value pairs as command-line arguments to configure the behavior of clusters. You can use the command-line arguments to set [zip defaults](https://code.kx.com/q/ref/dotz/#zzd-zip-defaults) for your clusters. For this, pass the following key-value pair: + **Key**: `AWS_ZIP_DEFAULT` + **Value**: `17,2,6` The value consists of comma separated three numbers that represent logical block size, algorithm, and compression level respectively. For more information, see [compression parameters](https://code.kx.com/q/kb/file-compression/#compression-parameters). You can also add the key-value pair when you [update code configuration on a cluster](update-cluster-code.md). **Note** You can only add up to 50 key-value pairs. To set compression default using AWS CLI, use the following command: ``` aws finspace create-kx-cluster \ ... --command-line-arguments '[{"key": "AWS_ZIP_DEFAULT", "value":"17,2,6"}]' \ ... ``` 1. Choose **Next**. **Note** In case of failure, to stop cluster creation from an initialization script, use the `.aws.stop_current_kx_cluster_creation` function in the script. # Step 3: Configure VPC settings You connect to your cluster using q IPC through an AWS PrivateLink VPC endpoint. The endpoint resides in a subnet that you specify in the AWS account where you created your Managed kdb environment. Each cluster that you create has its own AWS PrivateLink endpoint, with an elastic network interface that resides in the subnet you specify. You can specify a security group to be applied to the VPC endpoint. Connect a cluster to a VPC in your account. On the **Configure VPC settings** page, do the following: 1. Choose the VPC that you want to access. 1. Choose the VPC subnets that the cluster will use to set up your VPC configuration. 1. Choose the security group. 1. Choose **Next**. # Step 4: Configure data and storage Choose data and storage configurations that will be used for the cluster. The parameters on this page are displayed according to the cluster type that you selected in *Step 1: Add cluster details*. **Note** If you choose to add both the **Read data configuration** and **Savedown storage configuration**, the database name must be the same for both the configurations. ## For HDB cluster **Note** When you create a cluster with a database that has a changeset, it will autoload the database when you launch a cluster. If you choose **Cluster type** as *HDB*, you can specify the database and cache configurations as following: ------ #### [ Scaling group cluster ] 1. Choose the name of the database. 1. Choose a dataview for the database you selected. **Note** If a dataview is not available in the list, either choose **Create dataview** to create a new one for the database you selected or try changing the availability zone. 1. Choose **Next**. The **Review and create** page opens. ------ #### [ Dedicated clusters ] 1. Choose the name of the database. This database must have a changeset added to it. 1. Choose the changeset that you want to use. By default, this field displays the most recent changeset. 1. Choose whether you want to cache your data from your database to this cluster. If you choose to enable caching, provide the following information: 1. Choose the cache type, which is a type of read-only storage for storing a subset of your database content for faster read performance. You can choose from one of the following options: + **CACHE\$11000** – Provides a throughput of 1000 MB/s per unit storage (TiB). + **CACHE\$1250** – Provides a throughput of 250 MB/s per unit storage (TiB). + **CACHE\$112** – Provides a throughput of 12 MB/s per unit storage (TiB). 1. Choose the size of the cache. For cache type **CACHE\$11000** and **CACHE\$1250** you can select cache size as 1200 GB or increments of 2400 GB. For cache type **CACHE\$112** you can select the cache size in increments of 6000 GB. 1. Choose **Next**. The **Review and create** page opens. ------ ## For RDB cluster If you choose **Cluster type** as *RDB*, you can specify the savedown storage configurations for your cluster as following: ------ #### [ Scaling group cluster ] 1. **Savedown database configuration** Choose the name of the database where you want to save your data. 1. **(Optional) Savedown storage configuration** Choose the name of the storage volume for your savedown files that you created in advance. If a volume name is not available, choose **Create volume** to create it. 1. **(Optional) Tickerplant log configuration** Choose a **Volume name** to use the tickerplant logs from. 1. Choose **Next**. The **Review and create** page opens. ------ #### [ Dedicated clusters ] 1. **Savedown database configuration** Choose the name of the database where you want to save your data. 1. **(Optional) Savedown storage configuration** 1. Choose the writeable storage space type for temporarily storing your savedown data. Currently, only the **SDS01** storage type is available. This type represents 3000 IOPS and the Amazon EBS volume type `io2`. 1. Enter the size of the savedown storage that will be available to the cluster in GiB. 1. **Tickerplant log configuration** Choose one or more volume names to use the tickerplant logs from. 1. Choose **Next**. The **Review and create** page opens. ------ ## For Gateway cluster If you choose **Cluster type** as **Gateway**, you do not need to attach databases, cache configurations, or local storage in this step. ## For General purpose cluster If you choose **Cluster type** as *General purpose*, you can specify the database and cache configurations and savedown storage configurations as following: ------ #### [ Scaling group cluster ] 1. **(Optional) Read data configuration** 1. Choose the name of the database. 1. Choose a dataview for the database you selected. **Note** If a dataview is not available in the list, either choose **Create dataview** to create a new one for the database you selected or try changing the availability zone. 1. **(Optional) Savedown database configuration** Choose the name of the database where you want to save your data. 1. **(Optional) Savedown storage configuration** Choose the name of the storage volume for your savedown files that you created in advance. If a volume name is not available, choose **Create volume** to create it. 1. **(Optional) Tickerplant log configuration** Choose a **Volume name** to use the tickerplant logs from. 1. Choose **Next**. The **Review and create** page opens. ------ #### [ Dedicated clusters ] 1. **(Optional) Read data configuration** 1. Choose the name of the database. This database must have a changeset added to it. 1. Choose the changeset that you want to use. By default, this field displays the most recent changeset. 1. Choose whether you want to cache your data from your database to this cluster. If you choose to enable caching, provide the following information: 1. Specify paths within the database directory where you want to cache data. 1. Choose the cache type, which is a type of read-only storage for storing a subset of your database content for faster read performance. You can choose from one of the following options: + **CACHE\$11000** – Provides a throughput of 1000 MB/s per unit storage (TiB). + **CACHE\$1250** – Provides a throughput of 250 MB/s per unit storage (TiB). + **CACHE\$112** – Provides a throughput of 12 MB/s per unit storage (TiB). 1. Choose the size of the cache. For cache type **CACHE\$11000** and **CACHE\$1250** you can select cache size as 1200 GB or increments of 2400 GB. For cache type **CACHE\$112** you can select the cache size in increments of 6000 GB. 1. **(Optional) Savedown database configuration** Choose the name of the database where you want to save your data. 1. **(Optional) Savedown storage configuration** 1. Choose the writeable storage space type for temporarily storing your savedown data. Currently, only the **SDS01** storage type is available. This type represents 3000 IOPS and the Amazon EBS volume type `io2`. 1. Enter the size of the savedown storage that will be available to the cluster in GiB. 1. **Tickerplant log configuration** Choose one or more volume names to use the tickerplant logs from. 1. Choose **Next**. The **Review and create** page opens. ------ ## For Tickerplant cluster For both scaling groups clusters and dedicated clusters, you can choose a volume where you want to store the tickerplant data. 1. **Tickerplant log configuration** Choose a **Volume name** to store the tickerplant logs. 1. Choose **Next**. The **Review and create** page opens. # Step 5: Review and create 1. On the **Review and create** page, review the details that you provided. You can modify details for any step when you choose **Edit** on this page. 1. Choose **Create cluster**. The cluster details page opens where you can view the status of cluster creation. # Viewing kdb cluster detail Viewing a cluster **To view and get details of a kdb cluster** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Clusters** tab. The table under this tab displays a list of clusters. 1. Choose a cluster name to view its details. The cluster details page opens where you can view the cluster details and the following tabs. + **Configuration** tab – Displays the cluster configuration details like the node details, code, availability zones, savedown database configuration etc. + **Monitoring** tab – Displays the dashboard of cluster metrics. + **Nodes** tab – Displays a list of nodes in this cluster along with their status. All the nodes that are active will have a **Running** status and nodes that are being prepared or stuck due to lack of resources have the status as **Provisioning**. From here you could also delete a node. For this, select a node and choose **Delete**. + **Logs** section – Displays the activity logs for your clusters. + **Tags** tab – Displays a list of key-value pairs associated with the clusters. If you did not provide tags during cluster creation, choose **Manage tags** to add new tags. # Updating code configurations on a running cluster Updating code configurations Amazon FinSpace allows you to update code configurations on a running cluster. You can either use the console or the [UpdateKxClusterCodeConfiguration](https://docs.aws.amazon.com/finspace/latest/management-api/API_UpdateKxClusterCodeConfiguration) API to update the code. Both console and API allow you to choose how you want to update the code on a cluster by using different deployment modes. Based on the option you choose, you can reduce the time it takes to update the code on to a cluster. You can also add or delete default compression parameters for your files by using command-line arguments. **Note** The configuration that you update will override any existing configurations on the cluster. **To update code configurations on a cluster by using the console** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the list of environments, choose a kdb environment. 1. On the environment details page, choose the **Clusters** tab. 1. From the list of clusters, choose the one where you want to update the code. The cluster details page opens. 1. On the cluster details page, choose the **Details** tab. 1. Under **Code** section, choose **Edit**. **Note** This button is only available for an **Active** environment and when the cluster is in a **Running** state. 1. On the **Edit code configuration** page, choose how you want to update a cluster by choosing a deployment mode. The following options are available. + **Rolling** – (Default) Loads the code by stopping the exiting q process and starting a new q process with updated configuration. + **Quick** – Loads the code by stopping all the running nodes immediately. 1. Specify the **S3 URI** and the **Object version**. This allows you to choose the *.zip* file containing code that should be available on the cluster. 1. For **Initialization script**, enter the relative path that contains a q program script that will run at the launch of a cluster. 1. (Optional) Add or update the key-value pairs as command line arguments to configure the behavior of clusters. You can use the command-line arguments to set [zip defaults](https://code.kx.com/q/ref/dotz/#zzd-zip-defaults) for your cluster. The cluster has to be restarted for the changes to take effect. For this, pass the following key-value pair: + **Key**: `AWS_ZIP_DEFAULT` + **Value**: `17,2,6` The value consists of comma separated three numbers that represent logical block size, algorithm, and compression level respectively. For more information, see [compression parameters](https://code.kx.com/q/kb/file-compression/#compression-parameters). To update the compression default using AWS CLI, use the following command: ``` aws finspace update-kx-cluster-code-configuration \ ... --command-line-arguments '[{"key": "AWS_ZIP_DEFAULT", "value":"17,3,0"}]' \ --deploymentConfiguration deploymentStrategy=ROLLING|FORCE ... ``` 1. Choose **Save changes**. The cluster details page opens and the updated code configuration is displayed once the cluster updates successfully. # Updating a kdb cluster database Updating a cluster database You can update the databases mounted on a kdb cluster using the console. This feature is only available for HDB clusters types. With this feature, you can update the data in a cluster by selecting a changeset. You can also update the cache by providing database paths. You can't change a database name or add a new database if you created a cluster without one. You can also choose how you want to update the databases on the cluster by selecting a deployment mode. **To update a kdb cluster database** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the list of environments, choose a kdb environment. 1. On the environment details page, choose the **Clusters** tab. 1. From the list of clusters, choose the one where you want to update the database. The cluster details page opens. 1. On the cluster details page, choose the **Details** tab. 1. Under **Data management and storage** section, choose **Edit**. **Note** This button is not available for *RDB* and *Gateway* type clusters. 1. On the edit page, modify the changeset that you want to cache as needed. 1. Choose a deployment mode from one of the following options. + **Rolling** – (Default) To update the database, this option stops the existing q process and starts a new q process with the updated database configuration. The initialization script re-runs when the new q process starts. + **No restart** – This option updates the database but doesn't stop the existing q process. **No restart** is often quicker than the other deployment modes because it reduces the turnaround time to update the changeset configuration for a kdb database on your cluster. This option doesn't re-run the initialization script. **Note** After the update completes, you must re-load the updated database. However, if you use a historical database (HDB) cluster with a single database in a rolling deployment, FinSpace autoloads the database after an update. 1. Choose **Save changes**. The cluster details page opens and the updated information is displayed once the cluster updates successfully. # Deleting a kdb cluster Deleting a cluster **Note** This action is irreversible. Deleting a kdb cluster will delete all of its data from the local storage. **To delete a kdb cluster** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Clusters** tab. 1. From the list of clusters, choose the one that you want to delete. The cluster details page opens. 1. On the cluster details page, choose **Delete**. 1. On the confirmation dialog box, enter *confirm*. 1. Choose **Delete**. ## Deleting a cluster on scaling groups # Using Managed kdb Insights clusters Using Managed kdb clusters After you successfully create clusters in your kdb environment, you can use the clusters to do the following: + **Monitor cluster metrics** – You can view the available cluster metrics in CloudWatch for your clusters. Using the **Monitoring** tab you can adjust the date and time range and refresh frequency as you need. You can also add the graphs to CloudWatch dashboards from this tab. For more information, see the [Monitoring Managed kdb cluster metrics](kdb-cluster-logging-monitoring.md#kdb-cluster-monitoring-metrics) section. + **View logs** – You can view KDB application logs from Managed kdb Insights clusters using the **Logs** tab. You can view data directly in CloudWatch using CloudWatch reporting or CloudWatch Insights. For more information, see the [Logging](kdb-cluster-logging-monitoring.md#kdb-cluster-logging) section. + **Connect to clusters** – FinSpace provides you the ability to discover clusters in your dedicated account and connect to them. You can do this by using discovery API operations and q API operations. For more information on how to connect to a cluster, see the [Connecting to a cluster endpoint or node in a cluster](interacting-with-kdb-clusters.md#connect-kdb-clusters) section. + **Load code on to a cluster** – You can run your own KDB code on the cluster and perform analytics or query data in a database. For this, FinSpace provides a set of q API operations that you can use to perform required functions. For more information, see the [Running code on a Managed kdb Insights cluster](interacting-with-kdb-loading-code.md) section. # Managing kdb users The following sections provide a detailed overview of the operations that you can perform by using Managed kdb Insights users. A kdb user is required in order to establish a connection to a Managed kdb cluster. For more information, see [Interacting with a kdb cluster](interacting-with-kdb-clusters.md). ## Creating a kdb user **To create a user** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Users** tab. 1. Choose **Add user**. 1. On the **Add user** page, a unique name for the user. 1. Choose an IAM role available in your account to associate it with the user. This role will be used later when you connect to a cluster. **Note** The IAM role that you choose must have connect cluster permissions. 1. (Optional) Add a new tag to assign it to your kdb user. For more information, see [AWS tags](https://docs.aws.amazon.com/finspace/latest/userguide/create-an-amazon-finspace-environment.html#aws-tags). **Note** You can only add up to 50 tags to your user. 1. Choose **Add user**. The environment details page opens and the table under **Users** lists the newly added user. ## Updating a kdb user **Note** You can only modify the IAM role associated with a user. **To update a kdb user** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Users** tab. 1. From the list of users, choose the one that you want to update. 1. Choose **Edit**. 1. Choose a new IAM role to associate with this user. 1. Choose **Update user**. ## Deleting a kdb user **Note** This action is irreversible. **To delete a kdb user** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Users** tab. 1. From the list of users, choose the one that you want to delete. 1. Choose **Delete**. 1. On the confirmation dialog box, enter *confirm*. 1. Choose **Delete**. # Interacting with a kdb cluster To run commands on a Managed kdb Insights cluster, you must establish a q connection to a cluster endpoint or individual node in a cluster. If you don’t care which node in the cluster your connection is established with, use the cluster endpoint. The endpoint is an IP address (elastic network interface) that resides in your account. This provides a simple way to connect for a single-node cluster and for other scenarios. Alternately, from client code residing on a cluster node running with Managed kdb, you can also make a direct connection to an individual node. This gives you full control of which node in a cluster to use. This might be useful if you have custom allocation logic in your client code. You can use the Managed kdb list clusters and list node functionality to see what cluster and node resources are available in your environment. Then, you can use the cluster connection functionality to obtain a connection string that you can use to establish a q IPC connection to a cluster or node. As a part of cluster discovery, FinSpace provides you the following capabilities: + Listing all clusters running in your Managed kdb environment. + Listing all nodes running in a kdb cluster. For more information on this, see [Listing clusters and cluster nodes](#finding-kdb-clusters). + Connect to the underlying node from an existing cluster. For more information on this, see [Connecting to a cluster endpoint or node in a cluster](#connect-kdb-clusters). ## Listing clusters and cluster nodes There are three ways to view a list of clusters and nodes running in a cluster: + **FinSpace API operations** – You can call the `ListKxClusterNodes` API operation to get a list of nodes in a cluster. For more information, see the [ListKxClusterNodes](https://docs.aws.amazon.com/finspace/latest/management-api/API_ListKxClusterNodes.html) in the *Management API Reference Guide*. + **q API operations** – You can use the `.aws.list_kx_cluster_nodes()` and `.aws.list_kx_clusters()` API operations to get a list of nodes or clusters. For more information, see [Discovery APIs](interacting-with-kdb-q-apis.md#q-apis-discovery). + **Console** – You can view a list of nodes from the cluster details page in the FinSpace console. **To view a list of clusters by using the console** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Clusters** tab. 1. Choose a cluster name to view its details. On the cluster details page, you can see details about a cluster. **To view a list of nodes available in a cluster by using the console** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. Choose **Kdb environments**. 1. From the list of environments, choose a kdb environment. 1. On the environment details page, choose the **Clusters** tab. 1. From the list of clusters, choose the one where you want to view nodes. 1. On the cluster details page, choose **Nodes** tab. All the nodes running in the cluster are displayed along with the information about the node ID , the Availability Zone ID where the node is running, and the time when the node was started. You can use the `nodeId` to call the `GetKxConnectionString` API operation, which returns a signed connection string. ## Connecting to a cluster endpoint or node in a cluster Amazon FinSpace uses the model based on AWS Identity and Access Management that allows users to control access to clusters and their associated kdb databases using IAM roles and policies. Administrators create users in the FinSpace kdb environment using the existing `CreateKxUser` API operation, and associate these users with an IAM principal. Only users that will be connecting to a kdb cluster need to be registered as a FinSpace user. Next, using their IAM credentials, connecting users will request a SigV4 signed authentication token to connect to the cluster. Additionally, each cluster can be associated with an IAM execution role in the customer account when a cluster is created. This role will be used when a cluster connects to other clusters, or makes requests to other AWS resources in the customer’s account. **To connect to a cluster endpoint or cluster node** 1. **Create IAM role for a new user.** 1. Sign in to AWS Management Console, and open IAM Identity Center. 1. Create an IAM role. 1. Assign the following policy to the IAM role that you created. In the following example, replace each **user input placeholder** with your own values. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "finspace:ConnectKxCluster", "Resource": "arn:aws:finspace:us-east-1:111122223333:kxEnvironment/sdb3moagybykax4oexvsq4/kxCluster/testhdb-cluster" }, { "Effect": "Allow", "Action": "finspace:GetKxConnectionString", "Resource": "arn:aws:finspace:us-east-1:111122223333:kxEnvironment/sdb3moagybykax4oexvsq4/kxCluster/testhdb-cluster" } ] } ``` ------ 1. Associate the IAM role to the following trust policy that allows FinSpace to assume the role, as well as the account itself. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "finspace.amazonaws.com", "AWS": "arn:aws:iam::111122223333:root" }, "Action": "sts:AssumeRole" } ] } ``` ------ 1. **Create a kdb user with the environment id, username, and the IAM role that you created in the previous step.** ``` aws finspace create-kx-user --environment-id "sdb3moagybykax4oexvsq4" --user-name alice --iam-role arn:aws:iam::111122223333:role/user-alice [--tags {tags}] ``` 1. **Federate the user that you created into its user role.** 1. To get a kdb connection string for a user, you must first federated into the role associated with the user. How you assume this role depends on what federation tool you use. If you use AWS Security Token Service, you could run the following command and use the credentials of the customer account. ``` export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \ $(aws sts assume-role \ --role-arn arn:aws:iam::111122223333:role/user-alice \ --role-session-name "alice-connect-to-testhdb" \ --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \ --output text)) ``` 1. Verify that the role has been assumed. ``` aws sts get-caller-identity | cat ``` 1. **Get connection string for the user.** Get signed connection strings for connecting to kdb clusters or nodes. These connection strings are valid only for 60 minutes. To connect to a cluster endpoint, use `get-kx-connetion-string` to obtain a connection string. ``` aws finspace get-kx-connection-string --environment-id "sdb3moax4oexvsq4" --user-arn arn:aws:finspace:us-east-1:111122223333:kxEnvironment/sdb3moax4oexvsq4/kxUser/alice --cluster-name "testhdb-cluster" --region us-east-1 ``` Example of the signed connection string that you get. `:tcps://vpce-06259327736e61c9d-uczv1va3.vpce-svc-0938de45abc1ce4d8.us-east-1.vpce.amazonaws.com:443:testuser:Host=vpce-06259327736e61c9d-uczv1va3.vpce-svc-0938de45abc1ce4d8.us-east-1.vpce.amazonaws.com&Port=5000&User=testuser&Action=finspace%3AConnectKxCluster&X-Amz-Security-Token=IQoJb3JpZ2luX2Vj&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230524T150227Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIAR2V4%2Fus-east-1%2Ffinspace-apricot%2Faws4_request&X-Amz-Signature=28854cc2f97f8f77009928fcdf15480dd10b43c61dda22b0af5f0985d38e7114` 1. Connect to a cluster using the signed connection string. ``` hopen :tcps://vpce-06259327736e61c9d-uczv1va3.vpce-svc-0938de45abc1ce4d8.us-east-1.vpce.amazonaws.com:443:testuser:Host=vpce-06259327736e61c9d-uczv1va3.vpce-svc-0938de45abc1ce4d8.us-east-1.vpce.amazonaws.com&Port=5000&User=testuser&Action=finspace%3AConnectKxCluster&X-Amz-Security-Token=IQoJb3JpZ2luX2Vj&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230524T150227Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIAR2V4%2Fus-east-1%2Ffinspace-apricot%2Faws4_request&X-Amz-Signature=28854cc2f97f8f77009928fcdf15480dd10b43c61dda22b0af5f0985d38e7114 ``` **Note** The connection handles to the cluster VPC endpoint have an idle timeout period of 350 seconds. If you don't send commands or data by the time that the idle timeout period elapses, the connection closes and you will need to reopen it. To keep the connection open, use a timer that periodically sends a ping message to the cluster through an active handle. For this, you can run the following code. ``` .aws.start_keepalive: {\t 10000;.z.ts:{con "-1 \"Ping\"";}} .aws.stop_keepalive: {\t 0;.z.ts:{}} ``` # Running code on a Managed kdb Insights cluster Q is the programming system for working with kdb\$1. This corresponds to SQL for traditional databases, but unlike SQL, q is a powerful interpreted programming language in its own right. Q expressions can be entered and run in the q console, or loaded from a q script, which is a text file with extension `.q`. For more information, see the [q documentation](https://code.kx.com/q/learn/startingkdb/language/). You can run your own kdb code on a Managed kdb cluster to perform analytics or query data in a database. The following sections describe how you can use q in FinSpace. ** **Topics** ** + [ # .z namespace override ](interacting-with-kdb-z-namespace.md) + [ # Supported system commands ](interacting-with-kdb-system-commands.md) + [ # FinSpace q API reference ](interacting-with-kdb-q-apis.md) # .z namespace override KX uses the `.z` namespace that contains environment variables and functions, and hooks for callbacks. A FinSpace Managed kdb Insights cluster doesn't support direct assignment for `.z` namespace callbacks because of security concerns. For example, the system denies access to the following direct `.z.ts` assignment. ``` q)con".z.ts:{[x]}" / con is the hopen filehandle 'access [0] con".z.ts:{[x]}" ``` Because some of the assignments for `.z` namespace callbacks are critical for business logic, FinSpace provides a reserved namespace `.awscust.z` for you to override functions within the `.z` namespace. By overriding the functions within the new `.awscust.z` namespace, you can achieve the same effect as if you were directly overriding allowlisted `.z` functions. For example, if you need to override the `.z.ts` function, you can set a value for `.awscust.z.ts`. The FinSpace Managed kdb cluster invokes the `.awscust.z.ts` function whenever you invoke the `.z.ts` function, which provides a safety wrapper. The following list shows the allowlisted callbacks for the `.awscust.z` namespace. ``` .awscust.z.ts .awscust.z.pg .awscust.z.ps .awscust.z.po .awscust.z.pc .awscust.z.ws .awscust.z.wo .awscust.z.wc .awscust.z.pd .awscust.z.ph .awscust.z.pi: .awscust.z.exit ``` If you override `.z` callbacks that aren't on the preceding list, you won't have any effects on the `.z` namespace callbacks. # Supported system commands System commands control the q environment. The following table shows a list of system commands that FinSpace supports. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/finspace/latest/userguide/interacting-with-kdb-system-commands.html) ## Helper environment variables You can quickly access user directories through the following environment variables that return a string of the folder path. | Helper environment variables | Use for | Directory | | --- | --- | --- | | .aws.akcp | Primary user code path. | /opt/kx/app/code | | .aws.akcsp | Secondary user code path that's available only for **General purpose** cluster. | /opt/kx/app/code\$1scratch | | .aws.akscp | Primarily used for handling savedown functionality with an RDB cluster. | /opt/kx/app/scratch | ## Loading databases relative to code directory We have added a symlink to the code directory to allow loading of database relative to the code path. For example, if the database is labeled as *kxDatabase* and the current working directory is /`opt/kx/app/code` then the database can be loaded as `\l /kxDatabase`. # FinSpace q API reference FinSpace provides a set of q APIs that you can use to interact with resources in your Managed kdb environment. These APIs reside in the *.aws Q* namespace. ## Ingestion APIs **Function**: `.aws.create_changeset[db_name;change_requests]` Creates a new changeset in the specified database. ------ #### [ Parameters ] `db_name` **Description** – The name of the FinSpace kdb database where you can create Managed kdb Insights changesets. This must be the same database that you used when you created the RDB cluster. **Type** – String **Required** – Yes `change_requests` **Description** – A q table representing the list of change requests for the Managed kdb Insights changesets. The table has 3 columns : + `input_path` – The input path of the local file system directory or file to ingest as a Managed kdb changeset. + `database_path` – The target database destination path of the Managed kdb changeset. This column maps to the `databasePath` field of the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset.html) API. + `change_type` – The type of the Managed kdb changeset. It can be either `PUT` or `DELETE`. This column maps to the `changeType` field of the [CreateKxChangeset](https://docs.aws.amazon.com/finspace/latest/management-api/API_CreateKxChangeset.html) API. **Type** – Q table **Required** – Yes ------ #### [ Result ] Returns the `changeset_id` of the created Managed kdb changeset, along with its current status. ------ **Function**: `.aws.get_changeset[db_name;changeset_id]` Retrieves information about a specific changeset. ------ #### [ Parameters ] `db_name` **Description** – The name of the FinSpace kdb database where you can create Managed kdb changesets. This must be the same database that you used when you created the RDB cluster. **Type** – String **Required** – Yes `changeset_id` **Description** – The identifier of the Managed kdb changeset. **Type** – string **Required** – Yes ------ #### [ Result ] Returns the `changeset_id` and the status of the Managed kdb changeset. ------ **Function**: `.aws.get_latest_sym_file[db_name;destination_path]` Retrieves the latest sym file from the specified database. ------ #### [ Parameters ] `db_name` **Description** – The name of the FinSpace kdb database where you can create Managed kdb changesets. This must be the same database that you used when you created the RDB cluster. **Type** – String **Required** – Yes `destination_path` **Description** – The directory in the local filesystem scratch location where you want to download the symfile. **Type** – String **Required** – Yes ------ #### [ Result ] Returns the destination path where the sym file was copied to. ------ **Function**: `.aws.s3.get_object[source_s3_path;destination_disk_path]` Copies Amazon S3 object from your S3 bucket account into a local disk location in a kdb cluster. ------ #### [ Permissions ] For this function, the `executionRole` of the cluster must have the `s3:GetObject` permission to access the object and `kms:Decrypt` permission for the key that you use to encrypt the S3 bucket. ------ #### [ Parameters ] `source_s3_path` **Description** – The source path in the customer account from where you want to copy an S3 object. This can be S3 object ARN or S3 URI path. **Type** – String **Required** – Yes `destination_disk_path` **Description** – The local disk location to copy the S3 object to. **Type** – String **Required** – Yes ------ #### [ Example ] The following code is an example request to copy S3 object to a local disk. ``` q) .aws.s3.get_object["s3://customer-bucket/reference_data.csv"; "/opt/kx/app/shared/VolumeName/common/"] ``` ------ #### [ Result ] Returns a table of S3 object path and local directory disk location. Sample response ``` s3ObjectPath containerFileDestinationPath -------------------------------------------------------------------------- s3://data-bucket/data.csv "/opt/kx/app/shared/test/common/data.csv" ``` Sample response for retrieving multiple files ``` .aws.copy_database_files["DATABASE_NAME"; "DESTINATION_PATH"; "PARTITION_NAME/*"; ""] database_name| "DATABASE_NAME" changeset_id | "CHANGESET_ID" result_paths | ("DESTINATION_PATH/PARTITION_NAME/file1"; "DESTINATION_PATH/PARTITION_NAME/file2"...) ``` ------ **Function**: `.aws.copy_database_files[database_name, destination_path, db_path, changeset_id]` Retrieves a specific file from a specific version of the database. The `changeset_id` provides the version of the database from where you want to retrieve the file. ------ #### [ Parameters ] `database_name` **Description** – The name of the FinSpace kdb database where you can create Managed kdb changesets. This must be the same database that you used when you created the RDB cluster. **Type** – String **Required** – Yes `destination_path` **Description** – The directory in the local filesystem scratch location where you want to download one or more files. **Type** – String **Required** – Yes `db_path` **Description** – The path within the database directory of the file you want to retrieve. This can be a single file or a path ending with the wildcard "\$1” to retrieve multiple files. Following are a few example values for `db_path`. + `sym` retrieves the file named **sym** located in the root directory of the database. + `sym*` retrieves all files starting with **sym** for a database. For example, **sym1** and **sym2**. + `2022.01.02/*` retrieves all files within the directory **2022.01.02**. For example, **2022.01.02/col1**, **2022.01.02/col2**, etc. Alternatively, you can use `2022.01.02/` to achieve the same result. + `2022.05.*` retrieves all files from May 2022 within a date-partitioned database. For example, all files from **2022.05.01**, **2022.05.02**, etc. **Type** – String **Required** – Yes `changeset_id` **Description** – The identifier of the Managed kdb changeset. You can specify an empty string `""` to use the latest changeset. **Type** – String **Required** – Yes ------ #### [ Result ] Returns the destination path where the files were copied to, along with the `database_name` and `changeset_id` used. Sample response for retrieving a file ``` .aws.copy_database_files["DATABASE_NAME"; "DESTINATION_PATH"; "DB_FILE_PATH"; ""] database_name| "DATABASE_NAME" changeset_id | "CHANGESET_ID" result_paths | ,"DESTINATION_PATH/DB_FILE_PATH" ``` Sample response for retrieving multiple files ``` .aws.copy_database_files["DATABASE_NAME"; "DESTINATION_PATH"; "PARTITION_NAME/*"; ""] database_name| "DATABASE_NAME" changeset_id | "CHANGESET_ID" result_paths | ("DESTINATION_PATH/PARTITION_NAME/file1"; "DESTINATION_PATH/PARTITION_NAME/file2"...) ``` ------ **Function**: `.aws.get_kx_dataview[db_name;dataview_name]` Retrieves information about a specific dataview. This operation is helpful especially when you update a dataview with `update_kx_dataview`, as it retrieves the latest status and reflects the updated `changeset_id`, `segment_configurations`, and `active_versions`. ------ #### [ Permissions ] For this function, the `executionRole` must have the `finspace:GetKxDataview` permission. ------ #### [ Parameters ] `db_name` **Description** – The name of the FinSpace kdb database where the specified dataview exists. This must be same as the database you used when you created a cluster. **Type** – String **Required** – Yes `dataview_name` **Description** – The name of the Managed kdb dataview you want to retrieve. **Type** – String **Required** – Yes ------ #### [ Result ] Returns the details of specified dataview, including its status, changeset id, and segment configurations. ``` dataview_name | "example-dataview-name" database_name | "example-db" status | "ACTIVE" changeset_id | "example-changeset-id" segment_configurations | +`db_paths`volume_name!(,,"/*";,"example-volume") availability_zone_id | "use1-az2" az_mode | "SINGLE" auto_update | 0b read_write | 0b active_versions | +`changeset_id`segment_configurations`attached_clusters`created_timestamp`version_id!(("example-changeset-id";"prior-changeset-id");(+`db_paths`volume_name!(,,"/*";,"example-volume");+`db_paths`volume_name!(,,"/*";,"example-volume"));(();,"example-cluster");1.717532e+09 1.716324e+09;("kMfybotBQNQl5LBLhDnAEA";"XMfOcGisErAFO9i1XRTdYQ")) create_timestamp | 1.716324e+09 last_modified_timestamp| 1.717779e+09 ``` ------ **Function**: `.aws.update_kx_dataview[db_name;dataview_name;changeset_id;segment_configurations]` Updates the changeset id and/or segment configurations of the specified dataview. Each update of the dataview creates a new version, with its own changeset details and cache configurations. If a dataview is created with auto-update set to false, when new changesets are ingested, this operation must be run to update the dataview with the latest changeset. This operation can also be used to update the segment configurations, which define which database paths are placed on each selected volume. ------ #### [ Permissions ] For this function, the `executionRole` must have the `finspace:UpdateKxDataview` permission. ------ #### [ Parameters ] `db_name` **Description** – The name of the Managed kdb database where the specified dataview exists. This must be the same database that you used when you created a cluster. **Type** – String **Required** – Yes `dataview_name` **Description** – The name of the Managed kdb dataview you want to update. **Type** – String **Required** – Yes `changeset_id` **Description** – The identifier of the Managed kdb changeset that the dataview should use. **Type** – String **Required** – Yes `segment_configurations` **Description** – The output of the `.aws.sgmtcfgs` function. **Required** – Yes ------ #### [ Example ] The following code is an example request to update the dataview. ``` .aws.update_kx_dataview["example-db"; "example-dataview-name"; "example-changeset-id"; .aws.sgmtcfgs[.aws.sgmtcfg[("/*");"example-volume"]]] ``` ------ #### [ Result ] This function does not return any value. ------ **Function**: `.aws.sgmtcfgs[segment_configurations]` This is a helper function to construct arguments for .`aws.update_kx_dataview`, defining the list of segment configurations for the dataview. ------ #### [ Parameters ] `segment_configurations` **Description** – Either a single output of `.aws.sgmtcfg` or a list of .`aws.sgmtcfg` outputs. **Required** – Yes ------ #### [ Example ] The following example shows how this function can take a single segment configuration. ``` .aws.sgmtcfgs[.aws.sgmtcfg[("/*");"example-volume"]] ``` Alternatively, you can use this function with multiple segment configurations as following. ``` .aws.sgmtcfgs[(.aws.sgmtcfg[("/2020.02.01/*");"example-volume-1"];.aws.sgmtcfg[("/2020.02.02/*");"example-volume-2"])] ``` ------ #### [ Result ] The output of this function is used as input for `.aws.update_kx_dataview`. ------ **Function**: `.aws.sgmtcfg[db_paths;volume_name]` This is a helper function to construct arguments for `.aws.sgmtcfgs`, defining a single segment configuration, the database path of the data that you want to place on each selected volume. Each segment must have a unique database path for each volume. ------ #### [ Parameters ] `db_paths` **Description** – The database path of the data you want to place on each selected volume for the segment. Each segment must have a unique database path for each volume. **Type** – Array of strings **Required** – Yes `volume_name` **Description** – The name of the Managed kdb volume where you would like to add data. **Type** – String **Required** – Yes ------ #### [ Example ] The following example shows how this function can take a single db path. ``` .aws.sgmtcfg[("/*");"example-volume"] ``` Alternatively, you can use this function with multiple db paths as following. ``` .aws.sgmtcfg[("/2020.01.06/*";"/2020.01.02/*");"example-volume"] ``` ------ #### [ Result ] The output of this function is used as input for `.aws.sgmtcfgs`. ------ ## Discovery APIs **Function**: `.aws.list_kx_clusters()` Returns a table of clusters in non-deleted state. ------ #### [ Parameters ] N/A ------ #### [ Result ] Returns a table of Managed kdb clusters that are in a non-deleted state. This table consists of the following fields – `cluster_name`, `status`, `cluster_type`, and `description`. ------ **Function**: `.aws.list_kx_cluster_nodes[cluster_name]` Retrieves a list of nodes within a cluster. ------ #### [ Parameters ] `cluster_name` **Description** – The name of the Managed kdb cluster that you specified when creating a kdb cluster. You can also get this by using the `.aws.list_kx_clusters()` function. **Type** – String **Required** – Yes ------ #### [ Result ] Returns a table of nodes in the Managed kdb cluster that consists of `node_id`, `az_id`, and `launch_time`. ------ ## Authorization APIs **Note** You must create clusters with the IAM `executionRole` field to use the q auth APIs. Clusters will assume this role when calling the auth APIs, so the role should have `GetKxConnectionString` and `ConnectKxCluster` permissions. **Function**: `.aws.get_kx_node_connection_string[cluster_name;node_id]` Retrieves the connection string for a given kdb cluster node. ------ #### [ Parameters ] `cluster_name` **Description** – The name of the destination Managed kdb cluster for the connection string. **Type** – String **Pattern** – `^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$` **Length** – 3-63 **Required** – Yes `node_id` **Description** – The node identifier of the target cluster. **Type** – String **Length** – 1-40 **Required** – Yes ------ #### [ Result ] Returns the connection string. ------ **Function**: `.aws.get_kx_connection_string[cluster_name]` Retrieves the connection string for a given kdb cluster. ------ #### [ Parameters ] `cluster_name` **Description** – The name of the destination cluster for the connection string. **Type** – String **Pattern** – `^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$` **Length** – 3-63 **Required** – Yes ------ #### [ Result ] Returns the connection string. ------ ## Cluster management APIs **Note** You must create clusters with the IAM `executionRole` field to use the cluster management APIs. **Function**: `.aws.stop_current_kx_cluster_creation[message]` Stops the current cluster creation and puts the cluster in the `CREATE_FAILED` state. You can only call this function from an initialization script. ------ #### [ Parameters ] `message` **Description** – A message to display in the `statusReason` field of the cluster after the cluster reaches the `CREATE_FAILED` state. **Type** – String **Pattern** – `^[a-zA-Z0-9\_\-\.\s]*$` **Length** – 0-50 **Required** – Yes ------ #### [ Example ] The following code is an example request to stop creation of current cluster with a message. ``` .aws.stop_current_kx_cluster_creation[""] ``` ------ #### [ Result ] This function does not return any value. ------ **Function**: `.aws.delete_kx_cluster[clusterName]` Deletes the specified cluster. If `clusterName` is an empty string, this function deletes the current cluster. ------ #### [ Permissions ] For this function, the `executionRole` must have the following permissions to delete the cluster: + `ec2:DescribeTags` + `ec2:DeleteVpcEndpoints` + `finspace:DeleteKxCluster` ------ #### [ Parameters ] `clusterName` **Description** – The name of the cluster that you want to delete. **Type** – String **Pattern** – `^[a-zA-Z0-9-_]*$` **Length** – 1-63 **Required** – Yes ------ #### [ Example ] The following example deletes the *samplecst* cluster. ``` .aws.delete_kx_cluster["samplecst"] ``` The following example deletes the current cluster. ``` .aws.delete_kx_cluster[""] ``` ------ #### [ Result ] This function does not return any value. ------ **Function**: `.aws.get_kx_cluster[clusterName]` Retrieves information about the specified cluster. ------ #### [ Permissions ] For this function, the `executionRole` must have the `finspace:GetKxCluster` permission. ------ #### [ Parameters ] `clusterName` **Description** – The name of the target cluster. **Type** – String **Pattern** – `^[a-zA-Z0-9-_]*$` **Length** – 1-63 **Required** – Yes ------ #### [ Result ] ``` status | "RUNNING" >>>>>>> mainline clusterName | "example-cluster-name" clusterType | "HDB" capacityConfiguration| `nodeType`nodeCount!("kx.s.xlarge";1f) releaseLabel | "1.0" vpcConfiguration | `vpcId`securityGroupIds`subnetIds`ipAddressType!("vpcId";,"securityGroupId";,"subnetId";"IP_V4") executionRole | "arn:aws:iam::111111111111:role/exampleRole" lastModifiedTimestamp| 1.695064e+09 azMode | "SINGLE" availabilityZoneId | "use1-az1" createdTimestamp | 1.695063e+09 ``` ------ **Function**: `.aws.update_kx_cluster_databases[clusterName;databases;properties]` Updates the database of the specified kdb cluster. ------ #### [ Permissions ] + For this function, the `executionRole` must have the `finspace:UpdateKxClusterDatabases` permission. + You must have `finspace:GetKXCluster` permission for the `clusterName`. ------ #### [ Parameters ] `clusterName` **Description** – The name of the target cluster. **Type** – String **Pattern** – `^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$` **Length** – 3-63 **Required** – Yes `databases` **Description** – The output of the `.aws.sdbs` function. **Required** – Yes `properties` **Description** – The output of the `.aws.sdep` function. **Required** – Yes ------ #### [ Example ] The following code is an example request to update the cluster database. ``` .aws.update_kx_cluster_databases["HDB_TAQ_2021H1"; .aws.sdbs[ .aws.db["TAQ_2021H1";"osSoXB58eSXuDXLZFTCHyg"; .aws.cache["CACHE_1000";"/"]; "" ] ]; .aws.sdep["ROLLING"]] ``` ------ #### [ Result ] This function does not return any value. ------ **Function**: `.aws.sdbs[databases]` This is a helper function to construct arguments for `.aws.update_kx_cluster_databases`. ------ #### [ Parameters ] databases **Description** – It is either a single output of `.aws.db` or a list of `.aws.db` outputs. **Required** – Yes ------ #### [ Example ] Here is an example of how you can use this function. ``` .aws.sdbs[ .aws.db["TAQ_2021H1";"osSoXB58eSXuDXLZFTCHyg"; .aws.cache["CACHE_1000";"/"]; "" ] ]; ``` ------ #### [ Result ] The output of this function is used as input for `.aws.update_kx_cluster_databases` function. ------ **Function**: `.aws.db[databaseName;changesetId;caches;dataviewName]` This is a helper function to construct arguments for `.aws.sdbs`. ------ #### [ Parameters ] databaseName **Description** – The name of the target database. **Type** – String **Pattern** – `^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$` **Length** – 3-63 **Required** – Yes changesetId **Description** – A unique identifier of the changeset. If you pass empty string “” for this parameter, the latest changeset of the database will be used. **Type** – String **Length** – 1-26 **Required** – No caches **Description** – It is either a single output of `.aws.cache` or a list of .`aws.cache` outputs. If there is no cache associated to the cluster, this list must be empty. **Required** – No dataviewName **Description** – The name of the dataview. **Type** – String **Pattern** – `^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$` **Length** – 3-63 **Required** – No ------ #### [ Example ] You can use this function to specify the changeset that you want to update, as following: ``` .aws.db["example-db";"example-changeset-id"; .aws.cache["CACHE_1000";"/"];""] ``` Alternatively, if the cluster is attached to a dataview, you can use this function to update the cluster to the latest version of the dataview with the specified `dataviewName`, as following: ``` .aws.db["example-db";""; .aws.cache["CACHE_1000";"/"];"example-dataview-name"] ``` ------ #### [ Result ] The output of this function is used as input for `.aws.sdbs` function. ------ **Function**: `.aws.cache[cacheType;dbPaths]` This is a helper function to construct arguments for `.aws.db`. ------ #### [ Parameters ] cacheType **Description** – The type of disk cache. This parameter is used to map the database path to cache storage. **Type** – String **Length** – 8-10 **Required** – Yes dbPaths **Description** – The portions of database that will be loaded into the cache for access. **Type** – Array of strings **Pattern** – `^\/([^\/]+\/){0,2}[^\/]*$` **Length** – 1-1025 **Required** – Yes ------ #### [ Example ] The following two examples show how you can send different requests by using this function. ``` .aws.cache["CACHE_1000";"/"] ``` ``` .aws.cache["CACHE_1000";("path1";"path2")] ``` ------ #### [ Result ] The output of this function is used as input for `.aws.db` function. ------ **Function**: `.aws.sdbs[deploymentStrategy]` This is a helper function to construct arguments for `.aws.update_kx_cluster_databases`. ------ #### [ Parameters ] deploymentStrategy **Description** – The type of deployment that you want on a cluster. The following types are available. + `ROLLING` – This options loads the updated database by stopping the exiting q process and starting a new q process with updated configuration. + `NO_RESTART` – This option loads the updated database on the running q process without stopping it. This option is quicker as it reduces the turn around time to update a kdb database changeset configuration on a cluster. **Type** – String **Required** – Yes ------ #### [ Result ] The output of this function is used as input for `.aws.update_kx_cluster_databases` function. ------ ## Pub/Sub APIs **Function**: `.aws.sub[table;sym_list]` Initializes the publish and subscribe function. ------ #### [ Parameters ] `table` **Description** – The symbol of the table that you want to subscribe to. The symbol ``` subscribes to all tables. **Type** – Symbol **Required** – Yes `sym_list` **Description** – The list of symbols to filter published records. Defaults to ``` if no filter is applied. **Type** – Symbol list **Required** – Yes ------ #### [ Result ] Returns table schema or table schemas list. **Example 1: Subscribes to ``tab` table and filtering ``AAPL`MSFT`**. ``` target_instance_connection_handle ".aws.sub[`tab;`AAPL`MSFT]" ``` **Result** ``` `tab +`sym`sales`prices!(`g#`symbol$();`long$();`long$()) ``` **Example 2: Subscribes to all tables with no filtering**. ``` target_instance_connection_handle ".aws.sub[`;`]" ``` **Result** ``` `tab +`sym`sales`prices!(`g#`symbol$();`long$();`long$()) `tab1 +`sym`sales`prices!(`g#`symbol$();`long$();`long$()) `tab2 +`sym`sales`prices!(`g#`symbol$();`long$();`long$()) `tab3 +`sym`sales`prices!(`g#`symbol$();`long$();`long$()) ``` ------ **Function**: `.aws.pub[table;table_records]` Publishes table records to table subscribers by calling `upd[table;table_records]` within the subscriber connection handle. ------ #### [ Parameters ] `table` **Description** – Publishes the records to the table subscribers. **Type** – Symbol **Required** – Yes `table_records` **Description** – The table records that you want to publish. **Type** – Table **Required** – Yes ------ #### [ Example ] This example publishes ``tab` table and values to the subscribers. ``` .aws.pub[`tab;value `tab] ``` ------ #### [ Result ] This function does not return any value. ------ ## Database maintenance APIs **Function**: `.aws.commit_kx_database[database_name]` Commits the database changes after performing database maintenance. ------ #### [ Parameters ] `database_name` **Description** – The name of the database where you performed database maintenance operations and whose changes you want to commit. **Type** – String **Required** – Yes ------ #### [ Example ] ``` .aws.commit_kx_database["welcomedb"] ``` ------ #### [ Result ] Returns the `id` and `status` of the changeset that the API creates. ------ # Logging and monitoring Amazon FinSpace provides a variety of Amazon CloudWatch metrics that you can monitor to determine the health and performance of your FinSpace resources and instances. You can view these metrics using various tools, including the FinSpace console, the AWS CLI, the Amazon CloudWatch console. ## Metrics and dimensions The `AWS/FinSpace` namespace includes the following metrics. | Metric | Description | | --- | --- | | CPUUtilization | The average CPU utilization across all the nodes in a cluster. | | MemoryUtilization | The average memory utilization across all the nodes in a cluster. | | DatabaseCacheDataReadBytes | The operations for reading database cache. | | DatabaseCacheDataWriteBytes | The operations for writing to database cache. | | DatabaseCacheDataReadOperations | The operations for reading database cache. | | DatabaseCacheDataWriteOperations | The operations for writing to database cache. | | DatabaseCacheFreeDataStorageCapacity | Database cache free storage. | | LocalStorageVolumeReadBytes | The volume for read operation in local storage. | | LocalStorageVolumeWriteBytes | The volume for read operation in local storage. | | LocalStorageVolumeReadOps | The operations for reading local storage volume. | | LocalStorageVolumeWriteOps | The operations for writing to local storage volume. | | MemoryHeapBytes | The memory available in the heap in bytes for the component container. | | MemoryHeapPeakBytes | The maximum heap size so far in bytes for the component container. | | MemoryHeapLimitBytes | The memory limit heap in bytes for the component container as set by `-w`. | | MemoryMappedBytes | The mapped memory in bytes for the component container. | | MemoryPhysicalBytes | The physical memory available in bytes for the component container. | | KdbSymsTotal | The number of symbols for the component container. | | KdbSymsMemoryBytes | The memory use of symbols in bytes for the component container. | | KdbHandlesTotal | The number of active connection handles. | | KdbIpcOpenedTotal | The total number of IPC sockets that have been opened to the component container. | | KdbIpcClosedTotal | The total number of open handles (IPC and WebSocket) for the component container. | | KdbWsOpenedTotal | The total number of WebSocket connections opened to the component container. | | KdbWsClosedTotal | The total number of WebSocket connections closed to the component container. | | KdbSyncTotal | The sync requests made to the component container. | | KdbHttpGetTotal | The HTTP `GET` requests made to the component container. | | KdbHttpPostTotal | The HTTP `POST` requests made to the component container. | | KdbWsTotal | The WebSocket messages received by the component container. | | KdbTsTotal | The timer calls made within the component container. | | KdbSyncErrTotal | The number of errors returned in sync requests. | | KdbAsyncErrTotal | The number of errors returned in async requests. | | KdbHttpGetErrTotal | The number of errors returned in http `GET` requests. | | KdbHttpPostErrTotal | The number of errors returned in http `POST` requests. | | KdbWsErrTotal | The number of errors returned in WebSocket calls. | | KdbTsErrTotal | The number of errors returned in timer calls. | | KdbSyncSeconds | The count and time taken by the sync requests. | | KdbAsyncSeconds | The count and time taken by the async requests. | The `AWS/Usage` namespace includes the following metrics. | Metric | Description | | --- | --- | | KxNodeSglargePerEnvironment | The maximum number of kx.sg.large Managed kdb scaling group nodes per environment. | | KxNodeSgxlargePerEnvironment | The maximum number of kx.sg.xlarge Managed kdb scaling group nodes per environment. | | KxNodeSg2xlargePerEnvironment | The maximum number of kx.sg.2xlarge Managed kdb scaling group nodes per environment. | | KxNodeSg4xlargePerEnvironment | The maximum number of kx.sg.4xlarge Managed kdb scaling group nodes per environment. | | KxNodeSg8xlargePerEnvironment | The maximum number of kx.sg.8xlarge Managed kdb scaling group nodes per environment. | | KxNodeSg16xlargePerEnvironment | The maximum number of kx.sg.16xlarge Managed kdb scaling group nodes per environment. | | KxNodeSg32xlargePerEnvironment | The maximum number of kx.sg.32xlarge Managed kdb scaling group nodes per environment. | | KxNodeSgOne16xlargePerEnvironment | The maximum number of kx.sg1.16xlarge Managed kdb scaling group nodes per environment. | | KxNodeSgOne24xlargePerEnvironment | The maximum number of kx.sg1.24xlarge Managed kdb scaling group nodes per environment. | | KdbFileStorageVolumesPerEnvironment | The maximum number of Managed kdb volumes per environment. | | ReadMountsPerKdbVolume | The maximum number of read mounts per Managed kdb volume per environment. | | WriteMountsPerKdbVolume | The maximum number of write mounts per Managed kdb volume per environment. | | ManagedKdbVolumeStorage | The maximum amount of storage for Managed kdb volumes per environment. | | KdbScalingGroupsPerEnvironment | The maximum number of Managed kdb scaling groups per environment. | | KdbDataviewsPerEnvironment | The maximum number of Managed kdb dataviews per environment. | | ConcurrentKdbDataviewsVersionsPerEnvironment | The maximum number of concurrent Managed kdb dataview version processing. | | TotalKdbEnvironments | The maximum number of environments per AWS account. | | ManagedKdbClusterUsers | The maximum number of cluster users per environment. | | ManagedKdbClusters | The maximum number of clusters allowed per environment. | | ManagedKdbDatabases | The maximum number of databases allowed per environment. | | ManagedKdbConcurrentChangesetIngestions | The maximum number of concurrent changeset ingestions allowed per environment. | | ManagedKdbDatabaseClusterCacheSize | The maximum amount of database cluster cache allowed per environment. | | ManagedKdbSavedownStorage | The maximum amount of savedown storage allowed per environment. | | KxSlargeNodes | The maximum number of kx.s.large nodes allowed per environment. | | KxSxlargeNodes | The maximum number of kx.s.xlarge nodes allowed per environment. | | KxS2xlargeNodes | The maximum number of kx.s.2xlarge nodes allowed per environment. | | KxS4xlargeNodes | The maximum number of kx.s.4xlarge nodes allowed per environment. | | KxS8xlargeNodes | The maximum number of kx.s.8xlarge nodes allowed per environment. | | KxS16xlargeNodes | The maximum number of kx.s.16xlarge nodes allowed per environment. | | KxS32xlargeNodes | The maximum number of kx.s.32xlarge nodes allowed per environment. | | ManagedKdbSingleAzClusters | The maximum number of Single-AZ clusters per environment. | | ManagedKdbMultiAzClusters | The maximum number of Multi-AZ clusters per environment. | **Note** Amazon CloudWatch aggregates these metrics at one-minute intervals. ## Monitoring Managed kdb cluster metrics You can monitor the health and activity of Managed kdb Insights clusters by using CloudWatch. When you interact with Managed kdb clusters, the following metrics are sent to CloudWatch in the AWS account that you used to create the Managed kdb environment. You can use the following procedures to view the metrics for Managed kdb clusters. **To view metrics using the CloudWatch console** Metrics are grouped first by the service namespace, and then by the various dimension combinations within each namespace. 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Under the **All metrics** tab, choose **FinSpace** and then choose the **KxClusterId**. ## Monitoring Managed kdb scaling groups metrics You can monitor the health and activity of Managed kdb scaling groups by using CloudWatch. When you interact with Managed kdb scaling groups, the following metrics are sent to CloudWatch in the AWS account that you used to create the Managed kdb environment. You can use the following procedures to view the metrics for Managed kdb scaling groups. **To view metrics using the CloudWatch console** Metrics are grouped first by the service namespace, and then by the various dimension combinations within each namespace. 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Under the **All metrics** tab, choose **FinSpace** and then choose the **KxClusterId**. FinSpace supports the following metrics for Managed Kdb scaling groups. | Metric | Description | | --- | --- | | ScalingGroupCpuUtilization | The total CPU utilization of the Managed kdb scaling group hosts across all clusters. | | ScalingGroupMemoryUtilization | The total memory utilization of the Managed kdb scaling group hosts across all clusters. | ## Monitoring Managed kdb volume metrics You can monitor the health and activity of Managed kdb volumes by using CloudWatch. When you interact with Managed kdb volumes, the following metrics are sent to CloudWatch in the AWS account that you used to create the Managed kdb environment. You can use the following procedures to view the metrics for Managed kdb volumes. **To view metrics using the CloudWatch console** Metrics are grouped first by the service namespace, and then by the various dimension combinations within each namespace. 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Under the **All metrics** tab, choose **FinSpace** and then choose the **KxVolumeName**. FinSpace supports the following metrics for Managed Kdb volumes. | Metric | Description | | --- | --- | | KdbVolumeDataReadBytes | The data in bytes read from managed kdb volumes. | | KdbVolumeDataWriteBytes | The data in bytes written from managed kdb volumes. | | KdbVolumeDataReadOperations | The operations for reading Managed kdb volumes. | | KdbVolumeDataWriteOperations | The operations for writing to Managed kdb volumes. | | KdbVolumeFreeDataStorageCapacity | The Managed kdb volumes free storage. | ## Logging The KDB application logs from Managed kdb clusters are captured in Amazon CloudWatch Logs. There is a separate CloudWatch Log Group created for each cluster. You can access cluster logs for FinSpace in the following CloudWatch log group. `/aws/vendedlogs/finspace//` You can view data directly in CloudWatch using CloudWatch reporting or CloudWatch Insights. You can also extract the data from CloudWatch into other logging tools like Splunk or Datadog. Using the **Logs**, you can view `stdout` and `stderr` logs the for KDB processes. From this tab, you can navigate to CloudWatch Logs Insights to run custom queries on the logs. **To view and query logs for Managed kdb clusters** 1. Sign in to the AWS Management Console and open the Amazon FinSpace console at [https://console.aws.amazon.com/finspace](https://console.aws.amazon.com/finspace/landing). 1. In the left pane, under **Managed kdb Insights**, choose **Kdb environments**. 1. From the kdb environments table, choose the name of the environment. 1. On the environment details page, choose the **Clusters** tab. The table under this tab displays a list of clusters. 1. Choose a cluster name to view its logs. The cluster details page opens. 1. Choose the **Logs** tab. The logs are displayed in a list. 1. Choose the **@logstream** links to open the CloudWatch Logs Insights where you can run custom queries on the logs.