# Monitor the guidance
Monitor the guidance

Follow the step-by-step instructions in this section for:
+  [Monitoring cost and manage portfolio using Service Catalog AppRegistry](monitoring-cost-portfolio-with-aws-service-catalog-appregistry.md), and
+  [Monitoring guidance performance and operational data using Amazon CloudWatch](monitoring-performance-ops-with-amazon-cloudwatch.md).

# Monitoring cost and portfolio with Service Catalog AppRegistry


This guidance includes a Service Catalog AppRegistry resource to register the CloudFormation template and underlying resources as an application in both [Service Catalog AppRegistry](https://docs.aws.amazon.com/servicecatalog/latest/arguide/intro-app-registry.html) and [AWS Systems Manager Application Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/application-manager.html).

AWS Systems Manager Application Manager gives you an application-level view into this guidance and its resources so that you can:
+ Monitor its resources, costs for the deployed resources across stacks and AWS accounts, and logs associated with this guidance from a central location.
+ View operations data for the resources of this guidance (such as deployment status, CloudWatch alarms, resource configurations, and operational issues) in the context of an application.

  The following figure depicts an example of the application view for the guidance stack in Application Manager.

 **Depicts an AWS guidance stack in Application Manager** 

![\[appregistry1\]](http://docs.aws.amazon.com/solutions/latest/scalable-analytics-using-apache-druid-on-aws/images/appregistry1.png)


# Activate CloudWatch Application Insights


1. Sign in to the [Systems Manager console](https://console.aws.amazon.com/systems-manager).

1. In the navigation pane, choose **Application Manager**.

1. In **Applications**, search for the application name for this guidance and select it.

   The application name will have App Registry in the **Application Source** column, and will have a combination of the guidance name, Region, account ID, or stack name.

1. In the **Components** tree, choose the application stack you want to activate.

1. In the **Monitoring** tab, in **Application Insights**, select **Auto-configure Application Insights**.

 **Application Insights dashboard showing no detected problems and option to auto-configure.** 

![\[appreg1\]](http://docs.aws.amazon.com/solutions/latest/scalable-analytics-using-apache-druid-on-aws/images/appreg1.png)


Monitoring for your applications is now activated and the following status box appears:

Application Insights dashboard showing successful monitoring activation message. image::images/appreg2.png[scaledwidth=100%]

# Confirm cost tags associated with the guidance


After you activate cost allocation tags associated with the guidance, you must confirm the cost allocation tags to see the costs for this guidance. To confirm cost allocation tags:

1. Sign in to the site:https://console.aws.amazon.com/systems-manager[Systems Manager console].

1. In the navigation pane, choose **Application Manager**.

1. In **Applications**, choose the application name for this guidance and select it.

1. In the **Overview** tab, in **Cost**, select **Add user tag**.

    **Screenshot depicting the Application Cost add user tag screen**   
![\[AppManager 1\]](http://docs.aws.amazon.com/solutions/latest/scalable-analytics-using-apache-druid-on-aws/images/AppManager_1.png)

1. On the **Add user tag** page, enter `confirm`, then select **Add user tag**.

The activation process can take up to 24 hours to complete and the tag data to appear.

# Activate cost allocation tags associated with the guidance


After you confirm the cost tags associated with this guidance, you must activate the cost allocation tags to see the costs for this guidance. The cost allocation tags can only be activated from the management account for the organization.

To activate cost allocation tags:

1. Sign in to the [AWS Billing and Cost Management and Cost Management console](https://console.aws.amazon.com/billing/home).

1. In the navigation pane, select **\$1ost Allocation Tags**.

1. On the **Cost allocation tags** page, filter for the `AppManagerCFNStackKey` tag, then select the tag from the results shown.

1. Choose **Activate**.

# AWS Cost Explorer


You can see the overview of the costs associated with the application and application components within the Application Manager console through integration with AWS Cost Explorer. Cost Explorer helps you manage costs by providing a view of your AWS resource costs and usage over time.

1. Sign in to the [AWS Cost Management console](https://console.aws.amazon.com/cost-management/home).

1. In the navigation menu, select **Cost Explorer** to view the guidance’s costs and usage over time.

# Monitoring performance and operations with Amazon CloudWatch


The guidance captures all the Druid data logs in Amazon CloudWatch for monitoring purposes. This includes alarms, logs and a dashboard for reporting purposes.

 **Amazon CloudWatch** gives you an application-level view into this guidance and its resources so that you can:
+ Monitor alarms, logs for your deployed clusters, and metrics associated with this guidance from a central location.
+ View operations data for the guidance’s AWS resources (such as deployment status, Amazon CloudWatch alarms, resource configurations, and operational issues) in the context of an application.

Sign in to the [AWS Management Console](https://console.aws.amazon.com/), and navigate to **CloudWatch**. Use the left hand navigation to view this data for your Druid deployment.

**Note**  
You must activate the CloudWatch Application Insights before you can use CloudWatch to monitor any alarms, logs, or dashboards for the guidance. For more information, refer to the [Activate CloudWatch Application Insights](activate-cloudwatch-application-insights.md) section.

## Dashboard


1. From the left, select **CloudWatch > Dashboards**.

1. On the **Custom Dashboards** tab, click to select the dashboard you want to view. For example, `druid-ec2-custom-name-ops` -dashboard.

 **CloudWatch dashboard for Scalable Analytics using Apache Druid on AWS.** 

![\[image4\]](http://docs.aws.amazon.com/solutions/latest/scalable-analytics-using-apache-druid-on-aws/images/image4.png)


The dashboard provides the following information for your Druid deployment.


| Item | Description | 
| --- | --- | 
|  Canary status  |  Displays the availability and latency of your web services and allows you to discover and troubleshoot any issues.  | 
|  Application Load Balancer (ALB) - Key Performance Indicators  |  Provides the following data in relation to your ALB application: \$1 Request Count \$1 Target Response Time \$1 HTTP Connection Count \$1 Response Code Count  | 
|  Druid - Key Performance Indicators  |  Provides the following data in relation to Druid core parameters: \$1 Deep Storage \$1 Ingestion Count \$1 Query Count \$1 Query Time  | 
|  Druid ZooKeeper - Key Performance Indicators  |  Displays the following data in relation to the Zookeeper cluster state management: \$1 CPU Utilization (%) \$1 Network In/Out (bytes) \$1 Memory Utilization (%) \$1 Disk Utilization (%)  | 
|  Druid data  |  Displays the following data in relation to the Druid ingestion jobs and queryable data: \$1 CPU Utilization (%) \$1 Network In/Out (bytes) \$1 Memory Utilization (%) \$1 Disk Utilization (%)  | 
|  Druid query  |  Displays the following data in relation to the Druid queries: \$1 CPU Utilization (%) \$1 Network In/Out (bytes) \$1 Memory Utilization (%) \$1 Disk Utilization (%)  | 
|  Druid master  |  Displays the following data in relation to the Druid data ingestion and availability: \$1 CPU Utilization (%) \$1 Network In/Out (bytes) \$1 Memory Utilization (%) \$1 Disk Utilization (%)  | 
|  Aurora Cluster  |  Provides the following data in relation to the Druid databases: \$1 CPU Utilization (%) \$1 Database connections \$1 Throughput \$1 Free Memory  | 

## Alarms


Amazon CloudWatch provides detailed alarms for your Druid deployment, and displays the different states, conditions, and relevant actions associated with these alarms. To view the **Alarms** page, from the left, select **CloudWatch > Alarms**.

For more information about alarms, alarm states, actions, and configuring alarms, refer to the [Amazon CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) page.

## Logs


You can use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, and other sources. CloudWatch Logs allow you to centralize the logs from all of your Druid guidance components, applications, and AWS services, in a single, highly scalable service.

To view logs for the Druid deployment, from the left, select **CloudWatch > logs > Log groups**.

For more information about working with log groups and log streams, refer to the [AWS CloudWatch logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html) page.

## Metrics


In addition to logs and alarms, CloudWatch allows to view, and analyze data about the performance of your Druid deployment via custom namespaces.

To view metrics for the Druid deployment:

1. From the left, select **CloudWatch > Metrics > All metrics**.

1. On the Metrics page, under Custom namespaces, select `AWSSolutions/Druid`. This will display all metrics for the Druid deployment.

    **Screenshot of metrics for Scalable Analytics using Apache Druid on AWS in CloudWatch.**   
![\[image5\]](http://docs.aws.amazon.com/solutions/latest/scalable-analytics-using-apache-druid-on-aws/images/image5.png)

1. Choose to select a relevant dimension to view additional information. The dimensions page provides a breakdown of individual Druid services, source, query, and other metrics.