# Using the CloudWatch agent with Container Insights enhanced observability enabled Using Container Insights enhanced observability enabled Use the instructions in one of the following sections to set up Container Insights on an Amazon EKS cluster or Kubernetes cluster by using the CloudWatch agent. The quick start instructions are supported only on Amazon EKS versions 1.24 and later. **Note** You can install Container Insights by following the instructions in any one of the following sections. You don't need to follow all three sets of instructions. **Topics** + [ # Quick start with the Amazon CloudWatch Observability EKS add-on ](Container-Insights-setup-EKS-addon.md) + [ # Quick Start setup for Container Insights on Amazon EKS and Kubernetes ](Container-Insights-setup-EKS-quickstart.md) + [ # Setting up the CloudWatch agent to collect cluster metrics ](Container-Insights-setup-metrics.md) # Quick start with the Amazon CloudWatch Observability EKS add-on You can use the Amazon EKS add-on to install Container Insights with enhanced observability for Amazon EKS. The add-on installs the CloudWatch agent to send infrastructure metrics from the cluster, installs Fluent Bit to send container logs, and also enables CloudWatch [Application Signals](CloudWatch-Application-Monitoring-Sections.md) to send application performance telemetry. When you use the Amazon EKS add-on version 1.5.0 or later, Container Insights is enabled on both Linux and Windows worker nodes in the cluster. Application Signals is not supported on Windows in Amazon EKS. The Amazon EKS add-on is not supported for clusters running Kubernetes instead of Amazon EKS. For more information about the Amazon CloudWatch Observability EKS add-on, see [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md). If you use version 3.1.0 or later of the add-on, you can use EKS Pod Identity to grant the required permissions to the add-on. EKS Pod Identity is the recommended option and provides benefits such as least privilege, credential rotation, and auditability. Additionally, using EKS Pod Identity allows you to install the EKS add-on as part of the cluster creation itself. **To install the Amazon CloudWatch Observability EKS add-on** 1. Follow the [EKS Pod Identity association](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html#pod-id-association-create/) steps to create the IAM role and set up the EKS Pod Identity agent. 1. Attach an IAM policy that grants the required permissions to your role. Replace *my-role* with the name of your IAM role from the previous step. ``` aws iam attach-role-policy \ --role-name my-role \ --policy-arn=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy ``` 1. Enter the following command, using with the IAM role you created in the previous step: ``` aws eks create-addon \ --addon-name amazon-cloudwatch-observability \ --cluster-name my-cluster-name \ --pod-identity-associations serviceAccount=cloudwatch-agent,roleArn=arn:aws:iam::111122223333:role/my-role ``` # Quick Start setup for Container Insights on Amazon EKS and Kubernetes **Important** If you are installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing networks, you must use the Amazon CloudWatch Observability EKS add-on. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). To complete the setup of Container Insights, you can follow the quick start instructions in this section. If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster. **Important** Before completing the steps in this section, you must have verified the prerequisites including IAM permissions. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md). Alternatively, you can instead follow the instructions in the following two sections, [Setting up the CloudWatch agent to collect cluster metrics](Container-Insights-setup-metrics.md) and [Send logs to CloudWatch Logs](Container-Insights-EKS-logs.md). Those sections provide more configuration details on how the CloudWatch agent works with Amazon EKS and Kubernetes, but require you to perform more installation steps. With the original version of Container Insights, metrics collected and logs ingested are charged as custom metrics. With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics and logs are charged per observation instead of being charged per metric stored or log ingested. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Note** Amazon has now launched Fluent Bit as the default log solution for Container Insights with significant performance gains. We recommend that you use Fluent Bit instead of Fluentd. ## Quick Start with the CloudWatch agent operator and Fluent Bit There are two configurations for Fluent Bit: an optimized version and a version that provides an experience more similar to Fluentd. The Quick Start configuration uses the optimized version. For more details about the Fluentd-compatible configuration, see [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](Container-Insights-setup-logs-FluentBit.md). The CloudWatch agent operator is an additional container that gets installed to an Amazon EKS cluster. It is modeled after the OpenTelemetry Operator for Kubernetes. The operator manages the lifecycle of Kubernetes resources in a cluster. It installs the CloudWatch Agent, DCGM Exporter (NVIDIA), and the AWS Neuron Monitor on an Amazon EKS cluster and manages them. Fluent Bit and the CloudWatch Agent for Windows are installed directly to an Amazon EKS cluster without the operator managing them. For a more secure and feature-rich certificate authority solution, the CloudWatch agent operator requires cert-manager, a widely-adopted solution for TLS certificate management in Kubernetes. Using cert-manager simplifies the process of obtaining, renewing, managing and using these certificates. It ensures that certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry. cert-manager also facilitates issuing certificates from a variety of supported sources, including AWS Certificate Manager Private Certificate Authority. **To deploy Container Insights using the quick start** 1. Install cert-manager if it is not already installed in the cluster. For more information, see [cert-manager Installation](https://cert-manager.io/docs/installation/). 1. Install the custom resource definitions (CRD) by entering the following commmand. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl apply --server-side -f - ``` 1. Install the operator by entering the following command. Replace *my-cluster-name* with the name of your Amazon EKS or Kubernetes cluster, and replace *my-cluster-region* with the name of the Region where the logs are published. We recommend that you use the same Region where your cluster is deployed to reduce the AWS outbound data transfer costs. ``` ClusterName=my-cluster-name RegionName=my-cluster-region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f - ``` For example, to deploy Container Insights on the cluster named `MyCluster` and publish the logs and metrics to US West (Oregon), enter the following command. ``` ClusterName='MyCluster' RegionName='us-west-2' curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f - ``` **Migrating from Container Insights** If you already have Container Insights configured in an Amazon EKS cluster and you want to migrate to Container Insights with enhanced observability for Amazon EKS, see [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md) **Deleting Container Insights** If you want to remove Container Insights after using the quick start setup, enter the following commands. ``` ClusterName=my-cluster-name RegionName=my-cluster-region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f - curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f - ``` # Setting up the CloudWatch agent to collect cluster metrics **Important** If you are installing Container Insights on on Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). To set up Container Insights to collect metrics, you can follow the steps in [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) or you can follow the steps in this section. In the following steps, you set up the CloudWatch agent to be able to collect metrics from your clusters. If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster. ## Step 1: Create a namespace for CloudWatch Use the following step to create a Kubernetes namespace called `amazon-cloudwatch` for CloudWatch. You can skip this step if you have already created this namespace. **To create a namespace for CloudWatch** + Enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml ``` ## Step 2: Create a service account in the cluster Use one of the following methods to create a service account for the CloudWatch agent, if you do not already have one. + Use `kubectl` + Use a `kubeconfig` file ### Use `kubectl` for authentication **To use `kubectl` to create a service account for the CloudWatch agent** + Enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml ``` If you didn't follow the previous steps, but you already have a service account for the CloudWatch agent that you want to use, you must ensure that it has the following rules. Additionally, in the rest of the steps in the Container Insights installation, you must use the name of that service account instead of `cloudwatch-agent`. The CloudWatch agent requires a ClusterRole for cluster-wide access and a namespace-scoped role for ConfigMap operations in the `amazon-cloudwatch` namespace. **ClusterRole (cluster-scoped permissions):** ``` rules: - apiGroups: [""] resources: ["pods", "nodes", "endpoints"] verbs: ["list", "watch"] - apiGroups: [""] resources: ["services"] verbs: ["list", "watch", "get"] - apiGroups: ["apps"] resources: ["replicasets", "daemonsets", "deployments", "statefulsets"] verbs: ["list", "watch"] - apiGroups: ["batch"] resources: ["jobs"] verbs: ["list", "watch"] - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get"] - apiGroups: [""] resources: ["nodes/stats", "events"] verbs: ["create", "get"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get"] - nonResourceURLs: ["/metrics"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["list", "watch", "get"] ``` **Role (namespace-scoped permissions for amazon-cloudwatch namespace):** ``` rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["create", "update"] ``` ### Use `kubeconfig` for authentication Alternatively, you can use a `kubeconfig` file for authentication. This method allows you to bypass the need for a service account b directly specifying the `kubeconfig` path in your CloudWatch agent configuration. It also allows you to remove your dependency on the Kubernetes control plane API for authentication, streamlining your setup and potentially increasing security by managing authentication through your kubeconfig file. To use this method, update your CloudWatch agent configuration file to specify the path to your `kubeconfig` file, as in the following example. ``` { "logs": { "metrics_collected": { "kubernetes": { "cluster_name": "YOUR_CLUSTER_NAME", "enhanced_container_insights": false, "accelerated_compute_metrics": false, "tag_service": false, "kube_config_path": "/path/to/your/kubeconfig" "host_ip": "HOSTIP" } } } } ``` To create a `kubeconfig` file, create a Certificate Signing Request (CSR) for the `admin/{create_your_own_user}` user with the `system:masters` Kubernetes role. Then sign with Kubernetes cluster’s Certificate Authority (CA) and create the `kubeconfig` file. ## Step 3: Create a ConfigMap for the CloudWatch agent Use the following steps to create a ConfigMap for the CloudWatch agent. **To create a ConfigMap for the CloudWatch agent** 1. Download the ConfigMap YAML to your `kubectl` client host by running the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap-enhanced.yaml ``` 1. Edit the downloaded YAML file, as follows: + **cluster\$1name** – In the `kubernetes` section, replace `{{cluster_name}}` with the name of your cluster. Remove the `{{}}` characters. Alternatively, if you're using an Amazon EKS cluster, you can delete the `"cluster_name"` field and value. If you do, the CloudWatch agent detects the cluster name from the Amazon EC2 tags. 1. (Optional) Make further changes to the ConfigMap based on your monitoring requirements, as follows: + **metrics\$1collection\$1interval** – In the `kubernetes` section, you can specify how often the agent collects metrics. The default is 60 seconds. The default cadvisor collection interval in kubelet is 15 seconds, so don't set this value to less than 15 seconds. + **endpoint\$1override** – In the `logs` section, you can specify the CloudWatch Logs endpoint if you want to override the default endpoint. You might want to do this if you're publishing from a cluster in a VPC and you want the data to go to a VPC endpoint. + **force\$1flush\$1interval** – In the `logs` section, you can specify the interval for batching log events before they are published to CloudWatch Logs. The default is 5 seconds. + **region** – By default, the agent published metrics to the Region where the worker node is located. To override this, you can add a `region` field in the `agent` section: for example, `"region":"us-west-2"`. + **statsd** section – If you want the CloudWatch Logs agent to also run as a StatsD listener in each worker node of your cluster, you can add a `statsd` section to the `metrics` section, as in the following example. For information about other StatsD options for this section, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md). ``` "metrics": { "metrics_collected": { "statsd": { "service_address":":8125" } } } ``` A full example of the JSON section is as follows. If you're using a `kubeconfig` file for authentication, add the `kube_config_path` parameter to specify the path to your kubeconfig file. ``` { "agent": { "region": "us-east-1" }, "logs": { "metrics_collected": { "kubernetes": { "cluster_name": "MyCluster", "metrics_collection_interval": 60, "kube_config_path": "/path/to/your/kubeconfig" //if using kubeconfig for authentication } }, "force_flush_interval": 5, "endpoint_override": "logs.us-east-1.amazonaws.com" }, "metrics": { "metrics_collected": { "statsd": { "service_address": ":8125" } } } } ``` 1. Create the ConfigMap in the cluster by running the following command. ``` kubectl apply -f cwagent-configmap-enhanced.yaml ``` ## Step 4: Deploy the CloudWatch agent as a DaemonSet To finish the installation of the CloudWatch agent and begin collecting container metrics, use the following steps. **To deploy the CloudWatch agent as a DaemonSet** 1. + If you do not want to use StatsD on the cluster, enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml ``` + If you do want to use StatsD, follow these steps: 1. Download the DaemonSet YAML to your `kubectl` client host by running the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml ``` 1. Uncomment the `port` section in the `cwagent-daemonset.yaml` file as in the following: ``` ports: - containerPort: 8125 hostPort: 8125 protocol: UDP ``` 1. Deploy the CloudWatch agent in your cluster by running the following command. ``` kubectl apply -f cwagent-daemonset.yaml ``` 1. Deploy the CloudWatch agent on Windows nodes in your cluster by running the following command. The StatsD listener is not supported on the CloudWatch agent on Windows. ``` kubectl apply -f cwagent-daemonset-windows.yaml ``` 1. Validate that the agent is deployed by running the following command. ``` kubectl get pods -n amazon-cloudwatch ``` When complete, the CloudWatch agent creates a log group named `/aws/containerinsights/Cluster_Name/performance` and sends the performance log events to this log group. If you also set up the agent as a StatsD listener, the agent also listens for StatsD metrics on port 8125 with the IP address of the node where the application pod is scheduled. ### Troubleshooting If the agent doesn't deploy correctly, try the following: + Run the following command to get the list of pods. ``` kubectl get pods -n amazon-cloudwatch ``` + Run the following command and check the events at the bottom of the output. ``` kubectl describe pod pod-name -n amazon-cloudwatch ``` + Run the following command to check the logs. ``` kubectl logs pod-name -n amazon-cloudwatch ```