Tutorial for adding a new Prometheus scrape target: Redis OSS on Amazon EKS and Kubernetes clusters

Set the namespace for the sample Redis OSS workload.

REDIS_NAMESPACE=redis-sample

If you are running Redis OSS on a cluster with the Fargate launch type, you need to set up a Fargate profile. To set up the profile, enter the following command. Replace MyCluster with the name of your cluster.

eksctl create fargateprofile --cluster MyCluster \
--namespace $REDIS_NAMESPACE --name $REDIS_NAMESPACE

Enter the following command to install the sample Redis OSS workload.

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/redis/redis-traffic-sample.yaml \
| sed "s/{{namespace}}/$REDIS_NAMESPACE/g" \
| kubectl apply -f -

The installation includes a service named my-redis-metrics which exposes the Redis OSS Prometheus metric on port 9121 Enter the following command to get the details of the service:

kubectl describe service/my-redis-metrics  -n $REDIS_NAMESPACE

In the Annotations section of the results, you'll see two annotations which match the Prometheus scrape configuration of the CloudWatch agent, so that it can auto-discover the workloads:

prometheus.io/port: 9121
prometheus.io/scrape: true

The related Prometheus scrape configuration can be found in the - job_name: kubernetes-service-endpoints section of kubernetes-eks.yaml or kubernetes-k8s.yaml.

Download the latest version of the of kubernetes-eks.yaml or kubernetes-k8s.yaml file by entering one of the following commands. For an Amazon EKS cluster with the EC2 launch type, enter this command.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml

For an Amazon EKS cluster with the Fargate launch type, enter this command.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml

For a Kubernetes cluster running on an Amazon EC2 instance, enter this command.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-k8s.yaml

Open the file with a text editor, and find the cwagentconfig.json section. Add the following subsection and save the changes. Be sure that the indentation follows the existing pattern.

{
  "source_labels": ["pod_name"],
  "label_matcher": "^redis-instance$",
  "dimensions": [["Namespace","ClusterName"]],
  "metric_selectors": [
    "^redis_net_(in|out)put_bytes_total$",
    "^redis_(expired|evicted)_keys_total$",
    "^redis_keyspace_(hits|misses)_total$",
    "^redis_memory_used_bytes$",
    "^redis_connected_clients$"
  ]
},
{
  "source_labels": ["pod_name"],
  "label_matcher": "^redis-instance$",
  "dimensions": [["Namespace","ClusterName","cmd"]],
  "metric_selectors": [
    "^redis_commands_total$"
  ]
},
{
  "source_labels": ["pod_name"],
  "label_matcher": "^redis-instance$",
  "dimensions": [["Namespace","ClusterName","db"]],
  "metric_selectors": [
    "^redis_db_keys$"
  ]
},

The section you added puts the Redis OSS metrics onto the CloudWatch agent allow list. For a list of these metrics, see the following section.

If you already have the CloudWatch agent with Prometheus support deployed in this cluster, you must delete it by entering the following command.

kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch

Deploy the CloudWatch agent with your updated configuration by entering one of the following commands. Replace MyCluster and region to match your settings.

For an Amazon EKS cluster with the EC2 launch type, enter this command.

kubectl apply -f prometheus-eks.yaml

For an Amazon EKS cluster with the Fargate launch type, enter this command.

cat prometheus-eks-fargate.yaml \
| sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \
| kubectl apply -f -

For a Kubernetes cluster, enter this command.

cat prometheus-k8s.yaml \
| sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \
| kubectl apply -f -

Create environment variables, replacing the values below to match your deployment.

DASHBOARD_NAME=your_cw_dashboard_name
REGION_NAME=your_metric_region_such_as_us-east-1
CLUSTER_NAME=your_k8s_cluster_name_here
NAMESPACE=your_redis_service_namespace_here

Enter the following command to create the dashboard.

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/redis/cw_dashboard_redis.json \
| sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \
| sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \
| sed "s/{{YOUR_NAMESPACE}}/${NAMESPACE}/g" \