# Work with EKS
Using Network Flow Monitor, you can collect performance metrics for workloads that use Amazon Elastic Kubernetes Service (Amazon EKS). This chapter shows you how to install the agent step-by-step and describes the different EKS scenarios you can monitor. You'll also find detailed descriptions of the metadata that Network Flow Monitor provides for Amazon EKS in the console to help you understand network performance.
To gain the benefits of Network Flow Monitor performance monitoring, you must first install the AWS Network Flow Monitor Agent add-on for Amazon EKS. For more information, see [Install the EKS AWS Network Flow Monitor Agent add-on](CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks.md).
If you want to monitor a single EKS cluster with enhanced visibility of workload traffic and performance insights within the cluster and with external destinations, see [Amazon EKS Network Observability](https://docs.aws.amazon.com/eks/latest/userguide/network-observability.html).
**Topics**
+ [Install the EKS AWS Network Flow Monitor Agent add-on](CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks.md)
+ [Additional network path metadata included for Amazon EKS](CloudWatch-NetworkFlowMonitor-work-with-eks.performance-metadata.md)
# Install the EKS AWS Network Flow Monitor Agent add-on
Follow the steps in this section to install the AWS Network Flow Monitor Agent add-on for Amazon Elastic Kubernetes Service (Amazon EKS), to send Network Flow Monitor metrics to the Network Flow Monitor backend from Kubernetes clusters. After you complete the steps, AWS Network Flow Monitor Agent pods will be running on all of your Kubernetes cluster nodes.
If you use self-managed Kubernetes clusters, the installation steps to follow are in the previous section: [Install agents for self-managed Kubernetes instances](CloudWatch-NetworkFlowMonitor-agents-kubernetes-non-eks.md).
Be aware that Customer Managed prefix lists [Customer Managed prefix lists](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-managed-prefix-lists.html) are not supported for Network Flow Monitor.
You can install the add-on by using the console or by using API commands with the AWS Command Line Interface.
**Topics**
+ [Prerequisites for installing the add-on](#CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks-prereq)
+ [Install the add-on by using the console](#CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks-console)
+ [Install the add-on by using the CLI](#CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks-cli)
+ [Configure for third party tools](CloudWatch-NetworkFlowMonitor-agents-kubernetes-eks-third-party.md)
## Prerequisites for installing the add-on
Regardless of whether you use the console or the CLI to install the AWS Network Flow Monitor Agent add-on, there are several requirements for installing the add-on with Kubernetes.
**Ensure that your version of Kubernetes is supported**
Network Flow Monitor agent installation requires Kubernetes Version 1.25, or a more recent version.
**Amazon EKS Pod Identity Agent add-on installation**
You can install the Amazon EKS Pod Identity Agent add-on in the console or by using the CLI.
Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Amazon EKS Pod Identity provides credentials to your workloads with an additional Amazon EKS Auth API and an agent pod that runs on each node.
To learn more and see the steps for installing the Amazon EKS Pod Identity add-on, see [Set up the Amazon EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html) in the Amazon EKS User Guide.
## Install the AWS Network Flow Monitor Agent add-on by using the console
Follow the steps in this section to install and configure the AWS Network Flow Monitor Agent add-on in the Amazon EKS console.
If you have already installed the add-on and have issues upgrading to a new version, see [Troubleshoot issues in EKS agents installation](CloudWatch-NetworkFlowMonitor-troubleshooting.md#CloudWatch-NetworkFlowMonitor-troubleshooting.ec2-agent-installation).
Before you begin, make sure that you have installed the Amazon EKS Pod Identity Agent add-on. For more information, see the [previous section](#NFMPodIdentity).
**To install the add-on using the console**
1. In the AWS Management Console, navigate to the Amazon EKS console.
1. On the page for installing add-ons, in the list of add-ons, choose **AWS Network Flow Monitor Agent**.
1. Configure the add-on settings.
1. For **Add-on access**, choose **EKS Pod Identity**.
1. For the IAM role to use with the add-on, use a role that has the following AWS managed policy attached: [CloudWatchNetworkFlowMonitorAgentPublishPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchNetworkFlowMonitorAgentPublishPolicy.html). This policy gives permission for an agent to send telemetry reports to the Network Flow Monitor endpoint. If you don't already have a role with the policy attached, create a role by choosing **Create recommended role**, and following the steps in the IAM console.
1. Choose **Next**.
1. On the **Review and add** page, make sure that the add-on configuration looks correct, and then choose **Create**.
## Install the AWS Network Flow Monitor Agent add-on by using the AWS Command Line Interface
Follow the steps in this section to install the AWS Network Flow Monitor Agent add-on for Amazon EKS by using the AWS Command Line Interface.
**1. Install the EKS Pod Identity Agent add-on**
Before you begin, make sure that you have installed the Amazon EKS Pod Identity Agent add-on. For more information, see the [earlier section](#NFMPodIdentity).
**2. Create the required IAM role**
The AWS Network Flow Monitor Agent add-on must have permission to send metrics to the Network Flow Monitor backend. You must attach a role with the required permissions when you create the add-on. Create a role that has the following AWS managed policy attached: [CloudWatchNetworkFlowMonitorAgentPublishPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchNetworkFlowMonitorAgentPublishPolicy.html). You need the ARN of this IAM role to install the add-on.
**3. Install the AWS Network Flow Monitor Agent add-on**
To install the AWS Network Flow Monitor Agent add-on for your cluster, run the following command:
`aws eks create-addon --cluster-name CLUSTER NAME --addon-name aws-network-flow-monitoring-agent --region AWS REGION --pod-identity-associations serviceAccount=aws-network-flow-monitor-agent-service-account,roleArn=IAM ROLE ARN`
The result should be similar to the following:
```
{
"addon": {
"addonName": "aws-network-flow-monitoring-agent",
"clusterName": "ExampleClusterName",
"status": "CREATING",
"addonVersion": "v1.0.0-eksbuild.1",
"health": {
"issues": []
},
"addonArn": "arn:aws:eks:us-west-2:000000000000:addon/ExampleClusterName/aws-network-flow-monitoring-agent/eec11111-bbbb-EXAMPLE",
"createdAt": "2024-10-25T16:38:07.213000+00:00",
"modifiedAt": "2024-10-25T16:38:07.240000+00:00",
"tags": {},
"podIdentityAssociations": [
"arn:aws:eks:us-west-2:000000000000:podidentityassociation/ExampleClusterName/a-3EXAMPLE5555555"
]
}
}
```
**4. Make sure that the add-on is active**
Review the installed AWS Network Flow Monitor Agent add-on to ensure that it's active for your cluster. Run the following command to verify that the status is `ACTIVE`:
`aws eks describe-addon --cluster-name CLUSTER NAME --addon-name aws-network-flow-monitoring-agent --region AWS REGION`
The result should be similar to the following:
```
{
"addon": {
"addonName": "aws-network-flow-monitoring-agent",
"clusterName": "ExampleClusterName",
"status": "ACTIVE",
"addonVersion": "v1.0.0-eksbuild.1",
"health": {
"issues": []
},
"addonArn": "arn:aws:eks:us-west-2:000000000000:addon/ExampleClusterName/aws-network-flow-monitoring-agent/eec11111-bbbb-EXAMPLE",
"createdAt": "2024-10-25T16:38:07.213000+00:00",
"modifiedAt": "2024-10-25T16:38:07.240000+00:00",
"tags": {},
"podIdentityAssociations": [
"arn:aws:eks:us-west-2:000000000000:podidentityassociation/ExampleClusterName/a-3EXAMPLE5555555"
]
}
}
```
# Configure add-on for third party monitoring tools
You can configure the Network Flow Monitor add-on to expose an OpenMetrics server during installation. This enables integration with third-party monitoring tools such as Prometheus, allowing you to collect and analyze network flow metrics alongside your existing monitoring infrastructure. [Learn more about about OpenMetrics](https://openmetrics.io/). This feature is available from add-on version v1.1.0.
To enable the OpenMetrics server, add OPEN\$1METRICS, OPEN\$1METRICS\$1ADDRESS, and OPEN\$1METRICS\$1PORT to the configuration values of the EKS Network Flow Monitor add-on. This guide will explain how to do this using both CLI and Console. See [Amazon EKS add-ons advanced configuration](https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-advanced-configuration/) for additional details about adding configuration values.
## CLI Configuration
When using AWS Command Line Interface, you can provide the configuration values as a parameter:
```
aws eks create-addon \
--cluster-name my-cluster-name \
--addon-name aws-network-flow-monitoring-agent \
--addon-version v1.1.0-eksbuild.1 \
--configuration-values '{"env":{"OPEN_METRICS":"on","OPEN_METRICS_ADDRESS":"0.0.0.0","OPEN_METRICS_PORT":9109}}'
```
## Console Configuration
When using the Amazon EKS Console, these values can be added under Optional configuration settings, as part of the Configuration values.
**Sample JSON:**
```
{
"env": {
"OPEN_METRICS": "on",
"OPEN_METRICS_ADDRESS": "0.0.0.0",
"OPEN_METRICS_PORT": 9109
}
}
```
**Sample YAML:**
```
env:
OPEN_METRICS: "on"
OPEN_METRICS_ADDRESS: "0.0.0.0"
OPEN_METRICS_PORT: 9109
```
## EKS Network Flow Monitor add-on OpenMetric Parameters
+ **OPEN\$1METRICS:**
+ Enable or disable open metrics. Disabled if not supplied
+ Type: String
+ Values: ["on", "off"]
+ **OPEN\$1METRICS\$1ADDRESS:**
+ Listening IP address for open metrics endpoint. Defaults to 127.0.0.1 if not supplied
+ Type: String
+ **OPEN\$1METRICS\$1PORT:**
+ Listening port for open metrics endpoint. Defaults to 80 if not supplied
+ Type: Integer
+ Range: [0..65535]
**Important:** When setting OPEN\$1METRICS\$1ADDRESS to 0.0.0.0, the metrics endpoint will be accessible from any network interface. Consider using 127.0.0.1 for localhost-only access or implement appropriate network security controls to restrict access to authorized monitoring systems only.
## Troubleshooting
If you encounter issues with the OpenMetrics server configuration, use the following information to diagnose and resolve common problems.
### Metrics not displaying
Problem: The OpenMetrics server is configured, but metrics are not appearing in your monitoring tool.
Troubleshooting Steps:
1. Verify that the OpenMetrics server is enabled in your add-on configuration:
+ Check that OPEN\$1METRICS is set to "on" in your configuration values. See [describe-addon](https://docs.aws.amazon.com/cli/latest/reference/eks/describe-addon.html).
+ Confirm that the add-on version is v1.1.0 or later in the *Configure selected add-ons settings*.
1. Test the metrics endpoint directly:
+ Access the metrics at http://*pod-ip:port*/metrics (replace pod-ip with the actual pod IP address and port with your configured port).
+ If you can't access the endpoint, verify your network configuration and security group settings.
1. Validate your monitoring tool configuration. Consult you monitoring tools user guide for details on how to do the following:
+ Ensure your monitoring tool (such as Prometheus) is configured to scrape the correct endpoint.
+ Check that the scraping interval and timeout settings are appropriate.
+ Verify that your monitoring tool has network access to the pod IP address.
### Metrics missing from specific pods
Problem: Metrics are available from some pods but not others in your cluster.
Troubleshooting Steps:
The Network Flow Monitor add-on doesn't support pods that use hostNetwork: true. If your pod specification includes this setting, metrics won't be available from those pods.
Workaround: Remove the hostNetwork: true setting from your pod specification if possible. If you require host networking for your application, consider using alternative monitoring approaches for those specific pods.
### Connection refused errors
Problem: You receive "connection refused" errors when trying to access the metrics endpoint.
Troubleshooting Steps:
1. Verify the OPEN\$1METRICS\$1ADDRESS configuration:
+ If set to 127.0.0.1, the endpoint is only accessible from within the pod.
+ If set to 0.0.0.0, the endpoint should be accessible from other pods in the cluster.
+ Ensure your monitoring tool can reach the configured address.
1. Check the OPEN\$1METRICS\$1PORT configuration:
+ Verify that the port number is not already in use by another service.
+ Ensure the port is within the valid range (1-65535).
+ Confirm that any security groups or network policies allow traffic on this port.
### Verification steps
To confirm your OpenMetrics configuration is working correctly:
1. Check the add-on status:
```
aws eks describe-addon --cluster-name your-cluster-name --addon-name aws-network-flow-monitoring-agent
```
1. Verify pod status:
```
kubectl get pods app.kubernetes.io/name=aws-network-flow-monitoring-agent
```
1. Test the metrics endpoint from within the cluster:
```
kubectl exec add-on-pod-name -- curl localhost:9109/metrics
```
Replace 9109 with your configured port number, and the pod name with an AddOn pod name.
# Additional network path metadata included for Amazon EKS
When Network Flow Monitor gathers performance metrics for network flows between Amazon EKS components, it includes additional metadata information about the network path, to help you better understand how the network paths for your workload are performing.
You can view detailed information about Amazon EKS network flow performance by creating a monitor for the network flows that you're interested in, and then viewing details on the **Historical explorer** tab.
With Network Flow Monitor, you can measure network performance between the following Amazon EKS components, to better understand how your workload is performing with your Amazon EKS configuration and determine where there are bottlenecks or impairments.
+ Pod to pod on the same node
+ Node to node on the same cluster
+ Pod to pod on a different cluster
+ Node to node on different clusters
+ With and without Network Load Balancer
The following table lists the information that Network Flow Monitor returns for each network flow scenario.
| **Connection information** | **Metadata information** | | **Local** | **Remote** | **Scenario** | **Initiated by** | **Local** | **Remote** | **Pod name** | **Service** | **Namespace** | **Pod name** | **Service** | **Namespace** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Local pod connecting to cluster IP of another internal cluster service | Local | Local pod IP address | Remote pod IP address (through cluster IP address) | ✓ | ✓ | ✓ | ✓ ¹ | ✓ | ✓ |
| Local pod in a node network namespace connecting to cluster IP of another internal cluster service | Local | Local node IP address | Remote pod IP address (through cluster IP address) | ✓ ² | ✓ ² | ✓ ² | ✓ ¹ | ✓ | ✓ |
| Local pod connecting to individual pod IP address of another pod (headless service) | Local | Local pod IP address | Remote pod IP address | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Local pod connecting to individual pod IP address of another pod in node network namespace (headless service) | Local | Local pod IP address | Remote node IP address | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Local pod connecting to remote pod in another cluster | Local | Local pod IP address | Remote pod IP address (another cluster) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Local pod connecting to an external network address | Local | Local pod IP address | External IP address | ✓ | ✓ | ✓ | N/A | N/A | N/A |
| Local pod operating in a node network namespace connecting to an external network IP address | Local | Local node IP address | External IP address | ✓ ² | ✓ ² | ✓ ² | N/A | N/A | N/A |
| Remote pod connecting to local pod through cluster IP address | Remote | Local pod IP address (through cluster IP address) | Remote pod IP address | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Remote pod in a node network namespace connecting to local pod | Remote | Local pod IP address (through cluster IP address) | Remote node IP address | ✓ | ✓ | ✓ | ✓ ³ | ✓ ³ | ✓ ³ |
| Remote pod connecting to local pod (headless service) | Remote | Local pod IP address | Remote pod IP address | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| External pod connecting to a local pod | Remote | Local pod IP address | Remote pod IP address | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| External resource connecting through NodePort or a Load Balancer to a local pod | Remote | Local pod IP address | External IP address ⁴ | ✓ | ✓ | ✓ | N/A | N/A | N/A |
| External resource connecting through NodePort or a Load Balancer to a local pod operating in a node network namespace | Remote | Local node IP address | External IP address ⁴ | ✓ | ✓ | ✓ | N/A | N/A | N/A |
Be aware of the following additional information corresponding to the items marked with footnotes in the preceding table.
1. Pod name is not visible in this scenario for pods with other owners, such as a Kubernetes service managed by the EKS control plane.
1. Local pod name, service, and namespace are not resolved if other pods are present in node network namespace.
1. Remote pod name, service, and namespace are not resolved if other pods are present in node network namespace.
1. If service is using NodePort or LoadBalancer in instance mode, and `ExternalTrafficPolicy` is set to `Cluster`, then this IP address will be reported as the IP address of the node that receives the NodePort connection.