# Relevant fields in performance log events for Amazon EKS and Kubernetes


For Amazon EKS and Kubernetes, the containerized CloudWatch agent emits data as performance log events. This enables CloudWatch to ingest and store high-cardinality data. CloudWatch uses the data in the performance log events to create aggregated CloudWatch metrics at the cluster, node, and pod levels without the need to lose granular details.

The following table lists the fields in these performance log events that are relevant to the collection of Container Insights metric data. You can use CloudWatch Logs Insights to query for any of these fields to collect data or investigate issues. For more information, see [Analyze Log Data With CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html).


| Type | Log field | Source | Formula or notes | 
| --- | --- | --- | --- | 
|  Pod |  `pod_cpu_utilization`  |  Calculated  |  Formula: `pod_cpu_usage_total / node_cpu_limit`  | 
|  Pod |  `pod_cpu_usage_total` `pod_cpu_usage_total` is reported in millicores.  |  cadvisor  |   | 
|  Pod |  `pod_cpu_limit`  |  Calculated  |  Formula: `sum(container_cpu_limit)`  `sum(container_cpu_limit)` includes already-completed pods. If any containers in the pod don't have a CPU limit defined, this field doesn't appear in the log event. This includes [ init containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resources).  | 
|  Pod |  `pod_cpu_request`  |  Calculated  |  Formula: `sum(container_cpu_request)` `container_cpu_request` isn't guaranteed to be set. Only the ones that are set are included in the sum.  | 
|  Pod |  `pod_cpu_utilization_over_pod_limit`  |  Calculated  |  Formula: `pod_cpu_usage_total / pod_cpu_limit`  | 
|  Pod |  `pod_cpu_reserved_capacity`  |  Calculated  |  Formula: `pod_cpu_request / node_cpu_limit`  | 
|  Pod |  `pod_memory_utilization`  |  Calculated  |  Formula: `pod_memory_working_set / node_memory_limit` It is the percentage of pod memory usage over the node memory limitation.  | 
|  Pod |  `pod_memory_working_set`  |  cadvisor  |   | 
|  Pod |  `pod_memory_limit`  |  Calculated  |  Formula: `sum(container_memory_limit)` If any containers in the pod don't have a memory limit defined, this field doesn't appear in the log event. This includes [ init containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resources).  | 
|  Pod |  `pod_memory_request`  |  Calculated  |  Formula: `sum(container_memory_request)` `container_memory_request` isn't guaranteed to be set. Only the ones that are set are included in the sum.  | 
|  Pod |  `pod_memory_utilization_over_pod_limit`  |  Calculated  |  Formula: `pod_memory_working_set / pod_memory_limit` If any containers in the pod don't have a memory limit defined, this field doesn't appear in the log event. This includes [ init containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resources).  | 
|  Pod |  `pod_memory_reserved_capacity`  |  Calculated  |  Formula: `pod_memory_request / node_memory_limit`  | 
|  Pod |  `pod_network_tx_bytes`  |  Calculated  |  Formula: `sum(pod_interface_network_tx_bytes)` This data is available for all the network interfaces per pod. The CloudWatch agent calculates the total and adds metric extraction rules.  | 
|  Pod |  `pod_network_rx_bytes`  |  Calculated  |  Formula: `sum(pod_interface_network_rx_bytes)`  | 
|  Pod |  `pod_network_total_bytes`  |  Calculated  |  Formula: `pod_network_rx_bytes + pod_network_tx_bytes`  | 
|  PodNet |  `pod_interface_network_rx_bytes`  |  cadvisor  | This data is network rx bytes per second of a pod network interface.  | 
|  PodNet |  `pod_interface_network_tx_bytes`  |  cadvisor  | This data is network tx bytes per second of a pod network interface. | 
|  Container |  `container_cpu_usage_total`  |  cadvisor  |   | 
|  Container |  `container_cpu_limit`  |  cadvisor  |  Not guaranteed to be set. It's not emitted if it's not set. | 
|  Container |  `container_cpu_request`  |  cadvisor  |  Not guaranteed to be set. It's not emitted if it's not set. | 
|  Container |  `container_memory_working_set`  |  cadvisor  |   | 
|  Container |  `container_memory_limit`  |  pod  |  Not guaranteed to be set. It's not emitted if it's not set. | 
|  Container |  `container_memory_request`  |  pod  |  Not guaranteed to be set. It's not emitted if it's not set. | 
|  Node |  `node_cpu_utilization`  |  Calculated  |  Formula: `node_cpu_usage_total / node_cpu_limit`  | 
|  Node |  `node_cpu_usage_total`  |  cadvisor  |   | 
|  Node |  `node_cpu_limit`  |  /proc  |   | 
|  Node |  `node_cpu_request`  |  Calculated  | Formula: `sum(pod_cpu_request)` For cronjobs,`node_cpu_request` also includes requests from completed pods. This can lead to a high value for `node_cpu_reserved_capacity`.  | 
|  Node |  `node_cpu_reserved_capacity`  |  Calculated  | Formula: `node_cpu_request / node_cpu_limit`  | 
|  Node |  `node_memory_utilization`  |  Calculated  | Formula: `node_memory_working_set / node_memory_limit`  | 
|  Node |  `node_memory_working_set`  |  cadvisor  |   | 
|  Node |  `node_memory_limit`  |  /proc  |   | 
|  Node |  `node_memory_request`  |  Calculated  |  Formula: `sum(pod_memory_request)`  | 
|  Node |  `node_memory_reserved_capacity`  |  Calculated  | Formula: `node_memory_request / node_memory_limit`  | 
|  Node |  `node_network_rx_bytes`  |  Calculated  | Formula: `sum(node_interface_network_rx_bytes)`  | 
|  Node |  `node_network_tx_bytes`  |  Calculated  | Formula: `sum(node_interface_network_tx_bytes)`  | 
|  Node |  `node_network_total_bytes`  |  Calculated  | Formula: `node_network_rx_bytes + node_network_tx_bytes`  | 
|  Node |  `node_number_of_running_pods`  |  Pod List  |   | 
|  Node |  `node_number_of_running_containers`  |  Pod List  |   | 
|  NodeNet |  `node_interface_network_rx_bytes`  |  cadvisor  |  This data is network rx bytes per second of a worker node network interface.  | 
|  NodeNet |  `node_interface_network_tx_bytes`  |  cadvisor  |  This data is network tx bytes per second of a worker node network interface.  | 
|  NodeFS |  `node_filesystem_capacity`  |  cadvisor  |   | 
|  NodeFS |  `node_filesystem_usage`  |  cadvisor  |   | 
|  NodeFS |  `node_filesystem_utilization`  |  Calculated  |  Formula: `node_filesystem_usage / node_filesystem_capacity` This data is available per device name.  | 
|  Cluster |  `cluster_failed_node_count`  |  API Server  |   | 
|  Cluster |  `cluster_node_count`  |  API Server  |   | 
|  Service |  `service_number_of_running_pods`  |  API Server  |   | 
|  `Namespace` |  `namespace_number_of_running_pods`  |  API Server  |   | 

## Metrics calculation examples


This section includes examples that show how some of the values in the preceding table are calculated.

Suppose that you have a cluster in the following state.

```
Node1
   node_cpu_limit = 4
   node_cpu_usage_total = 3
   
   Pod1
     pod_cpu_usage_total = 2
     
     Container1
        container_cpu_limit = 1
        container_cpu_request = 1
        container_cpu_usage_total = 0.8
        
     Container2
        container_cpu_limit = null
        container_cpu_request = null
        container_cpu_usage_total = 1.2
        
   Pod2
     pod_cpu_usage_total = 0.4
     
     Container3
        container_cpu_limit = 1
        container_cpu_request = 0.5
        container_cpu_usage_total = 0.4
        
Node2
   node_cpu_limit = 8
   node_cpu_usage_total = 1.5
   
   Pod3
     pod_cpu_usage_total = 1
     
     Container4
        container_cpu_limit = 2
        container_cpu_request = 2
        container_cpu_usage_total = 1
```

The following table shows how pod CPU metrics are calculated using this data.


| Metric | Formula | Pod1 | Pod2 | Pod3 | 
| --- | --- | --- | --- | --- | 
|  `pod_cpu_utilization` |  `pod_cpu_usage_total / node_cpu_limit`  |  2 / 4 = 50%  |  0.4 / 4 = 10%  |  1 / 8 = 12.5%  | 
|  `pod_cpu_utilization_over_pod_limit` |  `pod_cpu_usage_total / sum(container_cpu_limit)`  |  N/A because CPU limit for `Container2` isn't defined  |  0.4 / 1 = 40%  |  1 / 2 = 50%  | 
|  `pod_cpu_reserved_capacity` |  `sum(container_cpu_request) / node_cpu_limit`  |  (1 \$1 0) / 4 = 25%  |  0.5 / 4 = 12.5%  |  2 / 8 = 25%  | 

The following table shows how node CPU metrics are calculated using this data.


| Metric | Formula | Node1 | Node2 | 
| --- | --- | --- | --- | 
|  `node_cpu_utilization` |  `node_cpu_usage_total / node_cpu_limit`  |  3 / 4 = 75%  |  1.5 / 8 = 18.75%  | 
|  `node_cpu_reserved_capacity` |  `sum(pod_cpu_request) / node_cpu_limit`  |  1.5 / 4 = 37.5%  |  2 / 8 = 25%  |