# App Mesh observability troubleshooting
**Important**
End of support notice: On September 30, 2026, AWS will discontinue support for AWS App Mesh. After September 30, 2026, you will no longer be able to access the AWS App Mesh console or AWS App Mesh resources. For more information, visit this blog post [Migrating from AWS App Mesh to Amazon ECS Service Connect](https://aws.amazon.com/blogs/containers/migrating-from-aws-app-mesh-to-amazon-ecs-service-connect).
This topic details common issues that you may experience with App Mesh observability.
## Unable to see AWS X-Ray traces for my applications
**Symptoms**
Your application in App Mesh is not displaying X-Ray tracing information in the X-Ray console or APIs.
**Resolution**
To use X-Ray in App Mesh, you must correctly configure components to enable communication between your application, sidecar containers, and the X-Ray service. Take the following steps to confirm that X-Ray has been set up correctly:
+ Make sure the App Mesh Virtual Node listener protocol is not set as `TCP`.
+ Make sure that the X-Ray container that is deployed with your application exposes UDP port `2000` and runs as user `1337`. For more information, see the [Amazon ECS X-Ray example](https://github.com/aws/aws-app-mesh-examples/blob/main/walkthroughs/howto-ecs-basics/deploy/2-meshify.yaml#L374-L386) on GitHub.
+ Make sure that the Envoy container has tracing enabled. If you are using the [App Mesh Envoy image](envoy.md), you can enable X-Ray by setting the `ENABLE_ENVOY_XRAY_TRACING` environment variable to a value of `1` and the `XRAY_DAEMON_PORT` environment variable to `2000`.
+ If you’ve instrumented X-Ray in your application code with one of the [language-specific SDKs ](https://docs.aws.amazon.com/xray/index.html), then make sure that it is configured correctly by following the guides for your language.
+ If all of the previous items are configured correctly, then review the X-Ray container logs for errors and follow the guidance in [Troubleshooting AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-troubleshooting.html). A more detailed explanation of X-Ray integration in App Mesh can be found in [Integrating X-Ray with App Mesh](https://aws.amazon.com/blogs/compute/integrating-aws-x-ray-with-aws-app-mesh/).
If your issue is still not resolved, then consider opening a [GitHub issue](https://github.com/aws/aws-app-mesh-roadmap/issues/new?assignees=&labels=Bug&template=issue--bug-report.md&title=Bug%3A+describe+bug+here) or contact [AWS Support](https://aws.amazon.com/premiumsupport/).
## Unable to see Envoy metrics for my applications in Amazon CloudWatch metrics
**Symptoms**
Your application in App Mesh is not emitting metrics generated by the Envoy proxy to CloudWatch metrics.
**Resolution**
When you use CloudWatch metrics in App Mesh, you must correctly configure several components to enable communication between your Envoy proxy, CloudWatch agent sidecar, and the CloudWatch metrics service. Take the following steps to confirm that CloudWatch metrics for Envoy proxy have been setup correctly:
+ Make sure that you are using the CloudWatch agent image for App Mesh. For more information, see [App Mesh CloudWatch agent](https://github.com/aws-samples/aws-app-mesh-cloudwatch-agent) on GitHub.
+ Make sure that you have configured the CloudWatch agent for App Mesh appropriately by following the platform-specific usage instructions. For more information, see [App Mesh CloudWatch agent](https://github.com/aws-samples/aws-app-mesh-cloudwatch-agent#usage) on GitHub.
+ If all of the previous items are configured correctly, then review the CloudWatch agent container logs for errors and follow the guidance provided in [Troubleshooting the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html).
If your issue is still not resolved, then consider opening a [GitHub issue](https://github.com/aws/aws-app-mesh-roadmap/issues/new?assignees=&labels=Bug&template=issue--bug-report.md&title=Bug%3A+describe+bug+here) or contact [AWS Support](https://aws.amazon.com/premiumsupport/).
## Unable to configure custom sampling rules for AWS X-Ray traces
**Symptoms**
Your application is using X-Ray tracing, but you are unable to configure sampling rules for your traces.
**Resolution**
Since App Mesh Envoy currently does not support **Dynamic X-Ray sampling configuration**, the following workarounds are available.
If your Envoy version is `1.19.1` or later, you have the following options.
+ To only set the sampling rate, use the `XRAY_SAMPLING_RATE` environment variable on the Envoy container. The value should be specified as a decimal between `0` and `1.00` (100%). For more information, see [AWS X-Ray variables](envoy-config.md#envoy-xray-config).
+ To configure the localized custom sampling rules for the X-Ray tracer use the `XRAY_SAMPLING_RULE_MANIFEST` environment variable to specify a file path in the Envoy container file system. For more information, see [Sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-go-configuration.html#xray-sdk-go-configuration-sampling) in the *AWS X-Ray Developer Guide*.
If your Envoy version is prior to `1.19.1`, then do the following.
+ Use the `ENVOY_TRACING_CFG_FILE` environment variable to change your sampling rate. For more information, see [Envoy configuration variables](envoy-config.md). Specify a custom tracing configuration and define local sampling rules. For more information, see [Envoy X-Ray config](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/trace/v3/xray.proto.html#config-trace-v3-xrayconfig).
+ Custom tracing configuration for the `ENVOY_TRACING_CFG_FILE` environment variable example:
```
tracing:
http:
name: envoy.tracers.xray
typedConfig:
"@type": type.googleapis.com/envoy.config.trace.v3.XRayConfig
segmentName: foo/bar
segmentFields:
origin: AWS::AppMesh::Proxy
aws:
app_mesh:
mesh_name: foo
virtual_node_name: bar
daemonEndpoint:
protocol: UDP
address: 127.0.0.1
portValue: 2000
samplingRuleManifest:
filename: /tmp/sampling-rules.json
```
+ For details on configuration for the sampling rule manifest in the `samplingRuleManifest` property, see [Configuring the X-Ray SDK for Go](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-go-configuration.html#xray-sdk-go-configuration-sampling).
If your issue is still not resolved, then consider opening a [GitHub issue](https://github.com/aws/aws-app-mesh-roadmap/issues/new?assignees=&labels=Bug&template=issue--bug-report.md&title=Bug%3A+describe+bug+here) or contact [AWS Support](https://aws.amazon.com/premiumsupport/).