Troubleshooting VPC networking

Network monitoring and troubleshooting

CloudTrail logging

All Configuration API operations and workflow runs using VPC networking are logged in CloudTrail. Use CloudTrail to audit configuration changes and track which runs use VPC networking.

Troubleshooting with ENI flow logs

When your workflow runs access external resources over the internet, you can use VPC Flow Logs to verify connectivity and diagnose issues. HealthOmics provisions elastic network interfaces (ENIs) in your VPC subnets to route traffic from your workflow tasks. By examining flow logs on these ENIs, you can trace network traffic to and from external destinations.

Setting up flow logs for troubleshooting

Option 1: VPC-level flow logs (for ongoing monitoring)

Enable flow logs on your VPC to automatically capture traffic from all HealthOmics workflow runs. This is best when you have many workflow runs and want comprehensive visibility without tracking individual ENIs.

Query flow logs using CloudWatch Logs Insights by time window, destination IP, or traffic patterns. You don't need to identify specific ENI IDs.

Option 2: ENI-level flow logs (for targeted troubleshooting)

Enable flow logs on specific ENIs when you have only a few HealthOmics ENIs in your account. This is the most cost-effective approach and makes it easy to isolate traffic for specific workflow runs.

Understanding VPC Flow Log format

VPC Flow Logs use a space-separated format with the following fields:

version account_id interface_id srcaddr dstaddr srcport dstport protocol packets bytes start end action log_status

Field descriptions:

Example flow log entries:

2 074296239033 eni-0e57c5476efeac402 10.0.130.58 13.226.238.96 40565 443 6 13 1502 1774338927 1774338929 ACCEPT OK
2 074296239033 eni-0e57c5476efeac402 13.226.238.96 10.0.130.58 443 40565 6 8 1024 1774338928 1774338930 ACCEPT OK

These entries show successful bidirectional HTTPS communication. Key IPs: 10.0.130.58 is the customer ENI created by HealthOmics in your account, and 13.226.238.96 is the external public domain your workflow is accessing. The first entry is outbound traffic, and the second is the return traffic. Both show ACCEPT, indicating the traffic was allowed by security groups.

Querying flow logs in CloudWatch Logs Insights

When flow logs are published to CloudWatch Logs, use CloudWatch Logs Insights to query and analyze the data.

Find rejected traffic (start here)

fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter action = "REJECT"
| sort @timestamp desc

If this returns results, you may have a connectivity issue. The rejected entries show which traffic is being blocked by security groups or network ACLs.

Find traffic to a specific external IP

First, resolve the domain to an IP address using nslookup or dig:

$ nslookup ftp.ncbi.nlm.nih.gov
Server:  127.53.53.53
Address: 127.53.53.53#53

Non-authoritative answer:
ftp.ncbi.nlm.nih.gov    canonical name = ftp.wip.ncbi.nlm.nih.gov.
Name:    ftp.wip.ncbi.nlm.nih.gov
Address: 130.14.250.10
Name:    ftp.wip.ncbi.nlm.nih.gov
Address: 130.14.250.11

The "Server" and "Address" at the top are your DNS resolver. The addresses under "Non-authoritative answer" (130.14.250.10 and 130.14.250.11) are the actual IPs for the domain.

Query flow logs using a prefix to match any IP in that range:

fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter dstAddr like "130.14.250"
| sort @timestamp desc

This matches any IP starting with 130.14.250, capturing traffic to all IPs in that subnet.

Find HTTPS traffic to external destinations

fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter dstPort = 443 and protocol = 6
| filter not (dstAddr like /^10\./ or dstAddr like /^172\./ or dstAddr like /^192\.168\./)
| sort @timestamp desc

The second filter excludes private IP ranges, showing only traffic to external (public) destinations.

Common flow log patterns and issues

Best practices for flow log troubleshooting

Configuration troubleshooting

Configuration stuck in CREATING status

Cause: Network resource provisioning can take several minutes.

Solution: Wait up to 10 minutes. If the status doesn't change to ACTIVE, check the following:

Run fails to start with VPC networking

Cause: The configuration might not be ACTIVE, or there might be network connectivity issues.

Solution:

Cannot delete configuration

Cause: The configuration is in use by active workflow runs.

Solution: Wait for all runs using the configuration to complete, then retry the deletion.

Cannot delete service-linked role

Cause: Active VPC configurations exist in your account.

Solution: Delete all VPC configurations first, then delete the service-linked role.

Workflow cannot connect to external resource

Cause: Security group or route table misconfiguration.

Solution:

Network performance issues

Symptom: Slow data transfer or workflow timeouts.

Cause: Network throughput limitations or NAT Gateway saturation.

Solution:

Workflow cannot reach the internet

Cause: The private subnets might not have a route to a NAT gateway, or security group rules might be blocking outbound traffic.

Solution:

Workflow run fails with connectivity errors

Cause: Network traffic might be blocked or misconfigured.

Solution: