Flink application troubleshooting

Check the application status:

# List all CMS Flink applications and their status
STAGE=dev
aws kinesisanalyticsv2 list-applications \
  --query "ApplicationSummaries[?contains(ApplicationName, 'cms-$STAGE-flink')].[ApplicationName,ApplicationStatus]" \
  --output table

Check CloudWatch alarms:

aws cloudwatch describe-alarms \
  --alarm-name-prefix "cms-dev-flink" \
  --query "MetricAlarms[?StateValue=='ALARM'].[AlarmName,AlarmDescription]" \
  --output table

Check the application logs for errors:

APP=cms-dev-flink-trip-processor
aws logs tail /aws/kinesis-analytics/$APP --since 30m --follow

Delete and recreate the affected Flink application. Deleting and recreating has a higher success rate than updating in place:

APP=cms-dev-flink-fw-telemetry-processor
TIMESTAMP=$(aws kinesisanalyticsv2 describe-application \
  --application-name $APP \
  --query "ApplicationDetail.CreateTimestamp" --output text)
aws kinesisanalyticsv2 delete-application \
  --application-name $APP --create-timestamp $TIMESTAMP
# Redeploy via CDK
cd deployment && make phase4

After recreation, start the application with SKIP_RESTORE_FROM_SNAPSHOT to avoid restoring stale state that may contain cached SSL sessions.

Verify the upstream Kafka topic has data:

# Check topic offsets (requires kafka-tools or MSK client)
# Look at the IoT Rule metrics to confirm messages are reaching MSK
aws cloudwatch get-metric-statistics \
  --namespace "AWS/IoT" \
  --metric-name "RuleMessageThrottled" \
  --dimensions Name=RuleName,Value=cms_dev_iot_msk_rule \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum

Check if the Flink consumer group is lagging or stuck at an old offset.

Verify the Flink application’s runtime properties have the correct Kafka topic name and bootstrap servers.

Check if DynamoDB writes are being throttled:

aws cloudwatch get-metric-statistics \
  --namespace "AWS/DynamoDB" \
  --metric-name "ThrottledRequests" \
  --dimensions Name=TableName,Value=cms-dev-storage-trips \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum

If DynamoDB is throttled, the tables use on-demand billing and should auto-scale. Wait a few minutes for capacity to adjust.

If Redis writes are timing out, check ElastiCache connectivity from the Flink VPC configuration.

Increase the checkpoint interval if the application is processing high volumes:

Update the Flink application runtime properties to set checkpointing.interval=120000 (2 minutes instead of the default 60 seconds).

Verify the CampaignSyncProcessor Flink application is in RUNNING state:

aws kinesisanalyticsv2 describe-application \
  --application-name cms-dev-flink-campaign-sync-processor \
  --query "ApplicationDetail.ApplicationStatus"

Check that the agent’s checkin is reaching the fw-checkin Kafka topic by reviewing the CampaignSyncProcessor CloudWatch logs:

aws logs tail /aws/kinesis-analytics/cms-dev-flink-campaign-sync-processor --follow

Verify that active campaigns exist in DynamoDB for the vehicle’s VIN:

aws dynamodb query --table-name cms-dev-campaigns \
  --index-name targetArn-index \
  --key-condition-expression "targetArn = :arn" \
  --expression-attribute-values '{":arn": {"S": "vehicle:YOUR_VIN"}}'

Confirm the campaign includes signalsToCollect with valid signal IDs from the decoder manifest.

Check the FWE agent container logs for scheme receipt:

# If using the simulation service
curl http://localhost:5001/api/agent/logs/YOUR_VIN
# Or directly via Docker
docker logs fwe-YOUR_VIN --tail 100

Verify the IoT Rule fw_dev_iot_msk_rule is routing messages to the fw-telemetry-raw Kafka topic. Check IoT Core rule metrics in CloudWatch for error counts:

aws cloudwatch get-metric-statistics \
  --namespace "AWS/IoT" \
  --metric-name "Failure" \
  --dimensions Name=RuleName,Value=fw_dev_iot_msk_rule \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum

Verify the FWTelemetryProcessor Flink application is RUNNING and not showing SSL or connection errors:

aws logs tail /aws/kinesis-analytics/cms-dev-flink-fw-telemetry-processor --follow

If the processor shows stale SSL connection errors, delete and recreate the Flink application with a fresh JAR (see Problem: Flink processor Kafka SSL connection errors).

Verify the decoder manifest exists in DynamoDB with signal mappings:

aws dynamodb scan --table-name cms-dev-decoder-manifest --max-items 5

Check that the FWTelemetryProcessor output topic cms-telemetry-preprocessed is being consumed by downstream processors. If the SimulatorPreprocessor is also writing to this topic, verify there are no consumer group conflicts.

Check the CampaignSyncProcessor logs for IoT Core MQTT publish errors:

aws logs tail /aws/kinesis-analytics/cms-dev-flink-campaign-sync-processor \
  --filter-pattern "ERROR" --since 30m

Verify the Flink application’s IAM role has iot:Publish permission on the topic cms/fleetwise/vehicles/+/collection_schemes_and_decoder_manifests.

Verify the agent is subscribed to the correct MQTT topic. The agent subscribes to cms/fleetwise/vehicles/{vin}/collection_schemes_and_decoder_manifests where {vin} must match the VIN in the campaign’s target.

Check campaign status in DynamoDB. Campaigns in SUSPENDED state will cause the processor to push an empty collection scheme, clearing the agent’s active collection.