Amazon ECS canary deployments

Traffic shift - The process Amazon ECS uses to shift production traffic. For Amazon ECS canary deployments, traffic is shifted in two phases: first to the canary percentage, then to complete the deployment.

Canary percentage - The percentage of traffic routed to the new version during the evaluation period.

Canary bake time - The duration to monitor the canary version before proceeding with full deployment.

Deployment bake time - The time, in minutes, Amazon ECS waits after shifting all production traffic to the new service revision, before it terminates the old service revision. This is the duration when both blue and green service revisions are running simultaneously after the production traffic has shifted.

Lifecycle stages - A series of events in the deployment operation, such as "after production traffic shift".

Lifecycle hook - A Lambda function that runs at a specific lifecycle stage. You can create a function that verifies the deployment.

Target group - An Elastic Load Balancing resource used to route requests to one or more registered targets (for example, EC2 instances). When you create a listener, you specify a target group for its default action. Traffic is forwarded to the target group specified in the listener rule.

Listener - A Elastic Load Balancing resource that checks for connection requests using the protocol and port that you configure. The rules that you define for a listener determine how Amazon ECS routes requests to its registered targets.

Rule - An Elastic Load Balancing resource associated with a listener. A rule defines how requests are routed and consists of an action, condition, and priority.

Resource usage: Canary deployments run both original and canary task sets simultaneously during the evaluation period, increasing resource usage.

Traffic volume: Ensure the canary percentage generates sufficient traffic for meaningful validation of the new version.

Monitoring complexity: Canary deployments require monitoring and comparing metrics between two different versions simultaneously.

Rollback speed: Canary deployments enable quick rollback by shifting traffic back to the original task set.

Risk mitigation: Canary deployments provide excellent risk mitigation by limiting exposure to a small percentage of users.

Deployment duration: Canary deployments include evaluation periods that extend the overall deployment time but provide validation opportunities.

Initial - The deployment begins with 100% of traffic routed to the blue (current) service revision. The green (new) service revision receives test traffic but no production traffic initially.

Canary traffic shifting - This is a two step traffic shift strategy.

Canary bake time - Waits for a configurable duration (canary bake time) after canary traffic shift to allow monitoring and validation of the new revision's performance with the increased traffic load.

Lifecycle hooks - Optional Lambda functions can be executed at various stages during the deployment to perform automated validation, monitoring, or custom logic. Lambda functions or lifecycle hooks configured for PRODUCTION_TRAFFIC_SHIFT will be invoked at every production traffic shift step.

Canary percentage - The percentage of traffic to route to the new service revision during the canary phase. This allows testing with a controlled subset of production traffic.

Canary bake time - The duration to wait during the canary phase before shifting the remaining traffic to the new service revision. This provides time to monitor and validate the new version.

Original target group - Contains tasks from the current stable version and receives the majority of traffic.

Canary target group - Contains tasks from the new version and receives a small percentage of traffic for testing.

Weighted routing - The load balancer uses weighted routing rules to distribute traffic between the target groups based on the configured canary percentage.

Health checks - Both task sets must pass health checks before receiving traffic.

Metrics comparison - Compare key performance indicators between the original and canary versions, such as response time, error rate, and throughput.

Automated rollback - Configure CloudWatch alarms to automatically trigger rollback if the canary version shows degraded performance.

Manual validation - Use the evaluation period to manually review logs, metrics, and user feedback before proceeding.

Start small - Begin with 5-10% of traffic to minimize impact if issues occur.

Consider application criticality - Use smaller percentages for mission-critical applications and larger percentages for less critical services.

Account for traffic volume - Ensure the canary percentage generates sufficient traffic for meaningful validation.

Allow sufficient time - Set evaluation periods long enough to capture meaningful performance data, typically 10-30 minutes.

Consider traffic patterns - Account for your application's traffic patterns and peak usage times.

Balance speed and safety - Longer evaluation periods provide more data but slow deployment velocity.

Key metrics - Monitor response time, error rate, throughput, and resource utilization for both task sets.

Alarm-based rollback - Configure CloudWatch alarms to automatically trigger rollback when metrics exceed thresholds.

Comparative analysis - Set up dashboards to compare metrics between original and canary versions side-by-side.

Business metrics - Include business-specific metrics like conversion rates or user engagement alongside technical metrics.

Automated rollback - Configure automatic rollback triggers based on health checks and performance metrics.

Manual rollback procedures - Document clear procedures for manual rollback when automated triggers don't capture all issues.

Rollback testing - Regularly test rollback procedures to ensure they work correctly when needed.

Pre-deployment testing - Thoroughly test changes in staging environments before canary deployment.

Health check configuration - Ensure health checks accurately reflect application readiness and functionality.

Dependency validation - Verify that new versions are compatible with downstream and upstream services.

Data consistency - Ensure database schema changes and data migrations are backward compatible.

Deployment windows - Schedule canary deployments during business hours when teams are available to monitor and respond.

Communication channels - Establish clear communication channels for deployment status and issue escalation.

Role assignments - Define roles and responsibilities for monitoring, decision-making, and rollback execution.