View a markdown version of this page

GAMEOPS03-BP03 Load test early and often - Games Industry Lens
Implementation guidance

GAMEOPS03-BP03 Load test early and often

Load testing is the process of simulating real-world traffic on a system to assess its reliability and performance.

Level of risk exposed if this best practice is not established: High

Implementation guidance

Load testing is a key factor in developing a performance baseline for your resources and understanding your system's capacity, which can guide financial forecasting, architecture design, resource allocation, automated scaling configurations, and post-launch pre-scaling activities. Additional benefits include:

  • Optimized infrastructure: Resources might be over or under-provisioned. Understanding the resources needed will result in lower costs and less infrastructure to manage.

  • Scalability readiness: Certain mechanisms and features can drive users into a game quickly. Knowing when and how to scale can be the difference between appropriately meeting the increased demand and losing players. Use load test results to prepare runbooks with system thresholds, alert points, and critical alert points at different scaling levels.

  • Higher quality code: Issues such as excessive crosstalk between services, unbatched database calls, inefficient algorithms, memory leaks, and service degradation issues are sometimes simpler to identify at scale.

  • Behavior validation: Injecting different kinds of failures into your tests can validate the system's expected behavior or uncover error-handling issues that need to be corrected.

Ideally, developers should perform load testing at multiple points throughout the development process, as each can yield different benefits: Early on, they guide architectural decisions and refactoring efforts while it's cheaper and straightforward to make changes. At the end of each sprint or iteration, they validate the application's performance with the latest features and functionality.

Prior to deploying to production, large-scale load testing simulating expected real-world usage patterns confirms the system's ability to handle the production workload. After the deployment, periodic load tests monitor the system's performance and identify changes or bottlenecks that may arise over time.

To simulate player traffic, you need lightweight clients or bots that emulate the game client flows and transact with the game backend to simulate real-world player behavior. This data is generally captured through game play logs and data generated by human-driven QA tests, as well as through real-world limited scale alpha or beta tests where real players are invited to play an early-access build of the game.

It is important to record the system's behavior in an operational runbook to assist in troubleshooting possible failures in the future and to retain performance metrics that future load tests can be compared against. It is also recommended to have human QA personnel test the game while it is being load tested as they might discover issues that bots fail to identify and metrics do not reflect.

AWS Fault Injection Service is a fully managed service for running fault injection experiments that make it straightforward to improve an application's performance, observability, and resiliency. Fault injection experiments are used in chaos engineering, which is the practice of stressing an application in testing or production environments by creating disruptive events, such as sudden increase in CPU or memory consumption, observing how the system responds, and implementing improvements. Fault injection experiments assist teams to create the real-world conditions needed to uncover the hidden bugs, monitoring blind spots, and performance bottlenecks that are difficult to find in distributed systems.

Implementation steps

  • Set up a distributed load testing environment using Guidance for Kubernetes-Bases Game Load Testing.

  • Customize and deploy Locust control and worker pods within the EKS cluster using the provided deployment files, enabling scalable and manageable load generation.

  • Record system behavior and metrics during load testing in an operational runbook to assist with future troubleshooting and establish performance baselines.

  • Use fault injection experiments to simulate real-world disruptions and uncover hidden issues in system performance, observability, and resilience.