Automating Performance in CI/CD

Performance testing delivers the most value when it runs automatically on every change, not as a manual gate before major releases. The "Test After Every Change" principle requires pipeline integration that makes performance results as visible as unit test results.

Where Performance Tests Fit in the Pipeline

Integration Approach: Tool Agnostic

DLT provides multiple integration points that work with any CI/CD system:

Pipeline Integration Pattern

The general pattern for any CI/CD tool:

Concrete Example: GitHub Actions with DLT CLI

This workflow runs a performance gate on every push to main. It starts a DLT scenario, waits for completion, and fails the build if p99 exceeds the baseline by more than 15%.

# .github/workflows/perf-gate.yml
name: Performance Gate
on:
  push:
    branches: [main]

jobs:
  load-test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write    # For OIDC auth to AWS
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/dlt-ci-role
          aws-region: us-east-1

      - name: Install DLT CLI
        run: pip install distributed-load-testing-cli

      - name: Run load test
        run: |
          dlt run \
            --test-id "regression-suite" \
            --wait \
            --timeout 1200 \
            --output results.json

      - name: Evaluate results against baseline
        run: |
          # Extract p99 from results and compare to baseline
          P99=$(jq '.results.avg_lt_p99' results.json)
          BASELINE_P99=$(jq '.baseline.avg_lt_p99' results.json)
          THRESHOLD=$(echo "$BASELINE_P99 * 1.15" | bc)
          if (( $(echo "$P99 > $THRESHOLD" | bc -l) )); then
            echo "::error::Performance regression detected. p99=${P99}ms exceeds baseline+15%=${THRESHOLD}ms"
            exit 1
          fi
          echo "Performance gate PASSED. p99=${P99}ms within threshold=${THRESHOLD}ms"

      - name: Upload results artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: dlt-results
          path: results.json

Adapt the same pattern for GitLab CI (use script: blocks), Jenkins (use sh steps in a Declarative Pipeline), or AWS CodePipeline (invoke DLT via a Lambda action or CodeBuild step).

Baseline Management

DLT supports designating any test run as a "baseline" for comparison. Establish your baseline strategy:

Scheduling Recurring Tests

For the "Always Test" principle, configure DLT scheduled tests:

Handling Performance Test Failures in the Pipeline

When a performance test fails your gate criteria, the team needs a clear escalation path. Without one, teams either ignore the failure ("just override it") or block releases indefinitely.

Automate severity classification in your pipeline. Compare the regression percentage against thresholds and route notifications appropriately: minor failures go to a Slack channel, moderate failures page the team lead, critical failures trigger an automatic revert.

Dealing with Flaky Performance Tests

Performance tests have inherent variance. A 2% latency difference between runs is noise, not signal. Reduce false positives with these techniques:

Put It Into Practice