Retry Failed Activities

Activities sometimes fail for ephemeral reasons, such as a temporary loss of connectivity. At another time, the activity might succeed, so the appropriate way to handle activity failure is often to retry the activity, perhaps multiple times.

There are a variety of strategies for retrying activities; the best one depends on the details of your workflow. The strategies fall into three basic categories:

The following sections describe how to implement these strategies. The example workflow workers all use a single activity, unreliableActivity, which randomly does one of following:

Retry-Until-Success Strategy

The simplest retry strategy is to keep retrying the activity each time it fails until it eventually succeeds. The basic pattern is:

The following workflow implements the retry-until-success strategy. The workflow interface is implemented in RetryActivityRecipeWorkflow and has one method, runUnreliableActivityTillSuccess, which is the workflow's entry point. The workflow worker is implemented in RetryActivityRecipeWorkflowImpl, as follows:

public class RetryActivityRecipeWorkflowImpl
    implements RetryActivityRecipeWorkflow {

    @Override
    public void runUnreliableActivityTillSuccess() {
        final Settable<Boolean> retryActivity = new Settable<Boolean>();

        new TryCatch() {
            @Override
            protected void doTry() throws Throwable {
                Promise<Void> activityRanSuccessfully
                    = client.unreliableActivity();
                setRetryActivityToFalse(activityRanSuccessfully, retryActivity);
            }

            @Override
            protected void doCatch(Throwable e) throws Throwable {
                retryActivity.set(true);
            }
        };
        restartRunUnreliableActivityTillSuccess(retryActivity);
    }

    @Asynchronous
    private void setRetryActivityToFalse(
            Promise<Void> activityRanSuccessfully,
            @NoWait Settable<Boolean> retryActivity) {
        retryActivity.set(false);
    }

    @Asynchronous
    private void restartRunUnreliableActivityTillSuccess(
            Settable<Boolean> retryActivity) {
        if (retryActivity.get()) {
            runUnreliableActivityTillSuccess();
        }
    }
}

The workflow works as follows:

Exponential Retry Strategy

With the exponential retry strategy, the framework executes a failed activity again after a specified period of time, N seconds. If that attempt fails the framework executes the activity again after 2N seconds, and then 4N seconds and so on. Because the wait time can get quite large, you typically stop the retry attempts at some point rather than continuing indefinitely.

The framework provides three ways to implement an exponential retry strategy:

All approaches support the following configuration options, where time values are in seconds:

The following sections describe the various ways that you can implement an exponential retry strategy.

Exponential Retry with @ExponentialRetry

The simplest way to implement an exponential retry strategy for an activity is to apply an @ExponentialRetry annotation to the activity in the interface definition. If the activity fails, the framework handles the retry process automatically, based on the specified option values. The basic pattern is:

The ExponentialRetryAnnotationWorkflow workflow worker implements the exponential retry strategy by using an @ExponentialRetry annotation. It uses an unreliableActivity activity whose interface definition is implemented in ExponentialRetryAnnotationActivities, as follows:

@Activities(version = "1.0")
@ActivityRegistrationOptions(
    defaultTaskScheduleToStartTimeoutSeconds = 30,
    defaultTaskStartToCloseTimeoutSeconds = 30)
public interface ExponentialRetryAnnotationActivities {
    @ExponentialRetry(
        initialRetryIntervalSeconds = 5,
        maximumAttempts = 5,
        exceptionsToRetry = IllegalStateException.class)
    public void unreliableActivity();
}

The @ExponentialRetry options specify the following strategy:

The workflow interface is implemented in RetryWorkflow and has one method, process, which is the workflow's entry point. The workflow worker is implemented in ExponentialRetryAnnotationWorkflowImpl, as follows:

public class ExponentialRetryAnnotationWorkflowImpl implements RetryWorkflow {
    public void process() {
        handleUnreliableActivity();
    }

    public void handleUnreliableActivity() {
        client.unreliableActivity();
    }
}

The workflow works as follows:

If the activity fails by throwing IllegalStateException, the framework automatically runs the retry strategy specified in ExponentialRetryAnnotationActivities.

Exponential Retry with the RetryDecorator Class

@ExponentialRetry is simple to use. However, the configuration is static and set at compile time, so the framework uses the same retry strategy every time the activity fails. You can implement a more flexible exponential retry strategy by using the RetryDecorator class, which allows you to specify the configuration at run time and change it as needed. The basic pattern is:

If the activity fails, the framework retries the activity according to the ExponentialRetryPolicy object's configuration. You can change the retry configuration as needed by modifying this object.

The following workflow implementation shows how to use the RetryDecorator class to implement an exponential retry strategy. It uses an unreliableActivity activity that doesn't have an @ExponentialRetry annotation. The workflow interface is implemented in RetryWorkflow and has one method, process, which is the workflow's entry point. The workflow worker is implemented in DecoratorRetryWorkflowImpl, as follows:

public class DecoratorRetryWorkflowImpl implements RetryWorkflow {
   ...
  public void process() {
      long initialRetryIntervalSeconds = 5;
      int maximumAttempts = 5;
      ExponentialRetryPolicy retryPolicy = new ExponentialRetryPolicy(
              initialRetryIntervalSeconds).withMaximumAttempts(maximumAttempts);

      Decorator retryDecorator = new RetryDecorator(retryPolicy);
      client = retryDecorator.decorate(RetryActivitiesClient.class, client);
      handleUnreliableActivity();
  }

  public void handleUnreliableActivity() {
      client.unreliableActivity();
  }
}

The workflow works as follows:

If the activity fails, the framework retries it according to the configuration specified in Step 1.

Exponential Retry with the AsyncRetryingExecutor Class

The RetryDecorator class provides more flexibility in configuring the retry process than @ExponentialRetry, but the framework still runs the retry attempts automatically, based on the ExponentialRetryPolicy object's current configuration. A more flexible approach is to use the AsyncRetryingExecutor class. In addition to allowing you to configure the retry process at run time, the framework calls a user-implemented AsyncRunnable.run method to run each retry attempt instead of simply executing the activity.

The basic pattern is:

The following workflow shows how to use the AsyncRetryingExecutor class to implement an exponential retry strategy. It uses the same unreliableActivity activity as the DecoratorRetryWorkflow workflow discussed earlier. The workflow interface is implemented in RetryWorkflow and has one method, process, which is the workflow's entry point. The workflow worker is implemented in AsyncExecutorRetryWorkflowImpl, as follows:

public class AsyncExecutorRetryWorkflowImpl implements RetryWorkflow {
  private final RetryActivitiesClient client = new RetryActivitiesClientImpl();
  private final DecisionContextProvider contextProvider = new DecisionContextProviderImpl();
  private final WorkflowClock clock = contextProvider.getDecisionContext().getWorkflowClock();

  public void process() {
      long initialRetryIntervalSeconds = 5;
      int maximumAttempts = 5;
      handleUnreliableActivity(initialRetryIntervalSeconds, maximumAttempts);
  }
  public void handleUnreliableActivity(long initialRetryIntervalSeconds, int maximumAttempts) {

      ExponentialRetryPolicy retryPolicy = new ExponentialRetryPolicy(initialRetryIntervalSeconds).withMaximumAttempts(maximumAttempts);
      final AsyncExecutor executor = new AsyncRetryingExecutor(retryPolicy, clock);

      new TryCatch() {
          @Override
          protected void doTry() throws Throwable {
              executor.execute(new AsyncRunnable() {
                  @Override
                  public void run() throws Throwable {
                      client.unreliableActivity();
                  }
              });
          }
          @Override
          protected void doCatch(Throwable e) throws Throwable {
          }
      };
  }
}

The workflow works as follows:

For more discussion of how to use the TryCatch class to handle errors, see AWS Flow Framework for Java Exceptions.

Custom Retry Strategy

The most flexible approach to retrying failed activities is a custom strategy, which recursively calls an asynchronous method that runs the retry attempt, much like the retry-until-success strategy. However, instead of simply running the activity again, you implement custom logic that decides whether and how to run each successive retry attempt. The basic pattern is:

The following workflow shows how to implement a custom retry strategy. It uses the same unreliableActivity activity as the DecoratorRetryWorkflow and AsyncExecutorRetryWorkflow workflows. The workflow interface is implemented in RetryWorkflow and has one method, process, which is the workflow's entry point. The workflow worker is implemented in CustomLogicRetryWorkflowImpl, as follows:

public class CustomLogicRetryWorkflowImpl implements RetryWorkflow {
  ...
  public void process() {
      callActivityWithRetry();
  }
  @Asynchronous
  public void callActivityWithRetry() {
      final Settable<Throwable> failure = new Settable<Throwable>();
      new TryCatchFinally() {
          protected void doTry() throws Throwable {
              client.unreliableActivity();
          }
          protected void doCatch(Throwable e) {
              failure.set(e);
          }
          protected void doFinally() throws Throwable {
              if (!failure.isReady()) {
                  failure.set(null);
              }
          }
      };
      retryOnFailure(failure);
  }
  @Asynchronous
  private void retryOnFailure(Promise<Throwable> failureP) {
      Throwable failure = failureP.get();
      if (failure != null && shouldRetry(failure)) {
          callActivityWithRetry();
      }
  }
  protected Boolean shouldRetry(Throwable e) {
      //custom logic to decide to retry the activity or not
      return true;
  }
}

The workflow works as follows: