Class ProductionVariantRoutingConfig

java.lang.Object
software.amazon.awssdk.services.sagemaker.model.ProductionVariantRoutingConfig
All Implemented Interfaces:
Serializable, SdkPojo, ToCopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>

@Generated("software.amazon.awssdk:codegen") public final class ProductionVariantRoutingConfig extends Object implements SdkPojo, Serializable, ToCopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>

Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

See Also:
  • Method Details

    • routingStrategy

      public final RoutingStrategy routingStrategy()

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      If the service returns an enum value that is not available in the current SDK version, routingStrategy will return RoutingStrategy.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from routingStrategyAsString().

      Returns:
      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      See Also:
    • routingStrategyAsString

      public final String routingStrategyAsString()

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      If the service returns an enum value that is not available in the current SDK version, routingStrategy will return RoutingStrategy.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from routingStrategyAsString().

      Returns:
      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      See Also:
    • prefixAwareRoutingConfig

      public final PrefixAwareRoutingConfig prefixAwareRoutingConfig()

      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

      Returns:
      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.
    • toBuilder

      Description copied from interface: ToCopyableBuilder
      Take this object and create a builder that contains all of the current property values of this object.
      Specified by:
      toBuilder in interface ToCopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>
      Returns:
      a builder for type T
    • builder

      public static ProductionVariantRoutingConfig.Builder builder()
    • serializableBuilderClass

      public static Class<? extends ProductionVariantRoutingConfig.Builder> serializableBuilderClass()
    • hashCode

      public final int hashCode()
      Overrides:
      hashCode in class Object
    • equals

      public final boolean equals(Object obj)
      Overrides:
      equals in class Object
    • equalsBySdkFields

      public final boolean equalsBySdkFields(Object obj)
      Description copied from interface: SdkPojo
      Indicates whether some other object is "equal to" this one by SDK fields. An SDK field is a modeled, non-inherited field in an SdkPojo class, and is generated based on a service model.

      If an SdkPojo class does not have any inherited fields, equalsBySdkFields and equals are essentially the same.

      Specified by:
      equalsBySdkFields in interface SdkPojo
      Parameters:
      obj - the object to be compared with
      Returns:
      true if the other object equals to this object by sdk fields, false otherwise.
    • toString

      public final String toString()
      Returns a string representation of this object. This is useful for testing and debugging. Sensitive data will be redacted from this string using a placeholder value.
      Overrides:
      toString in class Object
    • getValueForField

      public final <T> Optional<T> getValueForField(String fieldName, Class<T> clazz)
    • sdkFields

      public final List<SdkField<?>> sdkFields()
      Specified by:
      sdkFields in interface SdkPojo
      Returns:
      List of SdkField in this POJO. May be empty list but should never be null.
    • sdkFieldNameToField

      public final Map<String,SdkField<?>> sdkFieldNameToField()
      Specified by:
      sdkFieldNameToField in interface SdkPojo
      Returns:
      The mapping between the field name and its corresponding field.