Class ProductionVariantRoutingConfig
- All Implemented Interfaces:
Serializable,SdkPojo,ToCopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>
Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.
- See Also:
-
Nested Class Summary
Nested Classes -
Method Summary
Modifier and TypeMethodDescriptionbuilder()final booleanfinal booleanequalsBySdkFields(Object obj) Indicates whether some other object is "equal to" this one by SDK fields.final <T> Optional<T> getValueForField(String fieldName, Class<T> clazz) final inthashCode()final PrefixAwareRoutingConfigThe configuration for prefix-aware routing.final RoutingStrategySets how the endpoint routes incoming traffic:final StringSets how the endpoint routes incoming traffic:static Class<? extends ProductionVariantRoutingConfig.Builder> Take this object and create a builder that contains all of the current property values of this object.final StringtoString()Returns a string representation of this object.Methods inherited from interface software.amazon.awssdk.utils.builder.ToCopyableBuilder
copy
-
Method Details
-
routingStrategy
Sets how the endpoint routes incoming traffic:
-
LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them. -
RANDOM: The endpoint routes each request to a randomly chosen instance. -
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches theConcurrencyThresholdvalue, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request. -
KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does withLEAST_OUTSTANDING_REQUESTS.KV_AWAREworks only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.
If the service returns an enum value that is not available in the current SDK version,
routingStrategywill returnRoutingStrategy.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available fromroutingStrategyAsString().- Returns:
- Sets how the endpoint routes incoming traffic:
-
LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them. -
RANDOM: The endpoint routes each request to a randomly chosen instance. -
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches theConcurrencyThresholdvalue, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request. -
KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does withLEAST_OUTSTANDING_REQUESTS.KV_AWAREworks only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.
-
- See Also:
-
-
routingStrategyAsString
Sets how the endpoint routes incoming traffic:
-
LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them. -
RANDOM: The endpoint routes each request to a randomly chosen instance. -
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches theConcurrencyThresholdvalue, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request. -
KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does withLEAST_OUTSTANDING_REQUESTS.KV_AWAREworks only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.
If the service returns an enum value that is not available in the current SDK version,
routingStrategywill returnRoutingStrategy.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available fromroutingStrategyAsString().- Returns:
- Sets how the endpoint routes incoming traffic:
-
LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them. -
RANDOM: The endpoint routes each request to a randomly chosen instance. -
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches theConcurrencyThresholdvalue, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request. -
KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does withLEAST_OUTSTANDING_REQUESTS.KV_AWAREworks only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.
-
- See Also:
-
-
prefixAwareRoutingConfig
The configuration for prefix-aware routing. Specify this parameter only when you set
RoutingStrategytoPREFIX_AWARE.- Returns:
- The configuration for prefix-aware routing. Specify this parameter only when you set
RoutingStrategytoPREFIX_AWARE.
-
toBuilder
Description copied from interface:ToCopyableBuilderTake this object and create a builder that contains all of the current property values of this object.- Specified by:
toBuilderin interfaceToCopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig> - Returns:
- a builder for type T
-
builder
-
serializableBuilderClass
-
hashCode
-
equals
-
equalsBySdkFields
Description copied from interface:SdkPojoIndicates whether some other object is "equal to" this one by SDK fields. An SDK field is a modeled, non-inherited field in anSdkPojoclass, and is generated based on a service model.If an
SdkPojoclass does not have any inherited fields,equalsBySdkFieldsandequalsare essentially the same.- Specified by:
equalsBySdkFieldsin interfaceSdkPojo- Parameters:
obj- the object to be compared with- Returns:
- true if the other object equals to this object by sdk fields, false otherwise.
-
toString
-
getValueForField
-
sdkFields
-
sdkFieldNameToField
- Specified by:
sdkFieldNameToFieldin interfaceSdkPojo- Returns:
- The mapping between the field name and its corresponding field.
-