Skip to content

/AWS1/CL_SGMPREFAWAREROUTNGCFG

The configuration for prefix-aware routing on a SageMaker real-time inference endpoint. Specify PrefixLength and ConcurrencyThreshold to control routing behavior.

CONSTRUCTOR

IMPORTING

Optional arguments:

iv_prefixlength TYPE /AWS1/SGMPREAWAREROUTPRELNGT /AWS1/SGMPREAWAREROUTPRELNGT

The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

  • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

  • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

The endpoint routes requests that share the same prefix to the same instance. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances.

iv_concurrencythreshold TYPE /AWS1/SGMPREAWAREROUTCONCTHRSH /AWS1/SGMPREAWAREROUTCONCTHRSH

The maximum number of in-flight requests on the target instance before the endpoint routes to another instance. Required when RoutingStrategy is PREFIX_AWARE. When in-flight requests on the prefix-selected instance reach this threshold, the endpoint routes the request to an instance with more available capacity.

Queryable Attributes

PrefixLength

The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

  • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

  • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

The endpoint routes requests that share the same prefix to the same instance. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances.

Accessible with the following methods

Method Description
GET_PREFIXLENGTH() Getter for PREFIXLENGTH, with configurable default
ASK_PREFIXLENGTH() Getter for PREFIXLENGTH w/ exceptions if field has no value
HAS_PREFIXLENGTH() Determine if PREFIXLENGTH has a value

ConcurrencyThreshold

The maximum number of in-flight requests on the target instance before the endpoint routes to another instance. Required when RoutingStrategy is PREFIX_AWARE. When in-flight requests on the prefix-selected instance reach this threshold, the endpoint routes the request to an instance with more available capacity.

Accessible with the following methods

Method Description
GET_CONCURRENCYTHRESHOLD() Getter for CONCURRENCYTHRESHOLD, with configurable default
ASK_CONCURRENCYTHRESHOLD() Getter for CONCURRENCYTHRESHOLD w/ exceptions if field has n
HAS_CONCURRENCYTHRESHOLD() Determine if CONCURRENCYTHRESHOLD has a value