routingStrategy

Sets how the endpoint routes incoming traffic:

• LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

• RANDOM: The endpoint routes each request to a randomly chosen instance.

• PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.