Interface GlobalInferenceGatewayRoutingWeights.Builder

  • Method Details

    • kvHeadroom

      The weight of the free key-value (KV) cache capacity on the cluster.

      Parameters:
      kvHeadroom - The weight of the free key-value (KV) cache capacity on the cluster.
      Returns:
      Returns a reference to this object so that method calls can be chained together.
    • queueDepth

      The weight of the number of requests that are waiting on the cluster.

      Parameters:
      queueDepth - The weight of the number of requests that are waiting on the cluster.
      Returns:
      Returns a reference to this object so that method calls can be chained together.
    • regionProximity

      GlobalInferenceGatewayRoutingWeights.Builder regionProximity(Float regionProximity)

      The weight of how close the cluster is to the caller.

      Parameters:
      regionProximity - The weight of how close the cluster is to the caller.
      Returns:
      Returns a reference to this object so that method calls can be chained together.
    • errorRate

      The weight of the recent error rate of the cluster.

      Parameters:
      errorRate - The weight of the recent error rate of the cluster.
      Returns:
      Returns a reference to this object so that method calls can be chained together.