Interface SageMakerGatewayRuntimeClient

All Superinterfaces:
AutoCloseable, AwsClient, SdkAutoCloseable, SdkClient

@Generated("software.amazon.awssdk:codegen") @SdkPublicApi @ThreadSafe public interface SageMakerGatewayRuntimeClient extends AwsClient
Service client for accessing Amazon SageMaker Gateway Runtime. This can be created using the static builder() method.

Use the Amazon SageMaker Gateway Runtime API to run inference through a Global Inference Gateway. A Global Inference Gateway routes each request to an inference gateway that serves the model named in the request.

The request body is an OpenAI-compatible JSON object, sent with Content-Type: application/json, and the model is selected by the model key in that body. The content of the body is passed to the model server unchanged, except for the stream flag, which each operation sets.

Requests are signed with Signature Version 4 under the sagemaker signing name. IAM actions are in the sagemaker namespace, for example sagemaker:InvokeGlobalInferenceGateway. The default regional endpoint is gateway-runtime.sagemaker.{region}.amazonaws.com.

  • Field Details

  • Method Details

    • invokeGlobalInferenceGateway

      Buffered (non-streaming) inference. Returns the full OpenAI completion body.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>. A BadGatewayException can instead read inference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      Every error status the inference gateway returns is a typed error, chosen by that status:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.
      Parameters:
      invokeGlobalInferenceGatewayRequest - The request for InvokeGlobalInferenceGateway.
      Returns:
      Result of the InvokeGlobalInferenceGateway operation returned by the service.
      See Also:
    • invokeGlobalInferenceGateway

      Buffered (non-streaming) inference. Returns the full OpenAI completion body.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>. A BadGatewayException can instead read inference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      Every error status the inference gateway returns is a typed error, chosen by that status:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.

      This is a convenience which creates an instance of the InvokeGlobalInferenceGatewayRequest.Builder avoiding the need to create one manually via InvokeGlobalInferenceGatewayRequest.builder()

      Parameters:
      invokeGlobalInferenceGatewayRequest - A Consumer that will call methods on InvokeGlobalInferenceGatewayRequest.Builder to create a request. The request for InvokeGlobalInferenceGateway.
      Returns:
      Result of the InvokeGlobalInferenceGateway operation returned by the service.
      See Also:
    • create

      Create a SageMakerGatewayRuntimeClient with the region loaded from the DefaultAwsRegionProviderChain and credentials loaded from the DefaultCredentialsProvider.
    • builder

      Create a builder that can be used to configure and create a SageMakerGatewayRuntimeClient.
    • serviceMetadata

      static ServiceMetadata serviceMetadata()
    • serviceClientConfiguration

      default SageMakerGatewayRuntimeServiceClientConfiguration serviceClientConfiguration()
      Description copied from interface: SdkClient
      The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration
      Specified by:
      serviceClientConfiguration in interface AwsClient
      Specified by:
      serviceClientConfiguration in interface SdkClient
      Returns:
      SdkServiceClientConfiguration