Interface SageMakerGatewayRuntimeAsyncClient

All Superinterfaces:
AutoCloseable, AwsClient, SdkAutoCloseable, SdkClient

@Generated("software.amazon.awssdk:codegen") @SdkPublicApi @ThreadSafe public interface SageMakerGatewayRuntimeAsyncClient extends AwsClient
Service client for accessing Amazon SageMaker Gateway Runtime asynchronously. This can be created using the static builder() method.The asynchronous client performs non-blocking I/O when configured with any SdkAsyncHttpClient supported in the SDK. However, full non-blocking is not guaranteed as the async client may perform blocking calls in some cases such as credentials retrieval and endpoint discovery as part of the async API call.

Use the Amazon SageMaker Gateway Runtime API to run inference through a Global Inference Gateway. A Global Inference Gateway routes each request to an inference gateway that serves the model named in the request.

The request body is an OpenAI-compatible JSON object, sent with Content-Type: application/json, and the model is selected by the model key in that body. The content of the body is passed to the model server unchanged, except for the stream flag, which each operation sets.

Requests are signed with Signature Version 4 under the sagemaker signing name. IAM actions are in the sagemaker namespace, for example sagemaker:InvokeGlobalInferenceGateway. The default regional endpoint is gateway-runtime.sagemaker.{region}.amazonaws.com.

  • Field Details

  • Method Details

    • invokeGlobalInferenceGateway

      default CompletableFuture<InvokeGlobalInferenceGatewayResponse> invokeGlobalInferenceGateway(InvokeGlobalInferenceGatewayRequest invokeGlobalInferenceGatewayRequest)

      Buffered (non-streaming) inference. Returns the full OpenAI completion body.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>. A BadGatewayException can instead read inference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      Every error status the inference gateway returns is a typed error, chosen by that status:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.
      Parameters:
      invokeGlobalInferenceGatewayRequest - The request for InvokeGlobalInferenceGateway.
      Returns:
      A Java Future containing the result of the InvokeGlobalInferenceGateway operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • GatewayTimeoutException 504 -- The inference gateway did not respond in time. The Message prefix says whose time limit expired:

        • Global Inference Gateway: timed out waiting for the inference gateway -- the gateway's limit on the wait for the first part of the response.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ThrottlingException The request was throttled before it was dispatched for inference. Retry the request with backoff, after the Retry-After seconds when the response carries one. A message starting inference gateway means the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.
      • BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway. The Message prefix says why:

        • Global Inference Gateway: could not reach the inference gateway: <reason> -- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).
        • Global Inference Gateway: timed out waiting for the inference gateway -- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)
        • Any other Global Inference Gateway: message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.
        • inference gateway: response ended unexpectedly -- the inference gateway's connection failed while it was sending the response.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
      • ServiceUnavailableException The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its Message, not as this error.
      • InternalServerException An unexpected internal failure occurred.
      • ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting inference gateway means the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.
      • AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the sagemaker:InvokeGlobalInferenceGateway permission, or sagemaker:InvokeGlobalInferenceGatewayWithResponseStream for the streaming operation, on the gateway ARN. A message starting inference gateway means the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeGlobalInferenceGateway

      default CompletableFuture<InvokeGlobalInferenceGatewayResponse> invokeGlobalInferenceGateway(Consumer<InvokeGlobalInferenceGatewayRequest.Builder> invokeGlobalInferenceGatewayRequest)

      Buffered (non-streaming) inference. Returns the full OpenAI completion body.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>. A BadGatewayException can instead read inference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      Every error status the inference gateway returns is a typed error, chosen by that status:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.

      This is a convenience which creates an instance of the InvokeGlobalInferenceGatewayRequest.Builder avoiding the need to create one manually via InvokeGlobalInferenceGatewayRequest.builder()

      Parameters:
      invokeGlobalInferenceGatewayRequest - A Consumer that will call methods on InvokeGlobalInferenceGatewayRequest.Builder to create a request. The request for InvokeGlobalInferenceGateway.
      Returns:
      A Java Future containing the result of the InvokeGlobalInferenceGateway operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • GatewayTimeoutException 504 -- The inference gateway did not respond in time. The Message prefix says whose time limit expired:

        • Global Inference Gateway: timed out waiting for the inference gateway -- the gateway's limit on the wait for the first part of the response.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ThrottlingException The request was throttled before it was dispatched for inference. Retry the request with backoff, after the Retry-After seconds when the response carries one. A message starting inference gateway means the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.
      • BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway. The Message prefix says why:

        • Global Inference Gateway: could not reach the inference gateway: <reason> -- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).
        • Global Inference Gateway: timed out waiting for the inference gateway -- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)
        • Any other Global Inference Gateway: message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.
        • inference gateway: response ended unexpectedly -- the inference gateway's connection failed while it was sending the response.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
      • ServiceUnavailableException The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its Message, not as this error.
      • InternalServerException An unexpected internal failure occurred.
      • ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting inference gateway means the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.
      • AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the sagemaker:InvokeGlobalInferenceGateway permission, or sagemaker:InvokeGlobalInferenceGatewayWithResponseStream for the streaming operation, on the gateway ARN. A message starting inference gateway means the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeGlobalInferenceGatewayWithResponseStream

      default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(InvokeGlobalInferenceGatewayWithResponseStreamRequest invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler)

      Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.

      The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      An InternalStreamFailure event's Message uses the same prefixes, with the same text the buffered operation uses for the same failure. Its inference gateway: message carries no status: inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response.

      Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.
      Parameters:
      invokeGlobalInferenceGatewayWithResponseStreamRequest - The request for InvokeGlobalInferenceGatewayWithResponseStream. It has the same members as the InvokeGlobalInferenceGateway request.
      Returns:
      A Java Future containing the result of the InvokeGlobalInferenceGatewayWithResponseStream operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • GatewayTimeoutException 504 -- The inference gateway did not respond in time. The Message prefix says whose time limit expired:

        • Global Inference Gateway: timed out waiting for the inference gateway -- the gateway's limit on the wait for the first part of the response.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ThrottlingException The request was throttled before it was dispatched for inference. Retry the request with backoff, after the Retry-After seconds when the response carries one. A message starting inference gateway means the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.
      • BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway. The Message prefix says why:

        • Global Inference Gateway: could not reach the inference gateway: <reason> -- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).
        • Global Inference Gateway: timed out waiting for the inference gateway -- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)
        • Any other Global Inference Gateway: message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.
        • inference gateway: response ended unexpectedly -- the inference gateway's connection failed while it was sending the response.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ModelStreamErrorException Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
      • PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
      • InternalStreamFailureException A failure after the stream started. The Message says where it came from: Global Inference Gateway: when a gateway time limit expired or the inference gateway was taken out of service, or inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response. The stream ends after this event.
      • ServiceUnavailableException The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its Message, not as this error.
      • InternalServerException An unexpected internal failure occurred.
      • ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting inference gateway means the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.
      • AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the sagemaker:InvokeGlobalInferenceGateway permission, or sagemaker:InvokeGlobalInferenceGatewayWithResponseStream for the streaming operation, on the gateway ARN. A message starting inference gateway means the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeGlobalInferenceGatewayWithResponseStream

      default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(Consumer<InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builder> invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler)

      Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.

      The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.

      Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:

      • Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
      • inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>.
      • inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message> . The component is one of:
        • config -- the inference gateway's configuration and proxy layer.
        • scheduler -- the inference gateway's request scheduler.
        • model selector -- the inference gateway's model selector.
        • model server -- the model server.

      An InternalStreamFailure event's Message uses the same prefixes, with the same text the buffered operation uses for the same failure. Its inference gateway: message carries no status: inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response.

      Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:

      • 400 and any other 4xx not listed here -- a 400 validation error.
      • 401 and 403 -- AccessDeniedException (403).
      • 404 -- ResourceNotFoundException.
      • 408 -- GatewayTimeoutException (504).
      • 429 -- ThrottlingException, with the Retry-After header.
      • 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
      • 504 -- GatewayTimeoutException.

      This is a convenience which creates an instance of the InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builder avoiding the need to create one manually via InvokeGlobalInferenceGatewayWithResponseStreamRequest.builder()

      Parameters:
      invokeGlobalInferenceGatewayWithResponseStreamRequest - A Consumer that will call methods on InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builder to create a request. The request for InvokeGlobalInferenceGatewayWithResponseStream. It has the same members as the InvokeGlobalInferenceGateway request.
      Returns:
      A Java Future containing the result of the InvokeGlobalInferenceGatewayWithResponseStream operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • GatewayTimeoutException 504 -- The inference gateway did not respond in time. The Message prefix says whose time limit expired:

        • Global Inference Gateway: timed out waiting for the inference gateway -- the gateway's limit on the wait for the first part of the response.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ThrottlingException The request was throttled before it was dispatched for inference. Retry the request with backoff, after the Retry-After seconds when the response carries one. A message starting inference gateway means the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.
      • BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway. The Message prefix says why:

        • Global Inference Gateway: could not reach the inference gateway: <reason> -- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).
        • Global Inference Gateway: timed out waiting for the inference gateway -- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)
        • Any other Global Inference Gateway: message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service.
        • inference gateway: <status> <message> -- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.
        • inference gateway: response ended unexpectedly -- the inference gateway's connection failed while it was sending the response.

        The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).

        Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.

      • ModelStreamErrorException Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
      • PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
      • InternalStreamFailureException A failure after the stream started. The Message says where it came from: Global Inference Gateway: when a gateway time limit expired or the inference gateway was taken out of service, or inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response. The stream ends after this event.
      • ServiceUnavailableException The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its Message, not as this error.
      • InternalServerException An unexpected internal failure occurred.
      • ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting inference gateway means the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.
      • AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the sagemaker:InvokeGlobalInferenceGateway permission, or sagemaker:InvokeGlobalInferenceGatewayWithResponseStream for the streaming operation, on the gateway ARN. A message starting inference gateway means the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • serviceClientConfiguration

      default SageMakerGatewayRuntimeServiceClientConfiguration serviceClientConfiguration()
      Description copied from interface: SdkClient
      The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration
      Specified by:
      serviceClientConfiguration in interface AwsClient
      Specified by:
      serviceClientConfiguration in interface SdkClient
      Returns:
      SdkServiceClientConfiguration
    • create

      Create a SageMakerGatewayRuntimeAsyncClient with the region loaded from the DefaultAwsRegionProviderChain and credentials loaded from the DefaultCredentialsProvider.
    • builder

      Create a builder that can be used to configure and create a SageMakerGatewayRuntimeAsyncClient.