[ aws . sagemaker-inference-router-runtime ]
Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target’s destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
InvokeEndpoint . If your account has a higher model execution timeout for InvokeEndpoint , it doesn’t apply through an inference router. For models that take longer, use InvokeInferenceRouterAsync .If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn’t retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers .
See also: AWS API Documentation
invoke-inference-router
--inference-router-name <value>
--routing-target-name <value>
[--content-type <value>]
[--accept <value>]
[--inference-id <value>]
[--prefix-aware-id <value>]
[--custom-attributes <value>]
--body <value>
<outfile>
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
[--cli-binary-format <value>]
[--no-cli-pager]
[--cli-auto-prompt]
[--no-cli-auto-prompt]
[--cli-error-format <value>]
[--help]
--inference-router-name (string) [required]
The name of the inference router to invoke.
Constraints:
- min:
1- max:
63- pattern:
[a-zA-Z0-9](-*[a-zA-Z0-9])*
--routing-target-name (string) [required]
The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.
Constraints:
- min:
1- max:
63- pattern:
[a-zA-Z0-9](-*[a-zA-Z0-9])*
--content-type (string)
The MIME type of the input data in the request body. The router passes it to the destination.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
--accept (string)
The MIME type that you want for the inference response. The router passes it to the destination.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
--inference-id (string)
An identifier for the inference request. The router passes it to the destination.
Constraints:
- min:
1- max:
64- pattern:
\S[\p{Print}]*
--prefix-aware-id (string)
An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.
Applies only to destination endpoints configured with a
RoutingStrategyofPREFIX_AWARE.Constraints:
- min:
1- max:
64- pattern:
\S[\p{Print}]*
--custom-attributes (string)
Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
--body (blob) [required]
The input data for the inference, in the format that your model expects. The body can’t be empty.
Constraints:
- min:
0- max:
6291456
outfile (string) [required]
Filename where the content will be saved
--debug (boolean)
Turn on debug logging.
--endpoint-url (string)
Override command’s default URL with the given URL.
--no-verify-ssl (boolean)
By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates. This option overrides the default behavior of verifying SSL certificates.
--no-paginate (boolean)
Disable automatic pagination. If automatic pagination is disabled, the AWS CLI will only make one call, for the first page of results.
--output (string)
The formatting style for command output.
--query (string)
A JMESPath query to use in filtering the response data.
--profile (string)
Use a specific profile from your credential file.
--region (string)
The region to use. Overrides config/env settings.
--version (string)
Display the version of this tool.
--color (string)
Turn on/off color output.
--no-sign-request (boolean)
Do not sign requests. Credentials will not be loaded if this argument is provided.
--ca-bundle (string)
The CA certificate bundle to use when verifying SSL certificates. Overrides config/env settings.
--cli-read-timeout (int)
The maximum socket read time in seconds. If the value is set to 0, the socket read will be blocking and not timeout. The default value is 60 seconds.
--cli-connect-timeout (int)
The maximum socket connect time in seconds. If the value is set to 0, the socket connect will be blocking and not timeout. The default value is 60 seconds.
--cli-binary-format (string)
The formatting style to be used for binary blobs. The default format is base64. The base64 format expects binary blobs to be provided as a base64 encoded string. The raw-in-base64-out format preserves compatibility with AWS CLI V1 behavior and binary values must be passed literally. When providing contents from a file that map to a binary blob fileb:// will always be treated as binary and use the file contents directly regardless of the cli-binary-format setting. When using file:// the file contents will need to properly formatted for the configured cli-binary-format.
--no-cli-pager (boolean)
Disable cli pager for output.
--cli-auto-prompt (boolean)
Automatically prompt for CLI input parameters.
--no-cli-auto-prompt (boolean)
Disable automatically prompt for CLI input parameters.
--cli-error-format (string)
The formatting style for error output. By default, errors are displayed in enhanced format.
--help (boolean)
Display help for the command/subcommand.
Body -> (blob)
The inference that the model returned.
Constraints:
- min:
0- max:
6291456
ContentType -> (string)
The MIME type of the inference returned in the response body.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
InvokedRoutedResourceArn -> (string)
The Amazon Resource Name (ARN) of the endpoint or inference component that served the request.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
RoutingReason -> (string)
The reason that the router chose the destination that served the request. Possible values:
local-preferred– The destination is a healthy destination in the router’s Region.capacity-available– No destination in the router’s Region could take the request, and two or more destinations in other Regions could.only-healthy– No destination in the router’s Region could take the request, and exactly one destination in another Region could.failover– The router moved the request away from destinations in its Region because they were overloaded or returning errors. You also see this value when the router can’t assess destination health, and when no destination is healthy and the router tries one anyway.Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
InvokedProductionVariant -> (string)
The production variant of the destination endpoint that served the request.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*
CustomAttributes -> (string)
Provides additional information in the response about the inference. This is the value that the model container of the destination returned, unchanged.
Constraints:
- min:
0- max:
1024- pattern:
\p{ASCII}*