Complete embeddings request and response schema

Complete synchronous schema

{
    "schemaVersion": "nova-multimodal-embed-v1",
    "taskType": "SINGLE_EMBEDDING",
    "singleEmbeddingParams": {
        "embeddingPurpose": "GENERIC_INDEX" | "GENERIC_RETRIEVAL" | "TEXT_RETRIEVAL" | "IMAGE_RETRIEVAL" | "VIDEO_RETRIEVAL" | "DOCUMENT_RETRIEVAL" | "AUDIO_RETRIEVAL" | "CLASSIFICATION" | "CLUSTERING",
        "embeddingDimension": 256 | 384 | 1024 | 3072,
        "text": {
            "truncationMode": "START" | "END" | "NONE",
            "value": string,
            "source": SourceObject,
        },
        "image": {
            "detailLevel": "STANDARD_IMAGE" | "DOCUMENT_IMAGE",
            "format": "png" | "jpeg" | "gif" | "webp",
            "source": SourceObject
        },
        "audio": {
            "format": "mp3" | "wav" | "ogg",
            "source": SourceObject
        },
        "video": {
            "format": "mp4" | "mov" | "mkv" | "webm" | "flv" | "mpeg" | "mpg" | "wmv" | "3gp",
            "source": SourceObject,
            "embeddingMode": "AUDIO_VIDEO_COMBINED" | "AUDIO_VIDEO_SEPARATE"
        }
    }
}

   

The following list includes all of the parameters for the request:

InvokeModel Response Body

When InvokeModel returns a successful result, the body of the response will have the following structure:

{
   "embeddings": [
      {
          "embeddingType": "TEXT" | "IMAGE" | "VIDEO" | "AUDIO" | "AUDIO_VIDEO_COMBINED",
          "embedding": number[],
          "truncatedCharLength": int // Only included if text input was truncated
      }
    ]                       
}
   

The following list includes all of the parameters for the response:

Complete asynchronous schema

You can generate embeddings asynchronously using the Amazon Bedrock Runtime API functions StartAsyncInvoke, GetAsyncInvoke, and ListAsyncInvokes. The asynchronous API must be used if you want to use Nova Embeddings to segment long content such as long passage of text or video and audio longer than 30 seconds.

When calling StartAsyncInvoke, you must provide modelId, outputDataConfig, and modelInput parameters.

response = bedrock_runtime.start_async_invoke(
    modelId="amazon.nova-2-multimodal-embeddings-v1:0",
    outputDataConfig=Data Config,
    modelInput=Model Input
)
  

outputDataConfig specifies the S3 bucket to which you'd like to save the generated output. It has the following structure:

{
    "s3OutputDataConfig": {
        "s3Uri": "s3://your-s3-bucket"
    }
} 
  

The s3Uri is the S3 URI of the destination bucket. For additional optional parameters, see the StartAsyncInvoke documentation.

The following structure is used for the modelInput parameter.

{
    "schemaVersion": "nova-multimodal-embed-v1",
    "taskType": "SEGMENTED_EMBEDDING",
    "segmentedEmbeddingParams": {
        "embeddingPurpose": "GENERIC_INDEX" | "GENERIC_RETRIEVAL" | "TEXT_RETRIEVAL" | "IMAGE_RETRIEVAL" | "VIDEO_RETRIEVAL" | "DOCUMENT_RETRIEVAL" | "AUDIO_RETRIEVAL" | "CLASSIFICATION" | "CLUSTERING",
        "embeddingDimension": 256 | 384 | 1024 | 3072,
        "text": {
            "truncationMode": "START" | "END" | "NONE",
            "value": string,
            "source": {
                "s3Location": {
                    "uri": "s3://Your S3 Object"
                }
            },
            "segmentationConfig": {
                "maxLengthChars": int
            }
        },
        "image": {
            "format": "png" | "jpeg" | "gif" | "webp",
            "source": SourceObject,
            "detailLevel": "STANDARD_IMAGE" | "DOCUMENT_IMAGE"
        },
        "audio": {
            "format": "mp3" | "wav" | "ogg",
            "source": SourceObject,
            "segmentationConfig": {
                "durationSeconds": int
            }
        },
        "video": {
            "format": "mp4" | "mov" | "mkv" | "webm" | "flv" | "mpeg" | "mpg" | "wmv" | "3gp",
            "source": SourceObject,
            "embeddingMode": "AUDIO_VIDEO_COMBINED" | "AUDIO_VIDEO_SEPARATE",
            "segmentationConfig": {
                "durationSeconds": int
            }
        }
    }
}
  

The following list includes all of the parameters for the request:

StartAsyncInvoke Response

The response from a call to StartAsyncInvoke will have the structure below. The invocationArn can be used to query the status of the asynchronous job using the GetAsyncInvoke function.

{
    "invocationArn": "arn:aws:bedrock:us-east-1:xxxxxxxxxxxx:async-invoke/lvmxrnjf5mo3",
}
  

Asynchronous Output

When asynchronous embeddings generation is complete, output artifacts are written to the S3 bucket you specified as the output destination. The files will have the following structure:

   amzn-s3-demo-bucket/
    job-id/
        segmented-embedding-result.json
        embedding-audio.jsonl
        embedding-image.json
        embedding-text.jsonl
        embedding-video.jsonl
        manifest.json
  

The segmented-embedding-result.json will contain the overall job result and reference to the corresponding jsonl files which contain actual embeddings per modality. Below is a truncated example of a file:

{
    "sourceFileUri": string, 
    "embeddingDimension": 256 | 384 | 1024 | 3072,
    "embeddingResults": [
        {
            "embeddingType": "TEXT" | "IMAGE" | "VIDEO" | "AUDIO" | "AUDIO_VIDEO_COMBINED",
            "status": "SUCCESS" | "FAILURE" | "PARTIAL_SUCCESS",
            "failureReason": string, // Granular error codes
            "message": string, // Human-readbale failure message
            "outputFileUri": string // S3 URI to a "embedding-modality.jsonl" file
        }
        ...
    ]
}
  

The embedding-modality.json will be jsonl files which contain the embedding output for each modality. Each line in the jsonl file will adhere to the following schema:

{
    "embedding": number[], // The generated embedding vector
    "segmentMetadata": {
        "segmentIndex": number,
        "segmentStartCharPosition": number, // Included for text only
        "segmentEndCharPosition": number, // Included for text only
        "truncatedCharLength": number, // Included only when text gets truncated
        "segmentStartSeconds": number, // Included for audio/video only
        "segmentEndSeconds": number // Included for audio/video only
    },
    "status": "SUCCESS" | "FAILURE",
    "failureReason": string, // Granular error codes
    "message": string // Human-readable failure message
}
  

The following list includes all of the parameters for the response. For text characters or audio/video times, all starting and ending times are zero-based. Additionally, all ending text positions or audio/video time values are inclusive.

File limitations for Nova Embeddings

Synchronous operations can accept both S3 inputs and inline chunks. Asynchronous operations can only accept S3 inputs.

When generating embeddings asynchronously, you'll need to ensure that your file is separated into an appropriate number of segments. For text embeddings you cannot have more than 1900 segments. For audio and video embeddings you cannot have more than 1434 segments.