Skip to content

/AWS1/CL_SGMS3DATASOURCE

Describes the S3 data source.

Your input bucket must be in the same Amazon Web Services region as your training job.

CONSTRUCTOR

IMPORTING

Required arguments:

iv_s3datatype TYPE /AWS1/SGMS3DATATYPE /AWS1/SGMS3DATATYPE

If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

If you choose Converse, S3Uri identifies an Amazon S3 location that contains data formatted according to Converse format. This format structures conversational messages with specific roles and content types used for training and fine-tuning foundational models.

iv_s3uri TYPE /AWS1/SGMS3URI /AWS1/SGMS3URI

Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

  • A key name prefix might look like this: s3://bucketname/exampleprefix/

  • A manifest might look like this: s3://bucketname/example.manifest

    A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.

    The following code example shows a valid manifest format:

    [ {"prefix": "s3://customer_bucket/some/prefix/"},

    "relative/path/to/custdata-1",

    "relative/path/custdata-2",

    ...

    "relative/path/custdata-N"

    ]

    This JSON is equivalent to the following S3Uri list:

    s3://customer_bucket/some/prefix/relative/path/to/custdata-1

    s3://customer_bucket/some/prefix/relative/path/custdata-2

    ...

    s3://customer_bucket/some/prefix/relative/path/custdata-N

    The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

Your input bucket must be located in same Amazon Web Services region as your training job.

Optional arguments:

iv_s3datadistributiontype TYPE /AWS1/SGMS3DATADISTRIBUTION /AWS1/SGMS3DATADISTRIBUTION

If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

it_attributenames TYPE /AWS1/CL_SGMATTRIBUTENAMES_W=>TT_ATTRIBUTENAMES TT_ATTRIBUTENAMES

A list of one or more attribute names to use that are found in a specified augmented manifest file.

it_instancegroupnames TYPE /AWS1/CL_SGMINSTGROUPNAMES_W=>TT_INSTANCEGROUPNAMES TT_INSTANCEGROUPNAMES

A list of names of instance groups that get data from the S3 data source.

io_modelaccessconfig TYPE REF TO /AWS1/CL_SGMMODELACCESSCONFIG /AWS1/CL_SGMMODELACCESSCONFIG

ModelAccessConfig

io_hubaccessconfig TYPE REF TO /AWS1/CL_SGMHUBACCESSCONFIG /AWS1/CL_SGMHUBACCESSCONFIG

The configuration for a private hub model reference that points to a SageMaker JumpStart public hub model.

Queryable Attributes

S3DataType

If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

If you choose Converse, S3Uri identifies an Amazon S3 location that contains data formatted according to Converse format. This format structures conversational messages with specific roles and content types used for training and fine-tuning foundational models.

Accessible with the following methods

Method Description
GET_S3DATATYPE() Getter for S3DATATYPE, with configurable default
ASK_S3DATATYPE() Getter for S3DATATYPE w/ exceptions if field has no value
HAS_S3DATATYPE() Determine if S3DATATYPE has a value

S3Uri

Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

  • A key name prefix might look like this: s3://bucketname/exampleprefix/

  • A manifest might look like this: s3://bucketname/example.manifest

    A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.

    The following code example shows a valid manifest format:

    [ {"prefix": "s3://customer_bucket/some/prefix/"},

    "relative/path/to/custdata-1",

    "relative/path/custdata-2",

    ...

    "relative/path/custdata-N"

    ]

    This JSON is equivalent to the following S3Uri list:

    s3://customer_bucket/some/prefix/relative/path/to/custdata-1

    s3://customer_bucket/some/prefix/relative/path/custdata-2

    ...

    s3://customer_bucket/some/prefix/relative/path/custdata-N

    The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

Your input bucket must be located in same Amazon Web Services region as your training job.

Accessible with the following methods

Method Description
GET_S3URI() Getter for S3URI, with configurable default
ASK_S3URI() Getter for S3URI w/ exceptions if field has no value
HAS_S3URI() Determine if S3URI has a value

S3DataDistributionType

If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

Accessible with the following methods

Method Description
GET_S3DATADISTRIBUTIONTYPE() Getter for S3DATADISTRIBUTIONTYPE, with configurable default
ASK_S3DATADISTRIBUTIONTYPE() Getter for S3DATADISTRIBUTIONTYPE w/ exceptions if field has
HAS_S3DATADISTRIBUTIONTYPE() Determine if S3DATADISTRIBUTIONTYPE has a value

AttributeNames

A list of one or more attribute names to use that are found in a specified augmented manifest file.

Accessible with the following methods

Method Description
GET_ATTRIBUTENAMES() Getter for ATTRIBUTENAMES, with configurable default
ASK_ATTRIBUTENAMES() Getter for ATTRIBUTENAMES w/ exceptions if field has no valu
HAS_ATTRIBUTENAMES() Determine if ATTRIBUTENAMES has a value

InstanceGroupNames

A list of names of instance groups that get data from the S3 data source.

Accessible with the following methods

Method Description
GET_INSTANCEGROUPNAMES() Getter for INSTANCEGROUPNAMES, with configurable default
ASK_INSTANCEGROUPNAMES() Getter for INSTANCEGROUPNAMES w/ exceptions if field has no
HAS_INSTANCEGROUPNAMES() Determine if INSTANCEGROUPNAMES has a value

ModelAccessConfig

ModelAccessConfig

Accessible with the following methods

Method Description
GET_MODELACCESSCONFIG() Getter for MODELACCESSCONFIG

HubAccessConfig

The configuration for a private hub model reference that points to a SageMaker JumpStart public hub model.

Accessible with the following methods

Method Description
GET_HUBACCESSCONFIG() Getter for HUBACCESSCONFIG