Using distributed training in AWS Clean Rooms ML

To run a distributed training job (console)

1. Sign in to the AWS Management Console and open the AWS Clean Rooms console at https://console.aws.amazon.com/cleanrooms.

2. In the left navigation pane, choose Collaborations.

3. On the Collaborations page, choose the collaboration where you want to create a trained model.

4. After the collaboration opens, choose the ML models tab.

5. Under Custom ML models, in the Trained models section, choose Create trained model.

6. On the Create trained model page, for Associated model algorithm, specify the algorithm

7. For Trained model details, enter the following:

   1. For Name, enter a unique name for the model in the collaboration.

   2. (Optional) For Description, enter a description of the trained model.

   3. For Training data input mode, choose one of the following:

      • Select File if you have a smaller dataset that can fit on the ML storage volume and you prefer traditional file system access for your training script.

      • Select Pipe for large datasets to stream data directly from S3, avoiding the need to download everything to disk, which can improve training speed and reduce storage requirements.

      • Select FastFile if you want to combine the benefits of streaming from S3 with file system access, especially for sequentially read data or when dealing with fewer files for faster startup times.

8. For ML input channel details, do the following:

   1. For ML input channel, specify the ML input channel that provides data to the model algorithm.

      To add another channel, choose Add another ML input channel. You can add up to 19 additional ML input channels.

   2. For Channel name, enter the name of the ML input channel.

   3. For Amazon S3 data distribution type, choose one of the following:

      • Select Fully replicated to give each training instance with a complete copy of your dataset. This works best when your dataset is small enough to fit in memory or when each instance needs access to all data.

      • Select Sharded by S3 key to divide your dataset across training instances based on S3 keys. Each instance receives about 1/n of the total S3 objects, where 'n' is the number of instances. This works best for large datasets that you want to process in parallel.

      Note

      Consider your dataset size and training requirements when selecting a distribution type. Fully replicated provides complete data access but requires more storage, while Sharded by S3 key enables distributed processing of large datasets.

9. For Maximum training duration, choose the maximum amount of time you want to train your model.

10. For Hyperparameters, specify any algorithm-specific parameters and their intended values. Hyperparameters are specific to the model being trained and are used to fine-tune model training.

11. For Environment variables, specify any algorithm-specific variables and their intended values. Environment variables are set in the Docker container.

12. For Encryption, to use a custom AWS KMS key, select the Encrypt secret with a custom KMS key checkbox.

13. For EC2 Resource configuration, specify information about the compute resources that are used for model training.

    1. For Instance type, choose the type of instance you want to run.

       Supported instance types for distributed training are:

       • ml.m5.4xlarge

       • ml.m5.12xlarge

       • ml.m5.2xlarge

       • ml.g5.12xlarge

       • ml.g5.24xlarge

    2. For Instance count, enter the number of instances.

    3. For Volume size in GB, enter the ML storage volume size.

14. Choose Create trained model.