TransferUtility.OpenStreamWithResponseAsync (TransferUtilityOpenStreamRequest, CancellationToken)

This method uses parallel downloads and intelligent buffering to significantly improve throughput compared to the standard Amazon.S3.Transfer.ITransferUtility.OpenStreamAsync(Amazon.S3.Transfer.TransferUtilityOpenStreamRequest,System.Threading.CancellationToken) method.

How it works:

Multipart Download Strategy:

The Amazon.S3.Transfer.BaseDownloadRequest.MultipartDownloadType property controls how parts are downloaded (default: MultipartDownloadType.PART):

When to use PART vs RANGE:

Configuration Options:

You can customize the download behavior using Amazon.S3.Transfer.TransferUtilityConfig:

var config = new TransferUtilityConfig
{
    // Control how many parts download in parallel (default: 10)
    ConcurrentServiceRequests = 20
};
var transferUtility = new TransferUtility(s3Client, config);
                

Use Amazon.S3.Transfer.TransferUtilityConfig.ConcurrentServiceRequests to control parallel download threads. Use Amazon.S3.Transfer.TransferUtilityOpenStreamRequest.MaxInMemoryParts to limit memory consumption by capping the number of buffered parts in memory. Use Amazon.S3.Transfer.TransferUtilityOpenStreamRequest.ChunkBufferSize to control the size of individual memory chunks allocated from the ArrayPool when buffering downloaded parts. The default is 64KB, chosen to avoid Large Object Heap allocations. Larger chunks may improve throughput but can increase memory fragmentation.

You can also customize the part size per request using Amazon.S3.Transfer.BaseDownloadRequest.PartSize:

var request = new TransferUtilityOpenStreamRequest
{
    BucketName = "my-bucket",
    Key = "my-key",
    PartSize = 16 * 1024 * 1024,  // Use 16MB parts instead of default 8MB
    MultipartDownloadType = MultipartDownloadType.RANGE  // Enable RANGE mode to use custom PartSize
};
var response = await transferUtility.OpenStreamWithResponseAsync(request);
                

Additional Performance Settings:

You can also tune the S3 client's Amazon.Runtime.ClientConfig.BufferSize property to control the buffer size used when reading from S3 response streams and writing to the buffered stream. The default buffer size is 8KB. Increasing this value may improve throughput for large downloads at the cost of increased memory usage.

var s3Config = new AmazonS3Config
{
    BufferSize = 64 * 1024  // Use 64KB buffer instead of default 8KB
};
var s3Client = new AmazonS3Client(s3Config);
var transferUtility = new TransferUtility(s3Client);
                

Memory Considerations: The buffering mechanism uses memory to store downloaded parts. Adjust Amazon.S3.Transfer.TransferUtilityOpenStreamRequest.MaxInMemoryParts if you need to limit memory usage, especially when downloading very large files or multiple files concurrently.