Importing files from Amazon S3

You can use the Amazon S3 Import connector to browse a source bucket, select files or folders, and import them into SDMA without downloading and re-uploading through the portal. SDMA copies the selected files into your library using AWS Deadline Cloud and verifies integrity during the transfer. When the job completes, the imported files appear as a new asset in your project.

Overview

The Amazon S3 Import connector is initially created during deployment as a disabled connector named S3 Import. To use it, an administrator configures the source bucket credentials, associates the connector with an asset template, and enables it. You can then select Import from S3 when creating an asset.

Prerequisites

SDMA needs an IAM role to access the source S3 bucket. Create a role in the account that owns the bucket, then store its ARN in the SDMA secret.

Step 1: Create an IAM role in the source account

Create a role that grants read access to the source bucket. The role name must start with SpatialDataManagementContentDerivation-.

Permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::<SOURCE_BUCKET_NAME>",
        "arn:aws:s3:::<SOURCE_BUCKET_NAME>/*"
      ]
    }
  ]
}

Trust policy:

The role must trust the SDMA roles that will assume it at runtime. There are two callers:

Find these ARNs in your SDMA deployment:

# Connector invocation Lambda role (for bucket browsing)
aws lambda get-function \
  --function-name SpatialDataManagement-ConnectorInvocationFunction \
  --query 'Configuration.Role' --output text

# S3 Import Deadline queue role (for file copy jobs)
aws iam list-roles \
  --query "Roles[?contains(RoleName,'S3ImportDeadlineQueue')].Arn" --output text

Add these ARNs as trusted principals:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<CONNECTOR_INVOCATION_ROLE_ARN>",
          "<S3_IMPORT_DEADLINE_QUEUE_ROLE_ARN>"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 2: Update the Secrets Manager secret

SDMA creates a secret named SpatialDataManagementContentSecret-S3ImportConnector-<suffix> during deployment with placeholder values. Update it with the role ARN from Step 1.

You can find the secret ARN in the Secrets Manager console (filter by prefix SpatialDataManagementContentSecret-) or in the connector configuration.

Update the secret value to the following JSON structure:

{
  "authType": "assumeRole",
  "roleArn": "arn:aws:iam::<SOURCE_ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-<YOUR_SUFFIX>",
  "region": "<SOURCE_BUCKET_REGION>"
}

Step 3: Configure VPC endpoint access (cross-account only)

If the source bucket is in a different AWS account, you must allowlist that account on the S3 VPC endpoint policy. Set the S3ImportSourceAccountIds CloudFormation parameter during deployment:

# Single account
--parameters S3ImportSourceAccountIds=111122223333

# Multiple accounts (comma-separated)
--parameters S3ImportSourceAccountIds=111122223333,111222333444

This adds the external account(s) to the aws:PrincipalAccount condition on the S3 VPC endpoint, allowing the SDMA Lambda (running inside the VPC) to access buckets in those accounts.

If the source bucket is in the same account as SDMA, this parameter is not needed.

Connector configuration

The Amazon S3 Import connector is initially created with placeholder values. Update the connector configuration in the Spatial Data Portal under Connectors > S3 Import > Edit.

Full configuration example

{
  "schemaVersion": "1.0.0",
  "name": "S3 Import",
  "defaultStepConfig": {
    "stepType": "deadlineJob",
    "deadlineConfig": {
      "farmId": "<DEADLINE_FARM_ID>",
      "queueId": "<S3_IMPORT_QUEUE_ID>",
      "builtInTemplate": "s3_import_asset"
    }
  },
  "resources": {
    "s3ListObjects": {
      "discover": {
        "type": "s3ListObjects",
        "securityConfig": {
          "type": "SecretCredentials",
          "secretArn": "<YOUR_SECRET_ARN>",
          "assumeRoleArn": "<S3_IMPORT_CONNECTOR_OPS_ROLE_ARN>"
        },
        "buckets": [
          {
            "region": "<SOURCE_BUCKET_REGION>",
            "bucketName": "<SOURCE_BUCKET_NAME>"
          }
        ]
      }
    }
  },
  "triggers": [
    {
      "resources": ["project"],
      "events": ["onDemand"],
      "parameterDefinitions": {
        "assetName": { "type": "string", "required": true },
        "sourceKeys": { "type": "string", "required": true }
      },
      "steps": [
        {
          "description": "Write sourceKeys manifest to CAS",
          "stepType": "casPutObject",
          "payload": { "fields": ["parameters"] }
        },
        {
          "description": "Stream copy files from source to CAS",
          "stepType": "deadlineJob",
          "template": "template.yaml",
          "parameters": {
            "ManifestKey": "${$temp.objectKey}",
            "ManifestBucket": "${$temp.bucket}",
            "ManifestHash": "${$temp.contentHash}",
            "SecretArn": "<YOUR_SECRET_ARN>",
            "AssumeRoleArn": "<S3_IMPORT_CONNECTOR_OPS_ROLE_ARN>",
            "SourceBucket": "<SOURCE_BUCKET_NAME>",
            "Region": "<SOURCE_BUCKET_REGION>",
            "TemplateId": "<ASSET_TEMPLATE_ID>",
            "CasRootPrefix": "SpatialDataManagementAssets"
          },
          "output": {
            "assets": [
              {
                "assetTemplateIds": ["<ASSET_TEMPLATE_ID>"],
                "filter": { "fileNameRegex": "^assets\\.json$" }
              }
            ]
          }
        }
      ]
    }
  ]
}

Configuration fields

The following fields are pre-filled by SDMA during deployment and do not require manual configuration:

The following fields require manual configuration:

Queue and fleet recommendations

The Amazon S3 Import connector uses a dedicated Deadline Cloud queue (spatial-data-management-s3-import-queue) that is separate from the default content derivation queue. This keeps import jobs isolated from format conversion, rendering, and other processing work. You can modify this queue’s fleet configuration in the AWS Deadline Cloud console based on your usage needs. Consider the following fleet configuration options:

Enabling the connector

After editing and saving the connector:

Importing files from S3

How it works

When you submit an import, SDMA submits a job to AWS Deadline Cloud that copies the selected files from the source bucket into your library in the background. The connector verifies each file’s integrity during transfer and automatically deduplicates any files that already exist, rather than copying them again.

Troubleshooting