本文為英文版的機器翻譯版本，如內容有任何歧義或不一致之處，概以英文版為準。

使用作業提交器分類

概觀

Amazon EMR on EKS StartJobRun 請求會建立作業提交器 Pod (也稱為 job-runner Pod) 以產生 Spark 驅動程式。您可以使用 emr-job-submitter分類來設定節點選取器、新增容錯、自訂記錄，以及對任務提交器 Pod 進行其他修改。

我們建議將任務提交者 Pod 放置在隨需執行個體上。如果任務提交器 Pod 執行的執行個體發生 Spot 執行個體中斷，則將任務提交器 Pod 放置在 Spot 執行個體上可能會導致任務失敗。您也可以將任務提交者 Pod 放在單一可用區域中，或使用套用至節點的任何 Kubernetes 標籤。

作業提交器分類範例

適用於作業提交器 Pod 的具有隨需節點放置的 StartJobRun 請求

cat >spark-python-in-s3-nodeselector-job-submitter.json << EOF
{
  "name": "spark-python-in-s3-nodeselector", 
  "virtualClusterId": "virtual-cluster-id", 
  "executionRoleArn": "execution-role-arn", 
  "releaseLabel": "emr-6.11.0-latest", 
  "jobDriver": {
    "sparkSubmitJobDriver": {
      "entryPoint": "s3://S3-prefix/trip-count.py", 
      "sparkSubmitParameters": "--conf spark.driver.cores=5  --conf spark.executor.memory=20G --conf spark.driver.memory=15G --conf spark.executor.cores=6"
    }
  }, 
  "configurationOverrides": {
    "applicationConfiguration": [
      {
        "classification": "spark-defaults", 
        "properties": {
          "spark.dynamicAllocation.enabled":"false"
        }
      },
      {
        "classification": "emr-job-submitter",
        "properties": {
          "jobsubmitter.node.selector.eks.amazonaws.com/capacityType": "ON_DEMAND"
        }
      }
    ], 
    "monitoringConfiguration": {
      "cloudWatchMonitoringConfiguration": {
        "logGroupName": "/emr-containers/jobs", 
        "logStreamNamePrefix": "demo"
      }, 
      "s3MonitoringConfiguration": {
        "logUri": "s3://joblogs"
      }
    }
  }
}
EOF
aws emr-containers start-job-run --cli-input-json file:///spark-python-in-s3-nodeselector-job-submitter.json

StartJobRun 任務提交器 Pod 具有單一可用區節點置放和 Amazon EC2 執行個體類型置放的請求

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.node.selector.topology.kubernetes.io/zone": "Availability Zone",
        "jobsubmitter.node.selector.node.kubernetes.io/instance-type":"m5.4xlarge"
      }
    }
  ]
}

StartJobRun 具有任務提交器 Pod 的標籤、註釋和自訂排程器的請求

"configurationOverrides": { 
  "applicationConfiguration": [ 
    {
      "classification": "emr-job-submitter", 
      "properties": {
        "jobsubmitter.label.label1": "value1",
        "jobsubmitter.label.label2": "value2",
        "jobsubmitter.annotation.ann1": "value1",
        "jobsubmitter.annotation.ann2": "value2",
        "jobsubmitter.scheduler.name": "custom-scheduler"
      }
    }
  ]
}

StartJobRun 請求，其公差套用到具有索引鍵 dedicated、值 graviton_machines、效果 NoExecute和 60 秒tolerationSeconds的作業提交器 Pod

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.node.toleration.dedicated":"graviton_machines",
        "jobsubmitter.node.toleration.dedicated.effect":"NoExecute",
        "jobsubmitter.node.toleration.dedicated.tolerationSeconds":"60"
      }
    }
  ]
}

StartJobRun 任務提交者 Pod 停用記錄的 請求

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.logging": "DISABLED"
      }
    }
  ], 
  "monitoringConfiguration": {
    "cloudWatchMonitoringConfiguration": {
      "logGroupName": "/emr-containers/jobs", 
      "logStreamNamePrefix": "demo"
    }, 
    "s3MonitoringConfiguration": {
      "logUri": "s3://joblogs"
    }
  }
}

StartJobRun 具有任務提交器 Pod 的自訂記錄容器映像、CPU 和記憶體的 請求

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.logging.image": "YOUR_ECR_IMAGE_URL",
        "jobsubmitter.logging.request.memory": "200Mi",
        "jobsubmitter.logging.request.cores": "0.5"
      }
    }
  ], 
  "monitoringConfiguration": {
    "cloudWatchMonitoringConfiguration": {
      "logGroupName": "/emr-containers/jobs", 
      "logStreamNamePrefix": "demo"
    }, 
    "s3MonitoringConfiguration": {
      "logUri": "s3://joblogs"
    }
  }
}

StartJobRun 具有自訂任務提交者容器映像和提取政策的請求

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.container.image": "123456789012.dkr.ecr.us-west-2.amazonaws.com/emr6.11_custom_repo",
        "jobsubmitter.container.image.pullPolicy": "kubernetes pull policy"
      }
    }
  ]
}

StartJobRun 為任務提交器 Pod 啟用正常終止的請求

"configurationOverrides": {
  "applicationConfiguration": [
    {
      "classification": "emr-job-submitter",
      "properties": {
        "jobsubmitter.gracefulTermination": "true"
      }
    }
  ]
}