aws-cdk-lib.aws_stepfunctions_tasks.SparkSubmitJobDriver

SparkSubmitJobDriver Struct

The information about job driver for Spark submit.

Source

aws-stepfunctions-tasks/lib/emrcontainers/start-job-run.ts

Type last changed

v2.1.0 · 8 Dec 2021

Example

new tasks.EmrContainersStartJobRun(this, 'EMR Containers Start Job Run', {
  virtualCluster: tasks.VirtualClusterInput.fromVirtualClusterId('de92jdei2910fwedz'),
  releaseLabel: tasks.ReleaseLabel.EMR_6_2_0,
  jobName: 'EMR-Containers-Job',
  jobDriver: {
    sparkSubmitJobDriver: {
      entryPoint: sfn.TaskInput.fromText('local:///usr/lib/spark/examples/src/main/python/pi.py'),
    },
  },
  applicationConfig: [{
    classification: tasks.Classification.SPARK_DEFAULTS,
    properties: {
      'spark.executor.instances': '1',
      'spark.executor.memory': '512M',
    },
  }],
});

Properties

entryPointTaskInputentry_pointTaskInputentryPointTaskInputEntryPointTaskInputEntryPointTaskInputrequired#

The entry point of job application.

Length Constraints: Minimum length of 1. Maximum length of 256.

entryPointArguments?optionalTaskInputentry_point_arguments?optionalTaskInputentryPointArguments?optionalTaskInputEntryPointArguments?optionalTaskInputEntryPointArguments?optionalTaskInput#

The arguments for a job application in a task input object containing an array of strings.

Default: No arguments defined

Length Constraints: Minimum length of 1. Maximum length of 10280.

sparkSubmitParameters?optionalstringspark_submit_parameters?optionalstrsparkSubmitParameters?optionalStringSparkSubmitParameters?optionalstringSparkSubmitParameters?optional*string#

The Spark submit parameters that are used for job runs.

Default: No spark submit parameters

Length Constraints: Minimum length of 1. Maximum length of 102400.