diff --git a/packages/@aws-cdk/aws-glue-alpha/README.md b/packages/@aws-cdk/aws-glue-alpha/README.md index 26cfe1b1e489a..19fe808d3fcf5 100644 --- a/packages/@aws-cdk/aws-glue-alpha/README.md +++ b/packages/@aws-cdk/aws-glue-alpha/README.md @@ -79,6 +79,17 @@ The following ETL features are enabled by default: You can find more details about version, worker type and other features in [Glue's public documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html). +> **Note on continuous logging and encryption:** Because continuous logging is +> enabled by default, job driver and executor stdout/stderr are streamed to +> CloudWatch. Unless you attach a [`SecurityConfiguration`](#securityconfiguration) +> with `cloudWatchEncryption`, these logs are written to the account-shared, +> default Glue log group (`/aws-glue/jobs/logs-v2/`), which is **not** encrypted +> with a customer-managed key. Since job logs can contain sensitive runtime data +> (SQL statements, row values, error stack traces), attach a `SecurityConfiguration` +> with `cloudWatchEncryption` for regulated workloads. The construct emits a +> synthesis-time warning when continuous logging is on and no `SecurityConfiguration` +> is attached. + Reference the pyspark-etl-jobs.test.ts and scalaspark-etl-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs. @@ -245,8 +256,9 @@ Python shell jobs support a Python version that depends on the AWS Glue version you use. These can be used to schedule and run tasks that don't require an Apache Spark environment. Python shell jobs default to Python 3.9 and a MaxCapacity of `0.0625`. Python 3.9 supports pre-loaded -analytics libraries using the `library-set=analytics` flag, which is -enabled by default. +analytics libraries, enabled by default (`librarySet: glue.LibrarySet.ANALYTICS`). +Set `librarySet: glue.LibrarySet.NONE` when your libraries are custom or +conflict with the pre-installed ones. Reference the pyspark-shell-job.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs. @@ -333,6 +345,47 @@ new glue.PySparkEtlJob(stack, 'SelectiveJob', { This feature is available for all Spark job types (ETL, Streaming, Flex). +### Job Arguments + +Glue jobs are configured through a map of name-value arguments (`DefaultArguments`). This construct +manages several of these arguments on your behalf and exposes each one through a dedicated, +strongly-typed prop: + +| Managed argument(s) | Prop | +|--------------------------------------------------------------------------|-----------------------------------------------------------------| +| `--enable-continuous-cloudwatch-log`, `--continuous-log-*` | `continuousLogging` | +| `--enable-metrics` | `enableMetrics` | +| `--enable-observability-metrics` | `enableObservabilityMetrics` | +| `--enable-spark-ui`, `--spark-event-logs-path` | `sparkUI` | +| `--job-language`, `--class` | job class / `className` | +| `--extra-jars`, `--user-jars-first`, `--extra-py-files`, `--extra-files` | `extraJars`, `extraJarsFirst`, `extraPythonFiles`, `extraFiles` | + +The `defaultArguments` prop is the escape hatch for arguments this construct does **not** model. +Use it for any argument without a dedicated prop: + +```ts +import * as cdk from 'aws-cdk-lib'; +import * as iam from 'aws-cdk-lib/aws-iam'; +declare const stack: cdk.Stack; +declare const role: iam.IRole; +declare const script: glue.Code; + +new glue.PySparkEtlJob(stack, 'PySparkETLJob', { + role, + script, + defaultArguments: { + // an argument this construct does not manage + '--enable-glue-datacatalog': 'true', + }, +}); +``` + +To keep a single, unambiguous way to express each intent, setting a **construct-managed** argument +(any argument in the table above) or a **Glue-reserved** argument (`--debug`, `--mode`, +`--JOB_NAME`) through `defaultArguments` throws at synthesis time. Configure those through their +dedicated prop instead — for example, use `continuousLogging: { enabled: false }` rather than +`defaultArguments: { '--enable-continuous-cloudwatch-log': 'false' }`. + ### Enable Job Run Queuing AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC). diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/constants.ts b/packages/@aws-cdk/aws-glue-alpha/lib/constants.ts index 55a04431675fb..0ae7d606eb893 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/constants.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/constants.ts @@ -261,6 +261,25 @@ export enum PythonVersion { } +/** + * The set of pre-installed Python libraries available to a Python shell job running Python 3.9. + * + * @see https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library + */ +export enum LibrarySet { + /** + * Include the common analytics libraries for Python 3.9 (e.g. pandas, numpy, scikit-learn, + * awswrangler). + */ + ANALYTICS = 'analytics', + + /** + * Do not install the common library set. Use this when your libraries are custom or conflict + * with the pre-installed ones. + */ + NONE = 'none', +} + /** * AWS Glue runtime determines the runtime engine of the job. * diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts index 1525994ba7826..b2d9d20ca452b 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts @@ -375,6 +375,15 @@ export interface JobProps { * The default arguments for every run of this Glue job, * specified as name-value pairs. * + * This map is the escape hatch for Glue job arguments that this construct does not model. It + * MUST NOT be used to set arguments that already have a dedicated prop — configure those through + * the corresponding prop instead (`continuousLogging`, `enableMetrics`, + * `enableObservabilityMetrics`, `sparkUI`, `className`, `extraJars`, `extraJarsFirst`, + * `extraPythonFiles`, `extraFiles`). Passing a construct-managed argument (e.g. + * `--enable-continuous-cloudwatch-log`, `--enable-metrics`, `--enable-spark-ui`, + * `--job-language`) or a Glue-reserved argument (`--debug`, `--mode`, `--JOB_NAME`) here throws + * at synthesis time, so there is exactly one way to express each intent. + * * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html * for a list of reserved parameters * @default - no arguments @@ -475,40 +484,98 @@ export abstract class Job extends JobBase { return new Import(scope, id); } + /** + * Argument keys that Glue reserves for its own use and that a caller must never set through + * `defaultArguments`. Unlike construct-managed arguments (which are derived from what each job + * class emits), these are owned by the Glue service itself and are reserved for every job type. + * + * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html + */ + private static readonly GLUE_RESERVED_ARGUMENTS = new Set(['--debug', '--mode', '--JOB_NAME']); + /** * The IAM role Glue assumes to run this job. */ public readonly abstract role: iam.IRole; /** - * Check no usage of reserved arguments. + * Merge the customer-supplied `defaultArguments` with the arguments this construct manages. + * + * The construct owns every argument it emits — whether the value comes from a dedicated typed + * prop (e.g. `continuousLogging`, `enableMetrics`, `sparkUI`) or from the job class itself + * (e.g. `--job-language`). Those arguments, plus the arguments Glue reserves for its own use, + * MUST be configured through their dedicated props rather than the untyped `defaultArguments` + * map, so there is exactly one way to express each intent. Passing such a key through + * `defaultArguments` therefore throws instead of silently winning or being silently dropped. + * + * A managed key whose supplied value is identical to the construct's value is not contradictory, + * so it is allowed rather than rejected (auto-correcting config is preferred over errors). + * Glue-reserved keys are never emitted by the construct, so there is no value to reconcile and + * they always throw. + * + * The reserved set is derived from `managedArguments` (the arguments the caller actually emits) + * rather than a hand-maintained list, so adding a new typed prop automatically reserves its + * argument key without a second place to update. + * + * Conflict detection relies on string equality of the argument keys, which cannot see through + * unresolved tokens (e.g. a key produced by `CfnJson` that only resolves at deploy time). If a + * key is a token, the check is skipped for that key and a synthesis-time warning is emitted, so + * the (rare) case where a token key resolves to a managed argument at deploy time — in which the + * construct-managed value would silently take precedence — is surfaced rather than hidden. * * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html */ - protected checkNoReservedArgs(defaultArguments?: { [key: string]: string }) { + protected mergeManagedArguments( + managedArguments: { [key: string]: string }, + defaultArguments?: { [key: string]: string }, + ): { [key: string]: string } { if (defaultArguments) { - const reservedArgs = new Set(['--debug', '--mode', '--JOB_NAME']); - Object.keys(defaultArguments).forEach((arg) => { - if (reservedArgs.has(arg)) { - throw new cdk.ValidationError(lit`ReservedArgumentUsed`, `The ${arg} argument is reserved by Glue. Don't set it`, this); - } - }); + if (Object.keys(defaultArguments).some((arg) => cdk.Token.isUnresolved(arg))) { + cdk.Annotations.of(this).addWarningV2( + 'aws-cdk/aws-glue-alpha:tokenJobArgumentKey', + 'defaultArguments contains an unresolved token as an argument key, so it cannot be checked for conflicts with construct-managed arguments. If it resolves to a managed argument at deploy time, the construct-managed value will take precedence. Configure managed arguments through their dedicated props (e.g. continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI).', + ); + } + const conflicts = Object.keys(defaultArguments).filter( + (arg) => Job.GLUE_RESERVED_ARGUMENTS.has(arg) + || (arg in managedArguments && defaultArguments[arg] !== managedArguments[arg]), + ); + if (conflicts.length > 0) { + throw new cdk.ValidationError( + lit`ManagedJobArgument`, + `the job argument(s) ${JSON.stringify(conflicts)} are managed by the construct or reserved by Glue; configure them through the corresponding props (e.g. continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI) instead of defaultArguments`, + this, + ); + } } - return defaultArguments; + return { ...defaultArguments, ...managedArguments }; } /** * Setup Continuous Logging Properties * @param role The IAM role to use for continuous logging * @param props The properties for continuous logging configuration + * @param securityConfiguration The security configuration attached to the job, if any * @returns String containing the args for the continuous logging command */ - protected setupContinuousLogging(role: iam.IRole, props: ContinuousLoggingProps | undefined) : any { + protected setupContinuousLogging(role: iam.IRole, props: ContinuousLoggingProps | undefined, securityConfiguration?: ISecurityConfiguration) : any { // If the developer has explicitly disabled continuous logging return no args if (props && !props.enabled) { return {}; } + // Continuous logging is on (explicitly or by default), but the logs will be written to an + // unencrypted CloudWatch log group unless a SecurityConfiguration is attached. We cannot + // introspect whether the attached SecurityConfiguration actually configures cloudWatchEncryption + // (the ISecurityConfiguration interface only exposes the name), so we only warn when none is + // attached at all to avoid false positives. + if (!securityConfiguration) { + cdk.Annotations.of(this).addWarningV2( + 'aws-cdk/aws-glue-alpha:unencryptedContinuousLogging', + 'Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html', + ); + } + // Else we turn on continuous logging by default. Determine what log group to use. const args: {[key: string]: string} = { '--enable-continuous-cloudwatch-log': 'true', diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts index 5af04c53c7a8c..3d147b4ea4d1f 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts @@ -93,10 +93,11 @@ export class PySparkEtlJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); this.resource = new CfnJob(this, 'Resource', { name: props.jobName, diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts index 7d8764223f817..a00cd2ceb2a7e 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts @@ -81,10 +81,11 @@ export class PySparkFlexEtlJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); this.resource = new CfnJob(this, 'Resource', { name: props.jobName, diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts index 949b423e334c8..648985b0404a0 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts @@ -85,10 +85,11 @@ export class PySparkStreamingJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); this.resource = new CfnJob(this, 'Resource', { name: props.jobName, diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.ts index ca26eb0c2cc27..4e61c1136676b 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.ts @@ -7,7 +7,7 @@ import type { Construct } from 'constructs'; import type { JobProps } from './job'; import { Job } from './job'; import type { Code } from '../code'; -import { JobType, GlueVersion, PythonVersion, MaxCapacity, JobLanguage } from '../constants'; +import { JobType, GlueVersion, PythonVersion, MaxCapacity, JobLanguage, LibrarySet } from '../constants'; /** * Properties for creating a Python Shell job @@ -27,6 +27,18 @@ export interface PythonShellJobProps extends JobProps { */ readonly maxCapacity?: MaxCapacity; + /** + * The set of pre-installed Python libraries to make available to the job. + * + * Only applies to jobs running Python 3.9. Set to `LibrarySet.NONE` when your libraries are + * custom or conflict with the pre-installed ones. + * + * @default LibrarySet.ANALYTICS when running Python 3.9, otherwise no library set is configured + * + * @see https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library + */ + readonly librarySet?: LibrarySet; + /** * Additional Python files that AWS Glue adds to the Python path before executing your script. * Only individual files are supported, directories are not supported. @@ -78,10 +90,13 @@ export class PythonShellJob extends Job { this.role = props.role; this.grantPrincipal = this.role; - // Enable CloudWatch metrics and continuous logging by default as a best practice - const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging); - const profilingMetricsArgs = { '--enable-metrics': '' }; - const observabilityMetricsArgs = { '--enable-observability-metrics': 'true' }; + // Enable continuous logging by default as a best practice. Note: the --enable-metrics and + // --enable-observability-metrics arguments are intentionally NOT set here. Those profiling + // metrics require the Spark/GlueContext instrumentation that Python shell jobs do not have, so + // Glue accepts but ignores them for the pythonshell command (verified: a pythonshell run with + // --enable-metrics emits no JobName-dimensioned CloudWatch metrics). See SparkJob/RayJob for + // the job types where these metrics apply. + const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging, props.securityConfiguration); // Gather executable arguments const executableArgs = this.executableArguments(props); @@ -93,14 +108,12 @@ export class PythonShellJob extends Job { } // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...executableArgs, ...extraPythonFilesArgs, ...continuousLoggingArgs, - ...profilingMetricsArgs, - ...observabilityMetricsArgs, - ...this.checkNoReservedArgs(props.defaultArguments), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); this.resource = new CfnJob(this, 'Resource', { name: props.jobName, @@ -143,10 +156,11 @@ export class PythonShellJob extends Job { const args: { [key: string]: string } = {}; args['--job-language'] = JobLanguage.PYTHON; - // If no Python version set (default 3.9) or the version is set to 3.9 then set library-set argument + // The library-set option only applies to Python 3.9 (the default version). Default to the + // common analytics libraries, but let the caller override it (e.g. LibrarySet.NONE) via the + // typed prop. Note: Glue names this argument `library-set`, without the `--` prefix. if (!props.pythonVersion || props.pythonVersion == PythonVersion.THREE_NINE) { - // Selecting this option includes common libraries for Python 3.9 - args['library-set'] = 'analytics'; + args['library-set'] = props.librarySet ?? LibrarySet.ANALYTICS; } return args; diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts index f77b9afed713b..b84d87edc4cbd 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts @@ -90,19 +90,19 @@ export class RayJob extends Job { this.grantPrincipal = this.role; // Enable CloudWatch metrics and continuous logging by default as a best practice - const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging); + const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging, props.securityConfiguration); // Conditionally include metrics arguments (default to enabled for backward compatibility) const profilingMetricsArgs = (props.enableMetrics ?? true) ? { '--enable-metrics': '' } : {}; const observabilityMetricsArgs = (props.enableObservabilityMetrics ?? true) ? { '--enable-observability-metrics': 'true' } : {}; // Combine command line arguments into a single line item - const defaultArguments = { - ...this.checkNoReservedArgs(props.defaultArguments), + const managedArguments = { ...continuousLoggingArgs, ...profilingMetricsArgs, ...observabilityMetricsArgs, }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); if (props.workerType && props.workerType !== WorkerType.Z_2X) { throw new ValidationError(lit`RayJobsOnlySupportZ2XWorkerType`, 'Ray jobs only support Z.2X worker type', this); diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts index 1d2adc98945db..411fb19f2beac 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts @@ -93,10 +93,11 @@ export class ScalaSparkEtlJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); if ((!props.workerType && props.numberOfWorkers !== undefined) || (props.workerType && props.numberOfWorkers === undefined)) { throw new ValidationError(lit`WorkerTypeAndNumberOfWorkersMustBothBeSet`, 'Both workerType and numberOfWorkers must be set', this); diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts index b08b44d238929..bd318813907f5 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts @@ -90,10 +90,11 @@ export class ScalaSparkFlexEtlJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); this.resource = new CfnJob(this, 'Resource', { name: props.jobName, diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-streaming-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-streaming-job.ts index e2d21f3411bce..2f6070fd3d180 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-streaming-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-streaming-job.ts @@ -85,10 +85,11 @@ export class ScalaSparkStreamingJob extends SparkJob { addConstructMetadata(this, props); // Combine command line arguments into a single line item - const defaultArguments = { + const managedArguments = { ...this.executableArguments(props), ...this.nonExecutableCommonArguments(props), }; + const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments); if ((!props.workerType && props.numberOfWorkers !== undefined) || (props.workerType && props.numberOfWorkers === undefined)) { throw new ValidationError(lit`WorkerTypeAndNumberRequired`, 'Both workerType and numberOfWorkers must be set', this); diff --git a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts index 531ae98b7b6ef..d14a9cb5f8683 100644 --- a/packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts +++ b/packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts @@ -151,9 +151,12 @@ export abstract class SparkJob extends Job { this.sparkUILoggingLocation = props.sparkUI ? this.setupSparkUILoggingLocation(props.sparkUI) : undefined; } + /** + * The arguments this construct manages for a Spark job. These are owned by the construct (derived from typed props). + */ protected nonExecutableCommonArguments(props: SparkJobProps): {[key: string]: string} { // Enable CloudWatch metrics and continuous logging by default as a best practice - const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging); + const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging, props.securityConfiguration); // Conditionally include metrics arguments (default to enabled for backward compatibility) const profilingMetricsArgs = (props.enableMetrics ?? true) ? { '--enable-metrics': '' } : {}; @@ -170,7 +173,6 @@ export abstract class SparkJob extends Job { ...profilingMetricsArgs, ...observabilityMetricsArgs, ...sparkUIArgs, - ...this.checkNoReservedArgs(props.defaultArguments), }; } diff --git a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.assets.json b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.assets.json index ed1302193a2a8..8c7813a8ca619 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.assets.json +++ b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.assets.json @@ -15,16 +15,16 @@ } } }, - "d996ee6f69653119f4876f32f829688b46bde5e1673b35dcf677f9081fa302f1": { + "e0d0f16b70b0febb5afda6736e6505b2c7e85fd7af8bb06bbcfc1cf8d243eb3a": { "displayName": "aws-glue-job-python-shell Template", "source": { "path": "aws-glue-job-python-shell.template.json", "packaging": "file" }, "destinations": { - "current_account-current_region-670b8db6": { + "current_account-current_region-da41f3c5": { "bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}", - "objectKey": "d996ee6f69653119f4876f32f829688b46bde5e1673b35dcf677f9081fa302f1.json", + "objectKey": "e0d0f16b70b0febb5afda6736e6505b2c7e85fd7af8bb06bbcfc1cf8d243eb3a.json", "assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-${AWS::Region}" } } diff --git a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.metadata.json b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.metadata.json index 1605aacb41d51..ecebd1f2248a0 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.metadata.json +++ b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.metadata.json @@ -75,24 +75,104 @@ { "type": "aws:cdk:analytics:construct", "data": "*" + }, + { + "type": "aws:cdk:warning", + "data": "Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html [ack: aws-cdk/aws-glue-alpha:unencryptedContinuousLogging]", + "trace": [ + "Annotations.addMessage (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:244:29)", + "Annotations.addWarningV2 (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:113:18)", + "WrappedClass.setupContinuousLogging (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.js:321:38)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.js:105:48)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/prop-injectable.js:32:13)", + "Object. (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js:63:1)", + "Module._compile (node:internal/modules/cjs/loader:1760:14)", + "Object..js (node:internal/modules/cjs/loader:1893:10)", + "Module.load (node:internal/modules/cjs/loader:1480:32)", + "Module._load (node:internal/modules/cjs/loader:1299:12)", + "TracingChannel.traceSync (node:diagnostics_channel:328:14)", + "wrapModuleLoad (node:internal/modules/cjs/loader:244:24)", + "Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)", + "node:internal/main/run_main_module:33:47" + ] } ], "/aws-glue-job-python-shell/BasicShellJob": [ { "type": "aws:cdk:analytics:construct", "data": "*" + }, + { + "type": "aws:cdk:warning", + "data": "Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html [ack: aws-cdk/aws-glue-alpha:unencryptedContinuousLogging]", + "trace": [ + "Annotations.addMessage (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:244:29)", + "Annotations.addWarningV2 (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:113:18)", + "WrappedClass.setupContinuousLogging (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.js:321:38)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.js:105:48)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/prop-injectable.js:32:13)", + "Object. (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js:67:1)", + "Module._compile (node:internal/modules/cjs/loader:1760:14)", + "Object..js (node:internal/modules/cjs/loader:1893:10)", + "Module.load (node:internal/modules/cjs/loader:1480:32)", + "Module._load (node:internal/modules/cjs/loader:1299:12)", + "TracingChannel.traceSync (node:diagnostics_channel:328:14)", + "wrapModuleLoad (node:internal/modules/cjs/loader:244:24)", + "Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)", + "node:internal/main/run_main_module:33:47" + ] } ], "/aws-glue-job-python-shell/ShellJobWithExtraPyFiles": [ { "type": "aws:cdk:analytics:construct", "data": "*" + }, + { + "type": "aws:cdk:warning", + "data": "Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html [ack: aws-cdk/aws-glue-alpha:unencryptedContinuousLogging]", + "trace": [ + "Annotations.addMessage (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:244:29)", + "Annotations.addWarningV2 (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:113:18)", + "WrappedClass.setupContinuousLogging (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.js:321:38)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.js:105:48)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/prop-injectable.js:32:13)", + "Object. (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js:73:1)", + "Module._compile (node:internal/modules/cjs/loader:1760:14)", + "Object..js (node:internal/modules/cjs/loader:1893:10)", + "Module.load (node:internal/modules/cjs/loader:1480:32)", + "Module._load (node:internal/modules/cjs/loader:1299:12)", + "TracingChannel.traceSync (node:diagnostics_channel:328:14)", + "wrapModuleLoad (node:internal/modules/cjs/loader:244:24)", + "Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)", + "node:internal/main/run_main_module:33:47" + ] } ], "/aws-glue-job-python-shell/DetailedShellJob39": [ { "type": "aws:cdk:analytics:construct", "data": "*" + }, + { + "type": "aws:cdk:warning", + "data": "Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html [ack: aws-cdk/aws-glue-alpha:unencryptedContinuousLogging]", + "trace": [ + "Annotations.addMessage (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:244:29)", + "Annotations.addWarningV2 (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/annotations.js:113:18)", + "WrappedClass.setupContinuousLogging (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.js:321:38)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.js:105:48)", + "new PythonShellJob (/Users/otaviom/projects/aws-cdk/packages/aws-cdk-lib/core/lib/prop-injectable.js:32:13)", + "Object. (/Users/otaviom/projects/aws-cdk/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js:80:1)", + "Module._compile (node:internal/modules/cjs/loader:1760:14)", + "Object..js (node:internal/modules/cjs/loader:1893:10)", + "Module.load (node:internal/modules/cjs/loader:1480:32)", + "Module._load (node:internal/modules/cjs/loader:1299:12)", + "TracingChannel.traceSync (node:diagnostics_channel:328:14)", + "wrapModuleLoad (node:internal/modules/cjs/loader:244:24)", + "Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)", + "node:internal/main/run_main_module:33:47" + ] } ], "/aws-glue-job-python-shell/BootstrapVersion": [ diff --git a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.template.json b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.template.json index 6da6df0c4db6a..78feb2aab4b16 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.template.json +++ b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/aws-glue-job-python-shell.template.json @@ -110,9 +110,7 @@ "DefaultArguments": { "--job-language": "python", "library-set": "analytics", - "--enable-continuous-cloudwatch-log": "true", - "--enable-metrics": "", - "--enable-observability-metrics": "true" + "--enable-continuous-cloudwatch-log": "true" }, "GlueVersion": "3.0", "JobRunQueuingEnabled": false, @@ -147,9 +145,7 @@ }, "DefaultArguments": { "--job-language": "python", - "--enable-continuous-cloudwatch-log": "true", - "--enable-metrics": "", - "--enable-observability-metrics": "true" + "--enable-continuous-cloudwatch-log": "true" }, "GlueVersion": "1.0", "JobRunQueuingEnabled": false, @@ -197,9 +193,7 @@ ] ] }, - "--enable-continuous-cloudwatch-log": "true", - "--enable-metrics": "", - "--enable-observability-metrics": "true" + "--enable-continuous-cloudwatch-log": "true" }, "GlueVersion": "3.0", "JobRunQueuingEnabled": false, @@ -233,13 +227,11 @@ } }, "DefaultArguments": { + "arg1": "value1", + "arg2": "value2", "--job-language": "python", "library-set": "analytics", - "--enable-continuous-cloudwatch-log": "true", - "--enable-metrics": "", - "--enable-observability-metrics": "true", - "arg1": "value1", - "arg2": "value2" + "--enable-continuous-cloudwatch-log": "true" }, "Description": "My detailed Python 3.9 Shell Job", "GlueVersion": "3.0", diff --git a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/manifest.json b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/manifest.json index b8c73e20f6f7c..62fd685e182ef 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/manifest.json +++ b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/manifest.json @@ -18,7 +18,7 @@ "validateOnSynth": false, "assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-deploy-role-${AWS::AccountId}-${AWS::Region}", "cloudFormationExecutionRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-cfn-exec-role-${AWS::AccountId}-${AWS::Region}", - "stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/d996ee6f69653119f4876f32f829688b46bde5e1673b35dcf677f9081fa302f1.json", + "stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/e0d0f16b70b0febb5afda6736e6505b2c7e85fd7af8bb06bbcfc1cf8d243eb3a.json", "requiresBootstrapStackVersion": 6, "bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version", "additionalDependencies": [ diff --git a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/tree.json b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/tree.json index bfbe1885999c8..981a8650231ef 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/tree.json +++ b/packages/@aws-cdk/aws-glue-alpha/test/integ.job-python-shell.js.snapshot/tree.json @@ -1 +1 @@ -{"version":"tree-0.1","tree":{"id":"App","path":"","constructInfo":{"fqn":"aws-cdk-lib.App","version":"0.0.0"},"children":{"aws-glue-job-python-shell":{"id":"aws-glue-job-python-shell","path":"aws-glue-job-python-shell","constructInfo":{"fqn":"aws-cdk-lib.Stack","version":"0.0.0"},"children":{"IAMServiceRole":{"id":"IAMServiceRole","path":"aws-glue-job-python-shell/IAMServiceRole","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.Role","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/IAMServiceRole/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.CfnRole","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::IAM::Role","aws:cdk:cloudformation:logicalId":"IAMServiceRole61C662C4","aws:cdk:cloudformation:props":{"assumeRolePolicyDocument":{"Statement":[{"Action":"sts:AssumeRole","Effect":"Allow","Principal":{"Service":"glue.amazonaws.com"}}],"Version":"2012-10-17"},"managedPolicyArns":[{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":iam::aws:policy/service-role/AWSGlueServiceRole"]]}]}}},"DefaultPolicy":{"id":"DefaultPolicy","path":"aws-glue-job-python-shell/IAMServiceRole/DefaultPolicy","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.Policy","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/IAMServiceRole/DefaultPolicy/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.CfnPolicy","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::IAM::Policy","aws:cdk:cloudformation:logicalId":"IAMServiceRoleDefaultPolicy379D1A0E","aws:cdk:cloudformation:props":{"policyDocument":{"Statement":[{"Action":["s3:GetBucket*","s3:GetObject*","s3:List*"],"Effect":"Allow","Resource":[{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":s3:::",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/*"]]},{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":s3:::",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"}]]}]}],"Version":"2012-10-17"},"policyName":"IAMServiceRoleDefaultPolicy379D1A0E","roles":[{"Ref":"IAMServiceRole61C662C4"}]}}}}}}},"BasicShellJob39":{"id":"BasicShellJob39","path":"aws-glue-job-python-shell/BasicShellJob39","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Coded194a11e6093124ac8ca0f26b7502843":{"id":"Coded194a11e6093124ac8ca0f26b7502843","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843","constructInfo":{"fqn":"aws-cdk-lib.aws_s3_assets.Asset","version":"0.0.0"},"children":{"Stage":{"id":"Stage","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843/Stage","constructInfo":{"fqn":"aws-cdk-lib.AssetStaging","version":"0.0.0"}},"AssetBucket":{"id":"AssetBucket","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843/AssetBucket","constructInfo":{"fqn":"aws-cdk-lib.aws_s3.BucketBase","version":"0.0.0"}}}},"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/BasicShellJob39/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"BasicShellJob39F2E7D12A","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"--job-language":"python","library-set":"analytics","--enable-continuous-cloudwatch-log":"true","--enable-metrics":"","--enable-observability-metrics":"true"},"glueVersion":"3.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"BasicShellJob":{"id":"BasicShellJob","path":"aws-glue-job-python-shell/BasicShellJob","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/BasicShellJob/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"BasicShellJobC7D0761E","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3"},"defaultArguments":{"--job-language":"python","--enable-continuous-cloudwatch-log":"true","--enable-metrics":"","--enable-observability-metrics":"true"},"glueVersion":"1.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"ShellJobWithExtraPyFiles":{"id":"ShellJobWithExtraPyFiles","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Coded194a11e6093124ac8ca0f26b7502843":{"id":"Coded194a11e6093124ac8ca0f26b7502843","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843","constructInfo":{"fqn":"aws-cdk-lib.aws_s3_assets.Asset","version":"0.0.0"},"children":{"Stage":{"id":"Stage","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843/Stage","constructInfo":{"fqn":"aws-cdk-lib.AssetStaging","version":"0.0.0"}},"AssetBucket":{"id":"AssetBucket","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843/AssetBucket","constructInfo":{"fqn":"aws-cdk-lib.aws_s3.BucketBase","version":"0.0.0"}}}},"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"ShellJobWithExtraPyFiles513D7C4D","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"--job-language":"python","library-set":"analytics","--extra-py-files":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"--enable-continuous-cloudwatch-log":"true","--enable-metrics":"","--enable-observability-metrics":"true"},"glueVersion":"3.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"DetailedShellJob39":{"id":"DetailedShellJob39","path":"aws-glue-job-python-shell/DetailedShellJob39","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/DetailedShellJob39/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"DetailedShellJob39CB370B41","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"--job-language":"python","library-set":"analytics","--enable-continuous-cloudwatch-log":"true","--enable-metrics":"","--enable-observability-metrics":"true","arg1":"value1","arg2":"value2"},"description":"My detailed Python 3.9 Shell Job","glueVersion":"3.0","jobRunQueuingEnabled":true,"maxCapacity":1,"maxRetries":0,"name":"My Python 3.9 Shell Job","role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]},"tags":{"key":"value"}}}}}},"BootstrapVersion":{"id":"BootstrapVersion","path":"aws-glue-job-python-shell/BootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnParameter","version":"0.0.0"}},"CheckBootstrapVersion":{"id":"CheckBootstrapVersion","path":"aws-glue-job-python-shell/CheckBootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnRule","version":"0.0.0"}}}},"aws-glue-job-python-shell-integ-test":{"id":"aws-glue-job-python-shell-integ-test","path":"aws-glue-job-python-shell-integ-test","constructInfo":{"fqn":"@aws-cdk/integ-tests-alpha.IntegTest","version":"0.0.0"},"children":{"DefaultTest":{"id":"DefaultTest","path":"aws-glue-job-python-shell-integ-test/DefaultTest","constructInfo":{"fqn":"@aws-cdk/integ-tests-alpha.IntegTestCase","version":"0.0.0"},"children":{"Default":{"id":"Default","path":"aws-glue-job-python-shell-integ-test/DefaultTest/Default","constructInfo":{"fqn":"constructs.Construct","version":"10.6.0"}},"DeployAssert":{"id":"DeployAssert","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert","constructInfo":{"fqn":"aws-cdk-lib.Stack","version":"0.0.0"},"children":{"BootstrapVersion":{"id":"BootstrapVersion","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert/BootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnParameter","version":"0.0.0"}},"CheckBootstrapVersion":{"id":"CheckBootstrapVersion","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert/CheckBootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnRule","version":"0.0.0"}}}}}}}},"Tree":{"id":"Tree","path":"Tree","constructInfo":{"fqn":"constructs.Construct","version":"10.6.0"}}}}} \ No newline at end of file +{"version":"tree-0.1","tree":{"id":"App","path":"","constructInfo":{"fqn":"aws-cdk-lib.App","version":"0.0.0"},"children":{"aws-glue-job-python-shell":{"id":"aws-glue-job-python-shell","path":"aws-glue-job-python-shell","constructInfo":{"fqn":"aws-cdk-lib.Stack","version":"0.0.0"},"children":{"IAMServiceRole":{"id":"IAMServiceRole","path":"aws-glue-job-python-shell/IAMServiceRole","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.Role","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/IAMServiceRole/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.CfnRole","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::IAM::Role","aws:cdk:cloudformation:logicalId":"IAMServiceRole61C662C4","aws:cdk:cloudformation:props":{"assumeRolePolicyDocument":{"Statement":[{"Action":"sts:AssumeRole","Effect":"Allow","Principal":{"Service":"glue.amazonaws.com"}}],"Version":"2012-10-17"},"managedPolicyArns":[{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":iam::aws:policy/service-role/AWSGlueServiceRole"]]}]}}},"DefaultPolicy":{"id":"DefaultPolicy","path":"aws-glue-job-python-shell/IAMServiceRole/DefaultPolicy","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.Policy","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/IAMServiceRole/DefaultPolicy/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_iam.CfnPolicy","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::IAM::Policy","aws:cdk:cloudformation:logicalId":"IAMServiceRoleDefaultPolicy379D1A0E","aws:cdk:cloudformation:props":{"policyDocument":{"Statement":[{"Action":["s3:GetBucket*","s3:GetObject*","s3:List*"],"Effect":"Allow","Resource":[{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":s3:::",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/*"]]},{"Fn::Join":["",["arn:",{"Ref":"AWS::Partition"},":s3:::",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"}]]}]}],"Version":"2012-10-17"},"policyName":"IAMServiceRoleDefaultPolicy379D1A0E","roles":[{"Ref":"IAMServiceRole61C662C4"}]}}}}}}},"BasicShellJob39":{"id":"BasicShellJob39","path":"aws-glue-job-python-shell/BasicShellJob39","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Coded194a11e6093124ac8ca0f26b7502843":{"id":"Coded194a11e6093124ac8ca0f26b7502843","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843","constructInfo":{"fqn":"aws-cdk-lib.aws_s3_assets.Asset","version":"0.0.0"},"children":{"Stage":{"id":"Stage","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843/Stage","constructInfo":{"fqn":"aws-cdk-lib.AssetStaging","version":"0.0.0"}},"AssetBucket":{"id":"AssetBucket","path":"aws-glue-job-python-shell/BasicShellJob39/Coded194a11e6093124ac8ca0f26b7502843/AssetBucket","constructInfo":{"fqn":"aws-cdk-lib.aws_s3.BucketBase","version":"0.0.0"}}}},"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/BasicShellJob39/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"BasicShellJob39F2E7D12A","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"--job-language":"python","library-set":"analytics","--enable-continuous-cloudwatch-log":"true"},"glueVersion":"3.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"BasicShellJob":{"id":"BasicShellJob","path":"aws-glue-job-python-shell/BasicShellJob","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/BasicShellJob/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"BasicShellJobC7D0761E","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3"},"defaultArguments":{"--job-language":"python","--enable-continuous-cloudwatch-log":"true"},"glueVersion":"1.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"ShellJobWithExtraPyFiles":{"id":"ShellJobWithExtraPyFiles","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Coded194a11e6093124ac8ca0f26b7502843":{"id":"Coded194a11e6093124ac8ca0f26b7502843","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843","constructInfo":{"fqn":"aws-cdk-lib.aws_s3_assets.Asset","version":"0.0.0"},"children":{"Stage":{"id":"Stage","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843/Stage","constructInfo":{"fqn":"aws-cdk-lib.AssetStaging","version":"0.0.0"}},"AssetBucket":{"id":"AssetBucket","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Coded194a11e6093124ac8ca0f26b7502843/AssetBucket","constructInfo":{"fqn":"aws-cdk-lib.aws_s3.BucketBase","version":"0.0.0"}}}},"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/ShellJobWithExtraPyFiles/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"ShellJobWithExtraPyFiles513D7C4D","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"--job-language":"python","library-set":"analytics","--extra-py-files":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"--enable-continuous-cloudwatch-log":"true"},"glueVersion":"3.0","jobRunQueuingEnabled":false,"maxCapacity":0.0625,"maxRetries":0,"role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]}}}}}},"DetailedShellJob39":{"id":"DetailedShellJob39","path":"aws-glue-job-python-shell/DetailedShellJob39","constructInfo":{"fqn":"@aws-cdk/aws-glue-alpha.PythonShellJob","version":"0.0.0"},"children":{"Resource":{"id":"Resource","path":"aws-glue-job-python-shell/DetailedShellJob39/Resource","constructInfo":{"fqn":"aws-cdk-lib.aws_glue.CfnJob","version":"0.0.0"},"attributes":{"aws:cdk:cloudformation:type":"AWS::Glue::Job","aws:cdk:cloudformation:logicalId":"DetailedShellJob39CB370B41","aws:cdk:cloudformation:props":{"command":{"name":"pythonshell","scriptLocation":{"Fn::Join":["",["s3://",{"Fn::Sub":"cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"},"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"]]},"pythonVersion":"3.9"},"defaultArguments":{"arg1":"value1","arg2":"value2","--job-language":"python","library-set":"analytics","--enable-continuous-cloudwatch-log":"true"},"description":"My detailed Python 3.9 Shell Job","glueVersion":"3.0","jobRunQueuingEnabled":true,"maxCapacity":1,"maxRetries":0,"name":"My Python 3.9 Shell Job","role":{"Fn::GetAtt":["IAMServiceRole61C662C4","Arn"]},"tags":{"key":"value"}}}}}},"BootstrapVersion":{"id":"BootstrapVersion","path":"aws-glue-job-python-shell/BootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnParameter","version":"0.0.0"}},"CheckBootstrapVersion":{"id":"CheckBootstrapVersion","path":"aws-glue-job-python-shell/CheckBootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnRule","version":"0.0.0"}}}},"aws-glue-job-python-shell-integ-test":{"id":"aws-glue-job-python-shell-integ-test","path":"aws-glue-job-python-shell-integ-test","constructInfo":{"fqn":"@aws-cdk/integ-tests-alpha.IntegTest","version":"0.0.0"},"children":{"DefaultTest":{"id":"DefaultTest","path":"aws-glue-job-python-shell-integ-test/DefaultTest","constructInfo":{"fqn":"@aws-cdk/integ-tests-alpha.IntegTestCase","version":"0.0.0"},"children":{"Default":{"id":"Default","path":"aws-glue-job-python-shell-integ-test/DefaultTest/Default","constructInfo":{"fqn":"constructs.Construct","version":"10.6.0"}},"DeployAssert":{"id":"DeployAssert","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert","constructInfo":{"fqn":"aws-cdk-lib.Stack","version":"0.0.0"},"children":{"BootstrapVersion":{"id":"BootstrapVersion","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert/BootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnParameter","version":"0.0.0"}},"CheckBootstrapVersion":{"id":"CheckBootstrapVersion","path":"aws-glue-job-python-shell-integ-test/DefaultTest/DeployAssert/CheckBootstrapVersion","constructInfo":{"fqn":"aws-cdk-lib.CfnRule","version":"0.0.0"}}}}}}}},"Tree":{"id":"Tree","path":"Tree","constructInfo":{"fqn":"constructs.Construct","version":"10.6.0"}}}}} \ No newline at end of file diff --git a/packages/@aws-cdk/aws-glue-alpha/test/managed-job-arguments.test.ts b/packages/@aws-cdk/aws-glue-alpha/test/managed-job-arguments.test.ts new file mode 100644 index 0000000000000..7bec69bf8183e --- /dev/null +++ b/packages/@aws-cdk/aws-glue-alpha/test/managed-job-arguments.test.ts @@ -0,0 +1,199 @@ +import * as cdk from 'aws-cdk-lib'; +import { Annotations, Match, Template } from 'aws-cdk-lib/assertions'; +import * as iam from 'aws-cdk-lib/aws-iam'; +import * as s3 from 'aws-cdk-lib/aws-s3'; +import * as glue from '../lib'; + +/** + * Invariant guard for the `defaultArguments` API. + * + * The construct owns every job argument it emits — whether the value comes from a dedicated typed + * prop (`continuousLogging`, `enableMetrics`, `sparkUI`, `className`, `extra*`) or from the job + * class itself (`--job-language`). The untyped `defaultArguments` map is the escape hatch for + * arguments the L2 does NOT model; it must NOT be a second channel for arguments that have a typed + * equivalent. + * + * These tests derive the managed-key set from what each class actually synthesizes and then assert, + * per class: + * - passing any synthesized (managed) key through `defaultArguments` throws — no silent win, no + * silent drop; and + * - passing a genuinely custom key still flows through untouched (escape hatch preserved). + * + * If a future change adds a typed prop that emits a new argument, this test starts rejecting that + * key from `defaultArguments` automatically — there is no separate reserved list to keep in sync. + */ + +// Glue service-reserved keys — owned by Glue, reserved for every job type regardless of props. +const GLUE_RESERVED = ['--debug', '--mode', '--JOB_NAME']; + +/** + * Each entry builds one job class with every args-producing prop set, so the synthesized + * `DefaultArguments` contains the full managed-key surface for that class. + */ +const JOB_CLASSES: Array<{ name: string; build: (scope: cdk.Stack, props: any) => void }> = [ + { + name: 'PySparkEtlJob', + build: (scope, extra) => new glue.PySparkEtlJob(scope, 'Job', { + ...baseSparkProps(scope), ...extra, + }), + }, + { + name: 'PySparkFlexEtlJob', + build: (scope, extra) => new glue.PySparkFlexEtlJob(scope, 'Job', { + ...baseSparkProps(scope), ...extra, + }), + }, + { + name: 'PySparkStreamingJob', + build: (scope, extra) => new glue.PySparkStreamingJob(scope, 'Job', { + ...baseSparkProps(scope), ...extra, + }), + }, + { + name: 'ScalaSparkEtlJob', + build: (scope, extra) => new glue.ScalaSparkEtlJob(scope, 'Job', { + ...baseSparkProps(scope), className: 'com.example.MyJob', ...extra, + }), + }, + { + name: 'ScalaSparkFlexEtlJob', + build: (scope, extra) => new glue.ScalaSparkFlexEtlJob(scope, 'Job', { + ...baseSparkProps(scope), className: 'com.example.MyJob', ...extra, + }), + }, + { + name: 'ScalaSparkStreamingJob', + build: (scope, extra) => new glue.ScalaSparkStreamingJob(scope, 'Job', { + ...baseSparkProps(scope), className: 'com.example.MyJob', ...extra, + }), + }, + { + name: 'PythonShellJob', + build: (scope, extra) => new glue.PythonShellJob(scope, 'Job', { + role: roleOf(scope), script: scriptOf(scope), jobName: 'Job', ...extra, + }), + }, + { + name: 'RayJob', + build: (scope, extra) => new glue.RayJob(scope, 'Job', { + role: roleOf(scope), script: scriptOf(scope), jobName: 'Job', ...extra, + }), + }, +]; + +let uid = 0; +function roleOf(scope: cdk.Stack): iam.IRole { + return iam.Role.fromRoleArn(scope, `Role${uid++}`, 'arn:aws:iam::123456789012:role/TestRole'); +} +function scriptOf(scope: cdk.Stack): glue.Code { + return glue.Code.fromBucket(s3.Bucket.fromBucketName(scope, `CodeBucket${uid++}`, 'bucketname'), 'script'); +} +/** Spark props with every args-producing typed prop set, to exercise the full managed surface. */ +function baseSparkProps(scope: cdk.Stack) { + return { + role: roleOf(scope), + script: scriptOf(scope), + jobName: 'Job', + sparkUI: {}, + enableMetrics: true, + enableObservabilityMetrics: true, + extraJars: [scriptOf(scope)], + extraFiles: [scriptOf(scope)], + extraPythonFiles: [scriptOf(scope)], + extraJarsFirst: true, + }; +} + +/** Synthesize a class with no `defaultArguments` and return the managed args (key→value) it emits. */ +function managedArgsOf(build: (scope: cdk.Stack, props: any) => void): { [key: string]: string } { + const stack = new cdk.Stack(new cdk.App(), 'S'); + build(stack, {}); + const jobs = Template.fromStack(stack).findResources('AWS::Glue::Job'); + const resource = Object.values(jobs)[0]; + return resource.Properties.DefaultArguments; +} + +describe('defaultArguments managed-key invariant', () => { + for (const jobClass of JOB_CLASSES) { + describe(jobClass.name, () => { + const managedArgs = managedArgsOf(jobClass.build); + const managedKeys = Object.keys(managedArgs); + + test('emits at least one managed argument', () => { + expect(managedKeys.length).toBeGreaterThan(0); + }); + + test.each(managedKeys)('rejects managed key %s passed via defaultArguments with a different value', (key) => { + const stack = new cdk.Stack(new cdk.App(), 'S'); + // Use a value guaranteed to differ from the managed value. + const differentValue = `${managedArgs[key]}-different`; + expect(() => jobClass.build(stack, { defaultArguments: { [key]: differentValue } })) + .toThrow(/managed by the construct or reserved by Glue/); + }); + + test('allows managed keys passed via defaultArguments when the value is identical', () => { + // Only literal-valued managed args can be reconciled by value at synth time. Token-valued + // args (e.g. --spark-event-logs-path, extra-* S3 URLs) synthesize to CloudFormation + // intrinsics, so equality cannot be proven and they are (correctly) always rejected — see + // the "different value" case, which covers them. + const literalArgs = Object.fromEntries( + Object.entries(managedArgs).filter(([, value]) => typeof value === 'string'), + ); + const stack = new cdk.Stack(new cdk.App(), 'S'); + // Passing the exact same values the construct would emit is not contradictory. + expect(() => jobClass.build(stack, { defaultArguments: literalArgs })).not.toThrow(); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: managedArgs, + }); + }); + + test.each(GLUE_RESERVED)('rejects Glue-reserved key %s passed via defaultArguments', (key) => { + const stack = new cdk.Stack(new cdk.App(), 'S'); + expect(() => jobClass.build(stack, { defaultArguments: { [key]: 'x' } })) + .toThrow(/managed by the construct or reserved by Glue/); + }); + + test('allows a genuinely custom argument to flow through (escape hatch preserved)', () => { + const stack = new cdk.Stack(new cdk.App(), 'S'); + jobClass.build(stack, { defaultArguments: { '--my-custom-arg': 'value' } }); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: { + '--my-custom-arg': 'value', + }, + }); + }); + }); + } + + describe('token-keyed defaultArguments', () => { + test('warns that a token argument key cannot be checked for conflicts', () => { + const stack = new cdk.Stack(new cdk.App(), 'S'); + // A Lazy.string resolves to a plain string, so it survives synthesis (unlike a CfnParameter + // key, which resolves to an intrinsic and fails). Its value is unknown at synth time, so the + // conflict check cannot see that it collides with a managed argument. + const tokenKey = cdk.Lazy.string({ produce: () => '--enable-continuous-cloudwatch-log' }); + new glue.PythonShellJob(stack, 'Job', { + role: roleOf(stack), + script: scriptOf(stack), + jobName: 'Job', + defaultArguments: { [tokenKey]: 'false' }, + }); + Annotations.fromStack(stack).hasWarning('/S/Job', Match.stringLikeRegexp('.*unresolved token as an argument key.*')); + }); + + test('construct-managed value takes precedence when a token key resolves to a managed key', () => { + const stack = new cdk.Stack(new cdk.App(), 'S'); + const tokenKey = cdk.Lazy.string({ produce: () => '--enable-continuous-cloudwatch-log' }); + new glue.PythonShellJob(stack, 'Job', { + role: roleOf(stack), + script: scriptOf(stack), + jobName: 'Job', + defaultArguments: { [tokenKey]: 'false' }, + }); + // The managed 'true' wins over the user-supplied 'false' (documented, and warned about). + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.objectLike({ '--enable-continuous-cloudwatch-log': 'true' }), + }); + }); + }); +}); diff --git a/packages/@aws-cdk/aws-glue-alpha/test/pyspark-etl-jobs.test.ts b/packages/@aws-cdk/aws-glue-alpha/test/pyspark-etl-jobs.test.ts index c512f81301d05..c5a837d72e443 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/pyspark-etl-jobs.test.ts +++ b/packages/@aws-cdk/aws-glue-alpha/test/pyspark-etl-jobs.test.ts @@ -1,5 +1,5 @@ import * as cdk from 'aws-cdk-lib'; -import { Template, Match } from 'aws-cdk-lib/assertions'; +import { Annotations, Template, Match } from 'aws-cdk-lib/assertions'; import * as iam from 'aws-cdk-lib/aws-iam'; import { LogGroup } from 'aws-cdk-lib/aws-logs'; import * as s3 from 'aws-cdk-lib/aws-s3'; @@ -147,6 +147,53 @@ describe('Job', () => { }); }); + describe('Continuous logging encryption warning', () => { + const warningId = 'aws-cdk/aws-glue-alpha:unencryptedContinuousLogging'; + + test('warns when continuous logging is enabled by default and no security configuration is attached', () => { + new glue.PySparkEtlJob(stack, 'PySparkETLJob', { + role, + script, + jobName: 'PySparkETLJob', + }); + + Annotations.fromStack(stack).hasWarning('/Default/PySparkETLJob', Match.stringLikeRegexp('Continuous CloudWatch logging is enabled but no SecurityConfiguration')); + }); + + test('warns when continuous logging is explicitly enabled and no security configuration is attached', () => { + new glue.PySparkEtlJob(stack, 'PySparkETLJob', { + role, + script, + jobName: 'PySparkETLJob', + continuousLogging: { enabled: true }, + }); + + Annotations.fromStack(stack).hasWarning('/Default/PySparkETLJob', Match.stringLikeRegexp('Continuous CloudWatch logging is enabled but no SecurityConfiguration')); + }); + + test('does not warn when a security configuration is attached', () => { + new glue.PySparkEtlJob(stack, 'PySparkETLJob', { + role, + script, + jobName: 'PySparkETLJob', + securityConfiguration: glue.SecurityConfiguration.fromSecurityConfigurationName(stack, 'SecurityConfig', 'securityConfigName'), + }); + + Annotations.fromStack(stack).hasNoWarning('/Default/PySparkETLJob', Match.stringLikeRegexp(warningId)); + }); + + test('does not warn when continuous logging is explicitly disabled', () => { + new glue.PySparkEtlJob(stack, 'PySparkETLJob', { + role, + script, + jobName: 'PySparkETLJob', + continuousLogging: { enabled: false }, + }); + + Annotations.fromStack(stack).hasNoWarning('/Default/PySparkETLJob', Match.stringLikeRegexp(warningId)); + }); + }); + describe('Create PySpark ETL Job with G2 worker type with 2 workers', () => { beforeEach(() => { job = new glue.PySparkEtlJob(stack, 'PySparkETLJob', { diff --git a/packages/@aws-cdk/aws-glue-alpha/test/python-shell-job.test.ts b/packages/@aws-cdk/aws-glue-alpha/test/python-shell-job.test.ts index 141b8f82f6a87..8b9282d8098d1 100644 --- a/packages/@aws-cdk/aws-glue-alpha/test/python-shell-job.test.ts +++ b/packages/@aws-cdk/aws-glue-alpha/test/python-shell-job.test.ts @@ -74,14 +74,21 @@ describe('Job', () => { test('Has Continuous Logging Enabled', () => { Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { DefaultArguments: Match.objectLike({ - '--enable-metrics': '', - '--enable-observability-metrics': 'true', '--enable-continuous-cloudwatch-log': 'true', '--job-language': 'python', 'library-set': 'analytics', }), }); }); + + test('does not set Spark-only profiling metrics args (not supported on Python shell)', () => { + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.not(Match.objectLike({ '--enable-metrics': Match.anyValue() })), + }); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.not(Match.objectLike({ '--enable-observability-metrics': Match.anyValue() })), + }); + }); }); describe('Create new Python Shell Job with log override parameters', () => { @@ -105,8 +112,6 @@ describe('Job', () => { test('Has Continuous Logging enabled with optional args', () => { Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { DefaultArguments: Match.objectLike({ - '--enable-metrics': '', - '--enable-observability-metrics': 'true', '--continuous-log-logGroup': Match.objectLike({ Ref: Match.anyValue(), }), @@ -135,14 +140,51 @@ describe('Job', () => { test('Has Continuous Logging Disabled', () => { Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { DefaultArguments: { - '--enable-metrics': '', - '--enable-observability-metrics': 'true', '--job-language': 'python', }, }); }); }); + describe('librarySet', () => { + test('defaults to analytics on Python 3.9', () => { + new glue.PythonShellJob(stack, 'PythonShellJob', { role, script }); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.objectLike({ 'library-set': 'analytics' }), + }); + }); + + test('can be overridden to none', () => { + new glue.PythonShellJob(stack, 'PythonShellJob', { + role, + script, + librarySet: glue.LibrarySet.NONE, + }); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.objectLike({ 'library-set': 'none' }), + }); + }); + + test('is not set for non-3.9 Python versions', () => { + new glue.PythonShellJob(stack, 'PythonShellJob', { + role, + script, + pythonVersion: glue.PythonVersion.TWO, + }); + Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { + DefaultArguments: Match.not(Match.objectLike({ 'library-set': Match.anyValue() })), + }); + }); + + test('rejects the managed `library-set` key passed via defaultArguments', () => { + expect(() => new glue.PythonShellJob(stack, 'PythonShellJob', { + role, + script, + defaultArguments: { 'library-set': 'none' }, + })).toThrow(/managed by the construct or reserved by Glue/); + }); + }); + describe('Create Python Shell Job with overridden Python verion and max capacity', () => { beforeEach(() => { job = new glue.PythonShellJob(stack, 'PythonShellJob', { @@ -231,8 +273,6 @@ describe('Job', () => { test('Verify Default Arguemnts', () => { Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { DefaultArguments: Match.objectLike({ - '--enable-metrics': '', - '--enable-observability-metrics': 'true', '--job-language': 'python', }), }); @@ -402,8 +442,6 @@ describe('Job', () => { test('Verify Default Arguemnts', () => { Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', { DefaultArguments: Match.objectLike({ - '--enable-metrics': '', - '--enable-observability-metrics': 'true', '--job-language': 'python', }), });