feat(glue-alpha): ensure job parameters consistency

packages/@aws-cdk/aws-glue-alpha/README.md

@@ -79,6 +79,17 @@ The following ETL features are enabled by default:
 You can find more details about version, worker type and other features in
 [Glue's public documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html).
 
+> **Note on continuous logging and encryption:** Because continuous logging is
+> enabled by default, job driver and executor stdout/stderr are streamed to
+> CloudWatch. Unless you attach a [`SecurityConfiguration`](#securityconfiguration)
+> with `cloudWatchEncryption`, these logs are written to the account-shared,
+> default Glue log group (`/aws-glue/jobs/logs-v2/`), which is **not** encrypted
+> with a customer-managed key. Since job logs can contain sensitive runtime data
+> (SQL statements, row values, error stack traces), attach a `SecurityConfiguration`
+> with `cloudWatchEncryption` for regulated workloads. The construct emits a
+> synthesis-time warning when continuous logging is on and no `SecurityConfiguration`
+> is attached.
+
 Reference the pyspark-etl-jobs.test.ts and scalaspark-etl-jobs.test.ts unit tests
 for examples of required-only and optional job parameters when creating these
 types of jobs.
@@ -245,8 +256,9 @@ Python shell jobs support a Python version that depends on the AWS Glue
 version you use. These can be used to schedule and run tasks that don't
 require an Apache Spark environment. Python shell jobs default to
 Python 3.9 and a MaxCapacity of `0.0625`. Python 3.9 supports pre-loaded
-analytics libraries using the `library-set=analytics` flag, which is
-enabled by default.
+analytics libraries, enabled by default (`librarySet: glue.LibrarySet.ANALYTICS`).
+Set `librarySet: glue.LibrarySet.NONE` when your libraries are custom or
+conflict with the pre-installed ones.
 
 Reference the pyspark-shell-job.test.ts unit tests for examples of 
 required-only and optional job parameters when creating these types of jobs.
@@ -333,6 +345,47 @@ new glue.PySparkEtlJob(stack, 'SelectiveJob', {
 
 This feature is available for all Spark job types (ETL, Streaming, Flex).
 
+### Job Arguments
+
+Glue jobs are configured through a map of name-value arguments (`DefaultArguments`). This construct
+manages several of these arguments on your behalf and exposes each one through a dedicated,
+strongly-typed prop:
+
+| Managed argument(s)                                                      | Prop                                                            |
+|--------------------------------------------------------------------------|-----------------------------------------------------------------|
+| `--enable-continuous-cloudwatch-log`, `--continuous-log-*`               | `continuousLogging`                                             |
+| `--enable-metrics`                                                       | `enableMetrics`                                                 |
+| `--enable-observability-metrics`                                         | `enableObservabilityMetrics`                                    |
+| `--enable-spark-ui`, `--spark-event-logs-path`                           | `sparkUI`                                                       |
+| `--job-language`, `--class`                                              | job class / `className`                                         |
+| `--extra-jars`, `--user-jars-first`, `--extra-py-files`, `--extra-files` | `extraJars`, `extraJarsFirst`, `extraPythonFiles`, `extraFiles` |
+
+The `defaultArguments` prop is the escape hatch for arguments this construct does **not** model.
+Use it for any argument without a dedicated prop:
+
+```ts
+import * as cdk from 'aws-cdk-lib';
+import * as iam from 'aws-cdk-lib/aws-iam';
+declare const stack: cdk.Stack;
+declare const role: iam.IRole;
+declare const script: glue.Code;
+
+new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
+  role,
+  script,
+  defaultArguments: {
+    // an argument this construct does not manage
+    '--enable-glue-datacatalog': 'true',
+  },
+});
+```
+
+To keep a single, unambiguous way to express each intent, setting a **construct-managed** argument
+(any argument in the table above) or a **Glue-reserved** argument (`--debug`, `--mode`,
+`--JOB_NAME`) through `defaultArguments` throws at synthesis time. Configure those through their
+dedicated prop instead — for example, use `continuousLogging: { enabled: false }` rather than
+`defaultArguments: { '--enable-continuous-cloudwatch-log': 'false' }`.
+
 ### Enable Job Run Queuing
 
 AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC).

packages/@aws-cdk/aws-glue-alpha/lib/constants.ts

@@ -261,6 +261,25 @@ export enum PythonVersion {
 
 }
 
+/**
+ * The set of pre-installed Python libraries available to a Python shell job running Python 3.9.
+ *
+ * @see https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library
+ */
+export enum LibrarySet {
+  /**
+   * Include the common analytics libraries for Python 3.9 (e.g. pandas, numpy, scikit-learn,
+   * awswrangler).
+   */
+  ANALYTICS = 'analytics',
+
+  /**
+   * Do not install the common library set. Use this when your libraries are custom or conflict
+   * with the pre-installed ones.
+   */
+  NONE = 'none',
+}
+
 /**
  * AWS Glue runtime determines the runtime engine of the job.
  *

packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts

@@ -375,6 +375,15 @@ export interface JobProps {
    * The default arguments for every run of this Glue job,
    * specified as name-value pairs.
    *
+   * This map is the escape hatch for Glue job arguments that this construct does not model. It
+   * MUST NOT be used to set arguments that already have a dedicated prop — configure those through
+   * the corresponding prop instead (`continuousLogging`, `enableMetrics`,
+   * `enableObservabilityMetrics`, `sparkUI`, `className`, `extraJars`, `extraJarsFirst`,
+   * `extraPythonFiles`, `extraFiles`). Passing a construct-managed argument (e.g.
+   * `--enable-continuous-cloudwatch-log`, `--enable-metrics`, `--enable-spark-ui`,
+   * `--job-language`) or a Glue-reserved argument (`--debug`, `--mode`, `--JOB_NAME`) here throws
+   * at synthesis time, so there is exactly one way to express each intent.
+   *
    * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
    * for a list of reserved parameters
    * @default - no arguments
@@ -475,40 +484,98 @@ export abstract class Job extends JobBase {
     return new Import(scope, id);
   }
 
+  /**
+   * Argument keys that Glue reserves for its own use and that a caller must never set through
+   * `defaultArguments`. Unlike construct-managed arguments (which are derived from what each job
+   * class emits), these are owned by the Glue service itself and are reserved for every job type.
+   *
+   * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
+   */
+  private static readonly GLUE_RESERVED_ARGUMENTS = new Set(['--debug', '--mode', '--JOB_NAME']);
+
   /**
    * The IAM role Glue assumes to run this job.
    */
   public readonly abstract role: iam.IRole;
 
   /**
-   * Check no usage of reserved arguments.
+   * Merge the customer-supplied `defaultArguments` with the arguments this construct manages.
+   *
+   * The construct owns every argument it emits — whether the value comes from a dedicated typed
+   * prop (e.g. `continuousLogging`, `enableMetrics`, `sparkUI`) or from the job class itself
+   * (e.g. `--job-language`). Those arguments, plus the arguments Glue reserves for its own use,
+   * MUST be configured through their dedicated props rather than the untyped `defaultArguments`
+   * map, so there is exactly one way to express each intent. Passing such a key through
+   * `defaultArguments` therefore throws instead of silently winning or being silently dropped.
+   *
+   * A managed key whose supplied value is identical to the construct's value is not contradictory,
+   * so it is allowed rather than rejected (auto-correcting config is preferred over errors).
+   * Glue-reserved keys are never emitted by the construct, so there is no value to reconcile and
+   * they always throw.
+   *
+   * The reserved set is derived from `managedArguments` (the arguments the caller actually emits)
+   * rather than a hand-maintained list, so adding a new typed prop automatically reserves its
+   * argument key without a second place to update.
+   *
+   * Conflict detection relies on string equality of the argument keys, which cannot see through
+   * unresolved tokens (e.g. a key produced by `CfnJson` that only resolves at deploy time). If a
+   * key is a token, the check is skipped for that key and a synthesis-time warning is emitted, so
+   * the (rare) case where a token key resolves to a managed argument at deploy time — in which the
+   * construct-managed value would silently take precedence — is surfaced rather than hidden.
    *
    * @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
    */
-  protected checkNoReservedArgs(defaultArguments?: { [key: string]: string }) {
+  protected mergeManagedArguments(
+    managedArguments: { [key: string]: string },
+    defaultArguments?: { [key: string]: string },
+  ): { [key: string]: string } {
     if (defaultArguments) {
-      const reservedArgs = new Set(['--debug', '--mode', '--JOB_NAME']);
-      Object.keys(defaultArguments).forEach((arg) => {
-        if (reservedArgs.has(arg)) {
-          throw new cdk.ValidationError(lit`ReservedArgumentUsed`, `The ${arg} argument is reserved by Glue. Don't set it`, this);
-        }
-      });
+      if (Object.keys(defaultArguments).some((arg) => cdk.Token.isUnresolved(arg))) {
+        cdk.Annotations.of(this).addWarningV2(
+          'aws-cdk/aws-glue-alpha:tokenJobArgumentKey',
+          'defaultArguments contains an unresolved token as an argument key, so it cannot be checked for conflicts with construct-managed arguments. If it resolves to a managed argument at deploy time, the construct-managed value will take precedence. Configure managed arguments through their dedicated props (e.g. continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI).',
+        );
+      }
+      const conflicts = Object.keys(defaultArguments).filter(
+        (arg) => Job.GLUE_RESERVED_ARGUMENTS.has(arg)
+          || (arg in managedArguments && defaultArguments[arg] !== managedArguments[arg]),
+      );
+      if (conflicts.length > 0) {
+        throw new cdk.ValidationError(
+          lit`ManagedJobArgument`,
+          `the job argument(s) ${JSON.stringify(conflicts)} are managed by the construct or reserved by Glue; configure them through the corresponding props (e.g. continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI) instead of defaultArguments`,
+          this,
+        );
+      }
     }
-    return defaultArguments;
+    return { ...defaultArguments, ...managedArguments };
   }
 
   /**
    * Setup Continuous Logging Properties
    * @param role The IAM role to use for continuous logging
    * @param props The properties for continuous logging configuration
+   * @param securityConfiguration The security configuration attached to the job, if any
    * @returns String containing the args for the continuous logging command
    */
-  protected setupContinuousLogging(role: iam.IRole, props: ContinuousLoggingProps | undefined) : any {
+  protected setupContinuousLogging(role: iam.IRole, props: ContinuousLoggingProps | undefined, securityConfiguration?: ISecurityConfiguration) : any {
     // If the developer has explicitly disabled continuous logging return no args
     if (props && !props.enabled) {
       return {};
     }
 
+    // Continuous logging is on (explicitly or by default), but the logs will be written to an
+    // unencrypted CloudWatch log group unless a SecurityConfiguration is attached. We cannot
+    // introspect whether the attached SecurityConfiguration actually configures cloudWatchEncryption
+    // (the ISecurityConfiguration interface only exposes the name), so we only warn when none is
+    // attached at all to avoid false positives.
+    if (!securityConfiguration) {
+      cdk.Annotations.of(this).addWarningV2(
+        'aws-cdk/aws-glue-alpha:unencryptedContinuousLogging',
+        'Continuous CloudWatch logging is enabled but no SecurityConfiguration with cloudWatchEncryption is attached. Job stdout and stderr will be written to an unencrypted CloudWatch log group. See https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html',
+      );
+    }
+
     // Else we turn on continuous logging by default. Determine what log group to use.
     const args: {[key: string]: string} = {
       '--enable-continuous-cloudwatch-log': 'true',

packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts

@@ -93,10 +93,11 @@ export class PySparkEtlJob extends SparkJob {
     addConstructMetadata(this, props);
 
     // Combine command line arguments into a single line item
-    const defaultArguments = {
+    const managedArguments = {
       ...this.executableArguments(props),
       ...this.nonExecutableCommonArguments(props),
     };
+    const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments);
 
     this.resource = new CfnJob(this, 'Resource', {
       name: props.jobName,

packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts

@@ -81,10 +81,11 @@ export class PySparkFlexEtlJob extends SparkJob {
     addConstructMetadata(this, props);
 
     // Combine command line arguments into a single line item
-    const defaultArguments = {
+    const managedArguments = {
       ...this.executableArguments(props),
       ...this.nonExecutableCommonArguments(props),
     };
+    const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments);
 
     this.resource = new CfnJob(this, 'Resource', {
       name: props.jobName,

packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts

@@ -85,10 +85,11 @@ export class PySparkStreamingJob extends SparkJob {
     addConstructMetadata(this, props);
 
     // Combine command line arguments into a single line item
-    const defaultArguments = {
+    const managedArguments = {
       ...this.executableArguments(props),
       ...this.nonExecutableCommonArguments(props),
     };
+    const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments);
 
     this.resource = new CfnJob(this, 'Resource', {
       name: props.jobName,

packages/@aws-cdk/aws-glue-alpha/lib/jobs/python-shell-job.ts

@@ -7,7 +7,7 @@ import type { Construct } from 'constructs';
 import type { JobProps } from './job';
 import { Job } from './job';
 import type { Code } from '../code';
-import { JobType, GlueVersion, PythonVersion, MaxCapacity, JobLanguage } from '../constants';
+import { JobType, GlueVersion, PythonVersion, MaxCapacity, JobLanguage, LibrarySet } from '../constants';
 
 /**
  * Properties for creating a Python Shell job
@@ -27,6 +27,18 @@ export interface PythonShellJobProps extends JobProps {
    */
   readonly maxCapacity?: MaxCapacity;
 
+  /**
+   * The set of pre-installed Python libraries to make available to the job.
+   *
+   * Only applies to jobs running Python 3.9. Set to `LibrarySet.NONE` when your libraries are
+   * custom or conflict with the pre-installed ones.
+   *
+   * @default LibrarySet.ANALYTICS when running Python 3.9, otherwise no library set is configured
+   *
+   * @see https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library
+   */
+  readonly librarySet?: LibrarySet;
+
   /**
    * Additional Python files that AWS Glue adds to the Python path before executing your script.
    * Only individual files are supported, directories are not supported.
@@ -78,10 +90,13 @@ export class PythonShellJob extends Job {
     this.role = props.role;
     this.grantPrincipal = this.role;
 
-    // Enable CloudWatch metrics and continuous logging by default as a best practice
-    const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging);
-    const profilingMetricsArgs = { '--enable-metrics': '' };
-    const observabilityMetricsArgs = { '--enable-observability-metrics': 'true' };
+    // Enable continuous logging by default as a best practice. Note: the --enable-metrics and
+    // --enable-observability-metrics arguments are intentionally NOT set here. Those profiling
+    // metrics require the Spark/GlueContext instrumentation that Python shell jobs do not have, so
+    // Glue accepts but ignores them for the pythonshell command (verified: a pythonshell run with
+    // --enable-metrics emits no JobName-dimensioned CloudWatch metrics). See SparkJob/RayJob for
+    // the job types where these metrics apply.
+    const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging, props.securityConfiguration);
 
     // Gather executable arguments
     const executableArgs = this.executableArguments(props);
@@ -93,14 +108,12 @@ export class PythonShellJob extends Job {
     }
 
     // Combine command line arguments into a single line item
-    const defaultArguments = {
+    const managedArguments = {
       ...executableArgs,
       ...extraPythonFilesArgs,
       ...continuousLoggingArgs,
-      ...profilingMetricsArgs,
-      ...observabilityMetricsArgs,
-      ...this.checkNoReservedArgs(props.defaultArguments),
     };
+    const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments);
 
     this.resource = new CfnJob(this, 'Resource', {
       name: props.jobName,
@@ -143,10 +156,11 @@ export class PythonShellJob extends Job {
     const args: { [key: string]: string } = {};
     args['--job-language'] = JobLanguage.PYTHON;
 
-    // If no Python version set (default 3.9) or the version is set to 3.9 then set library-set argument
+    // The library-set option only applies to Python 3.9 (the default version). Default to the
+    // common analytics libraries, but let the caller override it (e.g. LibrarySet.NONE) via the
+    // typed prop. Note: Glue names this argument `library-set`, without the `--` prefix.
     if (!props.pythonVersion || props.pythonVersion == PythonVersion.THREE_NINE) {
-      // Selecting this option includes common libraries for Python 3.9
-      args['library-set'] = 'analytics';
+      args['library-set'] = props.librarySet ?? LibrarySet.ANALYTICS;
     }
 
     return args;

packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts

@@ -90,19 +90,19 @@ export class RayJob extends Job {
     this.grantPrincipal = this.role;
 
     // Enable CloudWatch metrics and continuous logging by default as a best practice
-    const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging);
+    const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging, props.securityConfiguration);
 
     // Conditionally include metrics arguments (default to enabled for backward compatibility)
     const profilingMetricsArgs = (props.enableMetrics ?? true) ? { '--enable-metrics': '' } : {};
     const observabilityMetricsArgs = (props.enableObservabilityMetrics ?? true) ? { '--enable-observability-metrics': 'true' } : {};
 
     // Combine command line arguments into a single line item
-    const defaultArguments = {
-      ...this.checkNoReservedArgs(props.defaultArguments),
+    const managedArguments = {
       ...continuousLoggingArgs,
       ...profilingMetricsArgs,
       ...observabilityMetricsArgs,
     };
+    const defaultArguments = this.mergeManagedArguments(managedArguments, props.defaultArguments);
 
     if (props.workerType && props.workerType !== WorkerType.Z_2X) {
       throw new ValidationError(lit`RayJobsOnlySupportZ2XWorkerType`, 'Ray jobs only support Z.2X worker type', this);