Feature(trace-deps, trace-pkg): User conditions

.changeset/lucky-cheetahs-relate.md

@@ -0,0 +1,8 @@
+---
+"trace-deps": minor
+"trace-pkg": minor
+---
+
+- Feature (`trace-pkg`): Add `conditions` option.
+- Feature (`trace-deps`): Add `conditions` parameter to `traceFile`/`traceFiles` to support user runtime loading conditions. (See [#trace-deps/56](https://github.com/FormidableLabs/trace-deps/issues/56))
+- BREAKING: Remove `production` and `development` from set of default user conditions in package resolution.

packages/trace-deps/README.md

@@ -26,6 +26,7 @@ _Parameters_:
 
 * `srcPath` (`string`): source file path to trace
 * `ignores` (`Array<string>`): list of package prefixes to ignore tracing entirely
+* `conditions` (`Array<string>`): list of Node.js runtime import [user conditions](https://nodejs.org/api/packages.html#resolving-user-conditions) to trace in addition to our default built-in Node.js conditions of `import`, `require`, `node`, and `default`.
 * `allowMissing` (`Object.<string, Array<string>`): Mapping of (1) absolute source file paths and (2) package name or relative file path keys to permitted missing module prefixes values.
     * Source file keys must match the entire file path (e.g., `/FULL/PATH/TO/entry.js`) while package keys are the start of package name either alone or with the rest of the relative path to ultimate file (e.g., `lodash`, `@scope/pkg` or `@scope/pkg/some/path/to/file.js`).
     * Missing module prefix values may be the package name or any part of the relative path thereafter (e.g., `pkg`, `pkg/some`, `pkg/some/path/to/file.js`)
@@ -69,6 +70,7 @@ _Parameters_:
 
 * `srcPaths` (`Array<string>`): source file paths to trace
 * `ignores` (`Array<string>`): list of package prefixes to ignore
+* `conditions` (`Array<string>`): list of Node.js runtime import user conditions to trace.
 * `allowMissing` (`Object.<string, Array<string>`): Mapping of source file paths and package names/paths to permitted missing module prefixes.
 * `bailOnMissing` (`boolean`): Throw error if missing static import.
 * `includeSourceMaps` (`boolean`): Include source map file paths from control comments
@@ -111,7 +113,7 @@ Examples:
 * **Modern Node.js ESM / `package.json:exports` Support**: Node.js v12 and newer now support modern ESM, and `trace-deps` will correctly package your application in any Node.js runtime. Unfortunately, the implementation of how to [resolve an ESM import](https://nodejs.org/api/packages.html) in modern Node.js is quite complex.
     * **It's complicated**: For example, for the same import of `my-pkg`, a `require("my-pkg")` call in Node.js v10 might match a file specified in `package.json:main`, while `require("my-pkg")` in Node.js v12 might match a second file specified in `package.json:exports:".":require`, and `import "my-pkg"` in Node,js v12 might match a _third_ file specified in `package.json:exports:".":import`. Then, throw in [conditions](https://nodejs.org/api/packages.html#packages_conditional_exports), [subpaths](https://nodejs.org/api/packages.html#packages_subpath_exports), and even subpath conditions, and it becomes exceedingly difficult to statically analyze what is actually going to be imported at runtime by Node.js ahead of time, which is what `trace-deps` needs to do. 🤯
     * **Our solution**: Our approach is to basically give up on trying to figure out the exact runtime conditions that will be used in module resolution, and instead package all reasonable conditions for a given module import. This means that maintain correctness at the cost of slightly larger zip sizes for libraries that ship multiple versions of exports.
-    * **Our implementation**: When `trace-deps` encounters a dependency, it resolves the file according to old CommonJS (reading `package.json:main`) and then in modern Node.js `package.json:exports` mode with each of the following built-in / suggested official conditions: `import`, `require`, `node`, `default`, `development`, and `production`. (We ignore `browser`).
+    * **Our implementation / conditions**: When `trace-deps` encounters a dependency, it resolves the file according to old CommonJS (reading `package.json:main`) and then in modern Node.js `package.json:exports` mode with each of the following built-in official conditions: `import`, `require`, `node`, `default`. We do not include any of the suggested user conditions (e.g., `production`, `development`, `browser`) by default. You can add additional user conditions using the `conditions` parameter.
     * **Missing Features**: `trace-deps` does not support the deprecated [subpath folder mappings](https://nodejs.org/api/packages.html#packages_subpath_folder_mappings) feature. Some advanced ESM features are still under development.
 
 * **Includes `package.json` files used in resolution**: As this is a Node.js-focused library, to follow the Node.js [module resolution algorithm](https://nodejs.org/api/modules.html#modules_all_together) which notably uses intermediate encountered `package.json` files to determine how to resolve modules. This means that we include a lot of `package.json` files that seemingly aren't directly imported (such as a `const pkg = require("mod/package.json")`) because they are needed for the list of all traced files to together resolve correctly if all on disk together.

packages/trace-deps/lib/trace.js

@@ -30,21 +30,33 @@ const RESOLVE_EXTS = [".js", ".json"];
 // https://github.com/FormidableLabs/trace-deps/issues/56
 const CONDITIONS = [
   // Node.js conditions
+  //
   // https://nodejs.org/api/packages.html#packages_conditional_exports
+  // > Node.js implements the following conditions, listed in order from most specific to
+  // > least specific as conditions should be defined:
+  // > - "node-addons"
+  // > - "node"
+  // > - "import"
+  // > - "require
+  // > - "default"
+  // NOTE: We don't include `node-addons` but could...
   "import",
   "require",
   "node",
   // Try `default` in both CJS + ESM modes.
   ["default", { require: true }],
-  ["default", { require: false }],
+  ["default", { require: false }]
 
   // Endorsed user conditions.
-  // https://nodejs.org/api/packages.html#packages_conditions_definitions
   //
-  // Note: We are ignoring
-  // - `browser`
-  "development",
-  "production"
+  // https://nodejs.org/api/packages.html#community-conditions-definitions
+  // > - "types"
+  // > - "deno"
+  // > - "browser"
+  // > - "development"
+  // > - "production"
+  //
+  // We do not default include any of these, but can be added with user-specified conditions.
 ];
 
 // Exports that disable modern ESM.
@@ -173,6 +185,7 @@ const _recurseDeps = async ({
   srcPaths,
   depPaths = [],
   ignores = [],
+  conditions = [],
   allowMissing = {},
   bailOnMissing = true,
   includeSourceMaps = false,
@@ -203,6 +216,7 @@ const _recurseDeps = async ({
       const traced = await traceFile({
         srcPath: depPath,
         ignores,
+        conditions,
         allowMissing,
         bailOnMissing,
         includeSourceMaps,
@@ -229,6 +243,7 @@ const _resolveDep = async ({
   depName,
   basedir,
   srcPath,
+  conditions,
   dependencies,
   extraDepKeys,
   addMisses,
@@ -339,13 +354,14 @@ const _resolveDep = async ({
           }
         });
 
-        CONDITIONS.forEach((cond) => {
+        CONDITIONS.concat(conditions).forEach((cond) => {
           let resolveOpts;
           if (Array.isArray(cond)) {
             resolveOpts = cond[1];
             cond = cond[0];
           }
 
+
           let relPath;
           try {
             relPath = resolveExports.resolve(pkg, depName, {
@@ -431,6 +447,7 @@ const _resolveDep = async ({
             depName,
             basedir,
             srcPath,
+            conditions,
             dependencies,
             extraDepKeys,
             addMisses,
@@ -487,6 +504,7 @@ const _resolveDep = async ({
  * @param {*}             opts                    options object
  * @param {string}        opts.srcPath            source file path to trace
  * @param {Array<string>} opts.ignores            list of package prefixes to ignore
+ * @param {Array<string>} opts.conditions         list of additional user conditions to trace
  * @param {Object}        opts.allowMissing       map packages to list of allowed missing package
  * @param {boolean}       opts.bailOnMissing      allow static dependencies to be missing
  * @param {boolean}       opts.includeSourceMaps  include source map paths in output
@@ -499,6 +517,7 @@ const _resolveDep = async ({
 const traceFile = async ({
   srcPath,
   ignores = [],
+  conditions = [],
   allowMissing = {},
   bailOnMissing = true,
   includeSourceMaps = false,
@@ -594,6 +613,7 @@ const traceFile = async ({
       depName,
       basedir,
       srcPath,
+      conditions,
       dependencies,
       extraDepKeys,
       addMisses,
@@ -628,6 +648,7 @@ const traceFile = async ({
   const recursed = await _recurseDeps({
     depPaths,
     ignores,
+    conditions,
     allowMissing,
     bailOnMissing,
     includeSourceMaps,
@@ -658,6 +679,7 @@ const traceFile = async ({
  * @param {*}             opts                    options object
  * @param {Array<string>} opts.srcPaths           source file paths to trace
  * @param {Array<string>} opts.ignores            list of package prefixes to ignore
+ * @param {Array<string>} opts.conditions         list of additional user conditions to trace
  * @param {Object}        opts.allowMissing       map packages to list of allowed missing package
  * @param {boolean}       opts.bailOnMissing      allow static dependencies to be missing
  * @param {boolean}       opts.includeSourceMaps  include source map paths in output
@@ -669,6 +691,7 @@ const traceFile = async ({
 const traceFiles = async ({
   srcPaths,
   ignores = [],
+  conditions = [],
   allowMissing = {},
   bailOnMissing = true,
   includeSourceMaps = false,
@@ -682,6 +705,7 @@ const traceFiles = async ({
   const results = await _recurseDeps({
     srcPaths,
     ignores,
+    conditions,
     allowMissing,
     bailOnMissing,
     includeSourceMaps,

packages/trace-deps/test/lib/trace.spec.js

@@ -2098,12 +2098,10 @@ describe("lib/trace", () => {
 
               expect(dependencies).to.eql(fullPaths([
                 "node_modules/complicated/default.js",
-                "node_modules/complicated/development.js",
                 "node_modules/complicated/import.mjs",
                 "node_modules/complicated/local/two.mjs",
                 "node_modules/complicated/main.js",
                 "node_modules/complicated/package.json",
-                "node_modules/complicated/production.mjs",
                 "node_modules/subdep/default.js",
                 "node_modules/subdep/from-default.js",
                 "node_modules/subdep/from-main.js",
@@ -2149,11 +2147,9 @@ describe("lib/trace", () => {
 
               expect(dependencies).to.eql(fullPaths([
                 "node_modules/complicated/default.js",
-                "node_modules/complicated/development.js",
                 "node_modules/complicated/local/one.js",
                 "node_modules/complicated/main.js",
                 "node_modules/complicated/package.json",
-                "node_modules/complicated/production.mjs",
                 "node_modules/complicated/require.js",
                 // Note: All of the import paths are to sub-paths, and **not**
                 // the root package, so no defaults in play.
@@ -2286,12 +2282,10 @@ describe("lib/trace", () => {
               const { dependencies, misses } = await traceFile({ srcPath });
 
               expect(dependencies).to.eql(fullPaths([
-                "node_modules/complicated/development.js",
                 "node_modules/complicated/import.mjs",
                 "node_modules/complicated/local/one.js",
                 "node_modules/complicated/local/two.mjs",
                 "node_modules/complicated/package.json",
-                "node_modules/complicated/production.mjs",
                 "node_modules/complicated/require.js",
                 "node_modules/subdep/default.js",
                 "node_modules/subdep/from-require.js",
@@ -2601,6 +2595,136 @@ describe("lib/trace", () => {
         expect(misses).to.eql({});
       });
     });
+
+    describe("user conditions", () => {
+      it("handles user conditions", async () => {
+        mock({
+          "first.js": `
+            const one = require("one");
+            const dynamicTwo = () => import("two");
+            const second = require("./second");
+          `,
+          "second.js": `
+            const one = require.resolve("one");
+
+            (async () => {
+              await import("three");
+            })();
+          `,
+          node_modules: {
+            one: {
+              "package.json": stringify({
+                name: "one",
+                main: "index.js",
+                exports: {
+                  ".": {
+                    bespoke: "./bespoke.js",
+                    production: "./production.js",
+                    "default": "./default.js"
+                  }
+                }
+              }),
+              "index.js": "module.exports = 'one';",
+              "default.js": "module.exports = 'default';",
+              "production.js": "module.exports = 'production';",
+              "bespoke.js": "module.exports = 'bespoke';"
+            },
+            two: {
+              "package.json": stringify({
+                name: "two",
+                main: "index.js",
+                exports: {
+                  ".": {
+                    bespoke: "./bespoke.js",
+                    production: "./production.js",
+                    require: "./require.js"
+                  }
+                }
+              }),
+              "index.js": "module.exports = 'two';",
+              "require.js": "module.exports = 'require';",
+              "production.js": "module.exports = 'production';",
+              "bespoke.js": "module.exports = 'bespoke';"
+            },
+            three: {
+              "package.json": stringify({
+                name: "three",
+                main: "index.mjs",
+                type: "module",
+                exports: {
+                  ".": {
+                    // NOTE: If first, `import` will override `production` with a `production`
+                    // condition in real Node.js runtime, so production _isn't_ matched here!
+                    "import": "./import.mjs",
+                    production: "./production.mjs"
+                  }
+                }
+              }),
+              "index.mjs": `
+                import three from "nested-three";
+                export default three;
+              `,
+              "import.mjs": `
+                import three from "nested-three";
+                export default three;
+              `,
+              node_modules: {
+                "nested-three": {
+                  "package.json": stringify({
+                    name: "nested-three",
+                    main: "index.mjs",
+                    type: "module",
+                    exports: {
+                      ".": {
+                        bespoke: "./bespoke.mjs",
+                        // NOTE: If a custom condition comes _before_ `import`, then it is actually
+                        // matched in real Node.js runtime.
+                        production: "./production.mjs",
+                        "import": "./import.mjs"
+                      }
+                    }
+                  }),
+                  "index.mjs": "export const three = 'nested-three';",
+                  "import.mjs": "export const msg = 'nested-import';",
+                  "production.mjs": "export const msg = 'nested-production';",
+                  "bespoke.mjs": "export const msg = 'nested-bespoke';"
+                }
+              }
+            }
+          }
+        });
+
+        const { dependencies, misses } = await traceFile({
+          srcPath: "first.js",
+          conditions: [
+            "bespoke",
+            "production"
+          ]
+        });
+        expect(dependencies).to.eql(fullPaths([
+          "node_modules/one/bespoke.js",
+          "node_modules/one/default.js",
+          "node_modules/one/index.js",
+          "node_modules/one/package.json",
+          "node_modules/one/production.js",
+          "node_modules/three/import.mjs",
+          "node_modules/three/index.mjs",
+          "node_modules/three/node_modules/nested-three/bespoke.mjs",
+          "node_modules/three/node_modules/nested-three/import.mjs",
+          "node_modules/three/node_modules/nested-three/index.mjs",
+          "node_modules/three/node_modules/nested-three/package.json",
+          "node_modules/three/node_modules/nested-three/production.mjs",
+          "node_modules/three/package.json",
+          "node_modules/two/bespoke.js",
+          "node_modules/two/index.js",
+          "node_modules/two/package.json",
+          "node_modules/two/production.js",
+          "node_modules/two/require.js",
+          "second.js"
+        ]));
+        expect(misses).to.eql({});
+      });
+    });
   });
 
   describe("traceFiles", () => {

packages/trace-pkg/README.md

@@ -70,6 +70,7 @@ Configuration options are generally global (`options.<OPTION_NAME>`) and/or per-
     - Can be overridden from CLI with `--concurrency <NUMBER>`
 - `options.includeSourceMaps` (`Boolean`): Include source map paths from files that are found during tracing (not inclusion via `include`) and present on-disk. Source map paths inferred but not found are ignored. (default: `false`). Please see [discussion below](#including-source-maps) to evaluate whether or not you should use this feature.
 - `options.ignores` (`Array<string>`): A set of package path prefixes up to a directory level (e.g., `react` or `mod/lib`) to skip tracing on. This is particularly useful when you are excluding a package like `aws-sdk` that is already provided for your lambda.
+- `options.conditions` (`Array<string>`): list of Node.js runtime import [user conditions](https://nodejs.org/api/packages.html#resolving-user-conditions) to trace in addition to our default built-in Node.js conditions of `import`, `require`, `node`, and `default`.
 - `options.allowMissing` (`Object.<string, Array<string>>`): A way to allow certain packages to have potentially failing dependencies. Specify each object key as either (1) an source file path relative to `cwd` that begins with a `./` or (2) a package name and provide a value as an array of dependencies that _might_ be missing on disk. If the sub-dependency is found, then it is included in the bundle (this part distinguishes this option from `ignores`). If not, it is skipped without error.
 - `options.dynamic.resolutions` (`Object.<string, Array<string>>`): Handle dynamic import misses by providing a key to match misses on and an array of additional glob patterns to trace and include in the application bundle.
     - _Application source files_: If a miss is an application source file (e.g., not within `node_modules`), specify the **relative path** (from the package-level `cwd`) to it like `"./src/server/router.js": [/* array of patterns */]`.
@@ -88,6 +89,7 @@ Configuration options are generally global (`options.<OPTION_NAME>`) and/or per-
 - `packages.<PKG_NAME>.trace` (`Array<string>`): A list of [fast-glob][] glob patterns to match JS files that will be further traced to infer all imported dependencies via static analysis. Use this option to include your source code files that comprises your application.
 - `packages.<PKG_NAME>.includeSourceMaps` (`Boolean`): Additional configuration to override value of `options.includeSourceMaps`.
 - `packages.<PKG_NAME>.ignores` (`Array<string>`): Additional configuration to merge with `options.ignores`.
+- `packages.<PKG_NAME>.conditions` (`Array<string>`): Additional configuration to merge with `options.conditions`.
 - `packages.<PKG_NAME>.allowMissing` (`Object.<string, Array<string>>`): Additional configuration to merge with `options.allowMissing`. Note that for source file paths, all of the paths are resolved to `cwd`, so if you provide both a global and package-level `cwd` the relative paths probably won't resolve as you would expect them to.
 - `packages.<PKG_NAME>.dynamic.resolutions` (`Object.<string, Array<string>>`): Additional configuration to merge with `options.dynamic.resolutions`.
 - `packages.<PKG_NAME>.dynamic.bail` (`Boolean`): Override `options.dynamic.bail` value.