Docusaurus slow on a monorepo site due to globbing node_modules

[hariabinesh-l-4219]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Prerequisites

• I'm using the latest version of Docusaurus.
• I have tried the npm run clear or yarn clear command.
• I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• I have tried creating a repro with https://new.docusaurus.io.
• I have read the console error message carefully (if applicable).

Description

The issue seems to be globby scanning all node_modules folders which was the main reason for slowness.

Before fixing log:
[PERF] CLI start > Load site > Load plugins > Load plugins content > Load docs@default > contentLoaded() - 39.57 ms - (Heap 115mb -> 117mb / Total 272mb)
[PERF] CLI start > Load site > Load plugins > Load plugins content > Load docs@default - 34.38 seconds! - (Heap 87mb -> 117mb / Total 272mb)

After fixing:
[PERF] CLI start > Load site > Load plugins > Load plugins content > Load docs@default > contentLoaded() - 41.76 ms - (Heap 101mb -> 104mb / Total 222mb)
[PERF] CLI start > Load site > Load plugins > Load plugins content > Load docs@default - 329.71 ms - (Heap 94mb -> 104mb / Total 222mb)
[PERF] CLI start > Load site > Load plugins > Load plugins content - 329.90 ms - (Heap 94mb -> 104mb / Total 222mb)

async function readCategoriesMetadata(contentPath) {
    const categoryFiles = await (0, utils_1.Globby)('**/_category_.{json,yml,yaml}', {
        cwd: contentPath,
        ignore: ['**/node_modules/**', '**/dist/**'], --> fix
    });

Need to add ignore to globby to fix the slowness

Reproducible demo

No response

Steps to reproduce

Have a monorepo with node_modules linked to each packages.

Expected behavior

Docusaurus should start within seconds.

Actual behavior

Docusaurus is very slow to start

Your environment

• Public source code:
• Public site URL:
• Docusaurus version used:
• Environment name and version (e.g. Chrome 89, Node.js 16.4):
• Operating system and version (e.g. Ubuntu 20.04.2 LTS):

Self-service

• I'd be willing to fix this bug myself.

[beanscg]

Bean Labs took a source/docs pass. The useful public surfaces for this look like packages/docusaurus, docs build plugins, performance tests.

The highest-signal next step would be to split timing into content discovery, plugin loading, route generation, and bundling so the slow stage is visible before proposing a fix. That keeps the issue scoped to observable source/docs behavior before opening a PR.

No PR opened from this pass.

[slorber]

Have a monorepo with node_modules linked to each package.

Could you please tell us more, giving an example tree of what your monorepo looks like? I'd like to understand better before applying a fix.

[hariabinesh-l-4219]

@slorber

Our MonoRepo consists of below pnpm-workspace.yaml config.

hoist: true
hoistWorkspacePackages: true
linkWorkspacePackages: deep

|packages
|--buttons
|----node_modules
|------icons-svg
|--------1k+ files
|--icons-svg
|----1k+files though symlinked
|--accordion
|----node_modules
|------icons-svg
|--------1k+ files
|------buttons
|--------button files again

We have around 180+ packages. Some packages like svg icons have around 1k+ files ( individual tsx files ). Since linked deeply, majority of packages will be dependencies of other packages.