Skip to content

Sync shared package via repository_dispatch and stop Dependabot duplicates#774

Open
edvilme wants to merge 14 commits into
mainfrom
shared-package-sync
Open

Sync shared package via repository_dispatch and stop Dependabot duplicates#774
edvilme wants to merge 14 commits into
mainfrom
shared-package-sync

Conversation

@edvilme

@edvilme edvilme commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Two related changes so the shared package (@vscode/common-python-lsp npm / vscode-common-python-lsp pip) is updated via a repository_dispatch event from vscode-common-python-lsp instead of by Dependabot.

1. New .github/workflows/shared-package-release.yml

Handles repository_dispatch (shared-package-release): branches off main, bumps the npm dependency, rewrites the pip pin in requirements.in and recompiles requirements.txt with uv, then pushes the branch and opens a tracking issue with a manual compare/PR link (org settings prevent the workflow from opening PRs automatically).

2. .github/dependabot.yml

Moves the shared package from the group exclude-patterns (which still produced a standalone Dependabot PR) into the npm and pip ignore lists, so Dependabot no longer opens duplicate PRs for it.

…icates

Adds .github/workflows/shared-package-release.yml to handle
repository_dispatch (shared-package-release) events from
vscode-common-python-lsp: it bumps the npm dep, recompiles the pip pin via
uv, pushes a branch, and opens a tracking issue with a manual-PR link
(org settings block auto-created PRs).

Removes @vscode/common-python-lsp (npm) and vscode-common-python-lsp (pip)
from Dependabot via the ignore lists so Dependabot no longer opens
duplicate update PRs for the shared package.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@edvilme edvilme added the debt Technical debt or repo cleanup label Jun 30, 2026
Replace the inline Python rewrite of requirements.in with a sed one-liner,
keeping uv pip compile --generate-hashes --upgrade to regenerate the
hash-locked requirements.txt (matching the command documented in
requirements.in).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@edvilme edvilme enabled auto-merge (squash) June 30, 2026 21:44
@rchiodo

rchiodo commented Jun 30, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR via the Review Center.

Other automated reviewers defer to this claim to avoid duplicate, conflicting feedback. The claim is refreshed while the review is active; if it is not refreshed within the ownership timeout it is treated as vacant and another reviewer may take over. This is an automated marker.

Comment thread .github/workflows/shared-package-submodule-sync.yml Outdated
Comment thread .github/workflows/shared-package-release.yml Outdated
Comment thread .github/workflows/shared-package-release.yml Outdated
Comment thread .github/workflows/shared-package-release.yml Outdated
Comment thread .github/workflows/shared-package-release.yml Outdated
Comment thread .github/workflows/shared-package-submodule-sync.yml
@rchiodo

rchiodo commented Jun 30, 2026

Copy link
Copy Markdown

Solid approach overall. Main themes to address: make the workflow idempotent/re-runnable (branch push + issue creation), validate the incoming payload, and align the Python version with the documented lockfile-compile process.

@rchiodo

rchiodo commented Jun 30, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR via the Review Center.

Other automated reviewers defer to this claim to avoid duplicate, conflicting feedback. The claim is refreshed while the review is active; if it is not refreshed within the ownership timeout it is treated as vacant and another reviewer may take over. This is an automated marker.

@edvilme edvilme disabled auto-merge June 30, 2026 22:33
- Pin uv pip compile to the Python version documented in requirements.in
  (--python-version) so markers like exceptiongroup are not dropped
- Scope the recompile with --upgrade-package so unrelated deps don't move
- Assert the pip pin actually changed after sed; fail fast otherwise
- Validate release_tag is present and version-shaped before any work
- Add concurrency group, --force-with-lease, tracking-issue dedup, and
  tolerate Issues being disabled (surface compare URL via job summary)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@edvilme

edvilme commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @rchiodo — addressed the feedback in the latest commit:

  • Python version / reproducibility: the workflow now pins PYTHON_VERSION to the version documented in this repo's requirements.in and passes --python-version to uv pip compile, so environment markers (e.g. exceptiongroup) aren't dropped when the lockfile is regenerated.
  • Blanket upgrade: replaced --upgrade with --upgrade-package "${PIP_DEP}" so only the shared package (and its forced transitives) move.
  • Silent sed no-op: the pin rewrite is now case-insensitive and is followed by a grep assertion that fails the job if requirements.in wasn't actually re-pinned.
  • Payload validation: a new step fails fast if release_tag is empty or not version/ref-shaped (^v?[0-9A-Za-z][0-9A-Za-z.+-]*$), preventing degenerate branches and npm install pkg@ resolving to latest.
  • Idempotency / re-runs: added a concurrency group keyed on release_tag, switched the push to --force-with-lease (fetching any existing remote branch first), de-duplicated the tracking issue (reuse/comment on an existing open one), and made issue creation tolerant of Issues being disabled by also writing the compare URL to the job summary.

Comment thread .github/workflows/shared-package-submodule-sync.yml Outdated
Comment thread .github/workflows/shared-package-release.yml Outdated
@rchiodo

rchiodo commented Jun 30, 2026

Copy link
Copy Markdown

Most of the first-pass concerns are already handled in this revision (concurrency group, payload validation, PYTHON_VERSION: '3.10', issue-dedup with fallback, and the post-sed pin assertion all exist). The one substantive remaining issue is that --force-with-lease no longer actually protects the branch because of the preceding fetch into its tracking ref.

edvilme and others added 2 commits July 1, 2026 14:48
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread .github/workflows/shared-package-submodule-sync.yml Outdated
Comment thread .github/workflows/shared-package-submodule-sync.yml Outdated
Comment thread package.json Outdated
Comment thread .github/workflows/shared-package-submodule-sync.yml
@rchiodo

rchiodo commented Jul 1, 2026

Copy link
Copy Markdown

GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for requirements.txt:L188.

📍 requirements.txt:185
The Architect observes this file was hand-edited: the pin was removed but lsprotocol/pygls/packaging still carry # via … vscode-common-python-lsp annotations, and the header still claims uv pip compile … ./requirements.in. Runtime is fine (those deps are pulled transitively via pygls), but the next genuine uv pip compile will produce an unexpected diff. Regenerate with uv pip compile so the header command and via annotations are truthful.

[verified]

@rchiodo

rchiodo commented Jul 1, 2026

Copy link
Copy Markdown

Main themes: (1) the submodule-sync workflow's --force-with-lease safety net is effectively a plain --force because the remote branch ref is fetched right before the push; (2) build/clone robustness — the file: link + postinstall build break plain (non---recurse-submodules) clones and the root lockfile goes stale when the gitlink advances, so the produced branch may fail npm ci. Worth tightening the push lease and guarding the postinstall.

@rchiodo

rchiodo commented Jul 1, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR.

Consolidate the sync steps into a single script, add a retry wrapper for
network operations, fail when the released tag is missing instead of
silently falling back to main, and surface failures as workflow annotations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread .github/workflows/shared-package-submodule-sync.yml
Add a Development section explaining that the shared vscode-common-python-lsp
library is a git submodule and how to initialize it (clone with
--recurse-submodules, or git submodule update --init --recursive for an
existing clone) before running npm install.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
echo 'exists=false' >>"${GITHUB_OUTPUT}"
fi

- name: Create the release branch from the latest main

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python-compat guard validates the wrong "minimum": EXT_MIN is parsed from runtime.txt (the bundling Python), not the extension's declared minimum supported interpreter in src/common/constants.ts/noxfile.py. If runtime.txt is 3.13 but the extension still supports 3.9, the guard computes 3.13 >= PKG_FLOOR and passes — yet a user on 3.9 hits an ImportError from the bundled lib, the exact failure this guard claims to prevent. The error message names three files but only inspects one. Compare PKG_FLOOR against the declared minimum, not runtime.txt.

echo 'exists=false' >>"${GITHUB_OUTPUT}"
fi

- name: Create the release branch from the latest main

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two structural issues in the compat guard: (1) PKG_FLOOR is parsed from ${SUBMODULE_PATH}/python/pyproject.toml's requires-python, so any upstream repackaging (moved file, setup.cfg, or a dropped requires-python) yields an empty value and hard-fails every future sync until this workflow is edited. (2) It runs before the no-op short-circuit (git diff --quiet), so a re-dispatch that genuinely has nothing to do can still fail loudly. Consider treating an unparseable floor as warn-and-skip, and reordering the no-op check ahead of the drift guard.

return 1
fi
echo "Attempt ${attempt} failed: $* — retrying..." >&2
attempt=$((attempt + 1))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Narrow notification gap: if git push succeeds but the runner dies before the tracking-issue/summary step, a re-dispatch hits this guard-and-skip and exit 0s before ever emitting a tracking issue or compare URL — the branch is left orphaned with no maintainer signal. On the skip path, consider checking for an existing tracking issue for ${BRANCH} and re-emitting the compare URL to $GITHUB_STEP_SUMMARY before exiting. Low priority; the guard-and-skip itself is correct.

break
fi
done
if [ -z "${TARGET}" ]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

npm install --package-lock-only --ignore-scripts is not wrapped in the retry helper used for fetch/push, so a flaky registry fails the job late. Because the root manifest carries ^/range deps, install (not ci) can also resolve newer transitives and commit lockfile churn beyond the gitlink bump. Consider wrapping it in retry and/or asserting the lockfile delta is limited to the linked package. Low priority.

@rchiodo

rchiodo commented Jul 2, 2026

Copy link
Copy Markdown

Overall the submodule-sync approach looks solid. A few robustness notes on the sync workflow's Python-floor guard and dependency-constraint coupling are worth a look, but nothing blocking.

rchiodo
rchiodo previously approved these changes Jul 2, 2026

@rchiodo rchiodo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

Add a guard that fails the sync if the extension's pinned Node (.nvmrc) is
older than the shared package's pinned Node (submodule .nvmrc), since the
extension builds the shared package's TypeScript during install.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rchiodo

rchiodo commented Jul 2, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR.

working-directory: ${{ env.SUBMODULE_PATH }}
run: |
set -euo pipefail
retry git fetch --tags --force origin

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Node guard validates against the wrong "minimum." It parses PKG_NODE from the submodule's .nvmrc (an exact dev pin) and requires EXT_NODE >= PKG_NODE, but the submodule's actual buildability floor is engines.node (>=18.0.0). If the submodule bumps its .nvmrc above the extension's Node version while still declaring engines.node >=18, the extension can build it fine yet this step hard-fails the whole sync. The strict full-X.Y.Z regex also hard-fails every future sync if the submodule .nvmrc is absent or partial (22, lts/*). Prefer comparing against engines.node (or drop the guard and let npm ci/postinstall fail loudly), and treat an unparseable floor as warn-and-skip.

@@ -0,0 +1,221 @@
name: Shared Package Submodule Sync

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description is stale and now misleading: it narrates a shared-package-release.yml doing uv pip compile/sed/requirements.in rewriting, but the shipped file is shared-package-submodule-sync.yml doing git-submodule vendoring. Only the Dependabot section is accurate. Please rewrite the description to match the submodule-sync mechanism before merge.

TARGET="${CANDIDATE}"
break
fi
done

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two structural points on the compat guards. (1) They reach across the repo boundary into the submodule's internal layout (${SUBMODULE_PATH}/python/pyproject.toml, ${SUBMODULE_PATH}/.nvmrc), so any upstream repackaging (moved file, setup.cfg, dropped requires-python) hard-fails every future sync until this workflow is edited. (2) Both guards run before the step-5 git diff --quiet no-op check, so a genuinely no-op re-dispatch does the extra work and can fail loudly with nothing to sync. Consider reordering the no-op detection ahead of the drift guards, and treating an unparseable floor as warn-and-skip.

node-version-file: .nvmrc

# Expose a `retry` helper to every later bash step via BASH_ENV so the
# network operations below (fetch/push) stay readable one-liners.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The branch-exists guard treats any git ls-remote failure as "branch absent" — a transient network error falls into the elseexists=false → the workflow proceeds to create and push. Unlike the later network ops, this ls-remote is not retried. Consider wrapping it in retry, or distinguishing exit code 2 (no ref) from a genuine failure, so a network blip can't bypass the guard.

@rchiodo

rchiodo commented Jul 2, 2026

Copy link
Copy Markdown

Overall this is sound. Two themes worth addressing before merge: (1) the Node/Python compatibility guards in the sync workflow validate against the submodule's exact dev pins (.nvmrc, runtime.txt) rather than its declared build/support floors (engines.node, declared minimum Python), which can hard-fail future syncs on benign upstream changes; and (2) the PR description still describes a different mechanism (shared-package-release.yml doing uv pip compile) than what shipped (shared-package-submodule-sync.yml doing git-submodule vendoring) — please update it.

rchiodo
rchiodo previously approved these changes Jul 2, 2026

@rchiodo rchiodo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

Address reviewer feedback on the sync workflow:
- Retry the branch-existence ls-remote and only treat a definitive
  'no such ref' (exit 2) as absent, so a transient network error can no
  longer bypass the guard and clobber an existing branch.
- Run the no-op detection before the compatibility guards so a re-dispatch
  with nothing to sync exits cleanly instead of failing loudly.
- Compare the Python floor against the extension's declared minimum in
  src/common/constants.ts (not the bundling interpreter in runtime.txt),
  and the Node floor against the shared package's engines.node build floor
  (not the exact .nvmrc dev pin).
- Treat an unparseable package floor as a warning instead of hard-failing
  every future sync.
- Wrap 'npm install --package-lock-only' in the retry helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rchiodo

rchiodo commented Jul 2, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR.

set +e
git ls-remote --exit-code --heads origin "refs/heads/${BRANCH}" >/dev/null 2>&1
rc=$?
set -e

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📍 .github/workflows/shared-package-submodule-sync.yml:88
Notification gap on the guard-skip path: if git push succeeds but the runner dies before the "Open or update the tracking issue" step, a re-dispatch hits the branch-exists guard (exists=true) and skips all gated steps — so the pushed branch is left with no tracking issue or compare URL. The guard conflates "maintainer is editing" with "prior run crashed mid-flight." On the exists=true path, consider re-emitting the compare URL to $GITHUB_STEP_SUMMARY (and/or checking for an existing tracking issue) before finishing. Low priority; the guard-and-skip itself is correct.

@rchiodo

rchiodo commented Jul 2, 2026

Copy link
Copy Markdown

Overall this is sound. One real structural gap worth tracking (submodule Python runtime deps are satisfied only by hand-pins in requirements.in with no drift enforcement) plus a stale PR description. Nothing blocking.

rchiodo
rchiodo previously approved these changes Jul 2, 2026

@rchiodo rchiodo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@edvilme edvilme requested a review from bschnurr July 2, 2026 21:40
@edvilme edvilme enabled auto-merge (squash) July 2, 2026 22:09
Point external/vscode-common-python-lsp at the v0.8.1 release and refresh
package-lock.json to keep the branch npm ci-mergeable. Python (>=3.10) and
Node (engines.node >=18.0.0) compatibility floors are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rchiodo

rchiodo commented Jul 3, 2026

Copy link
Copy Markdown

🔒 Automated review in progress — @rchiodo is auto-reviewing this PR.


# The extension must not claim support for an older Python than the shared
# package requires, or the bundled library would fail to import at runtime.
# Compare the package floor against the extension's *declared* minimum in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📍 .github/workflows/shared-package-submodule-sync.yml:145
The Node guard parses PKG_NODE as the first integer sequence in engines.node, which has two latent edge cases: (1) sort -V orders a bare-major .nvmrc (e.g. 18) before a full 18.0.0 floor, spuriously hard-failing a compatible sync; and (2) a compound range like >=20 || ^18.18.0 or >=18 <=20 is mis-classified (first-integer-wins, no upper-bound check). Neither trips today (.nvmrc is full 24.17.0, engines.node is open-ended >=18.0.0). Normalize both operands to a common component count (or use semver) if you want to harden the floor comparison.

@rchiodo

rchiodo commented Jul 3, 2026

Copy link
Copy Markdown

Overall this looks solid and is safe to merge. A few optional hardening notes are left inline (Python dep drift, workflow version-parsing edge cases, and a crash-window notification gap); none are blocking.

@rchiodo rchiodo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debt Technical debt or repo cleanup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants