issue #7547: a Hop marketplace by mattcasters · Pull Request #7577 · apache/hop

mattcasters · 2026-07-19T22:04:47Z

Summary

Implements apache/hop#7547: optional Wave 1 plugins out of the fat client, a bundled marketplace (CLI + GUI) that installs Maven GAV plugin zips into the client plugins/ tree, declarative hop-env.yaml, and local Nexus tooling for SNAPSHOT development.

Lean hop-client (~590 MB measured; under ~850 MB ASF limit); Wave 1 plugins optional (Spark, Beam plugin only, script, Cassandra, Tika, Drools, Parquet, Stanford NLP, Arrow, Dropbox, edi2xml). lib/beam stays in the core client.
Marketplace plugin: install/uninstall/list, repository management (ASF repository.apache.org primary + Maven Central fallback chain), GUI Tools → Marketplace with Manage repositories.
hop-env.yaml apply/validate and optional hop-run drift enforcement.
Local Sonatype Nexus OSS stack (docker/marketplace-nexus/) + smoke test; IT path re-injects Wave 1 via tools/install-wave1-plugins.sh.
Docs: user-manual marketplace page; release notes for deploying optional plugin zips; assembly size check guidance.

Done in this PR

Intentionally out of scope / follow-up

Maven Central search-driven catalog (GUI still uses curated Wave 1 list; searchApiUrl reserved)
True stage-before-plugin-scan on startup (install activates then restart for classloaders)
Re-enable Cassandra IT (disabled.txt left in place)
Full third-party plugin signing / hot-reload

Test plan

./mvnw -pl plugins/misc/marketplace test (28 tests)
./tools/check-assembly-size.sh — hop-client ~590 MB
Local Nexus: start / publish / smoke-test.sh and/or ./hop marketplace install hop-tech-parquet
Reviewer: rebuild client, ./hop marketplace repo list, install a Wave 1 plugin, restart, confirm plugin loads
Reviewer: ./hop marketplace apply -f sample hop-env against local or ASF repo when available

…lugins Split heavy plugins out of the default client so ASF Artifactory stays under the ~850MB package limit. Add a bundled marketplace plugin with CLI install/list/uninstall, keep lib/beam in core while making the Beam engine plugin optional, and add local Artifactory tooling plus a CI assembly size check.

Add Tools → Marketplace dialog for install/uninstall of optional plugins, a Wave 1 catalog, and install-wave1-plugins.sh wired into Jenkins and run-tests-docker so integration tests still get spark, beam, cassandra, and other marketplace-only plugins.

Add declarative environment files for marketplace plugins and JDBC dependencies: apply (--prune), validate, sample hop-env.yaml, and an optional HopRunStart enforcement when enforceOnRun is set.

…ce stack Generate and mount keys/master.key (openssl rand -hex 32, mode 600, owner 1030:1030) so JFrog Artifactory OSS no longer hangs at startup.

Single-node OSS still requires a join key; generate and mount both master.key and join.key (64-char hex, 600, 1030:1030) and point JF_SHARED_SECURITY_JOIN_KEY_FILE at the mount.

Modern Artifactory 7.x requires Postgres (no embedded DB). Add a postgres service, pass master/join keys via .env (not RO file mounts), fix master key size to AES-128 (hex-16), and add start.sh --reset.

…401) Send HTTP Basic credentials when installing plugins so private Maven repos work. Prefer HOP_MARKETPLACE_USERNAME/PASSWORD (or ARTIFACTORY_*) over storing secrets in hop-config.json.

…ce installs Add configure-anonymous-read.sh and document the intended security model: admin only for publish; anonymous read-only on hop-plugins-local for Hop downloads (no admin credentials in hop-config).

Repository create REST is Pro-only on Artifactory OSS. Rewrite configure-anonymous-read.sh to document UI steps and verify anonymous read instead of calling Pro APIs.

Clarify that create-repository REST is Pro-only and point users at the UI steps; keep configure-anonymous-read.sh as a verifier.

Document Administration module switching and deep links for repositories, anonymous access (Security → General), and permissions. Stop implying old "User Management → Settings" paths as primary.

Add docker/marketplace-nexus with scripted bootstrap (anonymous read, hosted hop-plugins repo), publish Wave 1 zips, and SNAPSHOT resolution via maven-metadata.xml. Remove docker/marketplace-artifactory entirely.

Relative HOP_HOME values were resolved against the current directory, doubling paths like assemblies/client/target/hop when already inside the install. Prefer any candidate that contains plugins/, including cwd.

…oke test Resolve the client install from process cwd (hop launchers already cd there), drop HOP_HOME/hop.home indirection, clarify hop-config is always under HOP_CONFIG_FOLDER, improve 401 auth errors, and add an anonymous Nexus CLI smoke script for install/list/validate/apply/uninstall.

…ll fallback Pre-configure repository.apache.org as primary with Maven Central fallback, add hop marketplace repo commands and Manage Repositories dialog, try enabled repos in order on install, and report disabled IT projects as skipped instead of a false docker environment failure.

…e gate Add user-manual marketplace page, release-manager notes for deploying optional plugin zips to ASF Nexus, and packaging/size guidance for the lean hop-client.

…v.yaml Expand the user-manual marketplace page with apply and validate options, file discovery, hop-run enforcement, and a full hop-env.yaml field reference with examples.

…gistry Add optional-plugins.yaml as the source of truth for marketplace-optional plugins, generate full-client-env.yaml at build time, package it at the Hop install root (and samples), load the GUI catalog from the registry, and document one-command restore of the previous full plugin set.

Add environment file path, Browse, Validate, and Apply to the Marketplace dialog (with Prune and Strict options matching the CLI), pre-fill full-client-env.yaml when present, and document the GUI flow.

Package Beam runners/libraries with the standard Beam marketplace plugin under lib-beam instead of lib/beam. Slim client libs keep spark-client and shared lib/core extras. Launch scripts add lib-beam only when present. Maven profile -Pbeam bakes the full Beam plugin into hop-client for IT/dev.

Add CLI query FILTER to search the local optional-plugins registry by case-insensitive substring across artifact id, name, category, description, and install path, with optional [installed] markers and docs.

… Nexus Add a developer-manual guide for packaging marketplace-compatible plugin zips, declaring Hop version and plugin dependencies, configuring Nexus, and automating releases with a template script for a specific plugin version.

hansva · 2026-07-20T10:05:20Z

We will have to take a look at our release process too; currently, we are not publishing zips to Maven Central.
Not 100% sure if this is/was a choice on our side or if it is part of the ASF publishing profile

…docs Improve the marketplace GUI with Plugins/Environment/Repositories tabs, inline repository and hop-env editors, search, and TableView lists. Add AttributesContext and IAttributes-based lifecycle environment hooks so marketplace can validate or enforce hop-env on project enable without a hard dependency between plugins. Include GUI screenshots, user manual updates, 13 locale translations, and related tooling.

Integrate upstream main (friction reduction, Databricks VFS, editor improvements, etc.) with marketplace work. Resolve conflicts in LifecycleEnvironmentDialog by keeping tabbed layout and marketplace XP tabs while retaining main's canvas text, import variables, and name suggestions.

Bring in ExecProcess single-command tokenization fix (apache#7581/apache#7584) and other main updates to unblock PR build.

…e checks Enable Jenkins wagon upload of marketplace plugin zips, keep release plugin assemblies, and add scripts/docs to verify file:// deploy layout before merging to main.

mattcasters · 2026-07-21T10:45:16Z

Marketplace plugin publishing — validation & testing overview

High-level steps taken to validate that optional marketplace plugins are packaged and can be published (including confidence for Jenkins SNAPSHOT deploy to repository.apache.org after merge to main).

1. Product packaging

Optional plugins are declared only in optional-plugins.yaml (catalog / full-client-env / publish lists).
Each module builds an installable artifactId-version.zip via the Maven assembly profile (appendAssemblyId=false).
Local package of registry modules produces all expected zips under plugins/**/target/.

2. Private / sandbox publish (data-hopper)

Deployed marketplace plugin SNAPSHOT zips to https://repository.data-hopper.com/repository/apache-hop-plugins/ via tools/deploy-snapshots-data-hopper.sh marketplace (credentials local only; not in git).
Fixed env precedence so local Docker Nexus .env does not override an explicit remote URL.
Confirmed HTTP 400 when the hosted repo was Release-only and SNAPSHOT was rejected; resolved by setting the repo version policy to Mixed.
Subsequent uploads of all optional plugins succeeded.
Marketplace install smoke against a Maven repo (install / list / validate / uninstall / apply) via docker/marketplace-nexus/dummy-staging.sh and related smoke tooling.

3. Jenkins-shaped SNAPSHOT layout (ASF CI confidence)

Apache Jenkins on main does not use the data-hopper script. It:

clean deploy into local-snapshots-dir (file-based alt deployment repository), then
on main, mvn -P deploy-snapshots wagon:upload to ASF snapshots.

Validation we ran for step (1) and the upload filter for step (2):

Root pom.xml deploy-snapshots wagon excludes: no longer drop plugin zips; still exclude hop-assemblies and core/engine/ui zips (so SNAPSHOT plugin zips can reach repository.apache.org).
maven-release-plugin: keeps per-plugin assembly (zips for releases/staging); still skips the hop-client assemblies reactor module.

Reproduced Jenkins deploy layout locally:

./mvnw -T 2 -B -DskipTests \
  -DaltDeploymentRepository=snapshot-repo::default::file:$(pwd)/local-snapshots-dir \
  clean deploy

→ BUILD SUCCESS.

Ran tools/verify-ci-snapshot-zips.sh against that tree → PASSED (ok=29 missing=0 for every artifact in optional-plugins.yaml).

This supports that a successful main CI Test & Build will place marketplace plugin .zip files where wagon upload can pick them up. It does not replace watching Jenkins Deploy after merge, or a post-merge check that a SNAPSHOT zip appears under e.g.
https://repository.apache.org/content/repositories/snapshots/org/apache/hop/hop-tech-parquet/.

4. Tooling added for repeatable checks

tools/list-marketplace-plugins.sh — registry → artifactId/modulePath
docker/marketplace-nexus/publish-marketplace-plugins.sh — package/deploy registry zips
docker/marketplace-nexus/dummy-staging.sh — publish + HTTP verify + install smoke
tools/deploy-snapshots-data-hopper.sh — private SNAPSHOT deploy helper
tools/verify-ci-snapshot-zips.sh — assert all optional plugins have zips in local-snapshots-dir

Residual risk (called out on purpose)

Full Jenkins test suite / agent environment was not fully re-run here (-DskipTests used for the file:// deploy layout check).
Actual ASF snapshot credentials and network upload only happen on Jenkins main Deploy.
Official release staging remains a separate RM path (release:perform + staging repo UI).

Resolve hop-gui.bat conflict: keep optional Beam classpath under plugins/engines/beam/lib-beam and take main's DEBUG log-level flag.

mattcasters · 2026-07-21T11:54:00Z

Adding plugin discovability and importing/exporting repositories through a YAML file, command line and GUI.

mattcasters added 16 commits July 19, 2026 21:22

issue apache#7547: hop-env.yaml apply/validate and hop-run drift check

202061f

Add declarative environment files for marketplace plugins and JDBC dependencies: apply (--prune), validate, sample hop-env.yaml, and an optional HopRunStart enforcement when enforceOnRun is set.

issue apache#7547: provide Artifactory master key for local marketpla…

6489579

…ce stack Generate and mount keys/master.key (openssl rand -hex 32, mode 600, owner 1030:1030) so JFrog Artifactory OSS no longer hangs at startup.

issue apache#7547: add Artifactory join.key so local stack can start

88b6d2c

Single-node OSS still requires a join key; generate and mount both master.key and join.key (64-char hex, 600, 1030:1030) and point JF_SHARED_SECURITY_JOIN_KEY_FILE at the mount.

issue apache#7547: run Artifactory OSS with PostgreSQL and env keys

f44eb6c

Modern Artifactory 7.x requires Postgres (no embedded DB). Add a postgres service, pass master/join keys via .env (not RO file mounts), fix master key size to AES-128 (hex-16), and add start.sh --reset.

issue apache#7547: Basic auth for marketplace downloads (Artifactory …

3c8a621

…401) Send HTTP Basic credentials when installing plugins so private Maven repos work. Prefer HOP_MARKETPLACE_USERNAME/PASSWORD (or ARTIFACTORY_*) over storing secrets in hop-config.json.

issue apache#7547: configure Artifactory anonymous read for marketpla…

40d19e0

…ce installs Add configure-anonymous-read.sh and document the intended security model: admin only for publish; anonymous read-only on hop-plugins-local for Hop downloads (no admin credentials in hop-config).

issue apache#7547: Artifactory OSS anonymous setup via UI (no Pro REST)

cbc5441

Repository create REST is Pro-only on Artifactory OSS. Rewrite configure-anonymous-read.sh to document UI steps and verify anonymous read instead of calling Pro APIs.

issue apache#7547: document Artifactory OSS UI-only repo/anonymous setup

f5f8a59

Clarify that create-repository REST is Pro-only and point users at the UI steps; keep configure-anonymous-read.sh as a verifier.

issue apache#7547: JFrog Platform UI guide with deep links for OSS setup

51bda71

Document Administration module switching and deep links for repositories, anonymous access (Security → General), and permissions. Stop implying old "User Management → Settings" paths as primary.

issue apache#7547: replace JFrog local stack with Sonatype Nexus OSS

2c39263

Add docker/marketplace-nexus with scripted bootstrap (anonymous read, hosted hop-plugins repo), publish Wave 1 zips, and SNAPSHOT resolution via maven-metadata.xml. Remove docker/marketplace-artifactory entirely.

issue apache#7547: fix Hop home when HOP_HOME is a relative path

14d24e1

Relative HOP_HOME values were resolved against the current directory, doubling paths like assemblies/client/target/hop when already inside the install. Prefer any candidate that contains plugins/, including cwd.

issue apache#7547: document marketplace, release plugin zips, and siz…

bef7c39

…e gate Add user-manual marketplace page, release-manager notes for deploying optional plugin zips to ASF Nexus, and packaging/size guidance for the lean hop-client.

github-actions Bot added Containers Documentation Integration Testing labels Jul 19, 2026

mattcasters added 6 commits July 20, 2026 00:10

issue apache#7547: document hop marketplace apply/validate and hop-en…

9f779b5

…v.yaml Expand the user-manual marketplace page with apply and validate options, file discovery, hop-run enforcement, and a full hop-env.yaml field reference with examples.

issue apache#7547: GUI Apply/Validate for hop-env environment files

6faaa9d

Add environment file path, Browse, Validate, and Apply to the Marketplace dialog (with Prune and Strict options matching the CLI), pre-fill full-client-env.yaml when present, and document the GUI flow.

issue apache#7547 : RAT

9e96b96

issue apache#7547: add hop marketplace query for optional plugin catalog

2dafe91

Add CLI query FILTER to search the local optional-plugins registry by case-insensitive substring across artifact id, name, category, description, and install path, with optional [installed] markers and docs.

mattcasters marked this pull request as ready for review July 19, 2026 23:16

mattcasters added 2 commits July 20, 2026 01:25

issue apache#7547 : RAT

b11d36c

issue apache#7547 : spotless

c1c4d8e

github-actions Bot added the Hop Gui label Jul 20, 2026

mattcasters changed the title ~~issue #7547: Plugin marketplace, lean client, and hop-env installs~~ issue #7547: a Hop marketplace Jul 20, 2026

mattcasters mentioned this pull request Jul 20, 2026

[Feature Request]: Validat the system Hop is running on #7583

Open

mattcasters added 2 commits July 20, 2026 17:33

Merge upstream/main into issue-7547

8a39bd1

Bring in ExecProcess single-command tokenization fix (apache#7581/apache#7584) and other main updates to unblock PR build.

issue apache#7547 : SNAPSHOT plugin zip publish path and CI confidenc…

8421f12

…e checks Enable Jenkins wagon upload of marketplace plugin zips, keep release plugin assemblies, and add scripts/docs to verify file:// deploy layout before merging to main.

Merge upstream/main into issue-7547

edc264b

Resolve hop-gui.bat conflict: keep optional Beam classpath under plugins/engines/beam/lib-beam and take main's DEBUG log-level flag.

mattcasters marked this pull request as draft July 21, 2026 11:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

issue #7547: a Hop marketplace#7577

issue #7547: a Hop marketplace#7577
mattcasters wants to merge 30 commits into
apache:mainfrom
mattcasters:issue-7547

mattcasters commented Jul 19, 2026

Uh oh!

hansva commented Jul 20, 2026

Uh oh!

mattcasters commented Jul 21, 2026

Uh oh!

mattcasters commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mattcasters commented Jul 19, 2026

Summary

Done in this PR

Intentionally out of scope / follow-up

Test plan

Uh oh!

hansva commented Jul 20, 2026

Uh oh!

mattcasters commented Jul 21, 2026

Marketplace plugin publishing — validation & testing overview

1. Product packaging

2. Private / sandbox publish (data-hopper)

3. Jenkins-shaped SNAPSHOT layout (ASF CI confidence)

4. Tooling added for repeatable checks

Residual risk (called out on purpose)

Uh oh!

mattcasters commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants