[Feature Request]: Plugins from a marketplace

[mattcasters]

What would you like to happen?

Since Apache Hop is an ASF project, we already have access to the premier open-source artifact hosting infrastructure. The ASF provides a managed Sonatype Nexus instance (repository.apache.org) that automatically syncs with Maven Central.

How it works: Instead of bundling the plugins into the final assembly zip, we change the packaging of our plugins to separate, standalone artifacts (e.g., .zip or .jar files with their dependencies shaded or explicitly targeted). During your standard ASF release process, these are deployed to Maven Central as distinct modules.

How Hop handles it: we can build a lightweight "Plugin Manager" or "Marketplace" UI/CLI inside Hop. When a user wants a plugin, Hop uses a standard HTTP client or Maven resolver library to fetch the plugin zip directly from Maven Central using its GAV (Group, Artifact, Version) coordinates, then unpacks it into the local plugins/ directory.

Why it’s great: Zero infrastructure management for our team, massive global bandwidth via Central's CDN, and perfect compliance with the ASF lifecycle.

Issue Priority

Priority: 2

Issue Component

Component: Other

[mattcasters]

Summary of Proposed Plugin Repository Architecture

To address the growing size of the Hop assembly, we should transition toward a decentralized, on-demand "Plugin Repository" architecture. Because Apache Hop is an ASF project and already has key architectural foundations in place, we can implement a zero-maintenance, automated marketplace without needing external hosting infrastructure.

1. Leveraging Existing ASF Infrastructure (Maven Central / Nexus)

• Zero Infrastructure Costs: We do not need to host or manage our own artifact repository. Official Hop plugins can be published as standalone .zip or .jar artifacts to the official ASF Nexus (repository.apache.org) during the standard release process, which automatically syncs to Maven Central.
• Predictable Resolution: Because Maven Central uses a strict directory structure, the Hop client can predictably deduce the exact download URL for any given plugin using its standard GAV (Group, Artifact, Version) coordinates:
  [https://repo1.maven.org/maven2/](https://repo1.maven.org/maven2/)[group_path]/[artifactId]/[version]/[artifactId]-[version].zip

2. Fully Automated Discovery (Fixing the Kettle/Pentaho Flaw)

• The Problem with Old Models: The old Kettle/Pentaho marketplace relied on a static central XML/JSON file in a Git repository. This required manual upkeep, frequently broke, and led to stale or obsolete entries.
• The Dynamic Solution: We can leverage our existing strict naming conventions (org.apache.hop:hop-plugin-[type]-[name]) to treat Maven Central as a decentralized database. When a user opens the marketplace, the Hop GUI can query the public Maven Central Solr Search API directly for the user's specific running version:
  GET [https://search.maven.org/solrsearch/select?q=g](https://search.maven.org/solrsearch/select?q=g):"org.apache.hop"+AND+v:"2.18.1"&wt=json
• Self-Cleaning: This ensures perfect expiration management. If a plugin is removed or fails a specific release build, it simply won't appear in the API query results for that version. Dead plugins vanish gracefully without manual intervention.

3. Installation Workflow & UX

• Isolated Classloaders: Since Hop already utilizes isolated classloaders per plugin, we are protected against "dependency hell" from dynamically added libraries.
• The Staging & Restart Pattern: Instead of implementing risky runtime hot-reloading (which introduces memory leaks and file-locking issues, especially on Windows), we should lean into a simple GUI restart.

1. The user clicks "Install".
2. Hop downloads and extracts the plugin into a plugins/.staging/ directory.
3. Hop prompts the user to restart.
4. On reboot, the core launcher clears the old plugin directory (if updating), moves the staged files to active plugins/, and loads the application cleanly in a few seconds.

4. Enterprise-Ready Considerations

• Air-Gapped Environment Support: Many enterprise users run Hop in secure environments without internet access. We should abstract the search provider behind an interface and make the repository base URL a configurable property in hop-config.json. This allows corporate IT departments to point the marketplace to their internal JFrog Artifactory or Sonatype Nexus mirrors.
• API Caching: Implement a local 24-hour cache for marketplace search queries to keep the UI snappy and avoid spamming public repository APIs on every startup.

[mattcasters]

Tied to this concept I think I also want a way to describe hop installation configurations...

In modern DataOps and CI/CD pipelines, nobody wants to manually configure a server or click around a GUI to install plugins. They want to define the environment in a text file, commit it to Git, and have the deployment pipeline build it deterministically.

Here is an evaluation of why this idea is a game-changer, how the metadata could look, and a few architectural tips to make it bulletproof.

---

Why this is a Massive Win for Hop

1. The Ultimate CI/CD & Docker Companion

Right now, spinning up Hop in a Docker container or a Kubernetes pod requires custom Dockerfiles that curl zip files and extract them into directories. With a declarative file, a base Hop Docker image becomes incredibly lightweight. The container startup script simply runs:
hop install -f hop-env.yaml
The container configures itself on the fly based on the project's exact needs.

2. Solving the Eternal "JDBC Driver" Pain Point

In the data integration world, JDBC drivers are notorious. They aren't standard Hop plugins, but pipelines crash without them. By treating JDBC drivers as declared dependencies in this configuration file, Hop can fetch them from Maven Central/Artifactory just like a plugin and drop them into the correct shared classpath directory.

3. Production Governance ("Fail-Fast" Security)

Data engineers dread silent failures or partial runs caused by environment drift. If a developer upgrades a plugin locally but forgets to tell the DevOps team, the production pipeline might fail halfway through a critical data load. A startup check that says "Aborting execution: Environment mismatch detected" is worth its weight in gold.

---

Concept Design: hop-env.yaml

Here is a conceptual look at how elegant and readable this metadata file could be:

version: "1.0"
hopVersion: "2.18.1"

# Define where to look (Defaults to Maven Central if omitted)
repositories:
  - id: corporate-nexus
    url: "https://nexus.mycompany.internal/repository/maven-public/"

# The exact plugins this project requires
plugins:
  - artifactId: "hop-plugin-transform-rowgenerator"
    version: "2.18.1"
  - artifactId: "hop-plugin-database-postgresql"
    version: "2.18.1"
  - artifactId: "hop-plugin-transforms-aws-s3"
    version: "2.18.1"

# External dependencies like JDBC drivers
dependencies:
  - groupId: "org.postgresql"
    artifactId: "postgresql"
    version: "42.7.3"
  - groupId: "com.oracle.database.jdbc"
    artifactId: "ojdbc8"
    version: "19.3.0.0"

---

Architectural Deep Dive

If you implement hop install -f, keep these principles in mind:

The "Prune" (Sync) Capability

The CLI command should have two modes or a flag (e.g., --prune).

• Standard: Install anything missing from the file.
• Strict Sync: Install anything missing, and delete any plugin found in the plugins/ directory that is not listed in the file. This guarantees that the environment perfectly matches the Git-committed configuration, preventing developers from sneaking unapproved jars into production.

The Immutable Startup Check

When running a workflow via the CLI (hop-run), the very first lifecycle hook should be an environment validation check against the hop-env.yaml.

[hop-run triggered] ➔ [Scan local plugins/ directory] ➔ [Compare with hop-env.yaml]
                              │
               ┌──────────────┴──────────────┐
       [Matches Perfectly]            [Mismatch Found]
               │                             │
       [Execute Workflow]             [Hard Abort & Print Error]

If it detects a mismatch (e.g., a plugin is missing, or a version is 2.18.0 instead of 2.18.1), it should issue a hard abort before a single pipeline thread is spawned:

🛑 FATAL: Environment drift detected. Local plugin 'hop-plugin-transforms-aws-s3' is at version 2.18.0, but project metadata requires 2.18.1. Run 'hop install -f hop-env.yaml' to fix your environment.

Summary Verdict

This bridges the gap between old-school desktop ETL tools and modern cloud-native data engineering. It makes Hop behave like modern developer tools (think npm install, pip install -r, or cargo build). It provides corporate compliance, simplifies Docker deployments, and protects production data from configuration drift.

This would be a massive selling point for enterprise adoption.