[Task]: Move to a single classpath

[mattcasters]

What needs to happen?

Investigation: Migrating Apache Hop to a Single Unified Classpath

This document outlines the design, changes, and benefits of migrating Apache Hop from separate classpaths (and classloaders) per plugin to a single, unified classpath.

---

1. Executive Summary

Currently, Apache Hop uses a modular classloading architecture where each plugin directory (under plugins/) has its own dependencies and is loaded using a separate instance of HopURLClassLoader.

While this provides dependency isolation between plugins, it introduces:

• High classloader overhead and potential memory leaks (metaspace exhaustion).
• Complexity in containerized environments (Docker) and distributed executors (Apache Spark, Apache Flink, Google Cloud Dataflow via the Apache Beam engine), where classloading isolation causes issues.
• Difficulty in native compilation (GraalVM).

Proposed Solution: Manifest-only Pathing Jar

To move to a single classpath, we can pre-calculate the list of all jar files in the distribution (including core, beam, and all plugins) during the maven assembly build of hop-client and package them into a manifest-only jar: lib/hop-classpath.jar.

This jar contains only a META-INF/MANIFEST.MF file listing all individual jar relative paths under lib/ and plugins/ in its Class-Path header. This avoids OS-specific command-line length limits (particularly the 8,191-character limit on Windows) while achieving a single classpath.

---

2. Java Runtime Changes

To support running on a single classpath, we can introduce a system property/environment flag, e.g., HOP_SINGLE_CLASSPATH=Y. When set, the PluginRegistry bypasses separate classloader creation and delegates all plugin loading to the system (App) classloader.

Proposed Diff: PluginRegistry.java

diff --git a/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java b/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
--- a/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
+++ b/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
@@ -867,6 +867,10 @@
   public ClassLoader getClassLoader(IPlugin plugin) throws HopPluginException {
 
     if (plugin == null) {
       throw new HopPluginException(
           BaseMessages.getString(
               PKG, "PluginRegistry.RuntimeError.NoValidTransformOrPlugin.PLUGINREGISTRY001"));
     }
 
+    // If single classpath mode is active, use the main system classloader for all plugins
+    if (plugin.isNativePlugin() || "Y".equalsIgnoreCase(System.getProperty("HOP_SINGLE_CLASSPATH"))) {
+      return this.getClass().getClassLoader();
+    }
+
     try {

Jandex Plugin Scanning

Because plugin jars are on the system classpath, JarCache.getNativeJars() will automatically discover all plugin META-INF/jandex.idx files and register them. The system property HOP_SINGLE_CLASSPATH=Y ensures that even when standard folders are scanned, they map back to the system classloader at runtime.

---

3. Maven Build & Pre-Calculation Changes

To pre-calculate the list of jars during the client build process, we modify the build cycle of assemblies/client.

Workflow

1. Unpack Dependencies: Use maven-dependency-plugin to unpack all dependency zips (static, plugins, core, engine, etc.) into a staging directory ${project.build.directory}/stage/hop.
2. Pre-calculate Classpath: Execute a Groovy script using groovy-maven-plugin (or gmavenplus-plugin) during the prepare-package phase to scan ${project.build.directory}/stage/hop for all jars, compute their relative paths (excluding platform-specific SWT jars), and write a manifest-only jar lib/hop-classpath.jar.
3. Assemble Zip: Configure maven-assembly-plugin to pack files directly from the stage folder.

Proposed Configuration in assemblies/client/pom.xml

<build>
    <plugins>
        <!-- 1. Unpack all zip dependencies to a staging directory -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>unpack-dependencies</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>unpack-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeTypes>zip</includeTypes>
                        <outputDirectory>${project.build.directory}/stage/hop</outputDirectory>
                        <excludeTransitive>true</excludeTransitive>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <!-- 2. Pre-calculate classpath and generate manifest-only jar -->
        <plugin>
            <groupId>org.codehaus.gmavenplus</groupId>
            <artifactId>gmavenplus-plugin</artifactId>
            <version>3.0.2</version>
            <executions>
                <execution>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>execute</goal>
                    </goals>
                    <configuration>
                        <scripts>
                            <script><![CDATA[
                                import java.util.jar.JarOutputStream
                                import java.util.jar.Manifest
                                import java.util.jar.Attributes

                                File stageDir = new File(project.build.directory, "stage/hop")
                                List<String> relativePaths = []

                                // Scan staging directory recursively for jar files
                                stageDir.eachFileRecurse { file ->
                                    if (file.name.endsWith(".jar") && !file.path.contains("/swt/")) {
                                        // Compute path relative to lib/ directory
                                        String relPath = stageDir.toPath().relativize(file.toPath()).toString().replace('\\', '/')
                                        if (relPath.startsWith("lib/")) {
                                            relativePaths.add(relPath.substring(4))
                                        } else {
                                            relativePaths.add("../" + relPath)
                                        }
                                    }
                                }

                                // Build the manifest with the Class-Path attribute
                                Manifest manifest = new Manifest()
                                manifest.mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
                                manifest.mainAttributes.put(Attributes.Name.CLASS_PATH, relativePaths.join(" "))

                                // Write manifest-only pathing jar
                                File classpathJar = new File(stageDir, "lib/hop-classpath.jar")
                                classpathJar.parentFile.mkdirs()
                                new JarOutputStream(new FileOutputStream(classpathJar), manifest).close()
                                log.info("Generated pre-calculated classpath manifest jar: " + classpathJar)
                            ]]></script>
                        </scripts>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Proposed Changes to assemblies/client/src/assembly/assembly.xml

    <fileSets>
        <!-- Package from the staging dir which now includes hop-classpath.jar -->
        <fileSet>
            <directory>${project.build.directory}/stage/hop</directory>
            <outputDirectory>.</outputDirectory>
        </fileSet>
    </fileSets>

---

4. Startup Script Changes

In the shell and batch scripts, we replace the directory wildcards (lib/core/*:lib/beam/*) with lib/hop-classpath.jar, dynamically adding only the platform-dependent SWT library jar. We also pass the -DHOP_SINGLE_CLASSPATH=Y runtime option.

A. Linux & OSX Scripts (.sh scripts: hop-gui.sh, hop-run.sh, etc.)

-  if "${_HOP_JAVA}" -XshowSettings:properties -version 2>&1 | grep -q "os.arch = aarch64"; then
-    CLASSPATH="lib/core/*:lib/beam/*:lib/swt/linux/arm64/*"
-  else
-    CLASSPATH="lib/core/*:lib/beam/*:lib/swt/linux/$(uname -m)/*"
-  fi
+  if "${_HOP_JAVA}" -XshowSettings:properties -version 2>&1 | grep -q "os.arch = aarch64"; then
+    SWT_JAR="lib/swt/linux/arm64/swt.jar"
+  else
+    SWT_JAR="lib/swt/linux/$(uname -m)/swt.jar"
+  fi
+  CLASSPATH="lib/hop-classpath.jar:${SWT_JAR}"
+  HOP_OPTIONS="${HOP_OPTIONS} -DHOP_SINGLE_CLASSPATH=Y"

B. Windows Scripts (.bat scripts: hop-gui.bat, hop-run.bat, etc.)

-set CLASSPATH=lib\core\*;lib\beam\*;lib\swt\win64\*
+set CLASSPATH=lib\hop-classpath.jar;lib\swt\win64\swt.jar
+set HOP_OPTIONS=%HOP_OPTIONS% -DHOP_SINGLE_CLASSPATH=Y

---

5. Benefits & Drawbacks

Benefits

• Significantly Simplified Classloading: Standard class resolution is handled directly by the JVM system classloader.
• Massive Reduction in Overhead: Avoids spawning hundreds of HopURLClassLoader instances, reducing JVM Metaspace consumption and startup time.
• Distributed Compatibility: Facilitates executing pipelines on external clusters (Spark/Flink) since the entire Hop execution engine + plugins can be shipped using regular classpaths/jars.
• Solves Windows Command Limit: The manifest jar keeps the execution command short, preventing Windows 8,191-character limit crashes.

Drawbacks & Mitigation

• Dependency Conflicts: If two plugins include different versions of the same third-party library, classpath conflicts (NoSuchMethodError, etc.) may arise.
  • Mitigation: These conflicts are now surfaced at compile/build time when constructing the client rather than silently failing or causing classloader issues at runtime. Surfacing this enforces dependency alignment across the project.
• Dynamic Loading of JDBC Drivers: The driver installer hot-loading logic (DriverInstaller.hotLoad()) expects HopURLClassLoader to inject downloaded JDBC drivers.
  • Mitigation: Users should add custom JDBC drivers to the shared folders (HOP_SHARED_JDBC_FOLDERS), or place them directly in the classpath via options.

Issue Priority

Priority: 3

Issue Component

Component: API