[Bug]: JGit MissingObjectException in Hop 2.18.1 (Regression from 2.17.0) due to native Git index formats

[ch2dit-byte]

Apache Hop version?

2.18.1

Java version?

OpenJDK 21+35

Operating system

Windows

What happened?

When opening a project in Hop GUI that was cloned, fetched, or pulled using Native Git, a org.eclipse.jgit.errors.MissingObjectException is thrown, and the latest commits are not recognized in the Hop Git Perspective.

This issue occurs because the JGit engine bundled with Hop 2.18.1 seems unable to parse modern Native Git optimization files (such as commit-graph, multi-pack-index, or pack.useBitmaps).
Note: This issue did NOT occur in Apache Hop 2.17.0. It seems to be a regression related to a JGit version bump or how Hop handles git objects in 2.18.1.

Issue Priority

Priority: 3

Issue Component

Component: Hop Gui

[mattcasters]

Investigation notes (2.18.1 vs 2.17.0)

I looked into this report on current main / 2.18.x and tried to reproduce the stated JGit / native-git format cause.

Confirmed

1. Regression surface vs 2.17.0 is real in practice — not only a silent dependency bump:

   • JGit: 7.3.0.202506031305-r (2.17) → 7.5.0.202512021534-r (2.18 / 2.18.1), via the dependency update in fix delete mails after fetch, update dependencies, minor cleanup of docs, fixes #6713, #6719 #6721.
   • Git UI: 2.18 also added the full Git Perspective ([Feature Request]: Improvements to Git integration #6914 / Improvements to Git integration #6914 #7053). 2.17 did not have GitPerspective / PlotWalk-based history; it mainly used explorer git info + toolbar actions via UIGit / GitGuiPlugin.

2. The screenshots match Hop code. The log line:

   Error checking if commit is in current branch: Missing unknown …

   comes from GitPerspective.isCommitInCurrentBranch(), which does:

   RevCommit headCommit = walk.parseCommit(repository.resolve(Constants.HEAD));
   RevCommit commitToCheck = walk.parseCommit(commit.getId());
   return walk.isMergedInto(commitToCheck, headCommit);

   So the failure is RevWalk.parseCommit(...) → JGit object lookup, not a Hop-specific object format parser.

3. Missing unknown means the object was not found in the object database (type hint OBJ_ANY). That is “object absent / not readable via JGit’s ODB,” not “failed to decode a commit-graph chunk.”

Not confirmed (root-cause theory)

The suggestion that JGit 7.5 “can’t parse modern native Git optimization files” (commit-graph, multi-pack-index, pack bitmaps) was not reproduced here.

JGit 7.5 includes commit-graph / multi-pack-index / pack-bitmap support. Against local repos with those artifacts, both JGit 7.3 and 7.5 successfully:

• resolved HEAD and parseCommit
• ran PlotWalk (same style of walk the perspective uses)
• ran git.log
• walked history after concurrent native commit + repack -Ad --write-bitmap-index + commit-graph write while a JGit Repository stayed open
• handled shallow clones (--depth 1 / 5)

So a pure “7.5 can’t read commit-graph / midx / bitmaps” explanation is not supported by these tests. Those indexes are optional accelerators; the observed exception is specifically that an object id could not be opened from the ODB.

Better working model

• Symptom is consistent with object-not-found while the new 2.18 Git Perspective aggressively walks/parses commits (history refresh, branch membership checks, file diffs).
• The JGit bump is concurrent with the regression window, but format incompatibility with native optimization files has not been demonstrated.
• Other plausible causes still open: incomplete/corrupt local object store, refs pointing at objects not fully present locally, dual native-git / JGit use in a bad state, Windows-specific pack/path/locking issues (reporter is on Windows; not reproduced here).

Questions for the reporter (to pin this down)

If you can share any of the following from the same machine/repo that fails in Hop, that would help a lot:

1. Full object id from the Missing unknown … dialog/stack (complete 40-char SHA-1 if possible).
2. Does native git see that object?

   git cat-file -t <object-id>
   git rev-parse HEAD
   git fsck --no-full 2>&1 | head

3. After a failure, does removing optional indexes change Hop behavior? (backup first)

   # optional experiment
   mv .git/objects/info/commit-graph .git/objects/info/commit-graph.bak 2>/dev/null
   mv .git/objects/pack/multi-pack-index .git/objects/pack/multi-pack-index.bak 2>/dev/null

   Then reopen the project in Hop. If it still fails, the indexes are unlikely the cause.
4. How was the repo obtained/updated (clone/fetch/pull client, shallow clone, partial clone, worktree, submodule)?
5. Does git log -1 --oneline and the forge UI show commits that Hop’s Git Perspective does not list at all, or does history partially load and only some actions fail?

Happy to dig further once we have a concrete object id / repo shape.

[ch2dit-byte]

Thank you for the quick reply!

git cat-file -t
git rev-parse HEAD
git fsck --no-full 2>&1 | head

rm -f logs/HEAD
rm -f logs/refs/heads/main
git fsck --no-full 2>&1 | head

I cleared the corrupted logs using the commands above.
However, when I restarted Apache Hop and tried to switch projects, I encountered an error like edp_meta -> Test.

I tried implementing it again following my original steps

Gitea version: 1.26.4
Repositories: 2 repositories (admin/test and edp/edp_meta)
For admin/test, I pushed pipelines and workflows created in Apache Hop 2.17.0.
I set up a separate BAT script to sync git automatically when Hop starts.

The git sync script executes the following:
- Cleaned up residual Git lock files and graph indexes before pulling.
- Added automatic --unshallow fetch for shallow clones.
- Executed pull with --rebase --autostash, disabling auto-gc to prevent background lock collisions.
- Added a 1-second buffer delay to ensure file handles are safely released.
- Added post-pull repository optimizations (remote prune origin, pack-refs --all).
- Included error handling to automatically abort rebase on failure and detailed timestamp logging.

I added the following to my Git global options to fix compatibility issues between JGit and native Git. (Assisted by AI)

[Steps taken]

1. Reinstalled Apache Hop (using the zip file). However, I couldn't initialize/reset the Gitea repository.
2. Moved to the Hop repository folder and ran git clone

3. In Hop, registered the cloned folder as the home folder by selecting "Add Project" -> "In a folder".
   a. Added admin/test (successfully).
   b. An error occurred while adding edp/edp_meta.
   --> The error message was related to the last commit of admin/test.

Full SHA1 : 43d95f23121db40b3c8dddf78f272de30bee3d7c

[hansva]

Thank you for the clear reproduction path.

[mattcasters]

Quick synthesis from the OCR (not just transcription)

1. Native git on the server can resolve and type the object; fsck is clean after reflog cleanup.
2. Hop still reports Missing unknown 43d95f23121db40b3c8dddf78f272de30bee3d7c when enabling/adding projects.
3. Optimization formats are already disabled globally — so the earlier “commit-graph/midx/bitmap” theory is even weaker.
4. Stronger clue: the missing id is always the Test repo HEAD, including when project flow is default / edp_meta. That looks a lot like Git Perspective holding selection/state from the previous repo and re-parsing those commit ids against a different repository after rootChanged.

[mattcasters]

Root cause analysis (follow-up to the screenshots / logs)

Thank you for the detailed repro steps and screenshots — they were enough to pin this down more tightly than the original “JGit vs native optimization files” theory.

What the new evidence shows

Observation	Implication
Missing id is always 43d95f23121db40b3c8dddf78f272de30bee3d7c	That is admin/Test HEAD (as shown in Gitea), not a random corrupt id
On the Gitea bare repo: git cat-file -t → commit; fsck clean after reflog cleanup	Object exists on the server side
Global git config already has core.commitGraph=false, core.multiPackIndex=false, bitmap write/use disabled	commit-graph / midx / bitmap incompatibility is not required to explain the failure
Stack: GitGuiPlugin.rootChanged → GitPerspective.refresh → updateGui → isCommitInCurrentBranch → RevWalk.parseCommit	Failure is on project/root switch, not on opening a pack format
Adding edp_meta still reports the last commit of admin/Test	Cross-project / stale UI state

Note: the cat-file / fsck run in the screenshots was on the Gitea container bare repo. That proves the server has the object. It does not by itself prove the Windows working clone under F:\WorkSpace\Hop\hop_repo_2181\Test is healthy — still useful to confirm client-side if anything remains odd after the Hop-side issue is fixed — but the stack + multi-project behaviour already point at a Hop bug.

Primary root cause: stale Git Perspective selection after repository switch

The Git Perspective is a long-lived singleton. History table rows hold RevCommit instances (object ids) from whatever repository was active last.

On project change, GitGuiPlugin.rootChanged() replaces the active UIGit with a new repository, then calls GitPerspective.refresh(false).

refresh() currently calls updateGui() first, while the history table still has the previous project’s selection:

1. Project Test is active → history selection is HEAD 43d95f….
2. User switches to / adds edp_meta (small repo; does not contain Test’s commits).
3. rootChanged opens edp_meta’s .git.
4. refresh() → updateGui() → getSelectedCommit() still returns Test’s RevCommit.
5. isCommitInCurrentBranch() does walk.parseCommit(commit.getId()) against the new object database.
6. Result: MissingObjectException: Missing unknown 43d95f23121db40b3c8dddf78f272de30bee3d7c.

That matches:

• the log line Error checking if commit is in current branch
• the stack through isCommitInCurrentBranch / updateGui / refresh / rootChanged
• “error related to the last commit of admin/test” while dealing with edp_meta

Relevant code path (conceptually):

• GitGuiPlugin.rootChanged — opens new repo, then refreshes the perspective
• GitPerspective.refresh — calls updateGui() before clearing/rebuilding the history table
• GitPerspective.updateGui — reads getSelectedCommit() from the still-populated table
• GitPerspective.isCommitInCurrentBranch — re-parses that id against the current Repository

Missing unknown means JGit looked up the object id in the currently open ODB and did not find it. That is expected if the selected id belongs to another repository that was active a moment earlier.

This path is new in 2.18 with the Git Perspective (#6914 / #7053). It did not exist in 2.17 in this form. The concurrent JGit 7.3 → 7.5 bump is not required to explain this failure mode.

Secondary observations (not the main stack, but real)

1. Previous UIGit is not closed when rootChanged does git = null and opens a new repo. On Windows, abandoned file handles plus a startup native-git sync script (pull/rebase/gc/lock cleanup) can add noise or races — secondary to the selection bug above.
2. Invalid reflog entries on the bare repo (dc7d4355… / Initial commit) are a server hygiene issue; after cleanup, fsck was clean. Unlikely to be the direct cause of Hop’s parseCommit on the client path described above.
3. Error handling in isCommitInCurrentBranch logs a full ERROR stack and returns false. So a stale cross-repo selection looks like a severe JGit regression even though it is only used for toolbar enablement (e.g. cherry-pick).

What this is not

• Not primarily “JGit 7.5 cannot parse native Git commit-graph / multi-pack-index / pack bitmaps” — those are already disabled in the reporter’s global config, and the missing id is a concrete commit from another project’s history selection.
• Not explained solely by Gitea server object corruption: the same SHA is present and typed as commit on the server after the reported cleanup.

Summary

Root cause: On project / explorer root change, Git Perspective runs branch/commit UI logic against a stale history selection from the previous repository, using the newly opened JGit Repository. That produces MissingObjectException for object ids that are valid in the old repo (e.g. Test HEAD 43d95f…) but absent from the new one (e.g. edp_meta).

We’ll address the fix in a follow-up PR.

[mattcasters]

Fix available for testing

We opened a fix PR: #7531

@ch2dit-byte — would you be able to test this PR with your multi-project setup (e.g. switching between admin/Test and edp/edp_meta)? That would help confirm the MissingObjectException on project switch is gone on your Windows + Gitea environment.

How to evaluate a PR

Guide: Evaluating a pull request

Build (from the PR branch):

mvn -Pfast-build clean install

Thanks in advance if you get a chance to try it.

[ch2dit-byte]

I installed Apache Hop using the pre-compiled ZIP distribution from the official website, so I don't have a local Git/Maven build environment set up to compile the source code myself.

Is there a pre-built snapshot artifact (ZIP) available for this PR that I can download and test on my end? If not, I can wait and test it once it's merged into the next snapshot or release. Thanks for the quick fix!