CAMEL-24229: Fix flaky BacklogTracerActivityTest (volatile) and QuartzPersistentStore (MBean collision)#24713
Open
gnodet wants to merge 5 commits into
Open
CAMEL-24229: Fix flaky BacklogTracerActivityTest (volatile) and QuartzPersistentStore (MBean collision)#24713gnodet wants to merge 5 commits into
gnodet wants to merge 5 commits into
Conversation
gnodet
marked this pull request as ready for review
July 15, 2026 10:29
gnodet
commented
Jul 15, 2026
gnodet
left a comment
gnodet
left a comment
Contributor
Author
There was a problem hiding this comment.
Good batch of targeted flaky-test fixes — the Develocity analytics references are excellent for traceability.
Findings:
-
SSLContextParametersTest — incomplete JDK 24+ fix (medium): The first assertion block is correctly updated with
assertDefaultSignatureSchemes, but the "clear explicit schemes, keep filter" section at lines ~1035-1047 has the same JDK 24+ sensitivity. On JDK 24+,getSignatureSchemes()returns a non-null populated defaults array, so the.*include filter will match all of them andassertEquals(0, ...)will still fail. The comment at line 1044 ("JDK defaults are null -> filtering null gives empty array") is also stale on JDK 24+. This section likely needs the same JDK-version-aware treatment. -
BacklogTracer
volatileadditions — well-targeted.enabled/standbyare read on event notification threads (viaActivityEventNotifier.isDisabled()) while written from JMX threads.activityEnabledhas the same pattern. Correct and minimal fix. -
@Isolatedon Quartz restart test — right tool for the job. The test creates multiple CamelContexts with management names that can collide with concurrent test classes on JMX MBean registration. -
CXF
Thread.sleep(10000)— the increase from 2s to 10s is reasonable against the 100ms client timeout. The sleep runs in the background while the client times out quickly, so test duration isn't materially affected.
This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.
Claude Code review on behalf of @gnodet
oscerd
approved these changes
Jul 15, 2026
Contributor
|
🌟 Thank you for your contribution to the Apache Camel project! 🌟 🐫 Apache Camel Committers, please review the following items:
|
Contributor
|
🧪 CI tested the following changed modules:
🔬 Scalpel shadow comparison — Scalpel: 545 tested, 29 compile-only — current: 545 all testedMaveniverse Scalpel detected 574 affected modules (current approach: 545).
|
davsclaus
approved these changes
Jul 15, 2026
davsclaus
approved these changes
Jul 16, 2026
Contributor
Author
|
/retest |
apupier
requested changes
Jul 16, 2026
gnodet
force-pushed
the
fix/batch14-ssl-backlog-cxf-quartz
branch
from
July 16, 2026 14:25
db307d1 to
7149d16
Compare
gnodet
changed the title
Contributor
Author
…, QuartzPersistentStoreTest - BacklogTracer: Make enabled, standby, and activityEnabled fields volatile to prevent JIT register caching and stale cross-thread reads - CxfTimeoutTest: Increase GreeterImplWithSleep from 2s to 10s to reliably trigger the 100ms ReceiveTimeout under CI load - SpringQuartzPersistentStoreRestartAppChangeOptionsTest: Add @isolated to prevent JMX MBean name collisions with concurrent test classes SSLContextParametersTest fix dropped — already handled by PR apache#24734. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Timeout The CxfTimeoutTest was flaky because the 100ms ReceiveTimeout configured via the wildcard http-conduit in cxfConduitTimeOutContext.xml was not always being applied. The JAX-WS server published in @BeforeAll creates a default CXF Bus without the timeout configuration. Under CI load, the client conduit could resolve against this wrong Bus and miss the 100ms timeout entirely, causing the test to receive a successful response instead of the expected HttpTimeoutException. The previous fix (increasing GreeterImplWithSleep from 2s to 10s) would not resolve this because without the 100ms timeout, the effective timeout defaults to 60s (CXF default), and 10s < 60s. Fix: add a TimeoutCxfConfigurer that explicitly sets ReceiveTimeout=100 on each conduit, bypassing Bus-level configuration entirely. Revert GreeterImplWithSleep back to 2s since the sleep duration was never the root cause. Also add a comment on BacklogTracer volatile fields explaining why only enabled, standby, and activityEnabled need volatile (toggled at runtime via JMX) while other boolean fields do not (set during initialization). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
davsclaus
force-pushed
the
fix/batch14-ssl-backlog-cxf-quartz
branch
from
July 16, 2026 20:59
7149d16 to
918c7b5
Compare
gnodet
commented
Jul 16, 2026
gnodet
left a comment
gnodet
left a comment
Contributor
Author
There was a problem hiding this comment.
Updated PR looks good — the SSLContextParametersTest change was correctly dropped per @apupier's feedback (already handled in PR #24734).
Review of the 3 remaining fixes
1. BacklogTracer volatile fields ✅
core/camel-base-engine/src/main/java/org/apache/camel/impl/debugger/BacklogTracer.java
This is a production code fix and it's correct. enabled, standby, and activityEnabled are read by routing threads in shouldTrace() (line 122) and traceEvent() (line 212), while being written by JMX/management threads via setEnabled(), setStandby(), setActivityEnabled(). Without volatile, the JIT can hoist these reads out of loops and cache them in registers, causing routing threads to see stale values indefinitely.
The comment explaining why these three fields need volatile while others don't (set during init, not changed during routing) is a good addition — it prevents future maintainers from blindly adding or removing volatile on adjacent fields.
2. CxfTimeoutTest configurer approach ✅
components/camel-cxf/camel-cxf-spring-soap/src/test/java/org/apache/camel/component/cxf/CxfTimeoutTest.java + cxfConduitTimeOutContext.xml
The root cause is well-identified: the wildcard http-conf:conduit XML configuration is Bus-scoped, and when the conduit resolves against a different Bus (created by the JAX-WS server in @BeforeAll), the 100ms ReceiveTimeout is never applied. The TimeoutCxfConfigurer approach is the right fix — it applies the timeout directly on the conduit in configureClient(), making it independent of Bus lifecycle.
3. Quartz @Isolated ✅
components/camel-quartz/src/test/java/org/apache/camel/component/quartz/SpringQuartzPersistentStoreRestartAppChangeOptionsTest.java
Clean and minimal. JMX MBean name collisions are a well-known source of flakiness when Quartz-based tests run in parallel.
Checklist
| Check | Status |
|---|---|
| Tests | ✅ 3 flaky test fixes |
| Thread.sleep() | ✅ None introduced |
| Production code change | ✅ BacklogTracer volatile is a correctness fix, not a behavioral change |
| Backward compat | ✅ volatile has no API impact |
| Documentation | N/A — no user-visible behavior change |
| Commit convention | ✅ |
| Review feedback addressed | ✅ SSLContextParametersTest dropped per apupier |
Reviewed with Claude Code on behalf of gnodet. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
davsclaus
approved these changes
Jul 17, 2026
apupier
requested changes
Jul 17, 2026
Address review feedback: the comment said "these three flags" but only enabled and standby were adjacent. activityEnabled is further down the field list. Name all three fields explicitly in the comment and add a cross-reference near activityEnabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apupier
reviewed
Jul 17, 2026
… management names Replace @isolated with unique managementNamePattern suffixes on each Spring XML context file. The test creates multiple CamelContexts sequentially with the same context ID, and the previous `#name#` pattern produced identical JMX ObjectNames. When MBean unregistration from a stopped context lagged behind the next context's startup, the registration failed with VetoCamelContextStartException. Each XML file now uses a distinct suffix (-cron1, -cron2, -opts1, -opts2) so JMX names never collide, even under CI load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CXF timeout root cause investigation is tracked in CAMEL-24954. This PR now covers only BacklogTracer volatile fields and Quartz MBean name collision fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
gnodet
changed the title
Contributor
Author
|
Claude Code on behalf of gnodet Removed the CxfTimeoutTest fix from this PR — it's now tracked separately in #24954 for deeper root cause investigation. This PR now only covers:
|
gnodet
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude Code on behalf of gnodet
Summary
Fix two flaky tests by addressing their root causes:
BacklogTracerActivityTest
enabled,standby, andactivityEnabledfields inBacklogTracerwere notvolatile. When these flags are toggled via JMX while routing threads read them, JIT register caching can cause stale cross-thread reads.volatile.SpringQuartzPersistentStoreRestartAppChangeOptionsTest
managementNamePattern="#name#", producing identical JMX ObjectNames. When sequential CamelContexts share the same management name and one is still unregistering when the next starts, aVetoCamelContextStartExceptionis thrown.managementNamePatternsuffix (-cron1,-cron2,-opts1,-opts2) so JMX ObjectNames never collide.CxfTimeoutTest (removed — tracked in #24954)
CxfTimeoutTest fix has been moved to a separate issue for deeper root cause investigation.
Files changed
BacklogTracer.javaenabled,standby,activityEnabledvolatile with commentsSpringQuartzPersistent...CronExpressionTest1.xmlmanagementNamePattern="#name#-cron1"SpringQuartzPersistent...CronExpressionTest2.xmlmanagementNamePattern="#name#-cron2"SpringQuartzPersistent...ChangeOptionsTest1.xmlmanagementNamePattern="#name#-opts1"SpringQuartzPersistent...ChangeOptionsTest2.xmlmanagementNamePattern="#name#-opts2"Broader volatile analysis — systemic JMX visibility pattern
This PR fixes only
BacklogTracer, but an audit of the management layer revealed the same JMM-unsafe pattern across 12 engine classes (50+ fields). Fields writable via JMX@ManagedAttributesetters are stored as plain (non-volatile) fields in engine classes, but read on routing threads without any memory visibility guarantee.Why only these 3 fields cause a flaky test
The flaky test (
BacklogTracerActivityTest) fails becauseActivityEventNotifier.isDisabled()is a trivial one-liner with no opaque method calls:This is a prime candidate for JIT register caching — the method is tiny, hot (called on every exchange lifecycle event), and has no intervening calls that would force a memory reload. When the test enables
activityEnabledvia JMX and immediately sends exchanges, the routing thread may never see the update.Most other JMX-toggled fields are read through opaque method chains (e.g.,
expression.evaluate(),exchange.getProperty()) that act as natural compiler barriers — the JIT cannot prove the field hasn't changed across those calls, so it reloads from memory. This is why the other fields don't cause test failures despite having the same JMM violation.Other affected classes (for future work)
The following classes have non-volatile fields that are writable via JMX and read on routing threads. They are listed for completeness — none are known to cause flaky tests today, because on x86 (where CI runs) volatile reads compile to plain loads, and opaque method calls in the read paths prevent JIT hoisting in most cases.
HIGH severity — guard conditions or routing behavior on per-exchange hot paths:
BacklogTracerenabled,standby,activityEnabled,tracePattern+patterns,traceFilter+predicateshouldTrace(),traceEvent(),isDisabled()DefaultTracerenabled,tracePattern+patternsshouldTrace()viaTracingAdvice.before()BaseProcessorSupportdisabledCamelInternalProcessor.process()— every exchangeDefaultBacklogDebuggerfallbackTimeout(long — non-atomic on 32-bit JVMs)NodeBreakpoint.beforeProcess()TotalRequestsThrottlertimePeriodMillis(long),maxRequestsExpressionprocess()— every exchangeDefaultStreamCachingStrategyspoolThreshold(long),spoolUsedHeapMemoryThreshold,anySpoolRulesshouldSpoolCache()— every streaming writeThrottlingInflightRoutePolicymaxInflightExchanges+resumeInflightExchanges(compound non-atomic)throttle()viaonExchangeDone()ThrottlingExceptionRoutePolicyhalfOpenAfter(long),failureWindow(long),failureThresholdcalculateState()viaonExchangeDone()ManagedPerformanceCounterstatisticsEnabledDefaultInstrumentationProcessor.before()— every exchangeMEDIUM severity — data quality or non-guard behavior:
BacklogTracerformatting fields (bodyMaxChars,bodyIncludeStreams, etc.),DefaultBacklogDebuggerformatting fields,ScheduledPollConsumer(greedy,sendEmptyMessageWhenIdle),ThrottlingInflightRoutePolicy(scope).Recommended fix approach (not in this PR)
For the simple boolean/int fields: add
volatile. Cost is negligible — on x86, volatile reads compile to the same instruction as plain loads.For compound writes (
tracePattern+patterns,maxInflightExchanges+resumeInflightExchanges):volatilealone is insufficient. These need either a lock or an immutable holder object swapped atomically.synchronizedon MBean getters/setters is not recommended — it would add lock contention on the hottest paths (shouldTrace(),process()) for no benefit, since the fields are independent and don't require multi-field atomicity.Test plan
SpringQuartzPersistentStoreRestartAppChangeOptionsTest— 3/3 pass locally