Liquid cache vendored - #208
Closed
alchemist51 wants to merge 94 commits into
Closed
Conversation
…rage engine (opensearch-project#22178) [DFAE] Document count validation based on support from underlying storage engine Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
…row loss (opensearch-project#22209) Row-group / page statistics pruning resolved `StatisticsConverter` against the full union table schema. `StatisticsConverter` maps a column name to a parquet leaf positionally (parquet crate `parquet_column`), so under dynamic-mapping schema drift — where a segment's own parquet schema is narrower or reordered relative to the union schema — the lookup lands on the wrong leaf or off the end of the file. That yields all-null / wrong-column stats, and an always-true residual such as `severity >= 0` then evaluates to "cannot match" and the whole row group / segment is wrongly pruned, silently dropping matching rows (observed: `match() AND severity >= 0` returned ~⅓ of `match()` alone). Fix all three stats-resolution sites to build the arrow schema from the segment's own parquet descriptor via `parquet_to_arrow_schema(descr, kv)`, so the positional lookup lands on the right leaf: - page_pruner.rs `eval_leaf` (RG-level / whole-segment prune) - page_pruner.rs `PagePruner::prune_rg` (page-level prune) - dynamic_filter.rs `RgPruningContext::rg_provably_excluded` (dynamic-filter prefetch prune) Tests: - Two Rust unit guards (one per file) build a fixture where the column resolves positionally onto a different real column and assert the RG is not pruned; both fail without the fix. - SchemaDriftPrunePplIT: end-to-end REST IT over a composite index with a Lucene secondary and dynamic mapping, merge disabled (new integTestNoMerge cluster variant), 24 drifting segments. Asserts numeric-filter-only, sort+limit, match()+residual, and match()+filter+sort+limit all return correct counts. Verified to fail without the fix (match()+severity>=0 returned 6600 vs 7200). Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
…er, not an index-local flag (opensearch-project#22203) An index template carrying index.routing.allocation.total_primary_shards_per_node was rejected with a 400 ("can only be used with remote store enabled clusters") on an all-remote-store cluster, even though the same setting is accepted via PUT <index>/_settings and at index creation. Root cause: the template path called the create-index overload of validateIndexTotalPrimaryShardsPerNodeSetting(Settings), which reads the index-local index.remote_store.enabled flag. That flag is injected by create-index from a remote node's attributes and is never present in a template's own settings block — templates are validated before any index exists. It is also a private setting, so it cannot be added by hand. Fix: validate the template with the cluster/node-aware overload (MetadataUpdateSettingsService.validateIndexTotalPrimaryShardsPerNodeSetting(Settings, ClusterService)), matching the _settings update path — it checks whether every discovery node is a remote-store node. Harden that check so an empty node set is not treated as remote-store enabled (Stream.allMatch is vacuously true on an empty stream). Adds ShardsLimitAllocationDeciderRemoteStoreEnabledIT (template with the setting on a remote-store cluster propagates to a template-created index and is enforced at allocation) and MetadataIndexTemplateServiceRemoteStoreTests (mock-driven unit test for the remote-store-cluster template validation path, kept in its own class to avoid perturbing the randomized ordering of the shared-fixture suite). Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
…default (opensearch-project#22073) * Extend forbidden api for java serialization from build-time only to also enforce at runtime Signed-off-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Sandesh Kumar <sandeshkr419@gmail.com>
…roject#22231) The framework registers AnalyticsQueryTask before doExecute forks to the search executor and installs the cancellation callback via QueryScheduler.setCancellationCallback. A cancel arriving in that window (server-side timeout, HTTP disconnect from RestCancellableNodeClient, parent-task cascade) fires onCancelled() with no callback installed and silently no-ops; the task is marked cancelled but the analytics query runs to completion. Re-check isCancelled() after the callback is installed and run it inline when a cancel was already observed. Switch consumption to getAndSet(null) so the install-side and onCancelled() paths cannot both fire it. Mirrors AnalyticsShardTask.setCancellationListener, which already has this guard. Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
…ensearch-project#22234) When a coordinator query is cancelled while the reduce drain is in flight, the cancel thread closes the Arrow allocator before the reduce thread has released its in-flight batches, causing a spurious "Memory was leaked" IllegalStateException. Add a CountDownLatch to DatafusionReduceSink that fires when reduce's finally block completes teardown. When closeImpl() observes state=REDUCING (cancel during active drain), it fires cancelQuery then awaits the latch (up to 5s) so the reduce thread can release Arrow batches before the allocator closes. Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
…pensearch-project#22134) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
…oject#22207) After cancel_query aborts the CPU task, RepartitionExec's pull_from_input tasks (holding GroupedHashAggregateStream with GB-scale GroupValues buffers) remain in tokio's deferred drop queue until a worker processes them. On an idle runtime this can take seconds or never complete promptly. Store the DedicatedExecutor's runtime Handle in QueryTracker. In stream_close, after dropping the QueryStreamHandle (which drops the JoinSet and triggers the abort cascade), spawn yield tasks on the CPU runtime to give workers scheduling opportunities to process the deferred drops of pull_from_input futures and their captured GroupValues buffers. Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
…arch-project#22228) Replace IllegalStateException (500) with UnsupportedFunctionException (400) in planner rules when a function is not supported by any backend. The error message no longer leaks internal backend names. Before: 500 illegal_state_exception: No backend supports scalar function [MATCH] among [lucene, datafusion] After: 400 unsupported_function_exception: Function [MATCH] is not supported as a scalar function Signed-off-by: Finnegan Carroll <carMDroll@amazon.com> Signed-off-by: Finn Carroll <carrofin@amazon.com>
…project#22245) Signed-off-by: Craig Perkins <cwperx@amazon.com>
…lifecycle and buffers (opensearch-project#22244) Server-side Flight streaming frees off-heap Arrow buffers and terminates the gRPC call from two unsynchronized threads: the producer runs serially on the flight executor, while a client cancellation (onChannelCancelled -> close()) is delivered concurrently on a gRPC thread. The previous split ownership between FlightOutboundHandler and FlightServerChannel, plus a non-atomic close(), made several interleavings unsafe: - double-close: close() used a non-atomic get-then-set, so two callers (gRPC cancel and release()) could both run notifyCloseListeners(), double-firing the TaskManager untrack listener -> AssertionError that kills the node. - use-after-free / leak: the handler transferred the producer root into the reused stream root and called putNext while a concurrent close() could free that same root (gRPC may still hold a zero-copy ref), or the producer could refill the reused root after close() had already freed it -> stranded. - source-root orphan: a batch built but never sent (back-pressure gate threw, executor rejected) freed nothing. - consumer hang: a failed batch send dropped the batch with no error frame, so the consumer's FlightStream.getRoot() blocked forever. This makes FlightServerChannel the sole owner of every Arrow buffer and of the gRPC stream lifecycle, using a single-writer model rather than synchronization: - The stream root and every serverStreamListener call (start/putNext/ completed/error) happen only on the channel's single-threaded flight executor, which serializes batch sends, completeStream/sendError, and the root free among themselves so none can interleave. - close() is fire-once via open.compareAndSet(true,false) and is a signal, not a buffer op: from any thread it flips open, fires close listeners, and posts the root free onto the flight executor. Because the executor is FIFO, the free runs only after any in-flight send completes; close() never touches Arrow buffers or gRPC from the caller (cancel) thread, so it cannot race an in-flight putNext and never blocks. This is deadlock-free by construction -- nothing is held across a gRPC/Arrow/blocking call. - sendBatch(response, headerSupplier) is the single send entry. It owns the native source root and frees it exactly once on every exit; the handler frees a never-handed-off batch via releaseUnsent (handedOff discipline). Batches queued behind a cancel reject at the cancelled guard and free their own source root. - the byte-serialized stream root is reused across batches and freed once at close(), never per batch (fixes the zero-copy use-after-free). - completeStream/sendError are terminal and idempotent (no-op after close/cancel/terminal), so a failed batch send fails the stream instead of hanging the consumer. - close-listener registration and firing are atomic (a leaf mutex) and fault-isolated. The one trade-off is that close() frees the stream root asynchronously (on the executor) rather than inline; on node shutdown a dropped free task leaks the root until process exit. Both are documented. Adds docs/channel-lifecycle-ownership.md describing the contract, and unit tests for the cancel-vs-in-flight-send race, batches queued behind a cancel, fire-once close, source-root freeing on every exit, byte-path reuse, terminal-op idempotency, and the hand-off discipline. Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Mohit Godwani <mgodwan@amazon.com>
…project#22252) The opensearch-sql plugin nests the analytics QueryProfile under a "plan" node (profile envelope {summary, plan, phases}), so the SHARD_FRAGMENT stage list lives at profile.plan.stages. shardFragmentStage() now reads it there. Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
opensearch-project#22186) * fix: Correct StatsPruneTree rg_can_match absolute-to-relative index translation Add rg_index_to_pos reverse map (absolute RG index → position in rg_can_match vector) to fix incorrect lookups when chunks don't start at RG 0. Previously consumers indexed rg_can_match by absolute rg.index into a subset-relative vector, causing wrong pruning decisions or panics for non-zero-offset chunks. Changes: - Add rg_index_to_pos: HashMap<usize, usize> to TreeBitsetSource, SingleCollectorEvaluator, PredicateOnlyEvaluator - Thread rg_index_to_pos through TreeEvaluator::prefetch and bitmap_tree::prefetch_node - Build reverse map from chunk.row_group_indices at each factory site - Enable StatsPruneTree in fuzz harness (prune_tree_config: Some) Tests added: - page_pruner: offset rg_indices [2,3,4] with reverse map verification - bitmap_tree: offset rg_idx not in map (no prune), in map (prune), empty collector bitsets under pruned OR subtree with Predicate nodes - ITs: single/multi segment × single/multi partition with stats pruning enabled end-to-end, direct prefetch_rg assertions for RG prune, offset access, and empty collector bitset registration Signed-off-by: Somesh Gupta <someshgupta987@gmail.com> * Removed deep clone from StatsPruneTree Signed-off-by: Somesh Gupta <someshgupta987@gmail.com> --------- Signed-off-by: Somesh Gupta <someshgupta987@gmail.com>
opensearch-project#21803) Add version map and integrated get-by-id for document lookup. Signed-off-by: Koustubh Gupta <thorkous@amazon.com> Co-authored-by: Koustubh Gupta <thorkous@amazon.com> Co-authored-by: Mohit Godwani <81609427+mgodwan@users.noreply.github.com>
…ect#22249) Two leaks on the streaming-transport response path, each unit-tested (fails without the fix, passes with it): - FlightTransportResponse: close() racing the async prefetch could miss the stream the prefetch publishes, stranding the prefetched first-batch root. Set closed first and have the prefetch self-close on that flag. Test: FlightTransportResponseTests. - AnalyticsSearchTransportService: on an abnormal exit (consumer throw or stage failure), cancel() the flight stream instead of close() so the data-node producer tears down its FlightServerChannel rather than stranding its streamRoot. Test: AnalyticsSearchTransportServiceTests. Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
…pensearch-project#22247) * Fix coordinator cancel-teardown leaks in analytics-engine Two coordinator-side teardown leaks, each unit-tested (fails without the fix, passes with it): - StageExecution cascade: on a CANCELLED child, close the parent's input for that child so a COORDINATOR_REDUCE drain blocked on streamNext sees EOF and unwinds. Without this the reduce thread hangs and its borrowed batches are stranded (a mid-flight shard scan cancelled never EOFs its sender). Cancel is still not propagated to the parent's state. Test: AttachChildrenTests. - RowProducingSink: a feed() racing close() frees the batch instead of buffering it (close already freed the list and won't run again). Test: RowProducingSinkTests. Also defaults analytics.coordinator.buffer_limit to 0 (no per-query child allocator). Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * add todo Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
… settings and cache stats (opensearch-project#22257) * Datafusion - clear cache rest action, cache settings and cache stats Signed-off-by: G <bharath78910@gmail.com> * updating stats cache percent Signed-off-by: G <bharath78910@gmail.com> * addressing comments Signed-off-by: G <bharath78910@gmail.com> * removing unused settings Signed-off-by: G <bharath78910@gmail.com> --------- Signed-off-by: G <bharath78910@gmail.com>
Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Mohit Godwani <81609427+mgodwan@users.noreply.github.com>
…pensearch-project#22262) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
…#22187) * [analytics-datafusion] Native memory, spill, and threshold fixes for the DataFusion engine Squash of the datafusion-try-grow-fix work: - Clamp data-node fragment gate to CPU worker count. - Fix permanent CrossRtStream wedge by spawning the driver off the consumer. - Exempt self-liquidating spill reservations from the 85% RSS gate (bounded by SPILL_EXEMPT_CAP_BYTES). - Fix dynamic memory-guard threshold updates silently using stale values (single grouped settings-update consumer). - Default spill cap to 80% of spill volume capacity; validate operator overrides. - Make spill_exempt_cap_bytes a dynamic cluster setting (datafusion.memory_guard.spill_exempt_cap_bytes, raw bytes, default 512MB). - Add Rust unit tests for the spill_exempt_cap setter and FFI negative-clamp. - Add an end-to-end disk-spill test (GROUP BY under a small pool spills and returns correct results, asserted via DataFusion SpillCount/SpilledBytes metrics). The cross_rt_stream.rs and runtime_manager.rs changes were reverted; this squash reflects the net final state with those files unchanged from origin/main. Signed-off-by: snghsvn <snghsvn@amazon.com> * [analytics-datafusion] Fix DatafusionSettingsTests expected ALL_SETTINGS count Adding datafusion.memory_guard.spill_exempt_cap_bytes to ALL_SETTINGS raised the registered-setting count from 28 to 29, but testAllSettingsContainsAllExpectedSettings still asserted 28. Update the count and assert the new setting is registered. Signed-off-by: snghsvn <snghsvn@amazon.com> * [analytics-datafusion] Raise spill limit default to 90% of disk capacity SPILL_LIMIT_FRACTION 0.80 -> 0.90; update the derive-default test accordingly. Signed-off-by: snghsvn <snghsvn@amazon.com> * [analytics-datafusion] @AwaitsFix the failing composite-engine warm DFA ITs Mute 4 internalClusterTest failures unrelated to this PR (pulled in via the origin/main merge), all currently red on the sandbox-check: - DataFormatAwareDFASnapshotBlockingIT (3 tests, from opensearch-project#22011) - DataFormatAwareReadonlyGetByIdIT.testGetByIdFromWarmReadOnlyEngine (opensearch-project#21803) All tests in each class fail, so the annotation is applied at class level and points at the source PR for tracking. Signed-off-by: snghsvn <snghsvn@amazon.com> --------- Signed-off-by: snghsvn <snghsvn@amazon.com>
… Shard (opensearch-project#22266) * Detecting bottom most Filter in the plan for delegation to derive DelegationType Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Spotless Signed-off-by: Aniketh Jain <anijainc@amazon.com> --------- Signed-off-by: Aniketh Jain <anijainc@amazon.com>
…t startup validation (opensearch-project#22275) * Surface circuit breaker as HTTP 429 when it self-cancels the query A memory-gate trip (CircuitBreakingException) on the reduce stage fails the stage and then cancels the parent task via the child cancel sweep (ReduceStageExecution FAILED -> child.cancel -> parentTask.cancel). By the time QueryExecution.terminalCause reports, the parent task is already cancelled, so the original isCancelled()-first ordering returned a synthetic TaskCancelledException (HTTP 500) and discarded the real breaker (HTTP 429). terminalCause now peeks at the captured root failure inside the cancelled branch: if a CircuitBreakingException is in its cause chain, it is surfaced unwrapped so status() yields 429. Genuine top-down cancels record no stage failure (getFailure() == null), so they are unchanged. The non-cancel path is byte-for-byte the original. Uses ExceptionsHelper.unwrap (cause-only, no suppressed) to avoid a multi-shard suppressed-sibling false positive. Adds QueryExecutionTests coverage: bare breaker masquerading as a cancel, nested breaker under a cancel, genuine cancel stays 500, and non-breaker failure under a cancel stays 500. Signed-off-by: snghsvn <snghsvn@amazon.com> * Don't reject a non-zero spill limit when spill is disabled When datafusion.spill_directory is unset, spill is disabled and datafusion.spill_memory_limit_bytes has no effect. The validator previously threw IllegalArgumentException for any non-zero limit in that state, which aborted node startup (StartupException) on nodes configured with a spill cap but no spill volume. SpillLimitValidator now returns early when the directory is empty, accepting any value as an inert no-op. The volume-capacity check still applies when spill is enabled. Flips the corresponding unit test to assert acceptance. Signed-off-by: snghsvn <snghsvn@amazon.com> --------- Signed-off-by: snghsvn <snghsvn@amazon.com>
…elocation (opensearch-project#22279) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
* core page index cache changes Signed-off-by: G <bharath78910@gmail.com> * addressing comments Signed-off-by: G <bharath78910@gmail.com> * addressing comments Signed-off-by: G <bharath78910@gmail.com> * cleaning up code , removing surviving RGs Signed-off-by: G <bharath78910@gmail.com> * fixing policy wiring Signed-off-by: G <bharath78910@gmail.com> * fix for sort + head query Signed-off-by: G <bharath78910@gmail.com> * fixing perf by passing the object meta for get store ranges calls Signed-off-by: G <bharath78910@gmail.com> * fixing settings and IT Signed-off-by: G <bharath78910@gmail.com> * fixing test failures Signed-off-by: G <bharath78910@gmail.com> --------- Signed-off-by: G <bharath78910@gmail.com>
… engine open (opensearch-project#22281) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Signed-off-by: Lucas Kruger <88447897+krugerLucas@users.noreply.github.com>
…pensearch-project#22357) Signed-off-by: Rishabh Singh <sngri@amazon.com>
The prefetch thread re-read the closed flag in a finally block that ran after it completed the open future. A close() arriving in that window double-closed the Flight stream, which intermittently failed FlightTransportResponseTests and, when the stream's close threw, surfaced an uncaught exception on the virtual thread that in turn failed unrelated neighbor tests in the same class. Perform the closed check synchronously right after the stream is published and before the future completes, so any later close() always observes the stream and owns the close. Guard the physical FlightStream.close() with an AtomicBoolean so it is invoked at most once even when two close() calls race. Signed-off-by: Andrew Ross <andrross@amazon.com>
) planningTimeMs was gated behind the profile flag, so it was always 0 for non-profiled queries. The AnalyticsStatsCollector only recorded accurate planning time when profile=true was explicitly set. Remove the profile ternary so planning time is always computed and passed to recordExecution, enabling the stats endpoint to report accurate planning latency for all queries. Signed-off-by: Finn Carroll <carrofin@amazon.com>
…nsearch-project#22064) * Fix OpenSearchTimeoutException to return HTTP 504 instead of 500 Signed-off-by: Radhakrishnan Pachyappan <gingeekrishna@gmail.com>
Upgrades Apache Hadoop from 3.4.2 to 3.5.0 to resolve transitive dependency on org.eclipse.jetty/jetty-http 9.4.58.v20250814 which contains CVE-2026-2332. Also updates the thirdPartyAudit exclusion list: - Remove exclusions for classes that no longer have violations in 3.5.0 - Add exclusion for renamed AbstractFutureState$UnsafeAtomicHelper class Signed-off-by: Craig Perkins <cwperx@amazon.com>
Signed-off-by: Vishwas Garg <vishwasgarg14@gmail.com>
…estore (opensearch-project#20494) * fix: Separate internal ignore settings from user ignore settings in restore Internal ignore settings should override user-unremovable settings protection to allow proper filtering of settings like remote_store.* during cross-cluster restores. Signed-off-by: Piyush Kumar <piykumab@amazon.com> * test: Add unit tests for internal vs user ignore settings separation Add tests to verify that internal ignore patterns can filter protected settings while user ignore patterns respect protection. Also optimize the predicate to use Set.contains() instead of iteration. Signed-off-by: Piyush Kumar <piykumab@amazon.com> --------- Signed-off-by: Piyush Kumar <piykumab@amazon.com> Co-authored-by: Piyush Kumar <piykumab@amazon.com>
…ias members (opensearch-project#21838) * Scope checkBlock to write index only, skipping non-write alias members Signed-off-by: Mikhail Stepura <mstepura@apple.com> * Inline single-line method call in `checkBlock` for rollover action Signed-off-by: Mikhail Stepura <mstepura@apple.com> --------- Signed-off-by: Mikhail Stepura <mstepura@apple.com>
…ith PartialReduce using topk check (opensearch-project#22360) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Marc Handalian <marc.handalian@gmail.com> Co-authored-by: Aniketh Jain <anijainc@amazon.com>
…pensearch-project#22365) Signed-off-by: Lantao Jin <ltjin@amazon.com>
…not_corrupt (opensearch-project#22327) Signed-off-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: gaobinlong <gbinlong@amazon.com>
…pensearch-project#22376) ReplicationCheckpoint.compareTo returned a non-zero value for equal checkpoints (it never returned 0), which violates the Comparable/Comparator contract (antisymmetry: sgn(a.compareTo(b)) == -sgn(b.compareTo(a))). Signed-off-by: Harshita Kaushik <harycash@amazon.com>
…ypass (opensearch-project#22328) The fs repository base_path setting was read verbatim from REST input with no validation. An absolute base_path causes Path.resolve to discard the path.repo-validated location, redirecting all blob-store operations outside the repository (CWE-22) and enabling arbitrary filesystem deletion via the snapshot _cleanup API. Add two layers of defense: - BASE_PATH_SETTING validator rejects absolute and upward-escaping (..) values after normalization (benign interior '..' that cancels out is allowed). - validateBasePathWithinRepo() resolves base_path against the location and verifies the result stays within a configured path.repo directory. Signed-off-by: Aditya Khera <kheraadi@amazon.com> Co-authored-by: Aditya Khera <kheraadi@amazon.com>
) * Add support for overCommit for NativeAllocator Signed-off-by: rayshrey <rayshrey@amazon.com> * Fix javadocs Signed-off-by: rayshrey <rayshrey@amazon.com> * Review fixes Signed-off-by: rayshrey <rayshrey@amazon.com> * Add token base gating for the overCommit permits Signed-off-by: rayshrey <rayshrey@amazon.com> --------- Signed-off-by: rayshrey <rayshrey@amazon.com>
…nsearch-project#22359) * Add cross-cluster streaming support to the remote cluster client Enable a plugin to open an Arrow Flight stream to a *remote cluster* (mirroring the existing pull remote-client path), with producer-side backpressure and a poll-able cancellation signal so a long-lived producer does not leak when the consumer goes away. Also adds a synchronous option on send to block until the write is queued in the gRPC outbound buffer before returning. Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
* Add cluster level default for delayed allocation timeout Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> * mark "deprecated" the old getRemainingDelay method Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> --------- Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> Co-authored-by: Lamine Idjeraoui <lidjeraoui@apple.com>
…efore force merge completion (opensearch-project#22370) * fix(tiering): Track force merges in activeMerges to prevent tiering race condition MergeScheduler.forceMerge() runs merges by calling runMerge() directly, bypassing submitMergeTask() which is the only place that increments activeMerges. This makes in-flight force merges invisible to onMergesDrained(), causing tiering's prepare step to proceed immediately while a force merge is still running. When AutoForceMergeManager triggers a force merge on an idle shard and tiering is triggered before it completes, the merge finishes after shard relocation. The merged segment is never uploaded to remote store (RemoteStoreRefreshListener is already closed), leaving the replica permanently diverged with a linearly growing replication lag. Changes: - Increment activeMerges in forceMerge() so onMergesDrained correctly waits for in-flight force merges before proceeding with tiering - Fire drain listeners in forceMerge() finally block when all merges complete (mirrors submitMergeTask behavior) - Add waitForReplicaSync() to IndexShard to verify replicas are in sync after waitForRemoteStoreSync() in the tiering prepare step - Add unit tests verifying force merges block drain and are visible to getActiveMergeCount() Signed-off-by: bkhishor <bkhishor@amazon.com> * test: Add unit tests for waitForReplicaSync and minor refinements - Add IndexShardTests for waitForReplicaSync behavior - Refine integration test assertions Signed-off-by: bkhishor <bkhishor@amazon.com>
opensearch-project#22386) Signed-off-by: Andriy Redko <drreta@gmail.com>
…search-project#22382) * Add HTTP request body decompression to reactor-netty4 transport The reactor-netty4 HTTP transport plugin only configures response compression via .compress(true) but does not handle incoming request body decompression. This means requests with Content-Encoding: gzip header arrive as raw compressed bytes, causing JSON parse failures (e.g. JsonParseException on byte 0x1F, the gzip magic byte). The standard Netty4HttpServerTransport handles this via HttpContentDecompressor in its pipeline, but reactor-netty4 bypasses that pipeline entirely. This commit adds .doOnConnection() to inject Netty's HttpContentDecompressor into each connection's pipeline, enabling transparent decompression of gzip/deflate-encoded request bodies. This matches the behavior of the standard transport and is required for compatibility with clients like opensearch-java's AwsSdk2Transport, which auto-compresses request bodies larger than 8KB. Signed-off-by: Jiaping Zeng <jpz@amazon.com> * Update plugins/transport-reactor-netty4/src/main/java/org/opensearch/http/reactor/netty4/ReactorNetty4HttpServerTransport.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Jiaping Zeng <jpz@amazon.com> * address comments Signed-off-by: Jiaping Zeng <jpz@amazon.com> * fix build Signed-off-by: Jiaping Zeng <jpz@amazon.com> --------- Signed-off-by: Jiaping Zeng <jpz@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com>
…arch-project#22391) Adds A Setting.Validator for both balance factor settings that validates the sum > 0 constraint before the settings enter cluster state, preventing the poison pill from being published. Fixes opensearch-project#22305 Signed-off-by: Abhishek Dhanwani <f20170161h@alumni.bits-pilani.ac.in>
Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>
Signed-off-by: Craig Perkins <cwperx@amazon.com>
Apply `cargo fmt --all` across the sandbox native (Rust) workspace and wire a `cargo fmt --all -- --check` step into the sandbox-check GitHub Action so formatting regressions are caught in CI. Formatting uses rustfmt's default style (100-char width). No rustfmt.toml is added: rustfmt resolves config by walking up from each file's directory, and the workspace pulls in plugin crates from sibling paths (sandbox/plugins/...) that a config under dataformat-native/rust/ would not cover — so a width override there would apply only to the three lib crates and silently leave the plugin crates on the default. Relying on the default keeps every sandbox crate on one consistent style. The rustfmt component is added to the Rust toolchain setup and the format check runs before the build steps so it fails fast. Clippy enforcement will follow in a separate change. Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
…match (opensearch-project#21865) --------- Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> Co-authored-by: Lamine Idjeraoui <lidjeraoui@apple.com>
…rch-project#22058) --------- Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> Co-authored-by: Lamine Idjeraoui <lidjeraoui@apple.com>
Signed-off-by: Lamine Idjeraoui <lidjeraoui@apple.com> Co-authored-by: Lamine Idjeraoui <lidjeraoui@apple.com>
Vendor a minimal, in-memory-only subset of liquid-cache
(cocosz/liquid-cache branch lc-opensearch-df54-v2, commit 8311bcc)
into the sandbox native workspace as opensearch-liquid-cache-{core,
datafusion}, so OpenSearch takes no external Cargo dependency on the
liquid-cache package.
The vendored subset drops the disk tier (t4/io-uring), client/server
mode (arrow-flight/tonic), FSST string caching, variant shredding,
and the lineage optimizer. Under memory pressure, cached Arrow
batches are transcoded to the compressed liquid form and then
evicted. With io-uring gone the cache is no longer Linux-only.
Queries whose projections are numeric/date/timestamp/boolean engage
the cache through LiquidParquetSource, a drop-in ParquetSource
replacement wired in via parquet_bridge and the
LocalModeLiquidCacheOptimizer physical rule.
Gated behind opensearch.experimental.feature.liquid_cache.enabled.
New dynamic settings under datafusion.liquid_cache.* control
enablement, memory limit, eviction policy (lru/liquid), selectivity
threshold, and max projected columns. A REST endpoint
POST _plugins/analytics_backend_datafusion/liquid_cache/clear resets
the cache.
Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.