Add internalSearch flag for native get-by-id (no Substrait) - #1
Open
alchemist51 wants to merge 2 commits into
Open
Add internalSearch flag for native get-by-id (no Substrait)#1alchemist51 wants to merge 2 commits into
alchemist51 wants to merge 2 commits into
Conversation
Add version map and integrated get-by-id for document lookup. Signed-off-by: Koustubh Gupta <thorkous@amazon.com>
alchemist51
force-pushed
the
internal-search-getbyid
branch
2 times, most recently
from
June 19, 2026 13:18
2dc404b to
ad89c5b
Compare
alchemist51
force-pushed
the
internal-search-getbyid
branch
from
June 19, 2026 13:56
ad89c5b to
e0b2767
Compare
Route the pluggable-dataformat get-by-id and seq-no-scan through the existing df_execute_query FFM call with an internalSearch flag instead of synthesizing SQL->Substrait in GetService. When the flag is set, the native side builds a DataFrame plan with a pushed-down filter (__row_id__ = n for get-by-id, _seq_no > floor for the recovery scan), executes it, and returns a stream drained through the same df_stream_next/df_stream_close path as a normal query. - ffm.rs: df_execute_query gains internal_search_mode + internal_search_bound - datafusion_query_config.rs: InternalSearch enum + from_wire + unit test - api.rs: execute_internal_search (DataFrame .filter, single-file assert, pushdown on); early dispatch from execute_query - helpers.rs (new): consolidate shared query setup used by both the Substrait and internal-search paths -- build_query_runtime_env (list-files cache + per-query memory pool overlay + object-store registration), build_session_context (with register_udfs flag), new_shard_query_context, register_listing_table, execute_plan_to_stream_ptr, run_cancellable_to_handle_ptr, wrap_stream_as_handle; repoint query_executor/indexed_executor/session_context - NativeBridge.java: executeQueryAsync gains internalSearchMode/Bound + INTERNAL_SEARCH_* constants; FFM descriptor updated - GetService.java: drop sqlToSubstrait + WireConfigSnapshot; call the flagged path
alchemist51
force-pushed
the
internal-search-getbyid
branch
from
June 19, 2026 14:06
e0b2767 to
8d70736
Compare
thorkous
force-pushed
the
resolver-doc
branch
8 times, most recently
from
June 20, 2026 10:28
ff48bfc to
0f0daf4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Route the pluggable-dataformat get-by-id and seq-no-scan through the existing df_execute_query FFM call with an internalSearch flag instead of synthesizing SQL->Substrait in GetService. When the flag is set, the native side builds a DataFrame plan with a pushed-down filter (row_id = n for get-by-id, _seq_no
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.