Staging - #230
Merged
Merged
Conversation
…ng (#212) * feat: implement node assembler for page memory hierarchy to support shared page content and VLM-based node summarization * feat: implement page asset integration into node assembly and chunk connections with support for custom probes * feat: enhance document profiling and page memory integration with skip shard plan option - Added `skip_shard_plan` parameter to `run_lightweight_anatomy` and `profile_document` to allow bypassing LLM shard decisions during profiling. - Updated `memory_service` to utilize `skip_shard_plan` for page memory processing. - Introduced asset annotation functionality for visualizing extracted assets on pages. - Refactored page section mapping and node assembly to support internal section body pages. - Improved environment configuration for Java dependencies in Docker setup. * feat: enhance retrieval and document ingestion with new chunk types and effective parse track handling - Updated `data_type` in `RetrievalQueryRequest` to support new chunk types: page (7) and text+image+table (8). - Refactored document ingestion service to apply effective parse track based on file extension and user-defined settings. - Introduced new validation for parse track handling in oversized PDF processing. - Enhanced tests to cover new chunk types and parse track logic, ensuring robust validation and functionality. * refactor: decouple page memory from image URIs and simplify node metadata structure * feat: integrate summary engine for enhanced document parsing and asset handling - Updated document parser to utilize the new summary engine for images, tables, and text, improving the extraction of titles, summaries, and entities. - Refactored image and table handling to streamline the summarization process, replacing legacy methods with unified calls to the summary engine. - Enhanced markdown parsing to support new entity and asset title fields, ensuring comprehensive data capture during document processing. - Introduced serialization for typed entities to maintain structured data integrity across parsed documents. * feat: implement concurrent page tagging and enrich image and table asset metadata with LLM-extracted entities and titles * feat: standardize page_memory track for PDF/PPTX, implement page-count limits, and refactor section summary publication to initialization. * chore: align parser env defaults * refactor: replace legacy data_type integer with flexible chunk_types set across retrieval services * feat: implement intermediate node chain collapse and add utilities for cross-page table continuity analysis * Add thread-safe trace logging, remove unused heading dataframes, and implement a text-only tagging mode for page memory processing. * refactor: remove budget tracking dependency from page memory services * chore: fix lint and type errors caught by ruff and pyright - Remove unused pandas import in docx/parser.py (F401) - Remove unused collapse_page_ranges import in fine_hierarchy.py (F401) - Rename unused ctx variable to _ctx in debug_page_memory.py (F841) - Remove unused tempfile import in test_page_memory_cross_page_table_merge.py (F401) - Fix greenlet value type: filter None from scope_results list in memory_service.py (reportAssignmentType) - Fix BeautifulSoup Tag narrowing in page_assets.py: use isinstance(x, Tag) guards for find_all/append calls (reportAttributeAccessIssue, reportOptionalMemberAccess) * fix: resolve alembic revision ID conflict causing cycle detection failure The feature branch introduced a migration with revision ID 'a1b2c3d4e5f6' (replace_data_type_with_chunk_types), which collided with an existing main branch migration sharing the same ID (add_qstash_tracking_columns). Alembic detected this as a cycle and blocked all migration tests. Fix: Assigned a unique revision ID 'f0d85d209e68' to the chunk_types migration and rebased its down_revision onto 'f9d0e1f2a3b4' (add_document_metadata_to_documents), which is the current main head. The Alembic graph now has a single clean head with no cycles. * test: fix failing tests from page-memory native hierarchy branch - test_agentic_evidence_renderer_contract.py: Update expected string format to match 'Pages 225-226' instead of 'Pages: 225, 226'. - test_page_memory_cross_page_table_merge.py: Remove unused 'budget' kwarg from merge_cross_page_tables() test calls. - test_doc_profile_anatomy_contract.py: Mock 'PDF_PROFILE_TOC_ENABLED' = True since the feature branch relies on it to test TOC profile attempting logic. * refactor: remove page_memory_work_summary_20260624.md file and streamline page memory processing documentation * feat: enhance skeleton extractor with native hierarchy support and agentic navigation improvements - Add utility functions for page memory processing in _utils.py - Refactor skeleton_extractor with enhanced hierarchy building and cross-page table merge - Improve agentic navigation with structured state management and tool enhancements - Add contract tests for agentic discovery selection - Integrate hierarchy locator improvements for document structure analysis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove dead commented code and propagate PageLocateConfig to pending TOC fallback - Remove old _estimate_page_offset code block (superseded by _calibrate_offset_via_vlm) - Pass page_memory_config through to _resolve_pending_tocs so its PageLocateResidualAgent fallback uses the same PageLocateConfig as the primary path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: remove PageLocateResidualAgent fallback from skeleton extractor The offset-guided bulk anchoring with binary search handles all breakpoint scenarios. The expensive grep+VLM residual agent fallback is no longer needed. When offset anchoring is unavailable or returns empty, we fall through with calibration_overrides only and let the hierarchy resolver handle the rest via physical hints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve PR #200 lint and typecheck failures - Remove unused imports flagged by ruff F401 across page_memory and document_agent modules - Fix 6 pyright errors in skeleton_extractor.py: - min() on list[int|None]: use explicit loop with type-narrowed append - arithmetic on TitleNode.printed_page after filter: extract local int variable with explicit None guard Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add toc_page_offset to PageAnatomyMap and AgentBlackboard - Introduced toc_page_offset to both PageAnatomyMap and AgentBlackboard to facilitate better handling of table of contents page offsets. - Updated relevant functions to utilize the new toc_page_offset for improved shard planning and TOC processing. - Removed the match_h1_pages tool as it was deemed unnecessary for current functionality. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: prune out-of-scope TOC nodes before offset-guided anchoring When a PDF is truncated (physical pages < TOC page references), leaves with printed_page + offset > page_count are pruned from the tree before anchoring. This eliminates unnecessary bisect/recalibrate cycles and prevents unlocated fallback noise. Also adjusts _verify_offset_tail to prefer non-boundary leaves (printed_page + offset < page_count) for more reliable VLM verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add chapter boundary detection and toc_chapter_boundary anchor type Add derive_chapter_boundaries() for L1/L2-aware shard planning with physical page ranges. Extend Shard.anchor_type to accept "toc_chapter_boundary" alongside existing types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use private _calibrate_offset_via_vlm import after merge rename Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: restore use_agentic switch with classic top-K retrieval route Default retrieval now uses the classic 3-channel BM25 + RRF top-K path. Only explicit use_agentic=True triggers the full agentic workflow (LLM doc-select + navigation). Removes dead RETRIEVAL_AGENTIC_ENABLED env var reference from contract test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update contract test to expect classic_topk when use_agentic=False The test previously asserted that use_agentic=False was ignored and still routed to the workflow. Now that the switch is functional, the test verifies that use_agentic=False correctly routes to classic_topk. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove dead RETRIEVAL pydantic config fields RETRIEVAL_DECOMPOSITION_ENABLED and RETRIEVAL_AGENTIC_INLINE_TABLE_CHAR_LIMIT are defined in AIConfig but never accessed at runtime. Remove them to reduce confusion. Agentic vs classic routing is now controlled per-request via use_agentic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: implement position-aware spatial constraints and boundary detection for VLM title extraction * refactor: simplify spatial orientation instructions and scope boundary definitions in prompt service * fix: resolve lint errors after merging origin/main (remove unused variables) * fix: accept title_ctx in page tagger contract test mocks Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: update document path formatting and improve chunk handling - Changed path formatting in chunks.json from arrow-delimited to slash-delimited for consistency. - Updated source_catalog.py to reflect new path format in citations. - Enhanced source_projection.py to utilize new path handling functions for better readability. - Removed outdated tests related to arrow-delimited paths in test_chunk_document_path_contract.py. - Improved path handling in various document parser modules to ensure compatibility with new path structure. - Added new functions for joining and splitting document paths to streamline path management across services. * fix: resolve path-escape lint/type issues before PR sync Use Sequence typing for join_document_path and escape-aware path splitting when dividing long leaf contents so make check passes. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…cate (#220) * refactor: remove residual page location agent and related tools This commit eliminates the page locate sub-agent and its associated tools, streamlining the hierarchy locator module. The VLM page verification process is now directly integrated into the hierarchy locator, enhancing efficiency and reducing complexity. Unused configurations and functions related to the residual agent have also been removed from the page memory configuration. * feat: null-page TOC parent locate and VLM-order skeleton sort Add window-bounded locate for coarse TOC parents whose printed_page is null (compact-strict text match with visual RTL fallback), emitting parent self-only skeletons so interstitial content is not swallowed. Sort skeletons by start_page only and build the hierarchy tree from list order, so same-page siblings keep VLM/TOC reading order instead of being reordered alphabetically by section_path. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: drop stale null-page TODO and duplicate debug import after merge The null-page parent locate is already implemented on this branch, so the placeholder TODO reintroduced by the merge is misleading. Also remove the duplicated page-memory serialization import left by the merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: drop fine-hierarchy boundary page when next-title anchor misses Prevent next-section titles from bleeding into the current fat leaf when the VLM fails to observe the coarse end anchor. Align next-title ordering with stable reading order so same-page parent/child anchors stay correct. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: enhance document navigation summaries with top_summary persistence - Introduced a new function to persist the document-level top_summary in doc_nav.json, ensuring it is saved after enrichment. - Updated `enrich_doc_nav_summaries` to support an opt-in flag for using LLM for top_summary generation, defaulting to true. - Refactored existing logic to prioritize loading enriched summaries from disk, improving efficiency and reducing unnecessary recomputation. - Adjusted related tests to validate the new functionality and ensure correct behavior of top_summary persistence. This update enhances the document navigation process by ensuring that summaries are consistently saved and retrieved, improving overall user experience. * feat: enhance document parsing and image handling with new features - Updated the summary builder to improve document-level top summary generation, ensuring child contributions are deterministically summarized without LLM unless specified. - Enhanced inline asset handling to support table-embedded images, allowing for dynamic type assignment based on image references. - Introduced image size filtering across various document formats to skip undersized images, improving overall processing efficiency. - Refactored markdown image asset handling to include new properties for better management of image renaming and deferred tasks. - Improved table asset handling to include image references, ensuring that images embedded in tables are correctly processed and displayed. These updates collectively enhance the document parsing capabilities, streamline image handling, and improve the overall user experience in document navigation and summary generation. * fix: accept model kwarg in summary_builder prompt unit test Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: enhance page memory serialization and title detection - Added `observed_titles` to the `serialize_page_tags` function for improved title tracking. - Introduced `serialize_scope_skeletons` to handle scope input artifacts, facilitating Stage3 to Stage4 handoff. - Updated `write_scope_artifacts` to conditionally write `tags` and `assets_by_page`, preventing overwrites when set to None. - Enhanced title detection logic in `tag_vlm_titles` to escalate token budgets dynamically when JSON responses are truncated, improving robustness in title extraction. This commit improves the handling of page memory artifacts and enhances the title detection process, ensuring better data integrity and extraction accuracy. * refactor: remove experimental chart and VLM bbox probing scripts - Deleted `chart_asset_probe.py` and `vlm_bbox_tabula_probe.py` as they are no longer needed. - These scripts were used for experimental purposes related to VLM-driven asset detection and table extraction, but have been superseded by more efficient implementations. This cleanup helps streamline the codebase by removing outdated experimental files. * feat: enable asset extraction in page memory configuration and update related tests - Changed `asset_extraction_enabled` to `True` in `PageMemoryConfig` to activate asset extraction functionality. - Updated the `page_asset_extraction_enabled` function to return `True`, reflecting the new configuration. - Modified a test to assert that asset extraction is enabled for page memory jobs. - Added `.gitignore` entry for the `experiments` directory in the worker app to exclude local debugging scripts. This commit enhances the asset extraction capabilities in the page memory system, ensuring that the feature is properly enabled and tested. * feat: integrate asset probing and enhance document profiling - Added `_ensure_asset_probe` method to `ProfileCoordinator` to ensure asset probing occurs during profile generation. - Introduced `has_asset` and `asset_bboxes` fields in `PageFeature` for tracking asset presence and bounding boxes. - Updated `aggregate_doc_stats` to include asset-related statistics when assets have been probed. - Refactored `probe_page_features` to support asset probing alongside existing features. - Modified `planner` and `tagger` logic to accommodate new asset-related data and improve document profiling accuracy. This commit enhances the document agent's ability to extract and utilize asset information, improving overall profiling and analysis capabilities. * fix: tighten image bbox key typing for pyright Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: streamline document agent structure and enhance profiling capabilities - Removed the `ProfileAgent` class to simplify the document agent architecture. - Updated `ProfileCoordinator` methods to include a `skip_shard_plan` parameter, allowing for more flexible profiling without LLM shard decisions. - Enhanced `PageAnatomyMap` to include `toc_hierarchies` for improved TOC handling. - Cleaned up unused fields and methods related to asset probing and text previews, ensuring a more efficient codebase. This commit improves the overall structure and efficiency of the document agent, facilitating better profiling and asset management. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: chengke <404835780@qq.com>
Reuse the request AsyncSession for tier lookup and API-key last_used updates (with Redis debounce), and wire the async engine to DB_POOL_SIZE/MAX_OVERFLOW defaults of 50/50 so legal poll bursts no longer exhaust the per-process pool via nested get_db_context holds. Co-authored-by: Cursor <cursoragent@cursor.com>
Use Redis SET NX for last-used debounce, require the request session in resolve_current_user, and avoid double-commit when TierService owns the session. Co-authored-by: Cursor <cursoragent@cursor.com>
Explain the best-effort rollback swallow for CodeQL, narrow pool pressure logging to QueuePool for pyright, and stop unit tests from shadowing the worker `app` package during shared pytest collection. Co-authored-by: Cursor <cursoragent@cursor.com>
Harden worker contract eviction to drop deepest API-shadowed modules first, and make job-poll unit tests reassert the API import root then clear cached API app modules so shared pytest runs stay green. Co-authored-by: Cursor <cursoragent@cursor.com>
…l-hygiene fix: QueuePool hygiene for job-poll auth (mainline)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.