feat(intelligent-assistant): add DOM text extraction for deep context awareness (RHIDP-14317) - #4307
Open
its-mitesh-kumar wants to merge 11 commits into
Conversation
…itive data redaction Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
- Fall back to original canvas when getContext('2d') returns null
- Add CSS.escape() to prevent selector injection in label lookup
- Remove redundant password type check (already in isSensitiveElement)
- Wrap element.matches() in try/catch for invalid selectors
- Use case-insensitive CSS attribute selectors for aria-label matching
- Convert html2canvas-pro to dynamic import for bundle size optimization
- Clear timeout timer in finally block to prevent resource leak
- Replace as-any casts with expect.objectContaining in tests
- Remove fragile index-based SECRET_PATTERNS tests
- Add JSDoc warning about stateful /g regexes
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… awareness Implements RHIDP-14317 - extracts structured page content (headings, tables, alerts, body text) from the visible RHDH page and sends it as a text/plain attachment alongside user messages when screen context is enabled. Key decisions over the PoC (innerHTML) approach: - No token waste: strips HTML tags/attributes that add zero semantic value - Priority-based extraction: alerts > headings > tables > body text - Configurable char budget (default 8000) to cap token usage - Zero backend changes: uses existing attachment pipeline Co-authored-by: Cursor <cursoragent@cursor.com>
its-mitesh-kumar
requested review from
a team,
karthikjeeyar,
rohitkrai03 and
yangcao77
as code owners
August 13, 2026 22:53
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
…nt-dom-text-extraction Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com> # Conflicts: # workspaces/intelligent-assistant/plugins/intelligent-assistant/config.d.ts # workspaces/intelligent-assistant/plugins/intelligent-assistant/package.json # workspaces/intelligent-assistant/plugins/intelligent-assistant/src/utils/screen-capture.ts # workspaces/intelligent-assistant/yarn.lock
|
Important This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior. Changed Packages
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4307 +/- ##
==========================================
+ Coverage 61.27% 61.31% +0.04%
==========================================
Files 2509 2510 +1
Lines 100516 100668 +152
Branches 28136 28205 +69
==========================================
+ Hits 61594 61728 +134
- Misses 37113 37135 +22
+ Partials 1809 1805 -4
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Co-authored-by: Cursor <cursoragent@cursor.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements RHIDP-14317 — extracts structured page content from the visible RHDH page and sends it as context alongside user messages when screen context is enabled.
dom-extractor.tswith priority-based extraction: page header → alerts → headings → tables → body text → Shadow DOM (TechDocs)sendMessage()flow as atext/plainattachment using the existing backend pipeline (zero backend changes)dom-extraction.enabledanddom-extraction.maxCharsconfig options underscreen-contextscreen-context.enabled: trueis already set inapp-config.yamlfor local testingWhy this approach over the PoC (
innerHTML)?maxChars(default 8000 ≈ 2000 tokens)redactText()before sendingFixed
UI after testing
S_.2026-08-14.at.4.mp4
Local Testing: Download Extracted DOM as
.txtTo inspect the exact DOM context being extracted and sent, cherry-pick the debug commit:
This adds:
console.debug('[DOM Extraction]', { chars, content })on every message senddom-context-<timestamp>.txtfile so you can inspect the extracted contentSteps:
yarn startin theintelligent-assistantworkspace.txtfile will auto-download — open it to verify extracted content[DOM Extraction]log with char countTest Scenarios
Basic Functionality
Tab: ...)Header Extraction
Page: /catalog/default/component/...pathPlugin: <name>(if.bui-PluginHeaderToolbarNameexists)Title: <title>(if.bui-HeaderTitleexists)Tab: <tab-name>(if[role="tab"][aria-selected="true"]exists)Document: <title>Priority & Truncation
.txtfile size)Noise Removal
<script>,<style>,<svg>,<noscript>content is NOT present in outputnav,[role="navigation"]) are excluded.lightspeed-chat-panel)[aria-hidden="true"],[hidden]) are excludedTable Extraction
| col1 | col2 |pipe-delimited format... (N more rows)indicator...columnSensitive Data Redaction
*****in outputConfiguration
screen-context.dom-extraction.enabled: false→ no DOM attachment is sentscreen-context.enabled: false→ no DOM or screenshot attachment is sentmaxCharsvalue (e.g. 2000) reduces output size accordinglyEdge Cases
Unit Tests
All 26 tests pass covering: noise removal, heading extraction, alert extraction, table formatting, body text extraction, redaction integration, truncation/budget, Shadow DOM, header parsing (BUI + tabs), and custom exclusion selectors.