feat: expand clickable elements - #131
Conversation
c513712 to
05bb640
Compare
ruocco-l
left a comment
ruocco-l
left a comment
There was a problem hiding this comment.
I will be pre-approving because I think it's correct, but let's all agree that this is a no-opt, non-customizable feature for the user. Some pages (it happened in WCC but I can't seem to find the issue related to it) may be broken by clicking (because they cause navigation) de facto making it impossible to scrape them with headless browser on RAG
| if (page) { | ||
| await expandClickableElements(page, CLICK_ELEMENTS_CSS_SELECTOR); | ||
| addTimeMeasureEvent(request.userData, 'playwright-expand-clickable-elements'); | ||
| } | ||
|
|
There was a problem hiding this comment.
If you move this up, before the wait for dynamic content function, you don't have to add a synthetic wait
There was a problem hiding this comment.
Rechecked with scripts to measure: on real pages (3 runs each): clicking before the wait loses content: openai.com −785 chars, github.com −422
Before the wait the page usually isn't hydrated (we go with domcontentloaded), so the collapsed elements have no click listeners yet, or don't exist at all.
| Note that clicking costs about half a second on pages that have such elements, and that content of | ||
| expanded navigation menus can end up in the output as well. | ||
|
|
There was a problem hiding this comment.
They don't have a way to set it off, so there is no use in saying this
4 participants
In Browser mode the Actor now clicks the collapsed elements (
[aria-expanded="false"]) before extracting the content, same as WCC. No new input field, and the Raw HTTP mode is unchanged.Measured on 12 real pages: clicking costs about 0.5 s on pages that have collapsed elements and ~7 ms on pages that don't.
Closes #112