feat(saved-jobs): add saved/bookmarked jobs scraping with pagination and progress

[IfThingsThenStuff]

Thanks for your work here - useful tool. Appreciate your efforts. I wanted the ability to read out my saved jobs - so, I added it. It will handle multiple pages.

Let me know if this is aligned to what you would like to include. Let me know of any changes you think are needed.

Summary

• Add scrape_saved_jobs to LinkedInExtractor — scrapes the LinkedIn jobs tracker page, extracts job IDs from link hrefs, and paginates through results using numbered page buttons
• Add get_saved_jobs MCP tool with progress reporting via on_progress callback
• Cap total_pages with max_pages for accurate progress percentages
• Use Set for O(1) job ID deduplication in the DOM polling function
• Add navigation delay between page clicks consistent with other scraping methods

Test plan

• test_scrape_saved_jobs_single_page — single page with progress callback
• test_scrape_saved_jobs_paginates — multi-page with progress and ID collection
• test_scrape_saved_jobs_timeout_stops_gracefully — timeout returns partial results
• test_scrape_saved_jobs_stops_at_max_pages_despite_more_buttons — respects max_pages cap
• test_scrape_saved_jobs_empty — empty results
• test_get_saved_jobs — tool-level success path
• test_get_saved_jobs_error — session expired error handling
• Full suite: 112/112 passing

Greptile Summary

Adds get_saved_jobs MCP tool to scrape saved/bookmarked jobs from LinkedIn's job tracker with pagination and progress reporting.

Key Changes:

• Pagination: Navigates through numbered page buttons, extracting job IDs from link hrefs (/jobs/view/<id>/)
• Deduplication: Uses Set for O(1) job ID lookups in both JavaScript extraction and Python filtering
• Progress Reporting: Implements on_progress callback with accurate page counts capped by max_pages
• Error Handling: Gracefully handles timeouts, missing buttons, and empty results
• Navigation: Adds 2s delay between page clicks consistent with other scraping methods

Implementation Quality:

• Exposes max_pages parameter (default 10) for user control
• Embeds job IDs in text sections for LLM visibility
• Returns both structured job_ids list and formatted text
• Comprehensive test suite: 6 new tests covering pagination, timeouts, edge cases
• Full test suite passing: 112/112
• Documentation updated in README.md, AGENTS.md, docs/docker-hub.md, and CLAUDE.md per development workflow

Previous Review Items Addressed:

• ✅ Set-based deduplication in _EXTRACT_JOB_IDS_JS (lines 389-390)
• ✅ Exposed max_pages parameter in tool signature (line 75)
• ✅ Documentation updates completed across all required files

Confidence Score: 5/5

• This PR is safe to merge with no identified issues
• Score reflects comprehensive test coverage (112/112 tests passing including 6 new tests), complete documentation updates per development workflow, robust error handling with graceful degradation, efficient O(1) deduplication using Sets, proper pagination logic with multiple safety breaks, and all previous review comments fully addressed
• No files require special attention

Important Files Changed

Filename	Overview
linkedin_mcp_server/scraping/extractor.py	Added scrape_saved_jobs method with robust pagination logic, Set-based O(1) deduplication, proper error handling, and progress callbacks
linkedin_mcp_server/tools/job.py	Added get_saved_jobs MCP tool with exposed max_pages parameter, progress reporting, and consistent error handling
tests/test_scraping.py	Added comprehensive test suite with 5 tests covering single-page, pagination, timeout, max_pages cap, and empty results scenarios
tests/test_tools.py	Added tool-level tests for get_saved_jobs success path and error handling

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Start([Start]) --> Navigate[Navigate to jobs-tracker]
    Navigate --> ExtractPage1[Extract page 1 text and IDs]
    ExtractPage1 --> CountButtons[Count pagination buttons]
    CountButtons --> CalcTotal[Calculate total_pages cap]
    CalcTotal --> ReportP1[Report progress page 1]
    ReportP1 --> CheckMore{More pages?}
    
    CheckMore -->|Yes| CheckButton{Button exists?}
    CheckButton -->|No| Append[Append ID summary]
    CheckButton -->|Yes| ClickButton[Click page button]
    ClickButton --> WaitDelay[Wait nav delay]
    WaitDelay --> WaitNewIDs{Wait for new IDs}
    
    WaitNewIDs -->|Timeout| Append
    WaitNewIDs -->|Success| Scroll[Scroll to bottom]
    Scroll --> ExtractText[Extract page text]
    ExtractText --> ExtractIDs[Extract job IDs]
    ExtractIDs --> FilterDups[Filter duplicates]
    FilterDups --> CheckNewIDs{New IDs?}
    
    CheckNewIDs -->|No| Append
    CheckNewIDs -->|Yes| AddIDs[Add to all_job_ids]
    AddIDs --> ReportProgress[Report progress]
    ReportProgress --> CheckMore
    
    CheckMore -->|No| Append
    Append --> BuildSections[Build sections dict]
    BuildSections --> Return([Return result])

Loading

_{Last reviewed commit: 5e68717}

[greptile-apps]

_{4 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[IfThingsThenStuff]

Hey there @stickerdaniel - hope you're doing well. Is there anything I can do to help get this merged in sir? Thanks in advance. Let me know.

[greptile-apps]

Greptile Summary

This PR adds a get_saved_jobs MCP tool that scrapes LinkedIn's job tracker page (/jobs-tracker/), paginates through results using numbered page buttons in a SPA, and returns deduplicated job IDs alongside extracted text. All previously raised concerns (Set-based deduplication in _EXTRACT_JOB_IDS_JS, exposing max_pages, and adding documentation to README/CLAUDE.md/AGENTS.md) have been addressed. Tests cover single-page, multi-page, timeout, max-pages cap, and empty cases (112 passing).

Confidence Score: 5/5

Safe to merge — all previously raised concerns are resolved and no new P0/P1 issues found.

All prior review threads (Set deduplication in _EXTRACT_JOB_IDS_JS, exposing max_pages, and documentation updates for README/CLAUDE.md/AGENTS.md/docker-hub.md) are fully addressed. The only remaining finding is a P2 style suggestion to align the loop upper bound with the already-computed total_pages. The implementation is well-tested (112/112 passing), follows codebase conventions, and has no logic or security issues.

No files require special attention.

Vulnerabilities

No security concerns identified. The scraper only navigates to first-party LinkedIn URLs, does not accept user-supplied URLs or execute user-provided code, and job IDs are extracted via a regex match on numeric digits only (\d+), preventing injection.

Important Files Changed

Filename	Overview
linkedin_mcp_server/scraping/extractor.py	Adds scrape_saved_jobs method with SPA pagination via button clicks, Set-based deduplication, progress callbacks, and graceful timeout handling.
linkedin_mcp_server/tools/job.py	Adds get_saved_jobs MCP tool with max_pages parameter exposed and progress reporting via _report callback; correctly caps progress at 99% until final 100% signal.
tests/test_scraping.py	Adds 5 well-structured tests for scrape_saved_jobs covering single-page, multi-page pagination, timeout, max-pages cap, and empty cases.
tests/test_tools.py	Adds test_get_saved_jobs and test_get_saved_jobs_error covering the tool-level success path and session-expired error handling.

Sequence Diagram

sequenceDiagram
    participant Tool as get_saved_jobs (tool)
    participant Extractor as LinkedInExtractor
    participant Page as Patchright Page
    participant LI as LinkedIn (jobs-tracker)

    Tool->>Extractor: scrape_saved_jobs(max_pages, on_progress)
    Extractor->>Page: goto(jobs-tracker/)
    Page->>LI: HTTP GET /jobs-tracker/
    LI-->>Page: SPA HTML
    Extractor->>Page: evaluate(_EXTRACT_JOB_IDS_JS)
    Page-->>Extractor: page 1 job IDs
    Extractor->>Page: locator('button[aria-label^=Page]').count()
    Page-->>Extractor: total_pages
    Extractor->>Tool: on_progress(1, total_pages, ...)

    loop for each page 2..total_pages while button exists
        Extractor->>Page: locator('button[aria-label=Page N]').click()
        Extractor->>Page: wait_for_function(new IDs appear, timeout=15s)
        Page-->>Extractor: new IDs detected (or TimeoutError - break)
        Extractor->>Page: scroll_to_bottom()
        Extractor->>Page: evaluate(_EXTRACT_MAIN_TEXT_JS)
        Page-->>Extractor: page text
        Extractor->>Page: evaluate(_EXTRACT_JOB_IDS_JS)
        Page-->>Extractor: all visible IDs (deduped with prev_ids)
        Extractor->>Tool: on_progress(page_num, total_pages, ...)
    end

    Extractor-->>Tool: url, sections, job_ids, pages_visited, sections_requested
    Tool->>Tool: ctx.report_progress(100, 100, Complete)
    Tool-->>MCP Client: result dict

Loading

Prompt To Fix All With AI

This is a comment left during a code review.
Path: linkedin_mcp_server/scraping/extractor.py
Line: 446

Comment:
**Loop upper bound ignores `total_pages`**

`total_pages` is already capped at `max_pages` via `min(…, max_pages)`, so the loop can use `total_pages + 1` as its upper bound instead of `max_pages + 1`. Both produce the same result (the button check stops the loop early either way), but using `total_pages` makes the relationship between the detected page count and the iteration range explicit and avoids iterating past the last real page.

```suggestion
        for page_num in range(2, total_pages + 1):
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (3): Last reviewed commit: "docs(saved-jobs): add docs, expose max_p..." | Re-trigger Greptile}

[greptile-apps]

Tip:

Greploops — Automatically fix all review issues by running /greploops in Claude Code. It iterates: fix, push, re-review, repeat until 5/5 confidence.

Use the Greptile plugin for Claude Code to query reviews, search comments, and manage custom context directly from your terminal.

[IfThingsThenStuff]

replaced by #338