Skip to content

fix(adapters,output): repair site drift and markdown/console output bugs - #235

Open
ankitranjan7 wants to merge 3 commits into
mainfrom
fix/adapter-drift-and-output-escaping
Open

fix(adapters,output): repair site drift and markdown/console output bugs#235
ankitranjan7 wants to merge 3 commits into
mainfrom
fix/adapter-drift-and-output-escaping

Conversation

@ankitranjan7

Copy link
Copy Markdown
Contributor

Repairs a batch of adapter breakages caused by upstream site changes, plus three
correctness bugs in the shared output/download layer. Adds resumable archive
support to the Twitter likes/bookmarks commands and a new bounded
twitter collection command.

Core

Fix Detail
Markdown table escaping Cell values containing | broke the rendered table layout. They are now escaped as \\|.
consoleMessages('error') The error level was also returning warning messages. It now returns errors only.
Row-less tables turndown-plugin-gfm reads table.rows[0].parentNode from both its table rule and its keep filter, so a <table> with no <tr> threw before either could decide — failing the entire article download. A rule now claims those tables first.

Twitter / X

  • profile — X relocated follower/following/tweet/like counts and the bio out
    of legacy into relationship_counts, tweet_counts, action_counts, and
    profile_bio. All stats were reporting 0 and the bio was empty. Reads the
    current containers with the legacy fields retained as fallbacks.
  • article — images were dropped entirely. Atomic Draft.js blocks were being
    skipped, and the response now serializes entityMap as an unordered
    [{key, value}] array rather than an object keyed by entity id. Both shapes
    are normalized and media is resolved through media_entities.
  • likes / bookmarks — new --all, --resume-file, --output-file, and
    --max-pages flags for full-archive runs. Results stream to JSONL and the
    resume file is only cleared once the timeline is provably exhausted, so an
    interrupt, a page-budget stop, or a partial API error all remain resumable.
    A repeated cursor is treated as an error rather than silently claiming
    completion.
  • collection (new) — bounded user-timeline fetch with relationship facts
    and an explicit completion receipt, backed by a new shared user-timeline.js
    transport helper. tweets is untouched.

Facebook

  • profile — the header no longer uses an h1, so the display name came back
    empty. Falls back to the profile-avatar link's aria-label, scoped to the
    requested profile path so page chrome can't win. Friends/followers links are
    now matched against the exact profile path instead of any href containing
    /friends, which was picking up global navigation.
  • feed — the bounded scroll stopped as soon as raw [role="article"] /
    action-menu counts reached --limit, but those markers also count comments and
    suggestions, so runs under-returned. It now stops on the extractor's valid row
    count, with a stall guard.
  • searchpermalink.php, story.php, photo.php, and watch/?v= encode
    identity in the query string, so deduping on pathname alone collapsed distinct
    posts into a single row. Identity params are now kept, but only those —
    per-render tracking nonces (__cft__, __tn__, ref) are dropped so the same
    post doesn't look unique on every render. l.facebook.com / lm.facebook.com
    outbound-redirect shims are excluded.

TikTok

  • explore/explore now hydrates from /api/explore/item_list/
    (categoryType=120, the default "All" tab). The previous recommend feed is kept
    as a fallback for regions where /explore still uses it, and both hasMore /
    has_more shapes are handled.

Amazon

  • product / discussion / shared URL builders — a URL from any non-US
    marketplace was rewritten to amazon.com, so a amazon.co.uk product was
    fetched from the wrong store and every emitted URL pointed at the US listing.
    The input's marketplace is now honored end to end. Bare ASINs still default to
    amazon.com. Host matching uses an explicit marketplace domain list rather than
    a shape-only amazon.<tld> pattern, which would otherwise accept unrelated
    registrable domains like amazon.shop or amazon.zip, and look-alikes such as
    amazon.evil.com or evilamazon.com.
  • Product/discussion failures now name the URL that was actually loaded.

Testing

All touched suites pass:

npx vitest run plugins/twitter plugins/facebook plugins/tiktok plugins/amazon \
  src/output.test.ts src/download/article-download.test.ts src/browser

One unrelated pre-existing failure remains in
plugins/facebook/test/notifications.test.js, which reads the generated
plugin-command-manifest.json build artifact; it fails identically on main.

Note: src/output.test.ts had a case pinning the old unescaped-pipe output
(preserves the legacy literal Markdown table cells). It has been rewritten to
assert escaping, since unescaped pipes genuinely corrupt the table.

🤖 Generated with Claude Code

Core:
- escape `|` in markdown table cells so values containing pipes no longer
  break the rendered table layout
- stop `consoleMessages('error')` from also returning warnings
- keep a row-less `<table>` from throwing inside turndown-plugin-gfm and
  taking the whole article download with it

Twitter/X:
- read profile counts and bio from the current response containers after X
  moved them out of `legacy`; stats no longer report 0
- render images from atomic article blocks, handling the current
  array-shaped `entityMap`
- add resumable `--all` archives for `likes` and `bookmarks`, with JSONL
  output and a resume file so interrupted runs continue
- add `twitter collection`: a bounded user-timeline fetch with relationship
  facts and a completion receipt

Facebook:
- recover the display name from the current profile-avatar header and scope
  friends/followers links to the profile region
- keep scrolling the feed until the extractor has enough valid rows rather
  than stopping on raw article-marker counts
- preserve query identity in search result URLs (permalink/story/photo/watch)
  while stripping per-render tracking nonces, and drop l./lm. redirect shims

TikTok:
- use the current /api/explore/item_list/ endpoint, keeping the recommend
  feed as a fallback

Amazon:
- honor the marketplace named by the input URL instead of rewriting every
  request and emitted URL to amazon.com; bare ASINs still default to the US
  store, and look-alike hosts are rejected against an explicit domain list
- name the landed URL when a product or review page exposes no content

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

🟠 Maintainer review suggested — low confidence

The automated review could not reach a fully supported conclusion.

Limitations

  • Some automated findings could not be verified against the pull request diff.

This review is advisory and does not block merging.

ankitranjan7 and others added 2 commits August 8, 2026 09:08
…ut bytes

The plugin parity gate compared `args` against the frozen v0.5.3 manifest with
strict deep-equality, so a migrated command could never gain a new optional
flag — the check was stricter than the guarantee it protects. It now asserts
what actually matters: every frozen argument is still present, invocable
identically (name, type, default, required, positional), and in the same
relative order, so positional invocations keep working. New arguments may be
added around them and help text may be reworded. Every other field is still
compared strictly.

Also update the hosted markdown parity case, which pinned the pre-escaping
bytes, and shrink the typed-error lint baseline by the two entries the likes /
bookmarks rewrite resolved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The likes / bookmarks archive tests hardcoded POSIX `/tmp/...` paths. The
command resolves whatever it is given, so on Windows those became `D:\tmp\...`
— a directory that does not exist. Fixture writes failed with ENOENT, and the
assertions compared the raw `/tmp` string against the resolved one. Building
the paths from `os.tmpdir()` keeps them absolute and platform-correct, so
`path.resolve` is a no-op on them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant