fix(adapters,output): repair site drift and markdown/console output bugs - #235
Open
ankitranjan7 wants to merge 3 commits into
Open
fix(adapters,output): repair site drift and markdown/console output bugs#235ankitranjan7 wants to merge 3 commits into
ankitranjan7 wants to merge 3 commits into
Conversation
Core:
- escape `|` in markdown table cells so values containing pipes no longer
break the rendered table layout
- stop `consoleMessages('error')` from also returning warnings
- keep a row-less `<table>` from throwing inside turndown-plugin-gfm and
taking the whole article download with it
Twitter/X:
- read profile counts and bio from the current response containers after X
moved them out of `legacy`; stats no longer report 0
- render images from atomic article blocks, handling the current
array-shaped `entityMap`
- add resumable `--all` archives for `likes` and `bookmarks`, with JSONL
output and a resume file so interrupted runs continue
- add `twitter collection`: a bounded user-timeline fetch with relationship
facts and a completion receipt
Facebook:
- recover the display name from the current profile-avatar header and scope
friends/followers links to the profile region
- keep scrolling the feed until the extractor has enough valid rows rather
than stopping on raw article-marker counts
- preserve query identity in search result URLs (permalink/story/photo/watch)
while stripping per-render tracking nonces, and drop l./lm. redirect shims
TikTok:
- use the current /api/explore/item_list/ endpoint, keeping the recommend
feed as a fallback
Amazon:
- honor the marketplace named by the input URL instead of rewriting every
request and emitted URL to amazon.com; bare ASINs still default to the US
store, and look-alike hosts are rejected against an explicit domain list
- name the landed URL when a product or review page exposes no content
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
🟠 Maintainer review suggested — low confidenceThe automated review could not reach a fully supported conclusion. Limitations
This review is advisory and does not block merging. |
…ut bytes The plugin parity gate compared `args` against the frozen v0.5.3 manifest with strict deep-equality, so a migrated command could never gain a new optional flag — the check was stricter than the guarantee it protects. It now asserts what actually matters: every frozen argument is still present, invocable identically (name, type, default, required, positional), and in the same relative order, so positional invocations keep working. New arguments may be added around them and help text may be reworded. Every other field is still compared strictly. Also update the hosted markdown parity case, which pinned the pre-escaping bytes, and shrink the typed-error lint baseline by the two entries the likes / bookmarks rewrite resolved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The likes / bookmarks archive tests hardcoded POSIX `/tmp/...` paths. The command resolves whatever it is given, so on Windows those became `D:\tmp\...` — a directory that does not exist. Fixture writes failed with ENOENT, and the assertions compared the raw `/tmp` string against the resolved one. Building the paths from `os.tmpdir()` keeps them absolute and platform-correct, so `path.resolve` is a no-op on them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repairs a batch of adapter breakages caused by upstream site changes, plus three
correctness bugs in the shared output/download layer. Adds resumable archive
support to the Twitter likes/bookmarks commands and a new bounded
twitter collectioncommand.Core
|broke the rendered table layout. They are now escaped as\\|.consoleMessages('error')errorlevel was also returningwarningmessages. It now returns errors only.turndown-plugin-gfmreadstable.rows[0].parentNodefrom both its table rule and its keep filter, so a<table>with no<tr>threw before either could decide — failing the entire article download. A rule now claims those tables first.Twitter / X
profile— X relocated follower/following/tweet/like counts and the bio outof
legacyintorelationship_counts,tweet_counts,action_counts, andprofile_bio. All stats were reporting0and the bio was empty. Reads thecurrent containers with the legacy fields retained as fallbacks.
article— images were dropped entirely. Atomic Draft.js blocks were beingskipped, and the response now serializes
entityMapas an unordered[{key, value}]array rather than an object keyed by entity id. Both shapesare normalized and media is resolved through
media_entities.likes/bookmarks— new--all,--resume-file,--output-file, and--max-pagesflags for full-archive runs. Results stream to JSONL and theresume file is only cleared once the timeline is provably exhausted, so an
interrupt, a page-budget stop, or a partial API error all remain resumable.
A repeated cursor is treated as an error rather than silently claiming
completion.
collection(new) — bounded user-timeline fetch with relationship factsand an explicit completion receipt, backed by a new shared
user-timeline.jstransport helper.
tweetsis untouched.Facebook
profile— the header no longer uses anh1, so the display name came backempty. Falls back to the profile-avatar link's
aria-label, scoped to therequested profile path so page chrome can't win. Friends/followers links are
now matched against the exact profile path instead of any href containing
/friends, which was picking up global navigation.feed— the bounded scroll stopped as soon as raw[role="article"]/action-menu counts reached
--limit, but those markers also count comments andsuggestions, so runs under-returned. It now stops on the extractor's valid row
count, with a stall guard.
search—permalink.php,story.php,photo.php, andwatch/?v=encodeidentity in the query string, so deduping on pathname alone collapsed distinct
posts into a single row. Identity params are now kept, but only those —
per-render tracking nonces (
__cft__,__tn__,ref) are dropped so the samepost doesn't look unique on every render.
l.facebook.com/lm.facebook.comoutbound-redirect shims are excluded.
TikTok
explore—/explorenow hydrates from/api/explore/item_list/(
categoryType=120, the default "All" tab). The previous recommend feed is keptas a fallback for regions where
/explorestill uses it, and bothhasMore/has_moreshapes are handled.Amazon
product/discussion/ shared URL builders — a URL from any non-USmarketplace was rewritten to
amazon.com, so aamazon.co.ukproduct wasfetched from the wrong store and every emitted URL pointed at the US listing.
The input's marketplace is now honored end to end. Bare ASINs still default to
amazon.com. Host matching uses an explicit marketplace domain list rather thana shape-only
amazon.<tld>pattern, which would otherwise accept unrelatedregistrable domains like
amazon.shoporamazon.zip, and look-alikes such asamazon.evil.comorevilamazon.com.Testing
All touched suites pass:
One unrelated pre-existing failure remains in
plugins/facebook/test/notifications.test.js, which reads the generatedplugin-command-manifest.jsonbuild artifact; it fails identically onmain.Note:
src/output.test.tshad a case pinning the old unescaped-pipe output(
preserves the legacy literal Markdown table cells). It has been rewritten toassert escaping, since unescaped pipes genuinely corrupt the table.
🤖 Generated with Claude Code