feat(web):Add web page to Markdown conversion tool#1048
Open
wondywang wants to merge 1 commit intojackwener:mainfrom
Open
feat(web):Add web page to Markdown conversion tool#1048wondywang wants to merge 1 commit intojackwener:mainfrom
wondywang wants to merge 1 commit intojackwener:mainfrom
Conversation
This file implements a CLI tool to convert web pages to Markdown format, utilizing the Readability library for content extraction and Turndown for HTML-to-Markdown conversion. It includes options for outputting to a file or stdout and handles various media URLs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This file implements a CLI tool to convert web pages to Markdown format, utilizing the Readability library for content extraction and Turndown for HTML-to-Markdown conversion. It includes options for outputting to a file or stdout and handles various media URLs.
Description
This PR introduces a new, dedicated CLI plugin for converting web pages into high-fidelity Markdown. It is designed to address the limitations of the existing web read command, which currently lacks robust support for rich content like embedded media and complex tables.
This plugin ensures a more complete conversion by strictly preserving the original URLs of images and videos and accurately rendering HTML tables into Markdown format, making it suitable for archiving or processing content-heavy web pages.
Motivation
The primary motivation for this feature is to overcome the shortcomings of the current web read functionality. While web read is useful for extracting plain text, it often fails to capture the full context of a webpage, specifically:
Multimedia Content: Images and videos are frequently omitted or their links are lost.
Tabular Data: HTML tables are not correctly converted, leading to a loss of structured information.
This plugin fills that gap by providing a specialized tool for users who need to preserve the integrity of rich media and structured data when converting HTML to Markdown.
Web MD — Convert any web page to Markdown with enhanced quality.
Uses @mozilla/readability for content extraction and Turndown + GFM
or HTML-to-Markdown conversion. Preserves image/video URLs.
Usage:
Related issue: None
Type of Change