About

Wanish is a lightweight Python package for extracting key information from web articles and generating concise summaries. It reduces full-length articles to a few essential sentences while preserving the core meaning.

Note

This project is in maintenance mode. No new features are being added, but the code remains stable and fully functional for production use.

Key Features

• Smart Summarization: Extracts the main idea of the text into a short summary.
• Content Extraction: Cleans HTML by removing ads, navigation, headers, and footers.
• Metadata Retrieval: Automatically detects: * Canonical URL * Article Title * Main Image URL * Language Code (e.g., 'en', 'de', 'es')
• Clean HTML Output: Returns structured, clean HTML based on schema.org data.

Installation

You can install Wanish via pip:

pip install wanish

Usage

Basic usage example:

from wanish import Wanish

# Initialize the parser
wanish = Wanish()

# Process the URL
wanish.perform_url("https://example.com/article")

# Access extracted data
print(wanish.url)           # Canonical URL
print(wanish.title)         # Article title
print(wanish.image_url)     # Main image URL
print(wanish.language)      # Language code (e.g., 'en')
print(wanish.clean_html)    # Cleaned HTML content
print(wanish.description)   # Generated summary (5 sentences by default)

Configuration

You can customize the behavior by passing optional arguments to the Wanish class:

wanish = Wanish(
    url="https://example.com/article",  # Optional: auto-processes URL on init
    positive_keywords=["main", "story"], # Class/ID patterns to keep
    negative_keywords=["banner", "ad"],  # Class/ID patterns to remove
    summary_sentences_qty=5,             # Length of summary
    headers={'user-agent': 'my-bot/1.0'} # Custom HTTP headers
)

Parameter Details

• url (str, optional): If provided, the URL is processed immediately upon initialization. Default: None.
• positive_keywords (list, optional): List of CSS class or ID substrings that indicate important content (e.g., ["article-body", "content"]). Default: None.
• negative_keywords (list, optional): List of CSS class or ID substrings to exclude (e.g., ["sidebar", "ads", "footer"]). Default: None.
• summary_sentences_qty (int, optional): Number of sentences in the generated summary. Default: 5.
• headers (dict, optional): Custom HTTP headers for the request. Useful for bypassing basic bot protection. Default: None.

Special Thanks

This project builds upon the excellent work of the following libraries:

• NLTK
• python-readability
• langid.py