Add a daily article publisher action for the blog

.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: ci
+
+on:
+  push:
+    branches: [main, dev_asfa]
+  pull_request:
+    branches: [main]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install dependencies
+        run: pip install -r requirements-dev.txt
+      - name: Run tests
+        run: bash test.sh

.github/workflows/publish-article.yml

@@ -0,0 +1,62 @@
+name: publish-article
+
+on:
+  schedule:
+    - cron: "23 7 * * *"   # daily at 07:23 UTC
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Render the article and log it without opening a pull request"
+        required: false
+        type: boolean
+        default: false
+      topic_slug:
+        description: "Force a specific topic slug (optional)"
+        required: false
+        type: string
+        default: ""
+
+concurrency:
+  group: vmx_blog_publish
+  cancel-in-progress: false
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Generate the article
+        id: gen
+        env:
+          BRIGHT_DATA_API_KEY: ${{ secrets.BRIGHT_DATA_API_KEY }}
+          SERP_API_KEY: ${{ secrets.SERP_API_KEY }}
+          TOPIC_SLUG: ${{ inputs.topic_slug }}
+          DRY_RUN: ${{ inputs.dry_run && '1' || '0' }}
+        run: python run_publish.py
+      - name: Open a pull request
+        if: ${{ steps.gen.outputs.created == 'true' }}
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BRANCH: ${{ steps.gen.outputs.branch }}
+          TITLE: ${{ steps.gen.outputs.title }}
+        run: |
+          set -euo pipefail
+          git config user.name "vmodal-blog-bot"
+          git config user.email "vmodal-blog-bot@users.noreply.github.com"
+          git checkout -b "$BRANCH"
+          git add articles index.html
+          git commit -m "Add article: $TITLE"
+          git push --force-with-lease origin "$BRANCH"
+          gh pr create --base main --head "$BRANCH" \
+            --title "New blog article: $TITLE" \
+            --body "Automated draft of a new blog article for review. Merge to publish it to the site, or close to skip it."

.gitignore

@@ -0,0 +1,5 @@
+.env
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/

publisher/README.md

@@ -0,0 +1,59 @@
+# Daily article publisher
+
+A scheduled GitHub Action that adds a new article to the blog each day. It picks a
+topic, generates the article text, renders it into the site's HTML template, writes it
+into `articles/`, links it from `index.html`, and opens a pull request with the change
+for review. Nothing goes live until that pull request is merged.
+
+## How it works
+
+1. `run_publish.py` selects a topic from `publisher/topics.py`. It prefers a topic that
+   has not been published yet, then rotates through the list, so articles stay varied.
+2. It generates the body text, preferring Bright Data's Google AI Mode, falling back to
+   SerpApi's Google AI Overview, and finally to the topic's own self-contained body, so
+   a well-formed article is produced even before any key is configured.
+3. It renders a standalone page under `articles/YYYYMMDD-<slug>.html` using the same
+   template as the existing articles and adds a list item to the top of `index.html`.
+4. The workflow commits the change on a new branch and opens a pull request. A run is
+   idempotent per day: if today's article already exists, it does nothing.
+
+## Configuration
+
+Set these as repository secrets (**Settings → Secrets and variables → Actions**).
+Both are optional; with neither set, the publisher uses each topic's built-in body.
+
+| Secret | Purpose |
+| --- | --- |
+| `BRIGHT_DATA_API_KEY` | Bright Data Google AI Mode content (preferred) |
+| `SERP_API_KEY` | SerpApi Google AI Overview content (fallback) |
+
+For the workflow to open pull requests, **Settings → Actions → General → Workflow
+permissions** must allow GitHub Actions to create and approve pull requests.
+
+## Running it manually
+
+From the **Actions** tab, pick **publish-article** and **Run workflow**. Tick the
+dry-run box to render and log an article without opening a pull request, or set a topic
+slug to force a specific topic.
+
+From the command line:
+
+```bash
+gh workflow run publish-article.yml                       # real run, opens a PR
+gh workflow run publish-article.yml -f dry_run=true       # render and log only
+gh workflow run publish-article.yml -f topic_slug=edge-inference
+```
+
+## Running locally
+
+```bash
+pip install -r requirements-dev.txt
+bash test.sh                 # offline tests
+DRY_RUN=1 python run_publish.py   # render an article and log it, write nothing
+```
+
+## Adding or editing topics
+
+Topics live in `publisher/topics.py`. Each has a stable slug, a title, tags, a one-line
+summary, the search prompt sent to the AI source, and a self-contained fallback body.
+Add an entry to the `TOPICS` list to expand the rotation.

publisher/__init__.py

@@ -0,0 +1,7 @@
+"""Daily article publisher for the V-Modal AI Blog.
+
+Picks a topic, generates the article text from Google's AI answer (Bright Data's
+Google AI Mode, with SerpApi's Google AI Overview as a fallback, and a self-contained
+fallback when neither is configured), renders it into the site's HTML template, writes
+it into `articles/`, and links it from `index.html`.
+"""

publisher/config.py

@@ -0,0 +1,63 @@
+"""Runtime configuration for the article publisher.
+
+Values come from environment variables. In Actions they are provided as repository
+secrets; for local runs a .env file (gitignored) is loaded here if present. Nothing in
+this module ever writes secrets to disk or logs them.
+"""
+import os
+from dataclasses import dataclass
+
+
+def load_dotenv(path=".env"):
+    """Load KEY=VALUE lines from a .env file into os.environ (does not overwrite
+    variables that are already set). Missing file is a no-op. No dependency needed."""
+    if not os.path.isfile(path):
+        return
+    with open(path, "r", encoding="utf-8") as fh:
+        for raw in fh:
+            line = raw.strip()
+            if not line or line.startswith("#") or "=" not in line:
+                continue
+            key, _, value = line.partition("=")
+            key = key.strip()
+            value = value.strip().strip('"').strip("'")
+            if key and key not in os.environ:
+                os.environ[key] = value
+
+
+@dataclass
+class Config:
+    brightdata_api_key: str = ""
+    serpapi_key: str = ""
+    articles_dir: str = "articles"
+    index_path: str = "index.html"
+    # Optional: force a specific topic by slug instead of the automatic rotation.
+    topic_slug: str = ""
+    dry_run: bool = False
+
+    @property
+    def brightdata_enabled(self) -> bool:
+        return bool(self.brightdata_api_key)
+
+    @property
+    def serpapi_enabled(self) -> bool:
+        return bool(self.serpapi_key)
+
+
+def _truthy(value: str) -> bool:
+    return str(value or "").strip().lower() in {"1", "true", "yes", "on"}
+
+
+def load_config(env=None) -> Config:
+    """Build a Config from the environment (after loading .env for local runs)."""
+    if env is None:
+        load_dotenv()
+        env = os.environ
+    return Config(
+        brightdata_api_key=env.get("BRIGHT_DATA_API_KEY", "").strip(),
+        serpapi_key=env.get("SERP_API_KEY", "").strip(),
+        articles_dir=(env.get("ARTICLES_DIR") or "articles").strip(),
+        index_path=(env.get("INDEX_PATH") or "index.html").strip(),
+        topic_slug=env.get("TOPIC_SLUG", "").strip(),
+        dry_run=_truthy(env.get("DRY_RUN")),
+    )

publisher/content.py

@@ -0,0 +1,126 @@
+"""Generate the article text for a topic.
+
+Content generation prefers Bright Data's Google AI Mode API when configured, and
+otherwise falls back to SerpApi's Google AI Overview (a two-step call: a Google search
+yields an ai_overview, either inline or via a short-lived page_token that is then
+fetched from the AI-overview engine). Google only returns an AI answer for some
+queries, so when there is none (or neither provider is configured) we fall back to the
+topic's own self-contained body so the publisher always produces a complete article.
+"""
+from typing import Any, Dict
+
+import requests
+
+SERPAPI_SEARCH = "https://serpapi.com/search"
+
+# Bright Data Google AI Mode dataset (synchronous scrape endpoint).
+BRIGHTDATA_SCRAPE = "https://api.brightdata.com/datasets/v3/scrape"
+BRIGHTDATA_AIMODE_DATASET = "gd_mcswdt6z2elth3zqr2"
+# Keys most likely to hold the AI answer in Bright Data's response, in priority order.
+_AIMODE_TEXT_KEYS = (
+    "answer_text", "answer", "ai_overview", "overview",
+    "markdown", "text", "content", "response", "result",
+)
+
+
+def _overview_text(ai_overview: Dict[str, Any]) -> str:
+    """Flatten SerpApi ai_overview.text_blocks into plain text (markdown-ish)."""
+    parts = []
+    for block in ai_overview.get("text_blocks") or []:
+        snippet = str(block.get("snippet") or "").strip()
+        if snippet:
+            parts.append(snippet)
+        for item in block.get("list") or []:
+            s = str(item.get("snippet") or "").strip()
+            if s:
+                parts.append(f"- {s}")
+    return "\n".join(parts).strip()
+
+
+def fetch_ai_overview(query: str, api_key: str, get=requests.get) -> str:
+    """Return Google's AI Overview text for `query`, or "" when there is none."""
+    if not api_key:
+        return ""
+    resp = get(SERPAPI_SEARCH, params={"engine": "google", "q": query, "api_key": api_key}, timeout=30)
+    resp.raise_for_status()
+    data = resp.json() or {}
+    ai = data.get("ai_overview") or {}
+    text = _overview_text(ai)
+    if text:
+        return text
+    token = str(ai.get("page_token") or "").strip()
+    if not token:
+        return ""
+    resp2 = get(
+        SERPAPI_SEARCH,
+        params={"engine": "google_ai_overview", "page_token": token, "api_key": api_key},
+        timeout=30,
+    )
+    resp2.raise_for_status()
+    data2 = resp2.json() or {}
+    return _overview_text(data2.get("ai_overview") or {})
+
+
+def _extract_aimode_text(data) -> str:
+    """Best-effort pull of the readable AI answer out of Bright Data's response.
+
+    The scrape endpoint returns a list of result objects; the exact schema for the
+    Google AI Mode dataset varies, so walk it and take the first non-empty string under
+    a known answer key (recursing into nested dicts/lists)."""
+    items = data if isinstance(data, list) else [data]
+    for item in items:
+        if isinstance(item, str):
+            if item.strip():
+                return item.strip()
+            continue
+        if not isinstance(item, dict):
+            continue
+        for key in _AIMODE_TEXT_KEYS:
+            val = item.get(key)
+            if isinstance(val, str) and val.strip():
+                return val.strip()
+            if isinstance(val, (list, dict)):
+                nested = _extract_aimode_text(val if isinstance(val, list) else [val])
+                if nested:
+                    return nested
+    return ""
+
+
+def fetch_ai_mode(query: str, api_key: str, post=requests.post) -> str:
+    """Return Google AI Mode text for `query` via Bright Data, or "" when unavailable.
+    `post` is injectable for tests."""
+    if not api_key:
+        return ""
+    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
+    payload = {
+        "input": [{"url": "https://google.com/aimode", "prompt": query, "hl": "en", "country": ""}],
+        "limit_per_input": None,
+    }
+    resp = post(
+        BRIGHTDATA_SCRAPE,
+        params={"dataset_id": BRIGHTDATA_AIMODE_DATASET, "notify": "false", "include_errors": "true"},
+        headers=headers, json=payload, timeout=60,
+    )
+    resp.raise_for_status()
+    return _extract_aimode_text(resp.json())
+
+
+def generate_article_body(topic, config, get=requests.get, post=requests.post) -> str:
+    """Return the article body (markdown-ish text) for a topic.
+
+    Uses the Google AI answer when available (Bright Data preferred, SerpApi as a
+    fallback), otherwise the topic's own self-contained body. Never raises on a
+    missing or failing provider."""
+    query = topic.prompt
+    overview = ""
+    if getattr(config, "brightdata_enabled", False):
+        try:
+            overview = fetch_ai_mode(query, config.brightdata_api_key, post=post)
+        except Exception:
+            overview = ""
+    if not overview and getattr(config, "serpapi_enabled", False):
+        try:
+            overview = fetch_ai_overview(query, config.serpapi_key, get=get)
+        except Exception:
+            overview = ""
+    return overview.strip() if overview else topic.fallback_body()

publisher/index_update.py

@@ -0,0 +1,30 @@
+"""Insert a new article's list item into index.html.
+
+The homepage lists articles inside `<ul class="article-list">`. A new article is
+added as the first item so the newest is shown at the top, leaving the rest of the
+page untouched.
+"""
+
+LIST_OPEN = '<ul class="article-list">'
+
+
+class IndexFormatError(ValueError):
+    """Raised when index.html does not contain the expected article list."""
+
+
+def insert_article(index_html: str, li_html: str, href: str = "") -> str:
+    """Return index_html with li_html inserted as the first list item.
+
+    If `href` is given and already present in the page, the html is returned
+    unchanged (idempotent — a re-run does not duplicate the entry)."""
+    if href and f'"{href}"' in index_html:
+        return index_html
+    pos = index_html.find(LIST_OPEN)
+    if pos == -1:
+        raise IndexFormatError(f"could not find {LIST_OPEN!r} in index.html")
+    # Insert right after the end of the line that opens the list.
+    line_end = index_html.find("\n", pos)
+    if line_end == -1:
+        raise IndexFormatError("malformed article list in index.html")
+    insert_at = line_end + 1
+    return index_html[:insert_at] + li_html + "\n" + index_html[insert_at:]