HN/YouTube sidebar cleanup + auto-discover integrations from GitHub profile social links

[Copilot]

HN was dumping all items into the main feed with cluttered formatting and no pagination strategy. YouTube playlist ingestion was broken and had no channel support. No zero-config story existed — forks required manual env var setup for any integrations.

GitHub profile social links as default config source

New blog/ingestors/github_profile.py calls GET /users/{owner} + GET /users/{owner}/social_accounts at build start. When env vars are absent, social links auto-configure integrations:

GitHub social link provider	Auto-configures
hackernews	HN username (parsed from ?id= query param)
youtube	Channel handle/ID → channel upload feed

Env vars (HN_USERNAME, YOUTUBE_CHANNEL_IDS, YOUTUBE_PLAYLIST_IDS) always take precedence. Auto-discovery is noted explicitly on the config/transparency page.

HN cleanup

• Sidebar replaces main-feed dumping: 5 most recent submissions + 5 most recent comments in separate panels
• Submission items: title → article link, points, comment count, date
• Comment items: story title, direct comment link, date, one-line excerpt
• "View all" links go to submitted?id= and threads?id= — pagination deferred to HN
• _MAX_HITS_PER_TYPE reduced 100 → 20 (individual post pages still generated for full content)

YouTube

• Added YOUTUBE_CHANNEL_IDS support alongside existing YOUTUBE_PLAYLIST_IDS; accepts @handle or UCxxxxxx
• @handle resolved to channel ID by scraping the RSS <link> tag from the channel page — no API key needed
• Channel feed (?channel_id=) fetches up to 15 most recent uploads; playlists remain a separate explicit config (YOUTUBE_PLAYLIST_IDS) — auto-discovery of all playlists on a channel is not possible without the YouTube Data API
• view_more_url in each video's metadata drives "View my channel" / "View full playlist" links in sidebar

Layout

• Index: two-column grid — main column (My Writing) + sticky sidebar (HN + YouTube); collapses to stacked on mobile (≤900px)
• Container widened to 1100px

Config/transparency page

• GitHub Profile panel: avatar, bio, all linked social accounts with per-link integration status
• Content Sources table annotates auto-discovered usernames/channels inline

[HanClinto]

@copilot Investigate what it would take to add LinkedIn, Facebook, and Twitter / X as data sources from the social links present on Github. Would each of those require an API key, or is there a way to pull useful information without one?

[Copilot]

@copilot Investigate what it would take to add LinkedIn, Facebook, and Twitter / X as data sources from the social links present on Github. Would each of those require an API key, or is there a way to...

Great question! Here's a thorough breakdown of what's feasible for each platform:

---

Twitter / X

Short answer: Not realistically usable without a paid API key.

Twitter removed its free public RSS feeds years ago, and in 2023 they eliminated even the free read tier of the API. As of 2025:

• The official X API v2 requires payment — the cheapest plan with meaningful read access is ~$100/month.
• Scraping alternatives exist (cookie-based auth via Playwright, or unofficial GraphQL endpoint clients) but they require the user to supply a valid browser auth_token cookie, which means they need a logged-in account. This is fragile, violates ToS, and isn't suitable for a zero-config static site build.
• Third-party RSS bridge services (RSS.app, etc.) exist but are themselves paid services — they don't eliminate the dependency, just outsource it.

Verdict: ❌ Not viable without a paid API key or a logged-in cookie. The twitter_username is already extracted from the GitHub profile and shown on the config page — the best we could do today is link to the user's profile page.

---

LinkedIn

Short answer: API key required for anything useful; scraping is fragile and against ToS.

• LinkedIn eliminated public RSS feeds years ago.
• The official LinkedIn API requires going through their partner program review (even for basic post reads), which is extremely difficult to get approved for outside of enterprise/integration partners.
• Scraping without login is severely limited — LinkedIn aggressively blocks bots and shows login walls after a few requests.
• The hiQ v. LinkedIn court case did confirm that scraping publicly-visible content is legal in the US, but the technical barriers make it impractical for a reliable CI-time build step.
• No stable, key-free, production-grade feed mechanism exists.

Verdict: ❌ Not viable without either an API key (hard to get) or fragile scraping that will break regularly.

---

Facebook

Short answer: No official key-free path; third-party scrapers exist but are unreliable.

• Facebook removed public page RSS feeds in 2015 and the Graph API now requires app review even for basic public page reads.
• The facebook-scraper Python library works on truly public pages without a key, but Facebook actively combats this — it breaks regularly as Facebook updates its front-end.
• There are paid services (RSS.app, FetchRSS) that generate feeds from public page URLs, but those aren't something this project can call at build time without introducing a paid dependency.

Verdict: ❌ Not reliably viable without a paid third-party service or an app-reviewed Graph API token.

---

Comparison Table

Platform	Official API	Key-Free Feed?	Scraping Feasibility	Recommendation
Twitter / X	Paid ~$100/mo	❌ None	Fragile (cookie auth)	Show profile link only
LinkedIn	Partner approval required	❌ None	Very fragile, ToS risk	Show profile link only
Facebook	App review required	❌ None	Fragile, breaks often	Show profile link only
YouTube ✅	Optional API key	✅ Public Atom/RSS feeds	N/A	Already implemented
Hacker News ✅	None needed	✅ Algolia search API	N/A	Already implemented

---

Practical Path Forward

All three platforms are already discovered from the GitHub profile social links (extract_twitter_url, extract_linkedin_url already exist in github_profile.py). The most honest thing the blog can do is:

1. Display the discovered profiles on the config/transparency page (link to Twitter/X, LinkedIn, Facebook profiles) — so visitors know where to find you.
2. Not attempt to ingest posts from any of these, since there's no reliable, API-key-free, ToS-compliant way to do so.

This keeps SimpleGitBlog's core value proposition intact: zero-config, no API keys required, no fragile scrapers. If you (or a fork owner) want to add Twitter embeds or LinkedIn post cards in the future, the cleanest path would be to add an optional env var (e.g., TWITTER_BEARER_TOKEN) that enables ingestion only when a key is explicitly provided.