feat: Context Drift Visualizer using BERT

gemini_web_ideas.md

@@ -0,0 +1,50 @@
+Check out this website I built: worddrift.xyz. NOw currently its just word2vec. Are there more sophistcated tools I could use to analyze language?
+
+---
+
+It is awesome that you built worddrift.xyz using Word2Vec! Word2Vec is a classic, foundational milestone in Natural Language Processing (NLP), and building a project with it is a great way to master the basics of vector spaces, cosine similarity, and word embeddings.
+
+However, Word2Vec has a major limitation: it generates static embeddings. This means the word "bank" has the exact same vector representation whether you are talking about a "river bank" or a "savings bank."
+
+If you want to take your website to the next level, NLP has evolved significantly. Here are the more sophisticated tools and models you can use, ranked from advanced word embeddings to cutting-edge Large Language Models (LLMs).
+
+1. Contextualized Word Embeddings (The Next Logical Step)
+If you want to keep the core concept of your website (drifting or analyzing words in a vector space) but make it way smarter, you should move from static to contextual embeddings.
+
+BERT (Bidirectional Encoder Representations from Transformers): Unlike Word2Vec, BERT looks at the words before and after a target word to generate an embedding. The vector for "bank" will change dynamically depending on its context.
+
+How to use it: You can use the transformers library by Hugging Face in Python to extract the hidden states (vectors) of a specific layer of a BERT model.
+
+RoBERTa or DeBERTa: These are optimized, more robust versions of BERT that generally provide higher-quality semantic representations.
+
+2. Large Language Model (LLM) Embeddings
+If you want the highest-quality, multi-dimensional semantic mapping available today, you can use the embedding endpoints of modern LLMs. These catch deep nuances, slang, and cultural contexts that Word2Vec completely misses.
+
+OpenAI Embeddings (text-embedding-3-small or 3-large): Extremely popular, very cheap, and outputs vectors up to 3072 dimensions that capture incredibly deep semantic relationships.
+
+Cohere Embeddings: Renowned for their performance in semantic search and multi-lingual tasks.
+
+Voyage AI: Specialized embeddings that often outperform others in specific domains like code or financial data.
+
+3. Advanced NLP Frameworks (The Toolkits)
+To implement these models without building everything from scratch, you should look into these industry-standard libraries:
+
+Sentence-Transformers (Highly Recommended for your project): This is a Python framework built on top of PyTorch and Hugging Face. It is specifically designed to compute dense vector representations for sentences, paragraphs, and images. If you want users to see how entire phrases or sentences "drift" or relate to one another, this is the perfect library.
+
+spaCy: A lightning-fast, production-ready NLP library. It has built-in pipelines for named entity recognition (NER), part-of-speech tagging, and dependency parsing. It also integrates seamlessly with Hugging Face transformers.
+
+4. Vector Databases (For Scale)
+If your website grows and you start comparing thousands of words or sentences simultaneously, calculating cosine similarity using standard matrix math (like NumPy or SciPy) will slow down your website. You will need a Vector Database to handle fast, approximate nearest neighbor (ANN) searches:
+
+Chroma / LanceDB: Great, lightweight, open-source vector databases that can run locally or right inside your application.
+
+Pinecone / Milvus / Qdrant: Production-grade, hosted vector databases designed to handle millions of vectors with sub-millisecond retrieval times.
+
+How you could upgrade "WordDrift"
+Given the name of your site, here are a few cool ways you could upgrade it using these tools:
+
+Context Drift: Allow a user to type a word and two different sentences. Show them how the vector for that same word drifts in space depending on the sentence context (using BERT).
+
+Semantic Search Exploration: Let users type a sentence and map out how close it is to clusters of other topics (using Sentence-Transformers and a 2D/3D visualization tool like t-SNE or UMAP).
+
+Cross-Lingual Drift: Use a multilingual embedding model (like Cohere's or Google's) to show how the word "Apple" in English maps closely to "Manzana" in Spanish in the exact same vector space.

plans/upgrades_roadmap.md

@@ -0,0 +1,118 @@
+# WordDrift Upgrade Roadmap & GitHub Issues Plan
+
+This document outlines a plan to upgrade **WordDrift** from static Word2Vec word embeddings to advanced contextual models (BERT), sentence embeddings, and cross-lingual vector spaces.
+
+---
+
+## 📋 Architectural Overview
+
+To support these advanced features, we need to transition from a purely static, precomputed assets site to a hybrid architecture:
+1. **Interactive Frontend:** The current Next.js application.
+2. **NLP Server API:** A lightweight Python microservice (FastAPI + Hugging Face `transformers` or `sentence-transformers`) that embeds text on the fly.
+3. **Local Vector Database:** Integrations with LanceDB/Chroma for quick nearest-neighbor lookups during runtime.
+
+---
+
+## 🛠️ GitHub Issue Drafts
+
+### Issue 1: Context Drift Visualizer (BERT)
+* **Type:** Feature Enhancement
+* **Estimated Effort:** Medium
+* **Target Milestone:** v2.0-Contextual
+
+#### Title:
+`feat: Implement Target Word Context Drift Visualizer using BERT`
+
+#### Body:
+```markdown
+### Description
+Word2Vec produces static vectors where "bank" (savings) and "bank" (river) share the same representation. This issue aims to build a feature where a user can enter a target word and two distinct sentences to visualize how the target word's embedding "drifts" in space based on semantic context.
+
+### Implementation Checklist
+- [ ] **Python API Endpoint:**
+  - Create a FastAPI endpoint `/api/context-drift` that accepts:
+    - `word` (string)
+    - `context_a` (string containing `word`)
+    - `context_b` (string containing `word`)
+  - Load a lightweight transformer model (e.g., `bert-base-uncased` or `distilbert-base-uncased`).
+  - Extract the token representation for the target word from the final layer in both contexts.
+  - Return the raw dimensions (projected via PCA/t-SNE to 2D) and the cosine similarity between the two contextual states.
+- [ ] **Next.js Frontend:**
+  - Build a clean interface at `/context-drift` with form inputs.
+  - Render a visual 2D vector comparison graph (using SVG or Framer Motion) showing the displacement/drift between the two contexts.
+  - Display the cosine similarity score with dynamic color-coding (closer = bright gold, further = faded cyan).
+```
+
+---
+
+### Issue 2: Cross-Lingual Semantic Bridge Explorer
+* **Type:** Feature Enhancement
+* **Estimated Effort:** Medium
+* **Target Milestone:** v2.0-Contextual
+
+#### Title:
+`feat: Add Cross-Lingual Semantic Alignment & Drift Visualizer`
+
+#### Body:
+```markdown
+### Description
+Enable users to map how semantic concepts align or drift across languages (e.g. comparing "apple", "manzana", "apfel" in a shared multilingual vector space).
+
+### Implementation Checklist
+- [ ] **Model Selection & Precomputation:**
+  - Use a multilingual sentence/word model (e.g. `sentence-transformers/LaBSE` or `multilingual-e5-small`) to generate aligned vectors for a shared lexicon (~1,000 common concepts in English, Spanish, French, German).
+  - Project the joint vectors to a shared 2D coordinate space using UMAP.
+- [ ] **Frontend Interface:**
+  - Create a `/multilingual` route that shows a dual-galaxy or overlay view of languages.
+  - Highlighting a word in English automatically draws connection lines to its translation equivalents in other languages, showing the distance/drift (e.g., "compromise" in English might sit differently from "compromiso" in Spanish due to cultural variations).
+```
+
+---
+
+### Issue 3: Sentence Semantic Space & Search Visualizer
+* **Type:** Core Feature Upgrade
+* **Estimated Effort:** High
+* **Target Milestone:** v2.5-SentenceSpace
+
+#### Title:
+`feat: Add Sentence Space Visualizer with Live Semantic Search`
+
+#### Body:
+```markdown
+### Description
+Upgrade the landing page concept from static individual words to a 2D sentence coordinate space, letting users search/type full sentences and see them plot in real-time next to existing topic clusters.
+
+### Implementation Checklist
+- [ ] **Precompute Sentence Corpus:**
+  - Embed a dataset of 5,000–10,000 sentences (e.g., from news headlines, quotes, or commonsense questions) using `sentence-transformers/all-MiniLM-L6-v2`.
+  - Project them to 2D coordinates using UMAP and write them to a lightweight binary vector sheet.
+- [ ] **FastAPI Search Engine:**
+  - Create a `/api/embed-query` endpoint to encode a user's typed sentence on-the-fly.
+  - Perform cosine similarity against the precomputed sentence coordinates.
+- [ ] **Frontend t-SNE Canvas:**
+  - Render the sentence space.
+  - When the user searches a phrase, plot a glowing beacon indicating where their phrase landed, showing the nearest sentence matches with connections.
+```
+
+---
+
+### Issue 4: Vector Storage Infrastructure (LanceDB)
+* **Type:** Technical Debt / Performance
+* **Estimated Effort:** Low
+* **Target Milestone:** Infrastructure Upgrade
+
+#### Title:
+`refactor: Integrate LanceDB for Fast Nearest-Neighbor Vector Storage`
+
+#### Body:
+```markdown
+### Description
+Replace in-memory array search loops with a local, embedded vector database (LanceDB) to enable sub-millisecond querying as our sentence and contextual databases grow.
+
+### Implementation Checklist
+- [ ] **Database Setup:**
+  - Install and initialize `lancedb` in the Python pipeline.
+  - Load precomputed vectors (multilingual and sentence bases) directly into local LanceDB tables.
+- [ ] **Query Refactor:**
+  - Rewrite query endpoints to execute vector search directly using LanceDB indices instead of raw matrix dot products.
+```

pyproject.toml

@@ -6,6 +6,7 @@ readme = "README.md"
 requires-python = ">=3.13"
 dependencies = [
     "datasets>=4.8.5",
+    "fastapi>=0.136.3",
     "numpy>=2.4.5",
     "pandas>=3.0.3",
     "plotly>=6.7.0",
@@ -16,5 +17,7 @@ dependencies = [
     "tensorboard>=2.20.0",
     "torch>=2.12.0",
     "tqdm>=4.67.3",
+    "transformers>=5.10.2",
     "umap-learn>=0.5.12",
+    "uvicorn>=0.47.0",
 ]