📉 LLM Semantic Drift Analysis ("The Telephone Game")

A scientific experiment to measure how meaning degrades ("drifts") when information is passed sequentially through a chain of different Large Language Models.

🧪 The Experiment

Just like the children's game of "Telephone," this project feeds the output of one AI model (e.g., GPT-4o) as the only input for the next model (e.g., Claude 3). We measure the degradation of the core concept across 6 steps, including local models via Ollama.

The Chain: GPT-4o → Claude 3.5 Sonnet → Gemini 1.5 → DeepSeek → Mixtral → Llama 3 (Local)

📊 Key Features

• 🔄 Universal Wrapper: A single Python function to handle API calls for OpenAI, Anthropic, Google, and Ollama.
• 🛡️ Strict System Prompts: Ensures models act as "repeaters" rather than conversational assistants.
• 📐 Hybrid Evaluation:
  • Quantitative: Cosine similarity scoring using SentenceTransformers embeddings.
  • Qualitative: GPT-4o acts as a "Judge" to score Concept Mutation and Hallucination.
• 📈 Visualization: Matplotlib charts correlating embedding distance with idea survival.

🚀 Quick Start

1. Clone the repo

git clone https://github.com/Sama-ndari/llm-semantic-drift-analysis.git

2. Install dependencies

pip install -r requirements.txt

3. Setup Environment Create a .env file with your keys:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
DEEPSEEK_API_KEY=...
GROQ_API_KEY=...

4. Run the Notebook Launch telephone_game.ipynb and run all cells.

📈 Sample Results

"Significant semantic drift was observed at Step 4 (DeepSeek), where the specific academic context was replaced by generalized advice."

🛠️ Tech Stack

• Languages: Python
• Models: GPT-4o, Claude 3.5, Gemini 1.5 Pro, Deepseek, Groq, Llama 3:8b
• Libraries: openai, anthropic, sentence-transformers, scikit-learn, pandas, matplotlib

---

Created by Sama-ndari