Introduction to Transformers - IOAI 2025

This repository contains materials for the workshop designed for participants of the International Olympiad in Artificial Intelligence (IOAI).

🕒 Workshop Structure

Part 1: Transformer Architecture Deep Dive

• Learn the building blocks of Transformer models, including embeddings, encoders, and position encodings.
• Understand how input sequences are processed through the model and how representations evolve.

Part 2: Attention Mechanism and Multi-Head Attention

• Inspect attention weights and hidden states across layers and heads.

Part 3: Embeddings and Weight Manipulation

• Restore model performance under embedding corruption constraints in a zero-shot setting. Inspired by NEOAI 2025 Task 5, this challenge requires recovering sentiment classification accuracy for a "Broken BERT" model without retraining or external resources.

Part 4: Cross-Lingual Adaptation

• Develop systems for zero-shot slot filling and intent detection in a cross-lingual setting using English training data and Romanian evaluation. Inspired by NEOAI 2025 Task 8, under strict constraints (no translation, no external models).

📚 Notebooks

Before starting the workshop, make a personal copy of the notebook. You will work on your own version during the session. To do so, go to the File menu in Colab and select Save a copy in Drive.

Resource	Description	Time
Main Notebook	Complete workshop content	~4 hours
Exercises	Hands-on challenges	Coming soon
Solutions	Detailed solution guides	Coming soon

⚙️ Local Development Setup

Coming soon...

1. Clone the repository:

   # Placeholder setup
   git clone https://github.com/YOUR_ORG/transformers-ioai-2025.git
   cd transformers-ioai-2025
   # TODO: Add virtualenv/poetry/conda instructions

The notebooks are tested with Python 3.10+.

🛠️ Contributing

Everyone is welcome to contribute! Whether it's fixing a typo, improving explanations, adding new insights, or suggesting new tasks.

Quick contribution: See CONTRIBUTING.md

Ways to Help:

• 🐛 Report Issues: Found something unclear or broken?
• 📝 Improve Content: Better explanations or examples
• 🎯 Add Exercises: Create new challenges
• 📖 Documentation: Setup guides, troubleshooting tips

🔗 References

• Vaswani et al., 2023 – "Attention is All You Need"

• Hugging Face – LLM Course

• UVADLC – Tutorial 6: Transformers and MH Attention

• Northern Eurasia OAI 2025 Kaggle Competition

📖 Additional Resources

• Jay Alammar – The Illustrated Transformer

• Hugging Face Transformers Notebooks

🏷️ Repository Structure

ioai-transformer-workshop/
├── notebooks/
│   └── IOAI_Transformer_Workshop.ipynb  # Main workshop content
├── src/                                  # Utility functions
├── data/                                # Sample datasets
├── docs/                                # Setup and troubleshooting
└── assets/                              # Images and diagrams

---