This repository presents a complete deep learning pipeline for detecting extremist content in social media text.
The task is formulated as a binary classification problem distinguishing between extremist and non-extremist messages.
The solution was developed for a Kaggle competition on digital extremism detection and achieved a Top-10 position on the public leaderboard.
The approach focuses on contextual understanding, robustness to noisy language, and stable generalization rather than keyword-based heuristics.
- Introduction
- Problem Definition
- Challenges in Extremism Detection
- Methodology
- Model Architecture
- Text Preprocessing
- Training Strategy
- Inference Pipeline
- Experimental Results
- Repository Structure
- Setup and Usage
- Limitations
- Conclusion
Social media platforms generate large volumes of user-created content, some of which may contain extremist ideology, propaganda, or incitement.
Automatic detection of such content is critical but difficult due to language ambiguity, context dependence, and adversarial writing styles.
This project explores a transformer-based solution designed to address these challenges using modern regularization and training techniques.
Given a short text message from a social media platform, the goal is to classify it into one of two categories:
- EXTREMIST
- NON_EXTREMIST
The model must rely on contextual meaning rather than isolated keywords.
Extremist terms frequently appear in:
- News reporting
- Academic analysis
- Condemnatory statements
Correct classification requires understanding intent, not vocabulary alone.
Many words have multiple meanings depending on:
- Cultural context
- Political discourse
- Temporal trends
Social media text often includes:
- Repeated characters
- Misspellings
- Slang and abbreviations
- Excessive punctuation
These patterns are sometimes intentionally used to bypass automated moderation.
Some samples are inherently ambiguous, leading to:
- Annotation noise
- Model overconfidence
- Reduced generalization
The solution is built around a large pretrained language model combined with several stability-focused enhancements:
- Contextual transformer representations
- Layer-wise feature aggregation
- Implicit ensembling via dropout
- Adversarial regularization
- Confidence smoothing during optimization
The goal is not only high accuracy but robust decision boundaries.
- Python (3.9+)
- PyTorch (2.x)
- Hugging Face Transformers
- DeBERTa-v3-base
- Weighted Layer Pooling
- Multi-Sample Dropout
- Adversarial Weight Perturbation (AWP)
- AdamW Optimizer
- Cosine Learning Rate Scheduler
- Gradient Accumulation
- Label Smoothing
- Pandas
- NumPy
- Custom text normalization pipeline
- NVIDIA GPU (single-GPU training)
- Optimized for low VRAM via accumulation
- Model:
microsoft/deberta-v3-base - Hidden size: 1024
- Full hidden-state outputs enabled
The final representation is obtained by:
- Extracting the
[CLS]token from the last four transformer layers - Averaging them to reduce layer-specific noise
- Five dropout layers with increasing dropout probabilities
- The classifier is applied multiple times
- Logits are averaged to form the final prediction
This improves robustness and reduces variance.
- Small gradient-aligned perturbations applied to model weights
- Activated after initial convergence
- Encourages flatter loss landscapes and improved generalization
A controlled cleaning strategy is applied:
- Lowercasing
- Removal of HTML-like tags
- Normalization of repeated characters
- Removal of excessive punctuation while preserving
?and! - Whitespace normalization
The intent is to reduce noise without destroying semantic signals.
- Optimizer: AdamW
- Learning rate: 1e-5
- Weight decay: 0.01
- Cosine learning rate schedule
- Warmup over the first 10% of training steps
- Label smoothing (0.05)
- Multi-sample dropout
- Adversarial weight perturbation
- Small batch size with gradient accumulation
- Enables training on GPUs with limited VRAM
Inference is separated from training and submission logic.
A CSV file containing:
Original_Message
A CSV file with an additional column:
Prediction(EXTREMISTorNON_EXTREMIST)
The inference script performs preprocessing, tokenization, and classification in evaluation mode.
- Achieved Top-10 placement in a Kaggle competition
- Stable validation performance across folds
- Improved robustness to noisy and adversarial samples
- Reduced overconfidence on borderline cases
.
├── train.py # Training pipeline
├── inference.py # Standalone inference script
├── model.py # Model definition
├── requirements.txt # Dependencies
└── README.md
python -m venv venv
source venv/bin/activate # Linux / macOS
venv\Scripts\activate # Windows
pip install -r requirements.txt👉The model may struggle with extremely short or context-free messages 👉Cultural and regional slang not present in training data can affect accuracy 👉Binary classification does not capture degrees or categories of extremism
This project demonstrates that robust extremism detection requires more than keyword filtering. Contextual modeling, regularization, and disciplined training strategies are essential for handling noisy, real-world social media text.
The results validate the effectiveness of combining large pretrained transformers with stability-focused enhancements.