Paraphrase Detector

A final-year undergraduate NLP thesis project that detects paraphrase relationships between sentence pairs by combining BERT embeddings, syntactic tree edit distance, and lexical similarity into a multi-feature logistic regression classifier.

Overview

Rather than fine-tuning a large transformer end-to-end, this project takes an interpretable feature-fusion approach: pre-trained Sentence-BERT embeddings, Zhang-Shasha tree edit distance on dependency parses, Jaccard lexical overlap, and sentence length ratio are each computed independently and combined via logistic regression. This makes each component's contribution measurable and the system's failure modes interpretable.

Results (MSRP test set, 1,725 pairs)

Metric	Fusion model	BERT alone
ROC AUC	0.790	0.775
F1	0.753	0.811
Accuracy	69.8%	—

The fusion model achieves the best ROC AUC — the more robust metric when threshold choice matters — despite BERT alone having a higher F1 at the default 0.5 threshold.

Dataset

Microsoft Research Paraphrase Corpus (MSRP)

• ~5,800 sentence pairs from online news articles
• 4,076 train / 1,725 test
• Binary labels: paraphrase (1) / non-paraphrase (0)
• Class distribution: ~67% paraphrase, ~33% non-paraphrase

Notebooks

Notebook	Stage
01_data_exploration.ipynb	Linguistic feature analysis, vocabulary statistics
02_baseline_experiments.ipynb	Ten lexical/string baseline methods
03_semantic_methods.ipynb	Word vectors, SIF weighting, BERT embeddings
04_fusion_model.ipynb	Feature extraction, Mann-Whitney tests, logistic regression, ablation
05_final_evaluation.ipynb	Held-out evaluation, ROC analysis, error analysis, threshold optimisation
06_report_figures_tables.ipynb	Thesis specific figures and tables

Features

Feature	Method	Measures
bert	Sentence-BERT (all-MiniLM-L6-v2) cosine similarity	Contextual semantic similarity
ted	Zhang-Shasha tree edit distance (normalised)	Syntactic structural similarity
jaccard	Jaccard similarity on word sets	Lexical surface overlap
len_ratio	min / max sentence length	Relative length difference

All features are statistically significant discriminators (Mann-Whitney U, p < 0.001, Benjamini-Hochberg corrected).

Setup

pip install spacy sentence-transformers scikit-learn rapidfuzz statsmodels pandas numpy matplotlib seaborn
python -m spacy download en_core_web_lg

Preprocess the raw MSRP TSV files into pickles:

python data_preprocessing/msr_data_to_pickle.py

Then run the notebooks in order (01 → 05).

Tech Stack

• NLP: spaCy (en_core_web_lg, en_core_web_trf), sentence-transformers
• ML: scikit-learn (LogisticRegression, StandardScaler)
• Stats: statsmodels (Mann-Whitney U, FDR correction)
• Baselines: RapidFuzz
• Language: Python 3.13, Jupyter Notebooks

Thesis

Full write-up: 22535753_Dowd.pdf