Fake or Real: The Imposter Hunt in Text
Abstract
This project, part of the Fake or Real: The Imposter Hunt in Text Kaggle competition, focused on identifying authentic news articles from pairs consisting of one real and one synthetically altered version. Using transformer-based models like BERT, SBERT, and Longformer, the approach generated deep embeddings for the texts, which were then converted into interaction-based features such as vector differences and element-wise products. These features were used to train various classical machine learning classifiers. The Random Forest model achieved the highest performance with a 93.6% cross-validation accuracy and a leaderboard score of 0.87, demonstrating the effectiveness of embedding-based semantic comparison in detecting AI-generated misinformation.
Problem statement
The increasing use of artificial intelligence to generate or modify news content has made it difficult to distinguish between real and fabricated information. Subtle manipulations in wording, tone, or structure can make fake texts appear authentic, leading to misinformation and reduced public trust in media sources. The competition addresses this by providing pairs of news articles (one real and one synthetically altered) and tasks participants with identifying which is genuine.
Objectives
- Develop an automated system capable of detecting semantic and contextual differences between real and fake news articles.
- Leverage transformer-based models to extract deep embeddings for texts.
- Generate interaction features to capture relationships between text embeddings.
- Train machine learning models to accurately classify the real document in each pair.
Methodology
The methodology involved an embedding-based paradigm. Transformer-based models, including BERT, SBERT, and Longformer, were used to process raw text and generate contextual embeddings. To handle long texts exceeding the 512-token limit, a custom 'extract_mean_pooling_vector()' function was implemented, dividing input text into overlapping chunks via a sliding window technique, encoding each chunk, and averaging the results into a single fixed-length vector. No aggressive text cleaning was performed. Interaction-based features were then derived from these embeddings: vector difference, element-wise product, cosine similarity, Euclidean distance, Manhattan distance, and dot product. These features served as inputs for classical classifiers, including Random Forest, XGBoost, CatBoost, SVM, and Logistic Regression. Training was performed using a stratified 5-fold cross-validation split.
Results
The best results were achieved using BERT embeddings combined with two interaction features: vector difference and element-wise product. The Random Forest classifier emerged as the best-performing model, achieving a cross-validation accuracy of 93.6% and an F1-score of 0.94404. Performance metrics for other models included: CatBoost (90.5% accuracy), Logistic Regression (90.5% accuracy), XGBoost (89.5% accuracy), and SVM (87.4% accuracy). The final model achieved a public leaderboard score of 0.87.
Conclusion
The project demonstrated that transformer-based embeddings combined with classical machine learning models can effectively detect fake news. By analyzing paired documents through interaction features that capture semantic relationships, the study achieved high performance. The results indicate that embedding-based features, even without fine-tuning, provide a strong and efficient approach for identifying manipulated texts and combating misinformation.