Your Training Data is Boring Anomaly Detection with LLMs
Your Training Data is Boring: Anomaly Detection with LLMs
🚀 Following on from research presented at Black Hat USA 2025, Sophos data scientists Ben Gelman, Tamas Nyiri, and Tibor Kristóf Lányi recently showcased new work on anomaly detection at BSides Las Vegas 2026. Supervised classifiers in cybersecurity are often trained on data that doesn’t capture the most unusual examples. Benign training sets are dominated by simple, common patterns, while malicious sets reflect known attacks from past investigations. The long tail of unusual files on both sides goes unlabeled. When these files appear in production, classifiers are more likely to misclassify them, generating false positives that overwhelm SOCs, and false negatives that let threats through. Every detection team is familiar with the core issue: supervised classifiers are only as good as their training data, and that data is systematically biased. The events that classifiers get wrong in production are precisely the unusual events missing from both sides of the training set.
But anomaly detection combined with LLMs is excellent at finding and labeling the unusual data that is missing from cybersecurity training sets. The team developed an automated pipeline that combines anomaly detection and LLMs to augment training data for a suite of cybersecurity models, using complementary anomaly detection methods, viable across feature representations, to surface distinctly unusual data. Then, an LLM classified each anomaly with format-specific prompts calibrated per data type. A key architectural decision was to treat anomaly detection not as the detection system itself, but as a prefilter that surfaces candidates for LLM labeling, which in turn produces training data for a supervised classifier whose precision can be controlled empirically. Because each anomaly detection method has blind spots - it catches certain types of unusual data and misses others - Sophos AI used three methods (isolation forest, k-means anomaly, and PCA reconstruction error) on different feature representations, producing broader coverage with low overlap between methods.
Critically, Ben and his colleagues used separate prompts that err toward malicious and benign respectively, achieving high precision on both label types. More specifically, one prompt instructed the LLM to only evaluate the input as benign when it was confident that it was benign; the other prompt instructed the same but with the malicious label. This way, they only accepted the labels when the two LLMs with the two different instructions agreed, otherwise they threw that example away and didn’t use it for training. In practice, this method produced labels with better precision (tested on labeled data). This allows both label types from anomaly detection to be usable as training data, which was not achievable in prior work that relied on a single labeling pass.
The Sophos AI team evaluated their method across three structurally different data types: command lines (~10,000,000 monthly ingest), HTML files (~1,000,000 monthly ingest), and RTF documents (~10,000 monthly ingest (testing under data scarcity)). RTF performed best with isolation forest, showing a 2.41% increase over the existing model. With HTML, K-means was the most effective with a 0.68% increase over baseline. They noted that PCA decreased the score, highlighting why three different anomaly detection methods were used. Classifiers’ failure cases are a training data diversity problem; anomaly detection finds the unusual files that rules and manual labeling miss. Multi-method anomaly detection (IF + KMeans + PCA) on different feature representations catches more than any single method, and dual-prompt LLM labeling achieves high precision on both malicious and benign labels, making both usable as training data.