When in Doubt, Chase Breadcrumbs

Something was still bothering me about the training dynamics despite achieving reasonable performance metrics on the 24th. The validation curves were still persistently unstable. My initial assumption was that we needed better hyperparameter tuning and/or a better architecture, but the sporadic nature of the validation curves suggested something far worse.

Huh...

This is a single EEG recording from the TUH dataset. The initial seconds look... strange. They do not at all match the activity that follows after. We had no knowledge of this at the time, but this image would lead us to a dire realization about the TUH dataset, a problem that could only be revealed by asking the model: *"Where are you failing? and why?"

Building the Data Inspection Pipeline

I decided to investigate what the model was actually learning. If the training was unstable, then maybe the model's mistakes could tell us why. This led me to develop a set of error analysis scripts that quickly became our primary data auditing tools.

Example output of the analysis script

inspect_data.py's purpose was to provide a visual analysis of a model's performance on individual data samples. Given an MLFlow run_id, it was made to operate in three modes:

• Report Mode: Finds the most confidently wrong predictions
• Confidence Mode: Find the predictions the model is least sure about
• File Mode: Analyze the model's predictions on every single epoch from any given source .edf file (from the test set)

At the end of the training pipeline, a detailed error report that documents every decision the model makes on the test set, alongside the confidence for each decision. An accompanying summary file is also generated, which clued me into the suspicious subject.

Truncated output of the summary file

Subject aaaaajrf stood out dramatically with a 64.9% error rate, contributing 3,158 errors out of 4,867 total mistakes across the entire test set... something was very wrong.

From the Optuna runs, around 20 models were generated at this point. The analysis script on its own was quite useless, since you need to know which model to use it with. Sifting through the models manually was going to be a nightmare.

Finding the Best Detective

The second script, rank_models_for_labeling.py flips the problem on its head. IF we have models that are good at finding artifacts, can we use them to find mislabeled data in the test set? Can we find "clean" epochs that a model confidently labels as artifacts?

Truncated output of rank_models_for_labeling.py

This script evaluated every single model checkpoint against a set of criteria:

1. Minimum Knowledge (Balanced Accuracy > 75%): Balanced accuracy matters a lot here because it accounts for imbalance, averaging the recall for each class. A model can get high regular accuracy by simply always predicting the majority class, but it will have a low balanced accuracy.

2. Artifact Recall > 85%: This determines that the model correctly identifies at least 85% of all known artifacts. Our goal is to find unlabeled artifacts, so the primary skill of a model is to be sensitive to such.

3. Max False Positives < 8000: Filters out any model that generates a VERY high number of false positives. This might seem counter intuitive, but do recall how a model could achieve 100% artifact recall by just calling every epoch an artifact.

For the record, these were guided by inspecting the summary files of about 3 of the best and worst models based on their F1-score.

The models that pass these filters are then ranked by the average confidence of their false positive predictions. The higher the confidence, the more likely the model is genuinely "seeing" an artifact pattern in data that was labeled as clean.

The Visual Investigation

Naturally, I began by running the inspect data script on the best performing model from an Optuna hyperparameter search (trial_8). A very disturbing pattern quickly became abundantly clear.

Strange pattern at the beginning of recordings

The model was consistently identifying patterns that were labeled as "clean" in the human labeled data. Our data inspection script showed us something we did not expect to find: systematic problems at the beginning of recordings. These appear to be some kind of calibration or settling process at the beginning of almost every single recording. The eerie frightening part is that these were labeled with utter inconsistency, and it sort of makes sense. In the TUH dataset, these are sometimes labeled as "elec" artifacts, meaning electrode pop/moving. Any human examiner, given the settings of the trial, can easily recognize that this is not genuine brain data, and they may or may not choose to label it as such. A model, however, is not human.

You may say "okay, just trim off the first couple secs. simple, right?" we thought so too. It is in fact, NOT that simple.

Our model, trained on thousands of examples of real artifacts, was completely thrown off by this unseen, weirdly patterned data. It was misclassifying them left and right, in turn tanking its overall metrics.

Clear "elec" artifact being mislabeled

Now that I knew the data had labeling issues, I looked further. Above is a typical "elec" artifact as it appears throughout the dataset. TThis particular patient was completely missing labels past a certain point in the recording, almost like the experts got bored...

Surely it was just this patient... right? You goddamn wish
I took a brief look at the other high error rate subjects. The pattern still held. Subjects with the highest error rates consistently showed labeling problems rather than genuinely difficult cases for the most part. Some recordings had entire sections where clear artifacts went unlabeled.

The hopecore Experiment

Patient aaaaajrf was clearly not representative of the problem. I therefore made the decision to fully remove all epochs from this subject, as relabeling would have been a hassle at such an early stage. The results speak for themselves:

EEGNet Hopecore

The new model achieved 85.4% accuracy, up from 78.4%. More importantly, the F1-score for artifacts jumped from 0.79 to 0.85. The precision and recall were far more balanced, and the number of high confidence incorrect predictions dropped significantly. By simply removing ~5,000 bad data points (out of ~38,000), we achieved a massive jump in performance.

This run also featured a simple "fix" in place in the preprocessing pipeline to skip the first 3 seconds of every recording:

The Broader Implication

We were quite happy with our results, but once you realize how broadly this dataset is trusted and used, all excitement quickly fades.

[The TUH dataset] presents the world's largest publicly available corpus of clinical EEG data, representing a grand total of 29.1 years (total duration summed over all EEG channels) of EEG data.

- frontiersin

The TUH dataset is arguably the most important and widely used public resource in modern EEG research. The original paper introducing the corpus has been cited over 350 times at the time of writing. It is the de facto benchmark for a large array of applications, including seizures, BCI research, sleep data scoring, and validating new signal processing techniques.

If our relatively simple CNN could identify labeling errors in what is considered the most well-known EEG dataset, it raises serious questions about the reliability of published results using the TUH EEG corpus.

How many published, state-of-the-art results are influenced by these data quality issues? Are we, as a research community, building models that are excellent at identifying the specific quirks of the TUH dataset?

The performance of a model heavily depends on the luck of the draw. What if aaaaajrf had never ended up in our test set? We would have continued optimizing a model on a completely flawed foundation. At the end of the day, it's garbage in, garbage out. We suspect there exists many instances of this scattered in the dataset, likely already contaminating our validation and training sets.

The ultimate goal of this research is to build tools that can be used in serious clinical settings to help patients. A model trained on this data should absolutely not be trusted. Imagine a seizure detection model trained on this dataset. If it learns that "highly rhythmic, high-amplitude signals" are simply calibration artifacts to be ignored, it could ABYSMALLY fail to detect a real, rhythmic tonic clonic seizure in a clinical setting.

The field NEEDS to start valuing work that identifies and documents dataset issues, and not view it as just a janitorial task. A "TUH-Clean" dataset would be an invaluable service to the entire community. Totally not foreshadowing

Looking Forward

Instead of treating predictions as errors, they can be viewed as potential indicators of data quality issues should i even say it at this point, which can be leveraged to even correct the mislabeled data. Sometimes the most valuable contribution of a model is not its final predictions (remember trial_8), rather its ability to reveal issues in the assumptions about the data itself.