7448 AI · Inside the model
What it does, and where it fails.
Every claim here is generated from the experiment outputs in the 7448 repository, not written by hand. If a number looks unimpressive, that is because it is the number the model actually produced.
The result that shaped this tool
An excellent number that is almost entirely artefact.
The obvious way to predict "which condition does this profile resemble" is to pool all fourteen cohorts and train one classifier. Done that way, it reaches a median AUROC 0.958: an excellent result, and the kind of number that gets published and productised.
It is almost entirely an artefact. Each cohort studied one disease, on one sequencing platform, in one lab. So "has Crohn's disease" is nearly perfectly correlated with "was processed by the lab that studied Crohn's disease", and a model can score brilliantly by recognising the lab's technical signature instead of anything biological.
The control that proves it: train a classifier on nothing but which cohort a sample came from, no bacteria at all. It scores a median AUROC 0.915. A model with zero biological information nearly matches the impressive one.
| Model | What it can see | Median AUROC |
|---|---|---|
| Pooled across cohorts | Bacteria plus, implicitly, which lab ran the sample | 0.958 |
| Cohort identity only | Only which lab ran the sample. No biology whatsoever. | 0.915 |
| Within cohort | Bacteria only: cases vs controls from the same lab and platform | 0.643 |
The honest number is the last one: 0.643. Of 17 condition-by-cohort models tested, only 6 clear chance once a 95% confidence interval is applied. That is why 7448 AI shows literature citations rather than risk scores, and says so on every signal it displays.
Cross-cohort transfer
It does not transfer to a lab it has never seen.
The headline accuracy, median r = 0.458, is measured on held-out patients drawn from the same fourteen cohorts the model trained on. A stricter test is to hold out an entire cohort: train on thirteen, predict the fourteenth, and repeat for all of them.
Under that test the median correlation is 0.0024. Not reduced: gone. Across all 14 held-out cohorts the best result was 0.126 and the worst was -0.162.
Pearson correlation is unaffected by rescaling, so this cannot be explained away as differing measurement units between mass-spectrometry platforms. The model genuinely fails to rank samples correctly in a cohort it was not trained on.
Leakage check
The pipeline passes.
A pipeline can produce excellent scores purely by accidentally leaking the answer into the inputs. The test: shuffle each metabolite's values across training patients, destroying any real relationship, and refit everything unchanged. A correct pipeline should then score zero.
Real data scored r = 0.599; shuffled data scored r = -0.0018 across 60 metabolites. The within-cohort signal is real, whatever its limits.
Metabolite prediction accuracy
Measured on people the model never saw.
Split by individual rather than by sample, so repeat visits from one person cannot leak across the train/test boundary.
- Median correlation (r)
- 0.458
- Well predicted (r ≥ 0.5)
- 418
- Usable (r ≥ 0.3)
- 621
- Poorly predicted (r < 0.2)
- 185
The neural network lost to the linear model.
A deep network was trained on the same data and the same splits. It reached a median r = 0.376 against the elastic net's r = 0.458. With roughly 2,000 training samples against 5,038 input features, there is not enough data for the deep model to earn its complexity, so the simpler model ships.
That turns out to be a feature. Because the served model is linear, each prediction decomposes exactly into per-microbe contributions that sum to the answer. The "why this number" breakdown in the simulator is not an approximation of the model's reasoning; it is the model's reasoning, read directly off the weights.
Open problem
Help close the transfer gap.
Cross-cohort transfer is the failure that matters most. If you hold paired microbiome and metabolomics data from a lab outside these fourteen cohorts, we would like to test against it.
Pipeline run log
Every run, including the ones that failed.
Every data pull, training run and backtest appends a record, and nothing is curated for presentation. This is a snapshot taken September 13, 2026 with file paths redacted.