TL;DR: A T-cell signal reached PP4 near 0.992 in one dataset. Three independent public blood T-cell resources did not reproduce it.

PP4 ≈ 0.992 looked exciting. Three independent public blood T-cell resources did not reproduce it. This case study, from our prospectively frozen analysis of public data, shows what it takes to stop one striking number from quietly becoming a causal story.

It is part of our technical report, Falsification-First AI for Biological Target Assessment. Technical report; not peer reviewed. Nothing here is a clinical or therapeutic claim.

The setup

While testing a different ulcerative colitis (UC) hypothesis, our workflow surfaced an unexpected signal for the gene TTPAL at the UC 20q13 locus.

Our result (discovery). In the discovery resource, the signal reached an apparent colocalization posterior of approximately PP4 = 0.992 in three gut-resident T-cell groups.

Source fact. PP4 is the posterior probability, under a Bayesian colocalization test, that two association signals (here, the disease association and the gene-expression association) share a single causal variant [1]. A value near 0.992 is the kind of number that ends up on a slide.

The safeguard

We treated that number as nomination evidence only, not as a finding. Before reading any independent outcome, we:

  • sealed the discovery dataset, so it could not be reused to "confirm" itself;
  • prospectively defined the independent confirmation: which public cohorts were admissible, in what order, at which locus and window, and what result would count as replication.

What happened

Our result (independent test).

  • Primary independent cohort. OneK1K is a peripheral-blood single-cell eQTL resource [2], with up to approximately 954 donors in the eligible cell classes used by our frozen study. TTPAL had no usable supporting signal at the frozen locus and window. The maximum PP4 across the evaluated T-cell classes was approximately 0.031.
  • Secondary independent resources. CEDAR [3] did not support the nominated TTPAL effect. Kasela 2017 [4] had no usable confirming signal.
  • No alternative winner. No other local gene clearly won the independent confirmation analysis.

Decision

SINGLE_DATASET_ONLY. The exciting discovery signal remained single-dataset-only.

The caveat that travels with this result

The discovery cells were gut-resident T cells. The admissible independent public cohorts were peripheral blood. That difference matters:

  • This result does not prove that a tissue-resident TTPAL effect is false.
  • It does not say the discovery result was wrong.
  • It shows only that the discovery signal did not independently reproduce on the public data available for this test.

Keeping that caveat next to the result is part of the method. Dropping it would turn an honest "not replicated here" into a misleading "disproven".

Why it matters

The purpose of scientific diligence is not to make an exciting number disappear. It is to prevent one exciting number from silently becoming a causal story. An impressive posterior in one dataset is not replication, and the time to find that out is before the next expensive step, not after it.

For the general version of this discipline, see what makes an AI system trustworthy, as NIST describes it. The companion case study, Not Evaluable Is a Result, shows the other outcome that matters: a causal bridge that public data simply cannot decide.

Try it on your own target

We have packaged this method as the Biodius Target Challenge: a fixed-scope, five-business-day adversarial review of one target or mechanism, ending in a decision memo. Send us one target + indication + rationale. We will tell you whether it deserves the next expensive step. You can reach us at info@firstaimovers.com, or learn more about First AI Movers.

References

  1. Giambartolomei et al. Bayesian test for colocalisation between pairs of genetic association studies using summary statistics. PLoS Genetics 10(5): e1004383 (2014). https://doi.org/10.1371/journal.pgen.1004383
  2. Yazar et al. Single-cell eQTL mapping identifies cell type-specific genetic control of autoimmune disease. Science 376(6589): eabf3041 (2022). https://doi.org/10.1126/science.abf3041
  3. Momozawa et al. IBD risk loci are enriched in multigenic regulatory modules encompassing putative causative genes. Nature Communications 9: 2427 (2018). https://doi.org/10.1038/s41467-018-04365-8
  4. Kasela et al. Pathogenic implications for autoimmune mechanisms derived by comparative eQTL analysis of CD4+ versus CD8+ T cells. PLoS Genetics 13(3): e1006643 (2017). https://doi.org/10.1371/journal.pgen.1006643