Causal Traps for Evaluating AI Scientists

In a real biobank, a protein can be strongly associated with disease without causing it. Confounding, reverse causation, selection into the measured subcohort, pleiotropy and measurement artifacts all produce associations that look like causal evidence. An AI scientist that ranks proteins by association will nominate them, and in real data there is no answer key to show that it was wrong.

DrugTargetWorld builds these failure modes into the data-generating process. Each bias alters the causal or measurement process rather than adding random noise, so the proteins it affects can appear scientifically plausible, and the evaluator knows exactly which ones are decoys.

The generator can plant nine predefined biases, T1 to T9. Each world carries a subset; the count below is the number of worlds in the 20-world version 1 panel that carry it.

In addition, 15 worlds contain a delayed-effect driver: a causal driver whose effect appears mainly over five years of follow-up, paired with a non-causal partner that tracks the later disease. Together with T4, this creates ambiguous pairs that observational data cannot separate; only an experiment can.

Because the structural causal model is known to the evaluator, each claim can be checked against truth. Bias identification (15 points) rewards rejecting a planted non-causal protein together with its correct source of bias, multiplying recall by precision so that indiscriminate rejection earns little. Causal confidence (25 points) rewards matching claims to evidence on ambiguous pairs: an experimentally tested causal choice earns full credit, an explicit statement that the pair cannot yet be resolved earns partial credit, and an unsupported pick earns none.

The most consequential error is penalised directly. Advancing the T8 harmful surrogate as beneficial costs 30 points, the full weight of target identification.

No agent averaged more than 2.87 of 15 points for bias identification. The reverse-causation protein, rejected most often, was rejected in 32.0% of eligible episodes and with the correct mechanism in 14.3%. All models nominated the confounding (T1) and selection (T3) proteins in fewer than 9% of eligible episodes, but the reverse-causation protein was nominated in half of the Haiku 4.5 and GPT-OSS-20B episodes.

The harmful surrogate was nominated in 44 of 60 Opus 5 and 25 of 60 GPT-5.6 Sol episodes. These agents usually labelled it harmful to outcomes (30 of 44 and 16 of 25), avoiding the penalty; every surrogate nomination by Haiku 4.5 and GPT-OSS-20B claimed it was beneficial. Instrument strength, which the leading agents checked almost always, does not establish the exclusion restriction that pleiotropy (T7) violates. See findings.