DrugTargetWorld: A Synthetic Biobank for Training and Benchmarking AI Scientists

An open benchmark for training and evaluating AI agents on end-to-end, biobank-based drug target discovery.

v1.0rubric v0.9 · run 1

Mean benchmark score and component scores for nine models over 540 episodes. Sortable by any numeric column.
#	Model							Serving
1	Opus 5api-alias@2026-09-07	39.98	14.80	7.19	13.05	86.64	$4.14	API
2	GPT-5.6 Solapi-alias@2026-09-05	35.38	15.81	3.16	13.00	83.19	$2.02	API
3	Sonnet 5api-alias@2026-09-04	21.33	10.24	1.16	6.96	83.88	$1.47	API
4	Haiku 4.5api-alias@2026-09-04	12.92	5.27	0.21	5.91	75.00	$0.26	API
5	gpt-oss-20b6cee5e81	7.41	2.94	0.68	3.43	75.00	$0.07	GPU
6	Qwen3-Coder-30Bb2cff646	5.90	3.26	0.14	1.94	75.00	$0.05	GPU
7	GLM-4-32B077b5c2f	1.46	1.12	0.08	0.25	25.00	$0.20	GPU
8	Qwen3-8Bb968826d	1.31	0.70	0.39	0.22	30.71	$0.22	GPU
9	Devstral-Smallbd165ab2	0.81	0.75	0.06	0.00	15.00	$0.20	GPU

Mean score across 20 worlds × 3 budget conditions, one replicate each. Scores are out of 100; measured ceiling = 85.5.

Qwen3-8B and GLM-4-32B ran a reduced 8,000-token output budget and are not a like-for-like comparison.

Disease states

Each synthetic world expresses hidden disease biology through participant-level medical data.

world dev_dcm_t4

Drag to compareCardiac MRI · 25 frames · 128×128

ControlDCM

‹›

Dilated cavity with visibly reduced contraction across the cine loop.

03 Environment

How it works

Biobank
Genetics, omics, imaging, ECG, EHR.
Phenotype
Segment the myocardium from raw arrays, engineer features, fit a model against a proxy outcome.
Causal targets
Screen the proteome, instrument it genetically, separate drivers from decoys.
Experiments
Spend a finite research budget.
Submission
Nominate targets and therapeutic direction.

Release

world_07/
├─ genotypes.vcf.gz
├─ proteomics.parquet
├─ transcriptomics.parquet
├─ metabolomics.parquet
├─ covariates.parquet
├─ ecg_features.parquet
├─ coronary_ct.parquet
├─ ehr_diagnoses.parquet
├─ mortality.parquet
├─ mace_events.parquet
├─ targetability.parquet
├─ imaging/
└─ imaging_visit2/

Episode

>>> d = np.load(f); cine, t1 = d['cine'], d['t1map']

(3, 25, 128, 128) uint8 (3, 128, 128) float32, 98.2% NaN

>>> mc = ndi.binary_closing(~np.isnan(t1[s])); filled = ndi.binary_fill_holes(mc)

>>> cav = filled & ~mc; thr = (np.median(cm[:, mc]) + np.median(cm[fED][cav])) / 2

>>> A[s, f] = np.bincount(ndi.label((cm > thr) & reg)[0].ravel())[1:].max()

SUBJ_00001 [494 387 277 212 174 140 132 ... 467 474 492] EF 0.775

>>> for fp in files: rows.append(feat(fp)) # 10,800 subjects

159.9 s elapsed # turn limit 300 s

>>> y = hf_icd | mace | cardiac_death # proxy label, prev 0.307

>>> lr = LogisticRegression(max_iter=2000).fit(Z[feats], y)

edvi 0.111 esvi 0.424 ef 0.427 myoi -0.502 wt 0.217 gfr -0.442

corr(score, y) 0.260 hand-built composite 0.241

>>> pd.DataFrame({"subject_id": ids, "phenotype": z}).to_csv(OUT_CSV)

>>> G = load_genotypes(); B = (Gs.T @ Yc) / M.sum(0) # 8,192 x 2,941

>>> b_iv, z = wald(g[:, j], y, mask) # cis Wald ratio

PROT_1606 rs104994 bx 0.436 MR -0.396 z -17.4

PROT_1858 rs102089 bx 0.153 MR 0.005 z 0.1 obs r 0.36

>>> request_experiment("knockdown", "PROT_1850")

delta {cavity_r: 0.00, wall_t: 0.00, ef: 0.000, surv5y: 0.00}

The agent receives the files, writes Python, and chooses analyses and experiments over 30 turns.

04 Method

Hidden causal worlds

Each world is procedurally generated from a sealed structural causal model linking genetics, molecular measurements, hidden disease state, observable phenotypes, and intervention outcomes.

Disease state

L_i = z\left[\sum_{j\in\mathcal D} w_j P_{ij} + \gamma^\top C_i + \delta G_{i,\mathrm{direct}} + \epsilon_i\right]

The identity and number of causal molecular drivers are hidden from the agent.

Research policy

a_t \sim \pi(a \mid s_t, B_t)

B_{t+1} = B_t - c(a_t)

Analyses and experiments are chosen against accumulated evidence and the remaining budget.

T1 Confounding · T2 Reverse causation · T3 Selection / collider · T4 Causal non-identifiability · T5 Batch effects · T6 Benign remodeling · T7 Instrument pleiotropy · T8 Surrogate-outcome discordance · T9 Assay unit mixing · A9 Slow effect

Citation

Margolis S, Schmiedmayer P, Huang A, et al. DrugTargetWorld: A Synthetic Biobank for Training and Benchmarking AI Scientists. arXiv:2610.09558 (2026).

arXiv DOI

DrugTargetWorld: A Synthetic Biobank for Training and Benchmarking AI Scientists

Leaderboard

02 Disease states

How it works

Biobank

Phenotype

Causal targets

Experiments

Submission

Hidden causal worlds

05 Citation

Disease states

Citation