ISLES'24 hypothesis program + evaluation of an agentic scientific pipeline — research internship, Paetzold Lab, Weill Cornell Medicine / Cornell Tech.
01 — Question
Do brain imaging features predict how well stroke patients recover, beyond what clinical variables already tell you?
The dataset is ISLES'24: 129 enrolled large-vessel-occlusion stroke patients, 126 analyzed, about 97% of them successfully reperfused, with recovery measured by the 90-day modified Rankin Scale. The method is incremental R² over a clinical baseline of age plus admission NIHSS — pre-registered with a smallest effect size of interest of ΔR² ≥ 0.10 fixed in advance, multiplicity correction, NumPy/SciPy only, and every result carrying an artifact path and independently recomputed from saved design matrices.
02 — Findings
Exactly two verified positives, both clinical rather than imaging: NIHSS at 24 hours (ΔR² = 0.257) and premorbid mRS (ΔR² = 0.151 — essentially a positive control). Every imaging feature nulled out once infarct volume was in the model. The explanation is the mediation thesis: lesion-intrinsic imaging features are null because infarct volume already carries their signal, and frailty features are null because age already carries theirs.
Then I tried to break my own explanation: a pre-registered falsification batch of ten hypotheses, seven of them deliberately chosen to be mediation-resistant, powered 0.85–0.96. All seven returned well-powered nulls. A follow-up made an ordered prediction about lesion location that failed on its own pre-registered terms. The thesis survived a test built to break it.
One live lead: corticospinal-tract lesion load (ΔR² = 0.033, correct direction, a third of the effect-size bar, dead under multiplicity). Advancing it needs an external cohort of n ≥ 152 — this one is exhausted at 126 analyzed of 129 enrolled.
One hypothesis looked like a genuine discovery — ΔR² = 0.108, adjusted p = 0.00045 — and was killed by its own pre-registered secondary analysis, because the exposure secretly embedded infarct volume. Pre-registration caught a circular feature in real time, and the original POSITIVE label stays in the record next to its refutation rather than being retro-edited.
03 — Evaluation
The lab's analysis pipeline is VERITAS (arXiv:2604.12144), an agentic multi-phase framework in which LLM agents formulate hypotheses, write statistical code, execute it, and interpret results. Its published metrics measure whether the evidence label is right — they never check whether the underlying number is right. I measured both, against my independently verified harness: 5 hypotheses × 5 replicates × 2 model configurations = 50 runs, with the scoring instrument validated and frozen before collection began.
I also migrated the pipeline off paid APIs onto self-hosted open-source models via Ollama on SLURM/Linux HPC, cutting inference spend by roughly $1,000/month.
The single clearest case: one run reproduced the statistic to 16 significant figures, its agent correctly reasoned that the finding contradicted the hypothesis and wrote the right label — and deterministic downstream code overruled it. The model was right, and got overruled.
I also logged 15 defects in my own scoring instrument, including one where it committed the exact error it was built to detect. That's in the report, not hidden.
04 — Limits
Single cohort, no external validation. The ~97% reperfusion rate removes much of the variance imaging features were meant to explain, and nulls at SESOI 0.10 don't exclude smaller real effects in the 0.02–0.05 band.
The evaluation is pilot-scale. Every rate rests on n ≤ 25, and it is a domain-transfer study — its numbers are not comparable to the paper's published benchmark, and it is a study of evaluation design, not a replication attempt.
05 — Toolkit
Pre-registration and falsification design, power analysis, multiplicity correction, SLURM/HPC on a university cluster, local LLM deployment (Ollama, models up to 116B), building and validating a measurement instrument, and artifact-level provenance for every number.