Researchers Introduce OpenDiscoveryTrace Dataset to Evaluate Scientific AI Workflows

Researchers have released OpenDiscoveryTrace, a public dataset designed to evaluate the internal reasoning processes of autonomous scientific AI agents. While traditional benchmarks assess only final outputs like code or papers, this framework captures the step-by-step methodologies models use during scientific discovery.

The dataset comprises 558 complete agent trajectories across 124 scientific tasks spanning drug discovery, materials science, genomics, and literature analysis. It benchmarks seven models, including frontier systems (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro) and open-weight architectures (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B).

Pilot analysis of the trajectories revealed that despite achieving similar overall success rates between 84% and 89%, frontier models exhibit distinct failure patterns. Claude Opus 4.6 suffered primarily from tool misuse (66.7% of errors), whereas GPT-5.4 predominantly made reasoning errors (83.6%). The dataset, schema, and baseline benchmarks are available under the CC BY 4.0 license.