A preprint posted to arXiv on July 29 by a team of 24 researchers has delivered the most direct empirical challenge yet to claims that artificial intelligence is close to replacing human scientists. The team ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute; the agents completed all of the engineering without human help, yet could not make substantial progress toward answering the research questions. "I don't think full automation of open-ended research is on the horizon right now," said Sayash Kapoor, a computer scientist at Princeton University and a co-author of the preprint. The study, titled "Can AI agents conduct open-ended AI research? Early evidence from two case studies" (arXiv:2607.27191), was covered by Nature in late August.
The preprint arrives months after the field's most celebrated benchmark achievement. The effort to fully automate the scientific process, from idea generation to the writing and self-evaluation of a paper, was pioneered by a team mostly from Sakana AI in Tokyo, which unveiled The AI Scientist in 2024 and later published results from an improved version in March in Nature. In that work, a manuscript generated by the AI system passed the first round of peer review for a workshop of a top-tier machine learning conference. Yet the Princeton-led team argues that existing evaluations have been structurally too lenient: evidence on whether agents can carry out open-ended research is thin, because current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality.
The co-authors, who include legal scholars, ethicists, and computer scientists alongside Princeton's Arvind Narayanan, identify five recurring failure modes, though they are careful about overgeneralizing. Their main finding is negative, and they note that their results are tentative, with acknowledged limitations in sample size and potential scaffold improvements. Nature's news desk summarized the upshot bluntly: an agentic system successfully developed concepts from two computer-science papers, but the original authors were not impressed. The broader landscape of AI-assisted research adds important context: the complexity of the natural world means that AI co-scientists will only be truly effective when they can go beyond connecting words together, to modeling the full complexity of the systems those words describe.
Proponents of AI-driven research counter that the goalposts are moving fast. The peer-reviewed Nature publication of The AI Scientist expanded on the preprint's description of the system's weaknesses, included more ethical considerations, and toned down original statements about automating the entire research process, noting that humans helped to filter the most promising outputs. Sakana AI has also pointed to a clear scaling law, observing that as underlying models improve, the quality of AI-generated science rises as well. The debate now centers less on whether AI can handle the mechanics of research and more on whether it can do the harder cognitive work: generating genuinely new questions, recognizing when its own results are wrong, and exercising the kind of judgment that distinguishes a finding from a fluke.
The honest read of the evidence so far is that AI is a very fast, tireless research assistant that still needs a scientist in the room to tell it what actually matters.