📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are rapidly automating core engineering skills essential to AI research, reaching saturation in benchmarks. Research remains less automated, but progress suggests it may soon become the residual task.
Recent empirical data shows that AI systems have achieved near-complete automation of core engineering tasks in AI research, marking a significant shift in the field.
According to Thorsten Meyer, recent benchmarks such as CORE-Bench and MLE-Bench demonstrate rapid progress, with AI systems reaching 95.5% and 64.4% performance respectively, within 15 to 16 months. CORE-Bench measures research reproduction, where AI can now reliably reproduce experimental papers, reducing the traditional bottleneck of manual replication. Similarly, in Kaggle competitions, AI agents are reaching mid-tier human performance, indicating automation of practical engineering tasks involved in data science and model optimization.
These advancements imply that the engineering component of AI research—installing dependencies, running experiments, optimizing kernels—is now largely automated. The remaining challenge lies in the research phase itself, involving hypothesis generation, creative problem-solving, and theoretical breakthroughs. Clark’s analysis suggests that while engineering is nearing full automation, research may be inherently more complex and less amenable to automation—though this residual may shrink if research becomes an extension of engineering at scale.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
CLAUDE AI UNLEASHED From First Prompts to Pro: The Complete Guide to Claude AI for Writing, Research, Coding, and Business (The Claude AI Mastery Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
AI Experiment Journal: Track AI tools, experiments, and results to improve your workflows and productivity.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Local LLM Inference Optimization: A Comprehensive Guide to Quantization, Hardware Acceleration, and Efficient Private AI Deployment
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
AI and Multimodal Services – AIMS 2025: 14th International Conference, Held as Part of the Services Conference Federation, SCF 2025, Hong Kong, China, … Notes in Computer Science Book 16151)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of Engineering Automation in AI Development
This shift has major implications for the AI field: the traditional bottleneck of reproducing and validating research is diminishing, potentially accelerating innovation cycles. It raises questions about the future role of human researchers, emphasizing creative and theoretical work over engineering. Institutional strategies may need to adapt, focusing more on hypothesis-driven research rather than routine experimentation, which AI now handles effectively.
Progress in AI-Driven Engineering Tasks
Over the past 18 months, multiple benchmarks have shown rapid improvements in AI capabilities related to core research engineering tasks. Notably, CORE-Bench has seen a 4.4× improvement, with AI systems now capable of reproducing research papers at near-human reliability. Kaggle competition performance has also surged, with AI reaching mid-tier human performance levels. Parallel advances in kernel design, such as automated CUDA and Triton kernel generation, further underscore the trend of AI automating technical engineering work essential to AI development.
This progress is part of a broader pattern where AI capabilities are approaching or surpassing measurement saturation points across multiple domains, suggesting that engineering in AI research is nearing full automation.
“AI can today automate vast swatches, perhaps the entirety, of AI engineering. It is not yet clear how much of AI research it can automate, given that some aspects of research may be distinct from the engineering skills.”
— Thorsten Meyer
Uncertainties About the Future of AI Research Automation
It remains unclear how much of the research process—beyond engineering—can be automated. While engineering tasks are nearing full automation, the inherently creative aspects of research, such as hypothesis generation and abstract reasoning, may still resist full automation. The pace of progress in automating research-level innovation is uncertain and depends on future breakthroughs in AI capabilities.
Next Steps in AI Research Automation and Human Role
In the coming months, researchers will monitor whether AI can extend automation into the research phase itself, potentially reducing the residual research tasks further. Institutional responses may shift towards leveraging AI for hypothesis testing and theoretical development. Additionally, the AI community will likely develop new benchmarks to measure progress in automating more complex, creative aspects of research.
Key Questions
What does automation of engineering tasks mean for human researchers?
It suggests that routine engineering work, such as reproducing experiments and optimizing models, will increasingly be handled by AI systems, allowing human researchers to focus more on theoretical and creative aspects of research.
Are there limits to what AI can automate in research?
Yes, aspects involving hypothesis generation, abstract reasoning, and innovative thinking are less amenable to automation, though ongoing progress may reduce these gaps over time.
How soon might AI fully automate the research phase?
It is uncertain; while engineering is nearing full automation, the timeline for automating the creative and theoretical components of research remains unclear and depends on future AI breakthroughs.
What are the implications for research institutions?
Institutions may need to adapt by focusing more on hypothesis-driven research and less on routine experimentation, leveraging AI to accelerate discovery and reduce costs.
Source: ThorstenMeyerAI.com
