Agent Laboratory endeavors to reconfigure how machine learning (ML) research is conducted by placing large language model (LLM) agents at the core of a workflow for literature review, experimentation, and report writing. It represents an autonomous system designed not to originate novel research directions from scratch but to help actual human scientists execute their own preexisting ideas. In that sense, Agent Laboratory is simultaneously ambitious in scope—because it intends to manage a wide range of tasks previously needing human labor—and circumscribed by intention, as it focuses on extending human creativity rather than substituting it. The creators propose that by giving researchers more time to engage with the conceptual dimension of scientific inquiry, while delegating the labor-intensive tasks of coding, data exploration, and iterative drafting, one might accelerate discovery and reduce the costs of trial and error.
The historical impetus behind Agent Laboratory rests on the observation that contemporary scientific processes require significant resources: from scanning countless research papers to writing code that might or might not yield significant results, the overhead can be immense. Even with the rise of LLMs such as ChatGPT, Claude, or Llama, harnessing these models in a truly orchestrated research pipeline remains challenging. Early attempts, including frameworks like ResearchAgent, The AI Scientist, and others, offered glimpses of agents autonomously brainstorming possible project directions. But they were prone to feasibility issues, misalignment, or incomplete knowledge. Agent Laboratory attempts to address those shortcomings by starting from the premise of a user-provided research concept, thereby directing the pipeline toward a concrete goal that truly matters to a human scientist.
Yet the authors caution that, while LLMs have shown promise in code generation, tutorial-style Q&A, editorial tasks, and more, their capacity for reliable, replicable, and logically cohesive scientific discovery can hinge on external oversight. Humans, they argue, must remain integral in guiding the big picture—choosing the questions to ask, deciding which sub-problems are worth the time-consuming grind of data wrangling, and ultimately judging the meaning of the results. Despite these caveats, Agent Laboratory tries to automate as much of the “low-level” process as possible.
The system orchestrates a pipeline that unfolds in three main stages: Literature Review, Experimentation, and Report Writing. Each stage is staffed by distinct agent personas—PhD Student, Postdoc, Machine-Learning Engineer, and Professor—who each carry out specialized tasks and feed information to one another. This layered structure, reminiscent of hierarchical labor division in a real laboratory, aims to push LLMs to operate with specialized roles while remaining integrated in a coherent workflow.
Literature Review
During the first phase, the Literature Review, an agent designated as the “PhD student” queries an online repository like arXiv, gathers summaries about possibly relevant papers, fetches full texts if needed, and curates them to form a reference corpus for the new research. This curation does not happen in a one-shot manner. Rather, the system queries the database iteratively, refines the selection of references, and eventually reaches a point where it caps the number of included works. By design, the pipeline limits the maximum size to avoid token overflow or spurious expansions that might degrade performance. The significance of this step is that it fosters the use of relevant knowledge from potentially up-to-date or specialized literature, offering a vantage for subsequent planning. Indeed, the authors note that prior systems sometimes rely on limited offline data or hallucinated references, diminishing utility. The PhD-student agent’s job here is to remain mindful of the user’s idea, locate relevant prior approaches, and weave them into a curated compendium from which the second stage can glean insights.
Experimentation
The second phase, Experimentation, is more intricate. First, a Plan Formulation sub-phase emerges, where a “PhD student” agent works with a “Postdoc” agent. The Postdoc has a slightly higher-level vantage, verifying that the plan is methodically sound or at least plausible. They discuss which ML models to use (convolutional, transformer-based, or otherwise), which dataset might test the stated hypothesis, and the broad blueprint for how to run experiments. After some back-and-forth discussion, typically culminating in a final “plan” command, the system transitions to Data Preparation. Here, the “ML Engineer” agent is summoned. It consults the plan and attempts to code data-loading scripts, dataset splitting, or transformations that align with the previously devised approach. Through a specialized protocol, the code is compiled and tested for errors. If any emerge, the engineer agent modifies the code accordingly. The authors note frequent pitfalls: token usage can balloon if the system prints too much debug information or tries to do everything in a single pass. The code must remain modular, minimal, and functional.
After data preparation is successful, the pipeline proceeds to Running Experiments, which is orchestrated by a specialized module called “mle-solver.” This module draws from the research plan, previously curated references, and the code from data preparation to generate an experimental script from scratch if needed. Then it can refine or overhaul that script by two types of operations: REPLACE, which jettisons the code wholesale, and EDIT, which modifies a specified line range. Each iteration of code generation is tested with a compiler or interpreter. Failing code triggers attempts at automatic repairs or line edits. Successful code leads to scoring by an internal reward function that examines how well the script’s output aligns with the plan. Repeatedly, the solver tries to improve that internal score by introducing more sophisticated model architectures, hyperparameter tuning, or different data augmentations. The process can collect multiple candidate code versions, only retaining the top performers. Over time, with each iteration, the solver hopefully zeroes in on an approach that yields meaningful results. The authors highlight that this approach, while automated, still benefits from “self-reflection” prompts, where the system re-examines results or failures, tries to interpret them, and decides on subsequent modifications. The overarching philosophy is reminiscent of evolutionary search, but driven by LLM-based code generation and an LLM-based reward mechanism.
The final sub-phase of Experimentation, Results Interpretation, has the “PhD” and “Postdoc” agents reconvene to inspect numeric findings. If the solver code prints out, for example, a certain accuracy or a cluster of metrics, the agents attempt to parse these outcomes, interpret their meaning within the context of the initial research question, and form a coherent summary. Such a summary might note that the baseline outperformed the newly proposed adaptation, or that the difference was negligible, or that the plan’s approach needs revision. Critically, the pipeline can chain back if the results are unsatisfactory—similar to the peer-review process in a real lab—though the authors do not claim indefinite re-iterations. They do mention the possibility of returning to the plan or rewriting code if the newly gleaned insight demands more thorough experimentation.

Report Writing
The third and final phase, Report Writing, delegates the main tasks to the “PhD” and “Professor” agents. This stage capitalizes on a specialized “paper-solver,” which constructs a LaTeX document in a multi-step fashion. The pipeline commences by generating a skeletal scaffold with placeholders for the classic academic sections: Abstract, Introduction, Background, Related Work, Methods, Experimental Setup, Results, and Discussion. That skeleton is compiled as a preliminary check. Then, systematically, the solver populates the sections with text. It can also search arXiv again to find references to cite specifically within the writing phase, though the authors note this might be optional. Edits proceed in a line-by-line manner, ensuring at each step that the LaTeX still compiles. If compilation errors appear, the paper-solver tries to remedy them. Once a complete draft is formed, the system triggers an automated paper review process reminiscent of a mini peer review. Three “reviewer agents” simulate the perspective of NeurIPS reviewers, rating the paper on standard academic metrics: “quality,” “significance,” “originality,” “clarity,” “soundness,” “presentation,” “contribution,” and so forth. They also produce a final “accept” or “reject” decision. The “PhD” agent inspects these reviews and decides whether to finalize the paper or go back to earlier workflow stages to address the critiques.
Usage Modes and Preliminary Results
In terms of usage modes, Agent Laboratory offers an “autonomous” and a “co-pilot” deployment. In the “autonomous” mode, the system simply executes each phase in turn, with minimal or no human involvement besides the initial research question. The authors tested the performance of Agent Laboratory on five sample topics, such as “Do language models exhibit cognitive biases, such as confirmation bias or anchoring bias?” or “Are image transformers more or less sensitive to pixel noise than convolutional networks?” They employed three LLM backends—gpt-4o, o1-mini, and o1-preview—and then asked human PhD reviewers to rate the resulting papers in terms of experimental rigor, clarity, and practical usefulness. The results: while each system performed decently, o1-mini tended to produce stronger experimental results, whereas o1-preview was rated as the most overall “useful.” Meanwhile, gpt-4o achieved the fastest runtimes and the lowest cost (around only $2.33 per entire pipeline run), but its results were typically less robust across metrics. That cost differential is significant compared to prior systems like The AI Scientist, which might cost upward of $15. Hence, Agent Laboratory demonstrates the capacity to curb expense while producing end-to-end research expansions in a fraction of the typical time.
Under the “co-pilot” mode, a human can intervene at the conclusion of each major sub-phase: they can direct the system to re-run the literature search if a relevant paper was overlooked or instruct the system to incorporate a specific model architecture absent from the code. The authors found that these interventions generally enhanced the final scores across some key criteria—particularly clarity and alignment with the user’s intentions. However, they also observed friction: the system does not always parse a co-pilot’s instructions precisely, leading to frustration. Another surprising finding was that while co-pilot mode typically produced better final outputs (especially in “soundness” and “quality”), participants themselves rated “usefulness” slightly lower than the system’s own autonomous attempts, presumably because micromanaging the agent can be time-consuming. Nevertheless, co-pilot mode was well received. Human participants also had the option to generate either a “custom” research question or choose from the set of preselected prompts. Interestingly, external evaluators found the preselected topics gave the system a more consistent baseline, so sometimes those papers ended up with higher external ratings.
MLE-Solver and Kaggle Comparisons
A cardinal highlight concerns how Agent Laboratory’s “mle-solver” measures up against other automated Kaggle-solving approaches. The authors tested it on a subset of MLE-Bench, a compilation of real Kaggle tasks. The solver outperformed or matched state-of-the-art Kaggle-bot frameworks like MLAB, OpenHands, and AIDE on multiple metrics, including the Kaggle “medal” system, gleaning more gold and silver medals overall. The successes are presumably owed to its iterative refinement approach, textual reflection of errors, and integrated scoring function that actually checks whether the code is nailing or ignoring the specified plan.
However, no advanced system is complete without enumerating its constraints. The authors are forthright about the limitations of Agent Laboratory. First, they warn that “literature review” can become stuck in loops, repeatedly summarizing the same set of papers rather than forging ahead with a new or deeper analysis. Sometimes the system collects too many references, then inadvertently truncates them when token limits are reached. Another frequently encountered predicament is hallucination: certain less powerful LLM backends, such as gpt-4o, occasionally produce ostensible experimental logs or references that were never truly generated. This can manifest as spurious statements about hyperparameters or “test accuracy” that the system never actually computed. Although the authors tried to mitigate it by having the system pass real code execution logs, they still sometimes see illusions creeping into the final write-ups. Additionally, the system’s enforced structure (like fixed paper sections or a requirement for only two visual figures) might stifle unorthodox research formats or hamper experiments that need more graphics or specialized data visualization approaches. Practical next steps could revolve around enabling more flexible tools to manage a repository of multiple files, to integrate deeper library usage without incurring additional complexities.
Another dimension of caution arises from ethical considerations. Agent Laboratory reduces friction for generating full-blown research artifacts—code, data analysis pipelines, and polished manuscripts—yet that ease might lead to an influx of sub-par or possibly even misleading work. The paper’s authors highlight that unscrupulous or naive usage could saturate academic channels with questionable publications. Or a malicious user might adapt the pipeline for destructive tasks. These concerns underscore the importance of accountability and a transparent declaration of AI involvement. The authors also caution that current LLM-based systems might inadvertently replicate societal biases or illusions from their training data, so the synergy between the agent and the human in the loop remains essential for ensuring moral, factual, or interpretive integrity.

Human Evaluations
Turning to the experimental evaluations, the paper provides deeper insights into how humans rated the system’s outputs. In autonomous mode, on a scale of 1–5, the average “experimental quality” rated by volunteer PhD readers was around 2.6 for gpt-4o, and up to 3.2 for o1-mini. The average “report quality” was around 3.0 for gpt-4o and 3.2 for o1-mini, while “usefulness” hovered around 4.0 for gpt-4o and 4.3 for o1-mini. The o1-preview model scored highest in “usefulness” but slightly trailed o1-mini on “experimental quality.” Independent of the underlying model, none of the best papers met the typical acceptance threshold at, say, an elite ML conference. Using a NeurIPS-like scale from 1–10, the typical overall rating for an automatically produced paper landed around 3.5 to 4.0, while an average accepted real NeurIPS paper might be near 5.8 or 6.0. This discrepancy points to ongoing limitations in the content’s incisiveness, originality, or rigor when left entirely to the pipeline. Nonetheless, co-pilot mode edges those numbers upward, indicating that timely human feedback ensures better clarity and stronger alignment with the researcher’s intentions—although “significance,” “contribution,” and “novelty” remain difficult to automate. Agent Laboratory helps with the mechanical tasks, but cannot conjure genuine novelty unassisted.
Runtime and Cost
In analyzing runtime and cost, the authors emphasize Agent Laboratory’s efficiency. With gpt-4o as the backbone for the intermediate steps, the entire pipeline from literature search to final report was about 1,165 seconds (less than 20 minutes) and cost about $2.33 for inference. The same pipeline soared to $7.51 with o1-mini and $13.10 with o1-preview. Meanwhile, the cost and runtime soared even higher in some older autonomous research frameworks. The authors tout their method’s superiority in cost-effectiveness, especially if speed is a priority: the cheaper models might produce less accurate results, but at least they do so swiftly. For users who want more methodologically sound or thorough exploration, it might be worth the higher expense entailed by a more powerful model. Overall, the system’s subtask success rate (the fraction of times each step successfully completes without exceeding iteration bounds or token limits) is generally high, exceeding 90–95% across the board but dipping significantly in the literature review phase for some models.
In a more granular reflection, the authors list the system’s typical failure modes: repeated calls to the same commands during literature review, zero-accuracy code that never recovers in time, code that tries to run unapproved system commands, or attempts to do partial tasks at lines that do not exist. The user or an external coordinator must watch out for these anomalies. On the plus side, many ephemeral bugs can be automatically patched during intermediate steps.
Significance and Future Directions
Why, then, does Agent Laboratory matter? The authors stress that the tool’s design is partially motivated by a longing to shift researchers’ time from laborious chores—data cleaning, code debugging, tedious literature hunts—to higher-level conceptual tasks like generating incisive hypotheses or mapping out deeper theoretical frameworks. By bridging agent-based code-writing modules (mle-solver) with agent-based text generation modules (paper-solver) and with flexible roles for the user (autonomous or co-pilot), the pipeline stands as a thorough demonstration that LLM-based approaches can coordinate multiple tasks in the style of a real laboratory. While the authors refrain from claiming that the pipeline yields publishable breakthroughs by itself, they highlight that the framework does produce workable code with recognized performance benchmarks on well-known challenges, and authors can glean a comprehensible draft summarizing what was done. Indeed, one can imagine that as LLMs improve, the gap in clarity, originality, or significance might narrow.
That said, the conclusion does not hide the reality that the overall acceptance-level quality remains out of reach. On average, the system obtains an approximate 3.8/10 from external reviewers on NeurIPS-like standards, or around 4.38/10 when humans provide iterative co-pilot feedback. These scores reflect a big leap forward from older simpler code-gen tools that produce no integrated narrative. But by conferences’ standards, the final outputs mostly remain below acceptance thresholds. The authors champion further integration of advanced methods, including agent generation frameworks, dynamic tool usage expansions, or better synergy between system code and real-time environment feedback. Another possibility is user-driven expansions, where a domain expert might teach the system specialized domain knowledge to avoid naive or infeasible attempts.
Finally, the authors consider future directions. They imagine a scenario in which tens or hundreds of research ideas can be run in parallel, each culminating in workable prototypes and partial write-ups, letting a scientist pick the most promising leads. Over time, the synergy between tool and user might shorten the cycle from discovery to publication drastically. Yet they note that more thorough realism in the pipeline—like automatic figure generation, advanced data visualization, or repository-level code management—still awaits. The system might handle only single-file scripts, or it might struggle when an experiment requires repeated calls to specialized GPU-based tools. Code is kept relatively constrained to scikit-learn or PyTorch, typically ignoring other frameworks.
Ethically, they emphasize that it is vital not to interpret these improvements as a panacea. The risk of saturating academic fora with auto-generated manuscripts remains real, as does the potential abuse in domains like cybersecurity or misinformation. The authors close with an appeal for robust oversight and guidelines, praising attempts like clearly labeling AI-generated contributions. Adherence to fundamental research ethics is essential if we are to harness the prospective acceleration that Agent Laboratory heralds, without jeopardizing the quality or trust inherent to scientific enterprise.
In summation, Agent Laboratory—Using LLM Agents as Research Assistants systematically orchestrates a pipeline for scholarly invention. It stands on three pillars: retrieving curated knowledge from relevant papers, coding and refining experiments via iterative self-analysis, and producing a full textual account of the findings in a methodical structure. Empirical evaluations illustrate that, depending on the LLM used, the pipeline can produce reasonably coherent, although not conference-qualifying, outputs within affordable time and cost constraints. The design choice to incorporate a “co-pilot” mode, in synergy with an “autonomous” mode, underscores the authors’ belief that humans remain the central guiding intelligence in forging relevant, creative breakthroughs. By adopting a suite of specialized agents—PhD, Postdoc, ML Engineer, and Professor—Agent Laboratory imitates actual academic hierarchies while making the best use of LLM-based reasoning. Although the system is not flawless and requires more advanced capabilities to match the rigorous standards of top-tier publications, it marks a significant stride toward automated research pipelines, potentially revolutionizing how scientists spend time and resources. If further refined, it could become foundational, letting domain experts push boundaries in an era where thoughtful collaboration between humans and LLMs may define the next wave of innovation.
References
- Agent Laboratory: Using LLM Agents as Research Assistants: https://arxiv.org/pdf/2501.04227
- ChatGPT: https://openai.com/blog/chatgpt/
- Claude: https://www.anthropic.com/
- Llama: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/
- arXiv: https://arxiv.org/
- Kaggle: https://www.kaggle.com/
- scikit-learn: https://scikit-learn.org/
- PyTorch: https://pytorch.org/
- NeurIPS: https://nips.cc/