AI Scientist Agent — A simple local Multi-Agent Research System

A multi-agent research assistant that runs on your machine. Give it a research topic and it autonomously completes the full loop:

Literature review → Research proposal → Experiment plan → Write & run code → Analyze results → Write paper → LLM review & improve

It fuses ideas from three well-known open-source projects: AgentLaboratory, AI-Scientist, ASI-Evolve

⚠️ The system executes LLM-generated Python code on your machine. Safe mode is on by default and experiments run in isolated subdirectories, but only use it for your own research tasks.

---

Requirements

• Windows / Linux / macOS, Python 3.10+
• Recommend the pylearn conda env (Python 3.10 + torch + CUDA, deps already installed)
• One LLM API key (DeepSeek / OpenAI / Anthropic)

Install dependencies

conda activate pylearn        # or your existing env
pip install openai anthropic pyyaml flask sentence-transformers arxiv scikit-learn numpy matplotlib

Quick Start

1. Configure the API key

Edit config.yaml, or set an environment variable (recommended):

set DEEPSEEK_API_KEY=sk-xxx          # Windows cmd
# or
export DEEPSEEK_API_KEY=sk-xxx       # Linux/macOS

Or switch to OpenAI / Anthropic in config.yaml (llm.provider).

2. Run a research task

# Single-round research (start here)
python main.py run --task "Robustness of deep learning under label noise"

# Iterative improvement (ASI-Evolve style, 2 rounds)
python main.py run --task "..." --iterations 2

# Multiple proposals
python main.py run --task "..." --num-ideas 2

# Resume from a checkpoint
python main.py run --task "..." --resume

# Resume a specific study; increase the total step budget if it was exhausted
python main.py run --task "..." --experiment-dir experiments/my_study --resume --max-steps 60

3. View results

Everything lands under experiments/<timestamp>/:

experiments/<timestamp>/
├── state/               # checkpoints and per-stage JSON artifacts
│   ├── study_state.json # full state (used for --resume)
│   ├── research.json    # research proposal
│   ├── plan.json        # experiment plan
│   ├── analysis.json    # result analysis
│   ├── paper.json       # the paper
│   └── review.json      # review feedback
├── runs/                # each experiment's code, output, result.json, figures
├── reports/             # final paper .md and .tex
└── paper_library/       # vector-indexed local paper library

4. Web UI (optional)

python main.py web
# open http://127.0.0.1:5000

Browse experiment progress and results, view papers and figures, and upload PDF papers (auto-indexed for the researcher to retrieve).

5. Import local PDFs

python main.py import-papers "D:\some_papers_folder"

---

Architecture

        ┌─────────────────────────────────────────────┐
        │                Orchestrator                 │
        │  staged pipeline + resume + artifact mgmt   │
        └─────────────────────────────────────────────┘
        │   │        │        │        │       │
        ▼   ▼        ▼        ▼        ▼       ▼
   ┌─────────┐ ┌───────┐ ┌────────┐ ┌───────┐ ┌──────┐ ┌────────┐
   │Researcher│ │Planner│ │Engineer│ │Analyst│ │Writer│ │Reviewer│
   └─────────┘ └───────┘ └────────┘ └───────┘ └──────┘ └────────┘
        │              │         │         │
        ▼              ▼         ▼         ▼
   arXiv/library    plan text   local     LLM reads
                                Python    results

Agent	Responsibility	Output
Researcher	Survey literature, find gaps, propose a testable hypothesis	title/hypothesis/plan/related work
Planner	Refine into dataset, model, metric, baseline	experiment plan
Engineer	Write code (single script OR multi-file project), run it locally, auto-fix (≤3 tries)	code + run results
CodeReviewer	Review each candidate before execution; rejected code must be corrected	verdict + concrete fixes
Analyst	Interpret results, judge the hypothesis, extract lessons	analysis + score
Writer	Integrate proposal + results into an academic paper	Markdown paper
Reviewer	Multiple independent reviews; low scores trigger improve→re-review	score + feedback + decision
Supervisor	Choose the next agent using evidence, failures, and configured budgets	validated action + guidance

Reliable agent handoffs and recovery

The Supervisor chooses improvements; deterministic checks enforce the workflow's prerequisites. Initial research, planning, execution, and missing analysis run directly, saving scheduling calls. Agent tag outputs are validated with one format-repair attempt. Invalid decisions raise a recoverable error instead of being interpreted as completion.

• Supervisor handoffs require concrete guidance for specialist actions. Provider-reported truncation and unclosed tags trigger a complete retry, with the output allowance growing from supervisor.max_tokens (4096) to at most supervisor.max_repair_tokens (8192). This is one bounded repair, separate from network retries. Completion metadata stays attached to each response, so concurrent agents cannot overwrite one another's truncation status.
• Current feedback comes from the latest successful run, with an explicit run_name in analysis.json. The historical best remains in best_run_name / best_score; it cannot replace the latest diagnosis. Equal quality scores prefer newer parent code, and reanalyzing an older run does not change experiment chronology. Resume refreshes legacy stale analysis views using saved per-run analyses without rerunning training or making another analysis call.
• Following AI-Scientist's per-run feedback and AgentLaboratory's execution-based reflection, the Engineer receives the latest measurements and diagnosis, and the Analyst receives the actual implementation plus earlier results. ASI-Evolve's separate historical-best tracking is retained. Agents are instructed to verify changed mechanisms with small measured checks, preserve verified repairs, and distinguish resolved bugs from untested explanations and honest negative findings. The Writer receives every successful run, with the latest diagnostics first.
• A new plan must be tested before another replan. A research pivot keeps the old experiment history but starts a new direction; old measurements and parent code cannot support the new hypothesis.
• Engineering retries receive the full plan, code, runtime/prescan errors, environment diagnosis and review feedback. Each execution attempt has its own directory, preserving diagnostics and preventing old metrics or modules from leaking into a retry. Every Python file is compiled without execution, catching errors that AST parsing alone misses (including misplaced future imports). Finite numeric result fields are checked before an experiment is considered successful.
• Code review blocks only on concrete unresolved issues. Optional observations are recorded separately as warnings. A contradictory verdict is sent back for correction; rejected code is never run just because its review budget is exhausted. The planner, engineer and reviewer share the numeric result contract: baseline_metric, proposed_metric and improvement are scalars; detailed comparisons go in metrics or improvement_details.
• Failed experiment rounds have up to execution.max_experiment_retries additional recovery rounds per idea (default 2), even with evolution disabled. Recovery reuses the last implementation and full failure feedback when the plan is unchanged. Checkpoints keep the failed rounds, retry links and review history; resume does not replenish this allowance. Failed engineering does not count toward a scientific plateau.
• Measurements are checkpointed before analysis. If the Analyst fails, resume analyzes those saved measurements instead of generating and running the experiment again. Checkpoints use atomic writes; missing fields in older checkpoints receive valid defaults, and corrupt checkpoints fail explicitly.
• Undefined secondary metrics (NaN, Infinity, -Infinity) are represented as JSON null in run summaries and checkpoints, with their paths recorded in non_finite_fields. Original experiment result files remain unchanged. Required headline metrics must still be finite; invalid headline results trigger repair rather than being accepted. The Analyst treats nulls as missing evidence.
• If execution completed before a checkpoint write was interrupted, --resume recovers the latest attempt for that pending run from its saved summary, matching result file and source files. It then analyzes the saved measurements. Incomplete/failed attempts and unrelated run directories are not treated as completed evidence.
• Changing a paper or its experimental evidence invalidates its review. Completion requires validated evidence from the current direction, analysis, a paper covering that evidence, and a review of that exact draft meeting review.min_score (unless review is disabled). A failed member of the review ensemble does not discard the valid reviews; the aggregate reports how many succeeded and the individual failures.
• Review feedback remains available after replanning, experiments, analysis and rewriting, with all new reviews retained in review_history. Resume can recover the last review from older checkpoints' state/review.json; a historical score is never treated as approval of a new draft.
• state/study_state.json records each step's requested/executed action, guidance, status and error. The Supervisor sees remaining budgets, unavailable actions, recent decisions and the latest error. An exhausted experiment budget never redirects experimental instructions to the Writer: the Supervisor must choose a feasible specialist or explicitly stop. Saved metric tables reach Supervisor, Analyst and Writer in full so existing evidence can be reanalyzed and reported. The Supervisor also sees the current paper and the exact step of its last review, so it can check which historical criticisms have been resolved. After two consecutive Writer steps without new analysis, experiments or review, another rewrite is blocked for the Supervisor to reconsider. The lessons bank preserves full history on disk and limits the injected snapshot to 12 entries / 6,000 characters.

The default step budget is 30, including setup and failed steps, with a stop after 3 consecutive agent errors or unavailable-action requests. Experiment iteration is enabled by default, allowing the initial run plus 2 Supervisor-selected follow-up rounds per direction, subject to plateau and failure limits. --iterations 0 explicitly selects a single scientific round. The review.max_review_rounds limit applies to each set of successful experimental evidence: a new successful experiment permits fresh reviews, while rewrites, replans, reanalysis, failed runs and resume do not reset the count. All work remains subject to the total step budget. These limits stop the study with a reason; they never mark an unfinished study as complete. The CLI exits with code 2 for a stopped study and 0 for successful completion.

--resume without a directory selects the most recently updated checkpoint for the same task. Step history and the total step count survive resume; raise --max-steps when that budget is exhausted. An explicit resume resets the consecutive-error counter, allowing a recovered API or environment to continue from saved artifacts. execution.run_experiments: false stops after planning, before executing generated code; enable it and resume to continue. --skip-review skips the review agent entirely.

Billing/quota exhaustion (including HTTP 402), invalid credentials, denied access and invalid requests stop immediately with a saved checkpoint instead of being retried by the client and scheduler. Connection failures, temporary rate limits and server errors still use the configured retry budget; SDK retries are disabled so they do not multiply this allowance. Restore the account balance or fix the provider configuration, then use --resume --experiment-dir <existing-study> to continue. Successful experiments awaiting analysis are displayed as run complete; analysis pending, not a zero evaluation score. Completed measurements and partial successful reviews are preserved.

Autonomous optimization harness

The experiment stage is driven by an optimization harness (aiscientist/harness.py, inspired by ASI-Evolve) that handles the control flow while the agents supply the intelligence:

• Parent selection — picks which past experiment to build on (best / greedy / ucb1).
• Autonomous decision — the Supervisor selects research / plan / experiment / analysis / paper / review / done; workflow checks validate prerequisites and completion.
• Convergence detection — stops on plateau (N rounds no improvement), repeated failures, or max rounds — so it neither quits too early nor loops forever.
• Best-so-far tracking + decision log — each run's score and selected parent are saved with its experiment record; actions, rationale and outcomes are saved in state/study_state.json. UCB1 parent visits are reconstructed from persisted parent links when the harness is recreated.

Tune it under evolution: in config.yaml (patience, max_fails, min_improve, selection).

Skill manuals for agents

Each agent is guided by an explicit role manual (in skills/<agent>/SKILL.md), styled after Microsoft ResearchStudio. The manual defines the agent's responsibilities, input/output contract, quality gates, and anti-patterns, and is injected into the agent's system prompt on every LLM call. Edit these files to tune how each agent behaves.

skills/
├── agent_researcher/SKILL.md   # literature grounding + falsifiable hypothesis
├── agent_planner/SKILL.md      # executable experiment spec
├── agent_engineer/SKILL.md     # vectorized code, runtime discipline, auto-repair
├── agent_analyst/SKILL.md      # honest result judgment + actionable lessons
├── agent_writer/SKILL.md       # data-driven, honest paper writing
└── agent_reviewer/SKILL.md     # calibrated scoring + actionable weaknesses

Configuration

All settings in config.yaml; each experiment's config.yaml overrides it.

Key	Meaning	Default
llm.provider	deepseek / openai / anthropic	deepseek
llm.model	model name	deepseek-chat
llm.base_url	OpenAI-compatible endpoint (local vLLM/Ollama/relay)	empty
execution.python_bin	interpreter used to run experiments	current env
execution.run_experiments	whether to run experiments locally	true
report.language	paper language	English
review.enabled	whether to run LLM review	true
review.min_score	below this score, trigger improve loop	5.0
evolution.enabled	allow Supervisor-selected follow-up experiments	true
review.max_review_rounds	reviews per set of successful experimental evidence	2
execution.max_code_attempts	total code generation/repair attempts per experiment	3
execution.max_code_review_rounds	allowed review-requested rewrites; every candidate is reviewed (0 disables code review)	2
execution.max_experiment_retries	extra recovery rounds for failed experiments per idea; persists across resume	2
supervisor.max_steps	total persisted step budget; 0 explicitly removes this limit	30
supervisor.max_consecutive_errors	consecutive agent errors before stopping for recovery	3
supervisor.max_tokens	initial output allowance for a concise scheduling handoff	4096
supervisor.max_repair_tokens	output ceiling for the one allowed truncation repair	8192
search.proxy	proxy for online access (e.g. Clash)	empty

FAQ

Q: "No API key configured" error Fill llm.api_key in config.yaml, or set DEEPSEEK_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY.

Q: Experiments take too long / want a quick sanity check Lower experiment.experiment_timeout and tell the task to use small synthetic data with bounded runtime.

Q: Want to use my own code/data Put data files under the experiment's runs/ dir and mention the path in the topic, or upload PDFs via the web UI.

Q: Offline? Set search.engine to local to only search uploaded PDFs. The LLM still needs API access.

Tests

python -m pip install pytest
python -X utf8 -m pytest tests -q
python -X utf8 tests/smoke_test.py   # Mock LLM + real local sklearn experiment and report artifacts

Tests use local/default configurations and mock model responses, so they make no LLM API calls and perform no literature downloads. The UTF-8 flag keeps the Windows console output portable.

Directory Layout

aiscientist_agent/
├── main.py                  # CLI entry
├── config.yaml              # global config
├── aiscientist/
│   ├── agents/              # 6 specialists + code reviewer + supervisor
│   ├── orchestrator/        # staged pipeline + resume
│   ├── tools/               # literature search / paper library / experiment runner
│   ├── utils/               # unified LLM / logging / prompts
│   └── web_app.py           # Flask web UI
├── web/templates/           # web pages
├── experiments/             # experiment workspaces
│   └── demo_noisy_label/    # sample config
└── tests/                   # tests