A co-scientist that does not pick a winner. It asks "why do you think so?", finds the evidence on each side, and names the result that would change minds.
In the lab agent on yaniguan.github.io, the co-scientist proposes the next simulation or experiment. Here it is a partner in a disagreement. It surfaces the evidence and assumptions behind each claim, then suggests the test that tells them apart. The first attempt finds the sentences behind a claim in real abstracts.
SciFact dev set (claims checked against paper abstracts; gold rationale sentences marked). For each claim and its evidence abstract, BM25 (plain Python and numpy) ranks the abstract's sentences against the claim. A hit means a gold rationale sentence is in the top k. Random sentence ranking (20 seeds) is the baseline. The script also prints three template "why" questions per claim.
pip install numpy
python demo/why_probe.pyOutput (Linux, CPU only, 2026-11-18):
SciFact dev: rationale sentence retrieval inside gold evidence abstracts
claims with evidence: 188 claim-doc pairs: 209 labels: {'SUPPORT': 138, 'CONTRADICT': 71}
sentences per abstract: mean 9.7, median 8
method recall@1 recall@3
BM25 0.493 0.789
random (20 seeds) 0.197 0.528
random std 0.024 0.029
BM25 SUPPORT 0.507 0.797 (n=138)
BM25 CONTRADICT 0.465 0.775 (n=71)
======================================================================
Claim: 1,000 genomes project enables mapping of genetic sequence variation consisting of rare variants with larger penetrance effects than common variants.
Abstract: Rare Variants Create Synthetic Genome-Wide Associations
Gold label: SUPPORT gold rationale sentences: [2, 5, 7]
Top-3 BM25 sentences:
1.*[s7] In conclusion, uncommon or rare genetic variants can easily create synthetic associations that are credited to common variants, and this possibility requires careful consideration in the interpretation and follow up of GWAS signals.
2. [s0] Genome-wide association studies (GWAS) have now identified at least 2,000 common variants that appear associated with common diseases or related traits (http://www.genome.gov/gwastudies), hundreds of which have been convincingly replicated.
3.*[s2] We propose as an alternative explanation that variants much less common than the associated one may create "synthetic associations" by occurring, stochastically, more often in association with one of the alleles at the common site versus the other allele.
Why questions:
- What evidence makes you think that 1,000 genomes project enables mapping of genetic sequence variation consisting of rare variants with larger penetrance effects than common variants?
- What are you assuming must be true for '1,000 genomes project enables mapping of genetic sequence variation consisting of rare variants with larger penetrance effects than common variants' to hold?
- What result would change your mind about '1,000 genomes project enables mapping of genetic sequence variation consisting of rare variants with larger penetrance effects than common variants'?
======================================================================
Claim: A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.
Abstract: Increased Microerythrocyte Count in Homozygous α+-Thalassaemia Contributes to Protection against Severe Malarial Anaemia
Gold label: CONTRADICT gold rationale sentences: [1, 9, 10]
Top-3 BM25 sentences:
1.*[s1] Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count.
2.*[s10] CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA.
3. [s7] This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia.
Why questions:
- What evidence makes you think that a high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects?
- What are you assuming must be true for 'A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects' to hold?
- What result would change your mind about 'A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects'?
(* = gold rationale sentence)
Plain BM25 finds a supporting or contradicting reason in the top 3 for 0.79 of claim-abstract pairs, against 0.53 for random ranking (abstracts are short, about 10 sentences). It works about equally for SUPPORT and CONTRADICT. Next: retrieve across many papers (not only the gold abstract) and turn retrieved reasons into a proposed discriminating experiment.
| Dataset | What | License |
|---|---|---|
| SciFact | 1.4k expert claims with rationale sentences | CC BY-NC 2.0 (non-commercial) |
| SciRIFF | Instruction data for reading scientific papers | Apache-2.0 (code); data see repo |
| LAB-Bench | Biology research tasks (LitQA, ProtocolQA) | CC BY-SA 4.0 |
Licenses marked non-commercial are fine for research but not for products.
Stars as of 2026-11-18.
| Repo | What | Stars | License |
|---|---|---|---|
| stanford-oval/storm | An LLM-powered knowledge curation system that researches a topic and generates a... | 31,907 | MIT |
| SakanaAI/AI-Scientist | The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery | 15,014 | custom (see repo) |
| Future-House/paper-qa | High accuracy RAG for answering questions from scientific documents with citations | 9,621 | Apache-2.0 |
| ur-whitelab/chemcrow-public | Chemcrow | 971 | MIT |
| Future-House/robin | Robin: A multi-agent system for automating scientific discovery | 760 | Apache-2.0 |
| composable-models/llm_multiagent_debate | ICML 2024: Improving Factuality and Reasoning in Language Models through Multiagent Debate | 557 | no license file |
| NoviScl/AI-Researcher | - | 415 | MIT |
| allenai/scifact | Data and models for the SciFact verification task. | 282 | custom (see repo) |
| gomesgroup/coscientist | - | 214 | custom (see repo) |
| Paper | Venue | Link |
|---|---|---|
| Accelerating scientific discovery with Co-Scientist | arXiv 2025 | arXiv:2502.18864 |
| The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery | arXiv 2024 | arXiv:2408.06292 |
| Robin: A multi-agent system for automating scientific discovery | arXiv 2025 | arXiv:2505.13400 |
| Emergent autonomous scientific research capabilities of large language models | arXiv 2023 | arXiv:2304.05332 |
| ChemCrow: Augmenting large-language models with chemistry tools | arXiv 2023 | arXiv:2304.05376 |
| AI safety via debate | arXiv 2018 | arXiv:1805.00899 |
| Improving Factuality and Reasoning in Language Models through Multiagent Debate | ICML 2024 | arXiv:2305.14325 |
| Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers | arXiv 2024 | arXiv:2409.04109 |
| Fact or Fiction: Verifying Scientific Claims | EMNLP 2020 | arXiv:2004.14974 |
- NeurIPS (main track, Datasets and Benchmarks, workshops)
- ICLR
- ACL 2026
- AI for Science workshop series (NeurIPS / ICML / ICLR)
Recent results, newest first. Date = first arXiv version; each line is a short summary of the abstract.
| Date | Paper | In one line |
|---|---|---|
| 2026-10-09 | ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence | ACED-Bench: tests whether agents choose experiments and act on causal evidence, with objective ground truth. |
| 2026-08-20 | Symposium: Trust via Auditable Records for Communities of AI Scientist Agents | Symposium: auditable records of claims, evidence and assumptions for communities of AI scientist agents. |
| 2026-05-11 | When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews | When Reviews Disagree: fine-grained contradiction analysis across scientific peer reviews. |
| 2026-02-12 | Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences | Benchmark Illusion: models with similar accuracy still disagree on 16-66% of items. |
Found a broken link or a missing dataset, repo or paper? Open an issue or a pull request.
Code in this repo: MIT. Listed datasets, repos and papers keep their own licenses.