AI as Area Chair: What Science Would AI Send to ICML?
We had language models act as an area chair (AC) by ranking 6,617 papers accepted to ICML 2026. Then, we compared this ranking against the decisions made by human ACs (which papers received orals and spotlights) and found notable differences.
The five papers that AI ACs ranked highest out of all 6,617 papers. The ranking comes from our six-stage pipeline described below. Note that we didn't show the models the author names, human reviews, or acceptance decisions during this ranking procedure.
Motivation
AI systems, referred to as "research agents" or "AI scientists", now take part in several steps of research, from searching for better algorithms to drafting entire papers (autoresearch). As more of that process is automated, we want to have a better understanding of what kind of research these systems "prefer", how their preferences differ from those of humans, and the reasons behind these preferences. In other words, we want to understand whether their research taste is similar to or different from that of humans!
However, research taste is hard to study in most cases, because it is not well documented (e.g., humans rarely leave artifacts like "I prefer this kind of research over others") and it is hard to quantify (e.g., how similar or different two researchers' tastes are).
Conference review is one of the few places where research is compared explicitly and the outcome is written down: papers are scored, and a small share are chosen as orals, spotlights, or award papers. Just as a human AC reads a set of papers, weighs them against each other, and decides which should receive orals and spotlights, we had a panel of AI models rank every paper accepted to ICML 2026. The result is a single ranking of all 6,617 papers, which we then compared with human ACs' decisions.
How we built the ranking
Having a frontier model compare all 6,617 papers directly would be very expensive, so we adopted a multi-stage pipeline. In the early stages, cheaper models narrow the pool, and in the later stages, more capable models spend more effort on fewer papers. The diagram shows the six stages and how many papers enter each one.
-
Stage 01 6,617 papers
Anonymize and label. We keep the first nine pages of each paper (the main text) and remove author names, affiliations, acknowledgements, and similar identifying text. Gemini 3.1 Flash Lite then labels each paper's contribution type, such as algorithm, theory, benchmark, or application. Later stages use the labels to mix types within each batch and to keep every type represented at each cut. No paper is removed here.
-
Stage 02 6,617 → 1,442
First-pass ranking. Four inexpensive models (Nemotron 3 Ultra, Gemini 3.5 Flash Lite, GPT-5.6 Luna, and Grok 4.3) rank papers in batches of eight. Every paper appears in two different, randomly drawn batches, so each model compares it against two different sets of seven other papers. That gives eight rankings per paper. About the top fifth (1,442 papers) moves on. We kept this cut generous because a paper dropped here cannot come back.
-
Stage 03 1,442 → 250
Second-pass ranking. GPT-5.6 Terra and Claude Sonnet 5 rank the remaining papers with the same batch design, and we average their normalized ranks. The finalists are the top of this ranking plus the leading papers of each contribution type and papers on which the two models, or the two stages, disagreed. That gives 250 finalists.
-
Stage 04 250 → 172
Full-paper judgments. GPT-5.6 Sol, Claude Fable 5, and Gemini 3.1 Pro each read every finalist as a PDF, figures and tables included, and write a structured judgment with scores on six axes, the strongest case for the paper's impact, and its main risk (750 judgments in all). We order the finalists two ways, by averaging the three models' scores and by having GPT-5.6 Sol rank all 250 from the judgments, and keep the union of the two top-150 lists, 172 papers. From here on, papers are compared through these written judgments instead of the PDFs.
-
Stage 05 172 → 60
Swiss-system rounds. GPT-5.6 Sol compares papers two at a time, reading both papers' judgments and picking the one it expects to have more impact. As in a Swiss-system chess tournament, each round pairs papers with similar records so far. After 10 rounds (860 comparisons), the top 60 advance.
-
Stage 06 60 papers
Round robin. GPT-5.6 Sol compares every pair of the 60 remaining papers, 1,770 comparisons in all, and orders them by number of wins. This is the final ranking shown in Section 2.
What the models could see. Note that during all the stages, the AI models had no access to web search, retrieval, or other tools, and the prompts told them not to use outside knowledge about the paper or its authors. We also never included the reviews, scores, acceptance decisions, or presentation tiers (oral/spotlight/poster) in their inputs. A model could still have recognized a paper from its training data (memorization), which we return to in Section 5.
The AI's top ten
These are the first ten papers in the final round robin (Stage 06). Under each title is a one-line summary of the models' reasoning, followed by what humans (at ICML 2026) decided: the paper's presentation tier and its mean reviewer score. Four of the ten were orals or spotlights at ICML.
The AI's ranking and human's ranking on ICML papers agree only weakly
Finding 1
The rank correlation with ICML is positive but small
ICML records two judgments of each accepted paper: its presentation tier (a small number of orals, a larger group of spotlights, and posters for the rest) and the mean score from its reviewers. We compare the AI ranking with each using Kendall's τ, which looks at every pair of papers and asks whether the AI and ICML put the pair in the same order. A value of 0 means no relationship and 1 means identical orderings. Against both of ICML's records the AI ranking gets 0.08.
The AI ranking orders all 6,341 main-track papers, so its top tenth is 634 papers. The dashed line (10%) is a random baseline of AI ACs ordering the papers randomly. As shown in the figure, P(human="oral" | AI="top 10%") is 25.2% and P(human AC="spotlight" | AI AC="top 10%") is 16.7%. Hence, the relationship is real but weak: three quarters of ICML's orals are outside the AI's top tenth.
This figure does the same for reviewer scores, which are finer-grained than the three tiers (spotlight/oral/poster). The vertical axis is the average position in the AI ranking as a percentile, where 100 is the top. Papers with a mean reviewer score of 3.25 (between weak reject and weak accept) sit at the 44th percentile on average, and papers with 5.25 (just above accept) sit at the 61st. Better-reviewed papers do rank higher with the AI, but going from the lowest mean scores to the highest moves a paper less than 18 percentile points on average.
Finding 2
AI and humans favor different kinds of papers
A low correlation could come from noise as much as from different preferences, so we also compare what each side picked. We take the 536 papers ICML chose as orals or spotlights (top 10% papers) and the AI's top 536, and compare their contribution types using the Stage 01 labels. Relative to ICML's picks, the AI's contain more core ML algorithms, more safety and evaluation work, and more infrastructure. ICML's contain more theory, more analysis and position papers, and many more application papers (11% against 1.5%). Both sets have fewer application papers than the main track as a whole (19%), but the AI's set almost drops them.
Finding 3
The models agree with each other far more than with the reviewers
The four Stage 02 models come from four different organizations. Across the 6,341 main-track papers, their rankings correlate with each other at 0.62 on average (Spearman's ρ), and the least similar pair still reaches 0.57. Each of them correlates with the mean reviewer score at between 0.07 and 0.10.
Because the models ranked papers in the same batches of eight, part of their agreement could come from sharing a batch layout rather than a judgment. To check, we look inside single batches, where every model saw exactly the same eight papers, and compare each model's order with the other models' orders and with the reviewer scores of those eight papers. Within a batch, models from different companies agree at 0.43 on average, and models agree with the reviewers at 0.07, so the gap remains when the context is identical. The agreement is also close to its ceiling. A model's two rankings of the same papers, made in two different batches, correlate at 0.51, and its ranking correlates with another company's model at 0.47.
Why would models from different companies agree so much more with each other than with reviewers? The models were trained on much of the same text and tuned in similar ways, so they may share a view of what good ML research looks like. They also received the same prompt, which pulls their rankings together. And the reviewer score is a noisy target. Each paper is scored by three or four people with different backgrounds and priorities, and when two independent committees reviewed the same NeurIPS 2021 submissions, they disagreed on 23% of accept/reject decisions (Beygelzimer et al., 2023). Within accepted papers, reviewer scores also span a narrow range, which makes any correlation with them harder to detect. Still, the pattern is informative: human reviewers disagree with each other in many directions, while the models disagree with the reviewers in largely the same direction.
The AI ranks on predicted impact; reviewer scores track novelty
Finding 4
AI ACs care most about "prediction of impact" and this doesn't correlate to the human reviewer scores
In Stage 04, the three frontier models scored 248 papers on six axes, along with an overall score. Also, we have the overall score given by human reviewers. With these, we measure two things: (1) the correlation between AI ACs' score on each axis and the overall score given by AI AC, and (2) the correlation between AI ACs' score on each axis and the overall score given by human reviewers. "Prediction of field impact" correlates most with the AI's overall score (0.80) and least with the human reviewer score (0.06, with a 95% interval that includes zero). Interestingly, "novelty of the method" shows the opposite trend: it correlates most with the reviewer score (0.24) and ranks fifth of six on the AI side.
The 248 finalists are a small slice of the corpus that the AI had already filtered, so we also checked whether the same pattern holds across all 6,341 main-track papers. The Stage 02 models wrote short reasons for and against each of these papers, and we matched the reasons against keyword patterns for six recurring arguments. The most common objection is that a result may not hold beyond the setting where it was shown, that is, that its impact will stay local. At least one model raised it for 74% of papers. The more models raise it, the lower the paper ranks: its average position falls from the 53rd percentile when no model raises it to the 43rd when all four do. It does not track ICML's decisions, though. Three or more models raised it for 22.0% of orals, 15.1% of spotlights, and 18.4% of posters. The other five arguments show the same pattern. Each is more strongly related to the AI ranking than to the reviewer score, whether it counts for a paper or against it.
Six papers where the AI and ICML disagreed most
Across the main track, 531 papers that ICML made posters reached the AI's top tenth, and 35 papers that ICML made orals or spotlights fell into its bottom tenth. The table shows the three most extreme cases in each direction, with each paper's position in the AI ranking of all 6,341 main-track papers and one sentence from the models' reasoning.
In both directions the reasons are about reach. The three papers the AI ranked high are credited with general theory, broad empirical claims, and consequences for the field. The three it ranked low are faulted for combining existing techniques, for not transferring beyond one family of benchmarks, and for depending on datasets built for the paper. Another of the eight judgments of PhenoBrain says its impact is "primarily in clinical neuroscience, not general ML", and ICML chose it as an oral.
Limitations
This is a pilot study on one conference, and the comparison has weaknesses that readers should keep in mind. We were not trying to build a replacement for area chairs, and the ranking should not be used as one.
- The models may have recognized some papers. The models had no web search, author names were removed, and the prompts told them not to use outside knowledge. Even so, papers that circulated widely as preprints, including our top-ranked one, may be in the models' training data together with the discussion around them. We did not test for this. A natural next step is to control for how well known a paper was before the conference, for example through social media attention or the prominence of its lead authors, and check whether the ranking changes.
- The human side is not ground truth. Presentation tier and reviewer score are different signals, and a ranking can move toward one while moving away from the other. Both are noisy: reviewers of the same paper often disagree, and a few weak reviews can decide a paper's tier. Readers who trust conference reviews less should read our agreement numbers as a comparison between two imperfect rankings, not as a measure of the AI's accuracy.
- The ranking is sensitive to presentation order. In Stage 02, three of the four models favored whichever of two papers came earlier in a batch. Correcting for this moves the average paper 230 places and keeps only 23 of the original top 50, which is why Stage 02 advanced papers from both the raw and the corrected ranking.
- The models may rely on shortcuts. A ranking can line up with outcomes for reasons unrelated to research quality, such as subfield, writing style, or formatting. We have not checked which surface features the models respond to.
- The ranking is a diagnostic, not a target. If papers were optimized against a judge like ours, their scores would likely rise without the research getting better, as Goodhart's law predicts, and recent work shows that AI reviewers can be swayed by presentation-only edits. We have not run this test on our own pipeline.
- Accepted papers only, at one moment. We ranked papers ICML had already accepted, so we cannot say anything about accept or reject decisions. The results also depend on these particular models and should be rerun as models and conferences change.
- Judging finished papers is one kind of taste. We studied how models evaluate completed work against shared criteria. Choosing what to work on, which prior work to build on, and when to change direction happen earlier and may call on different skills, so doing well at one does not imply doing well at the other.
Our position: research needs varied taste
As research agents take on more of the work of choosing and judging research, our main concern is that they will all apply roughly the same taste, whatever its quality.
A research field covers a lot of ground because its members choose problems for unrelated reasons: where they trained, what data or equipment they have, who they work with, and sometimes a stubborn belief in an idea others dismissed. Most of these bets do not pay off, and it is hard to tell in advance which ones will. Having many independent bets is how a field finds the ones that do.
Our results suggest that current models do not supply that kind of independence. Four models from four companies ranked the same papers much more like each other than like the reviewers, and a model's ranking agreed with another company's model almost as well as with its own second attempt. If many researchers ask models what to work on or which papers matter, their choices become correlated even when each answer is reasonable on its own. This is the concern behind work on algorithmic monoculture (Kleinberg and Raghavan, 2021; Bommasani et al., 2022) and on monocultures in AI-assisted science (Messeri and Crockett, 2024). It is also close to the case for pluralistic alignment (Sorensen et al., 2024), which argues that models should represent the range of views people hold rather than a single one.
The kind of taste the models share matters as well. Near the top of our ranking, the models favored papers that change how we understand a prominent method or metric within the current framing of a problem. Spurious Rewards, ranked first, shows that a widely used RL recipe can raise math benchmark scores for some models even with random rewards. This is close to what Kuhn called normal science, and since most papers are normal science, it is a reasonable thing to reward. But conferences also exist to surface other kinds of contribution, such as a new framing of a problem or a result that matters to a small community, and a single ranking built on forecast impact will tend to miss them.
What could help? On the model side, instead of having a few commercial models debate each other, one could train or condition smaller models on different slices of the literature and different research histories, so that they disagree for reasons closer to the ones people have, and then study how they interact. Whether that produces varied taste or only varied noise is an open question that we think is worth testing. On the human side, researchers who work on what the models rank low, such as results that may not generalize, problems that matter to a small community, or payoffs that take years, supply something the models currently do not.
Finally, we measured taste in evaluation, where finished papers are judged against shared criteria. Conference review already combines several reviewers per paper, partly because evaluation tastes differ. Taste in formation, meaning which problem to pursue, which prior work to build on, and when to pivot, comes before any results exist and probably matters more for where a field goes. We do not know whether models that agree in evaluation also agree in formation, and that is the question we most want to answer next. To make follow-up work easier, we are releasing the full ranking, the intermediate judgments, and the code.