It's time we start reviewing less

Not everyone deserves to get their paper peer-reviewed.

There I said it… It’s a controversial take, and definitely somewhat gatekeepy. But this year I’m co-chairing a “small” AI conference (The Conference on Secure and Trustworthy Machine Learning, or SaTML for short) and I’ve been radicalized.

I’ll first talk about what we’re seeing at SaTML specifically (and how we’re dealing with it). Then I’ll try to philosophize about how peer review and academic incentives might have to change.

SaTML’s slop tsunami

SaTML 2027 received about 450 submissions. Twice as many as last year. It’s a “small” conference in a “small” sub-field of AI and yet there was supposedly more than one interesting paper written per day in the past year (spoiler: there were not).

What definitely did not double is our reviewing capacity. We did increase it a lot in anticipation of the deluge coming our way. But in the end we figured that the sanest thing to do would just be to reject as many of these submissions as quickly as possible, to keep our reviewers’ last bit of sanity.

We used a few simple filters to try and flag potentially poor papers: very short papers, single authors, heavy use of the words language models love, etc. We didn’t reject any papers directly based on these signals as they tend to have quite a few false positives. But we decided to give these papers a closer look. Two or three senior committee members looked at the filtered papers (about half of the submissions) for 5-10min each. (This represents about 3 days of work in total; that’s a lot, but still a lot less than if we had given each paper a full review). If these committee members agreed a paper was “very unlikely to be accepted”, we rejected it with a short meta review.

Roughly two thirds of the papers reviewed in this initial phase were rejected. So with this one sweep (plus desk rejects), we cut about 40% of the reviewing load. Still a lot of papers left, but together with the increase of the PC we had done preemptively, it will work out “fine” (for this year at least).

A few numbers from the process that stick out:

• We got over 100 submissions with a single author (this used to be a rare occurrence…) We rejected 95% of them in initial review.
• For about 80% of the early rejects, the reason was the same: the writing was so poor we couldn’t tell what the paper was claiming (not all these were AI written, but a large fraction were).

The process “did it’s job”, but it is certainly not perfect. Specifically, since not all papers went into this initial review, the filtering introduces some unfairness. At the same time, peer review has never been fair (you get a lucky set of reviewers or you don’t) and so I personally don’t think this is a big deal. Especially since I think every single one of the early rejects was justified (my co-chair Fabio and I looked at every single one before rejecting).

Other venues are apparently starting to take similar measures. That’s great. It will keep the great illusion alive a little longer. But I think we need something bigger to change, and fast.

Peer review doesn’t work when there’s no peers

Peer review has always been hard, and it has never been especially good. It seemed to work anyway, when it stayed within a small-ish community. The people reviewing were the people submitting and everyone knew everyone. If you sent in bad work, people remembered. If a paper was genuinely good, it ended up accepted after a few tries. You reviewed “for the good of the community” because you were the community.

This was broken in our field long before AI. How can you keep people accountable when your “community” consists of thousands of people most of whom are complete strangers? I remember in 2018, Nicholas Papernot and I organized one of the first workshops on AI security. Already then we got about 100 submissions! But I knew everyone on the program committee, and most of the people who submitted. Over the years, things got bigger and bigger and I started recognizing a smaller and smaller fraction of people in the field. This year at SaTML I can put a face to about 15% of our program committee. (I’m bad with names, so the real number is probably higher.) Still, it definitely no longer feels like a community.

But hey, shouldn’t this be a good thing? More people doing science! Everything is open, no gatekeeping, everyone can participate! That’s great and all, but now that writing a paper is becoming almost free, the question is who pays? Newcomers aren’t the ones doing the reviewing, and for the most part we don’t want them to. So the community ends up shaped like South Korea’s population pyramid: more and more papers have to be paid for (the papers are the elderly in this analogy), and the reviewers (the young people) have to work more and more.

Not every paper deserves a review

We’re essentially facing a spam problem: a small amount of work (creating and submitting a bad AI-written paper) can cause a lot of downstream headaches. And so we need anti-spam filters: not every paper deserves a couple hours of reviewing by experts.

For now, some venues have mainly tried to limit the number of submissions: max 20 submissions at ICLR (lol, that’s more than I have in my entire career), max one submission for single-author newcomers, rate limits and endorsements on arXiv, etc. I don’t think this will be very effective. A submission is still cheap to produce, the incentive to produce one is still high, and there seems to be a near-endless stream of people who want to participate in our fun little game.

So I think the only option is that we start reviewing less. A cheap first-pass read should answer a single question: would anyone want to read this? Because that’s what science is really about, isn’t it? If a paper gets written in the forest and no one is around to read it, has science advanced? Not really. We see mathematicians struggling with this right now: sure we supposedly have 700 new amazing results written down in a near-incomprehensible manner using AI, but has humanity’s understanding of mathematics advanced as a result of this? Not yet: this will have to wait until people take the time to understand the results and make them digestible.

Maybe this will just be a temporary problem. Soon the models will write well and design good experiments, and the slop will go away. But then I’m not sure what we’ll still be reviewing for anyhow. If a paper took no real human effort to produce, why should it deserve real human effort to check? Unless the humans are genuinely interested, and then they’ll go and read it and understand it, as is now happening with OpenAI’s math results.

What should we value in such a world?

Maybe we should reward reads, not writes

Our current incentives reward writing papers and getting them cited. What if we rewarded reads instead: how many people actually read your work and learned something from it?

I don’t know how we’d measure that exactly, but it seems it could have some nice benefits. It would favor clear writing over flag planting. It would shrink the payoff for rushing out a half-finished result, since the first bad version of an idea is rarely the one people end up reading. And it could let good explanation count in whatever form it takes: a paper, a blog post, a lecture, a video, etc.

You might say citations are already a proxy for reads. But in CS at least, they really aren’t. Count the papers that open with “deep learning has been very successful [Hinton, Bengio, LeCun]” or “LLMs are powerful [Vaswani et al.]”. You think every author actually read those papers? The latter is clearly one of the most influential papers of the century. But if a student asks me how a transformer works, I’d tell them to go read “The Illustrated Transformer” instead.

Now I don’t think we should reward only “reads”. This also has bad incentives: clickbait, people who explain someone else’s result without giving credit, etc. What we need is probably a mix: some credit for finding new results, and some credit for making them understood by others. As AI gets better and better at the former, we might shift more towards rewarding humans for doing the latter.

Until AI beats us at both.

Then we can just all meet up at conferences where the AIs give talks to other AIs, while the humans drink the bad coffee in the hallway.