Top
Best
New

Posted by stefanpie 3 days ago

Asking authors about their own papers(medium.com)
149 points | 79 commentspage 2
jszymborski 18 hours ago|
FYI in case the author is reading, https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf is a dead link.

EDIT: I found a live link on arxiv https://arxiv.org/html/2609.20481v1

encyclopediai 18 hours ago|
In June 2026 I proposed a CAPTCHA for scientific publications

https://chorasimilarity.wordpress.com/2026/06/13/a-captcha-f...

At the moment this was seen as a tongue in cheek proposal.

doc_ick 18 hours ago||
I’d agree it’d be a funny proposal, wouldn’t have worked back then but funny.
encyclopediai 17 hours ago||
Thanks. IMO the most fun is in the CAPTCHA, which turns on its head the Turing test.

But it goes even further than their greCAPTCHA and it solves their consumed time problem.

Indeed, in their proposal they have a human bottleneck, but in the june 2026 proposal is suggested that one could use an AI to generate the results without the knowledge of the submitted article.

If the AI can generate a pretty close result, with the article fed gradually as a prompt, then reject.

And even further, that it might be not even a need to publish anymore.

Just use the article for training and make a public database with some numbers about the successful researcher, where we see an influence score (how many times an idea from an accepted article are used by other accepted articles), a publication score (how many articles the author had).

I wonder if the reality will be more or less surprising, my bet is on "more".

cgio 3 hours ago||
Curation is the new skill. The dismissal of a result on the basis of authorship is one of the reasons blind reviews are there. It sounds like we need more scientists. In al seriousness, this is a skill not just for science. Even at work, the amount of slop is rising exponentially and people are half-treating the symptom with its source, using AI to summarise. We will find our ways eventually.
Calazon 19 hours ago||
I wonder how the ratios would change for papers at different parts of the review process. For what fraction of published papers are the authors unable to answer basic questions about them?
N_Lens 19 hours ago||
I think authenticity and trust will command a (larger) premium in this new age of slop.

The article highlights how only one out of ten paper’s authors were able to answer questions thoroughly and at a high level. This indicates an overwhelming percentage of authors are slopping up their work with AI and submitting it without even reading it.

SoftTalker 18 hours ago|
No doubt this is happening, but I wonder how many authors of papers "slated for desk rejection" 10 years ago could answer questions about their papers? We'd need that comparison to understand if this is a new problem or if AI is just a new source of content that the authors of poorly-written papers are using.
brianpan 17 hours ago||
Evidence-based assertions are a good thing, but some things are so obvious that a "comparison" or "research" is not needed. This is already an obvious problem in so many places from high schoolers turning in assignments they don't understand, coders submitting code changes they don't understand, blog posts, and certainly to scientific papers.
SoftTalker 9 hours ago||
Yeah sadly you're probably right.
logicallee 5 hours ago||
I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have performed an experiment. At the time that I answered the journal requests, AI could hallucinate fake reports, but agents weren't powerful enough to run the experiments yet.

These days agents are able to really perform genuine experiments and write up the results. A prompt like this: "You'll work autonomously end to end to select a research task that meaningfully advances the state of the art in AI, is clearly defined and worth performing, that people would be interested in reading, and that you can perform on this hardware" (insert details) " in a week. Carefully log your steps so that your results can be replicated. Then, do a research review and write your paper about it up with correct, cited references. You must check all of your citations. Look up current lists of "Claudisms", (such as use of the word "genuinely", or "load-bearing"), and remove them from your writeup. After writing your writeup, edit it and pare it down, remove anything unnecessary, keep it fast paced and interesting. Also, try to tell a story, be engaging in your writeup. Don't use violent metaphors, remove references to killing, strangulation, etc. Your writeup should be ready to publish and accurately reflect a real experiment with a meaningful result that advances the state of the art and contributes to understanding. Be concise and focus on why it matters."

Okay, so there's the prompt. You can give it to any AI and have a journal-ready publication in a week. I guess you can ask it to add charts and stuff, if you want to be fancy.

If I gave my agent the above prompt, would I be one of the authors? Maybe it's fair to say I guided, facilitated, elicited, or advised it. But it's clear that the AI would be the one that is actually selecting and running the experiment and writing up the results.

Someone could probably get a publication without even reading the paper they wrote their name on. Their only contribution might be editing their name into the PDF.

kukkeliskuu 4 hours ago|
It can be turned around as well, for validating existing research, creating a bot that checks papers against all the well-known logical fallacies, issues with statistical methods, checks the images etc.
j16sdiz 3 hours ago||
I hate to say this, but a researcher in biology ain't a statistician or logician. Many math paper hand waves over many parts of a proof.. yet both of those can be and have been useful in practice.
kukkeliskuu 3 hours ago||
That may well be the case. My suspicion is that many research papers would not pass such a quality review. I am not saying anything about their usefulness, though.

You are not suggesting it, but as some might, I want to emphasize that I think it is, really, really, really bad idea to argue for limiting exposure of the bad practices because the papers may be useful even when not following good practices.

The science can work only when we can trust that the foundation it has been built on, including previous research, is solid.

The trust is partly based on the fact that science is self-correcting. As we know from charge of electron measurements, the existing narrative can work against the self-correction, even when everybody is trying to be as truthful as possible.

My impression is that to a large extend, on many fields, publication numbers (and money) have become so important that self-correction process may have become broken.

muh_gradle 4 hours ago||
completely necessary.
dovholuknf 16 hours ago||
Sounds like I shouldn't be so hard on AI when it hallucinates things based on this data? :)
trombuance 2 hours ago|
[dead]