Posted by raahelb 6 hours ago
Caveating I'm not a biologist, but my understanding of the way this kind of thing works right now is a basic three-step process:
1) Find molecules and DNA/RNA sequences in the wild and catalog them.
2) Discover interesting subsequences among these.
3) Figure out whether any useful applications can come from what was discovered.
All three of these generally take a long time. Systematic automatic analysis of known databases speeds up and removes some of the luck from 2. But 1 and 3 are still long poles. 1 has the further issue that we usually discover these in existing organisms. I recall much of the outcry over tropical deforestation back in the 90s and replacing of rainforests with palm oil monoculture today is that the vast majority of terrestrial biodiversity is found in rainforests, and destroying them at industrial scale risks losing potentially useful molecules forever. 3 has the problem that you need to conduct physical experiments, and are limited by the speed of biochemical reactions no matter what and by the speed at which human subjects can be found and ethically experimented on assuming we care about being ethical.
A lot of good can come of this, but I don't see a path to singularity here, assuming we're talking the original Kurzweil meaning there of all technological progress that will ever happen all happening at once. Data collection and experimentation on living subjects, human or not, can only happen so fast, regardless of automation. It's not computational. Whenever you have to interface with the real world, you're now working at the speed of the real world, not the speed of electricity. CRISPR was discovered in 1987 and first used to edit a gene sequence in a human zygote in 2015. I'm sure there are plenty of ways to make the candidate discovery to human application step not take three decades, but it's never going to be three months, either.
To me this strikes me as an incremental discovery that would have taken someone with time, interest, and expertise to make before. It could have cool applications or it could just be interesting biology. Molecular biology has progressed through many years and many rounds of automation and new tools, but the problems are still hard. This just strikes me as one more way we may be able to speed up one part of the process.
> All of the lab work is performed by human scientists.
A lot of molecular biology is noticing something that you can't explain or that seems weird and might be interesting. Once it's noticed the followup is often fairly straightforward and it either pans out or it doesn't. The exciting/scary/unlikely part is that the LLM on its own recognized something as being important to follow up.
From my skim of the paper, the work could only be done by someone with a pretty good understanding of the biology and an extremely good understanding of how to use LLMs and agents. LLMs are not going to take over biology yet.
I don't think there can be a coherent definition of RSI unless people lay out their theory for how intelligence scales. LLM-assisted coding is great but respectfully optimizing pytorch features or whatever is not gonna lead to exponential improvements. That approach to scaling diminished years ago, leading all the labs to switch to reasoning.
Now it seems reasoning is also yielding diminishing returns, so all the labs are pivoting to specializing in particular fields like math / infosec / biology. They're improving due to accessing new proprietary training data and doing RL with human experts. Again I don't really see any amount of "AI research interns" leading to an exponential improvement to this strategy, they're not the bottleneck in the first place.
Is the diminishing returns in the room with us?
>so all the labs are pivoting to specializing in particular fields like math / infosec / biology.
They're not pivoting to anything. The goal has always been creating a machine that could automate all or nearly all human work. They're just coming along on that mission.
As for RSI...I think the term is a bit odd in the modern context. It was created at a time when conventional wisdom was that generally intelligent machines would be these logic automatons that could "alter their own code". Instead we have massive neural networks that take months to train.
In this paradigm, the ways a LLM could "improve itself" would be altering its own weights directly or creating and training better, vastly more efficient architectures for the next generation of models.
The former is probably not happening but the latter is possible.
This is why I'm trying so hard to drill down on the theory of scaling, and not just talk about improvement in general, hand-wavy terms. If the bottleneck of current scaling strategies is training data, or something fundamental about the model architecture, then just throwing more harnessed chatbots at it won't lead to an exponential increase in performance.
Now you could argue that the AI we have now will help us find that change in architecture, and I would agree. But that means we're firmly outside the singularity for the time being, and what people are in fact talking about is a hypothetical.
That's not quite right. They are still scaling model size and have had several new base pre-trains, just nothing so big as 4.5 (as far as we're aware). o1/4o has not been the base for some time now.
Data is obviously a bottleneck for some regimes and LLMs will have to get their hands dirty experimenting but it doesn't look like an insurmountable wall either.
Not true. On the contrary, LLMs are developing faster than predicted. They were expected to solve a Millennium Prize by 2030... and here we are in 2026. Release cycles are getting faster. Just compare the most recent GPT or Claude with what they were an year ago.
> How is this different from arguing that Microsoft Clippy was RSI?
We can argue about semantics, but that's not really the point. The point is that what started now - which no doubt is in its infancy - will result in full autonomy quite soon (they project an year or so), with the risk of RSI causing agent development to slip (long term) outside human cognitive control/capacity.
I feel like I laid out several cases where other things were the limiting factor on improvement and more agents wouldn't have helped, and I didn't get a response to those cases.
What "they project" (the labs) is of minor interest to me. Aside from their incentives and track record of lying, in recent months they are laying out a story that is pretty much just the plot of Terminator, and directly referencing rationalist beliefs that were published long before LLMs even existed.
With the level of compute they have they aren't stuck with frozen models like you are.