Part of the problem here is how people use these models and expectations are formed.

If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.

I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.

It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.

If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.

It does seem like they are much better in the small as opposed to the large. I agree on the judgement being questionable, and every report on making large swarms seems to suggest it's relying on spectacularly diminishing returns on the performance of a single agent, especially a single agent with a human in the loop.

Innovation comes out of the human experimenting with solutions and learning new things that the llm knows.

It’s a dance, not an orchestration.

It does not look like the majority here realize that your coding AI + harness is very different from a standard LLM's output and capability. The majority of the comments here are referencing their coding harness and attributing that class of application behaviors and nature to a standard LLM. They are very different. Your coding agent and your desktop chatgpt app have more harness enabled capabilities unlocking aspects of the LLM that create interactions that a standard LLM is not capable, and would only engage with considered prompting and context tuning.

> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.

I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.

Honestly at first I was really impressed by AI progress because most of what I do day to day falls squarely into the training distribution - and the models seem to excel at that kind of work, literally in certain areas they are more competent than me.

But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.

For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.

They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.

I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".

After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:

-- It's possible for code in a bundle to execute without the bundle being activated.

We also have a new state (another kind of fact)

-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.

So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:

"which is definitely ACTIVE (since your code is running from it),"

Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.

Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.

For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.

It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.

It blows my mind how bad the standard of basic logical arguments are on lesswrong, which is supposed to be a rationalist website, and has the sort of smug name that would challenge me to hold myself to a higher standard than this.

Here we have an article which amounts to “I have used AI to do some unimpressive research. I have read some unimpressive AI-assisted research. So AIs are unimpressive at research.”

Extrapolate from a tiny sample and establish a universal conclusion. Then look yourself in the mirror and congratulate yourself for how much “less wrong” you are than all the irrational schmucks out there.

> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.

Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)

AGI right around the corner!