LLMs can really only mostly do trivial things still, they're always going to do very bad work outside of what your average web developer does day-to-day, and even those things aren't a slam dunk in many cases.

Instead of "pointing it" at docs, you need to paste the docs into context. Otherwise it will skim small parts by searching. Of course if you're using an obscure tool you need to supply more context.

Xhigh can also perform worse than High - more frequent compaction, and "overthinking".

I’ve noticed the models still can’t complete complex tasks

Such as:

Adding fine curl noise to a volumetric smoke shader

Fixing an issue with entity interpolation in an entity/snapshot netcode

Find some rendering bugs related to lightmaps not loading in particular cases, and it actually introduced this bug.

Just basic stuff.

While I’ve had tremendous success with Golang projects and Typescript Web Apps, when I tried to use Metal Mesh Shaders in January, both Codex and Claude both had issues getting it right.

That sort of GPU code has a lot of concepts and machinery, it’s not just a syntax to express, and everything has to be just right or you will get a blank screen. I also use them differently than most examples; I use it for data viz (turning data into meshes) and most samples are about level of detail. So a double whammy.

But once I pointed either LLM at my own previous work — the code from months of my prior personal exploration and battles for understanding, then they both worked much better. Not great, but we could make progress.

I also needed to make more mini-harnesses / scaffolds for it to work through; in other words isolating its focus, kind of like test-driven development.

It works somewhat well with trivial things. That's where most of these success stories are coming from.

My impression is that it always comes down to how well what you’re trying to do pattern-matches the training set.

Nah, it only lives up to the hype for crud apps and web ui. As soon as you stop doing webshit it becomes way less useful.

(Don’t get mad at me, I’m a webshit developer)

" or if it just can't solve everything yet."

Obviously it cannot. But if you give the AI enough hints, clear spec, clear documentation and remove all distracting information, it can solve most problems.

Most of the folks are building CRUD apps with AI and that works fine.

What you're doing is more specialized and these models are useless there. It's not intelligence.

Another NFT/Crypto era is upon us so no you're not holding it wrong.

Enemy centered decision making

Cannot you use Codex (which is open source, unlike Claude Code) with Claude, even via Amazon Bedrock?

I'm with you on the ethical part, but everything is a spectrum. All the AI leadership are some shade of evil. There's no way the product would be effective if they weren't. I don't like that Sam Altman is a lunatic, but frankly they all are. I also recognize that these are massive companies filled with non shitty engineers who are actually responsible for a lot of the magic. Conflating one charlatan with the rest of it is a tragedy of nuance.

[flagged]

You'd think they would have dashboards for all of this stuff, to easily notice any change in metrics and be able to track down which release was responsible for it.

can a druggie stop using when the quality is too poor? I get your analogy, but it doesn't apply here