This matches my experience. Just Talk To It is the best method for working with LLMs if you're an expert.

I've seen this at work (as eng manager/lead/principal/whoevenknowsanymore) – all the big APIs give you stats. We see how much people burn in tokens and we know how much output they produce. There is a pretty strong inverse correlation between token burn and output.

The more tokens people burn, the less likely they are to produce a good outcome.

I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?

The interesting thing is, most messages were ending with just one question of his. Examples:

- ..Does this polynomial map have any symmetry or other structure that makes this cancelation less miraculous?

- Given this structure can you see the non injectivity in a transparent way?

- ..But why is the jacobian from x u r to P Q R just a monomial?

- ..Is there a general theory of such twisted jacobians and do you have any sense why those particular dilation weights were used?

- Given this weight structure, why exactly is x given by a cubic equation from P,Q,R?

Also, looks like Terence Tao was doing lot of work and asking LLM to verify. This is inverse of the LLM trend, where LLM does the work and humans verify.

A good moment to remind everyone that if we took the promise for granted, that AI will in fact prevail and prompting is the one skill that will rule them all... we'll lose all domain experts in one generation.

It's less of "signaling expertise" and more about actually having said "expertise".

In my experience with LLMs it's not uncommon to be having a deep conversation about making pasta, only to be told, after asking for a sample recipe, to get a bucket of paint and a bag of concrete. Of course these hallucinations are way more subtle and easy to miss for someone who doesn't have deep domain knowledge.

I've experienced this firsthand and 100% agree.

The more cracked you are in a domain, the more you can squeeze out of an LLM. If you already know what "good" looks like, you can steer it, call out its BS, and iterate way faster than someone who's using it to learn the domain itself.

Agree with this. LLMs multiply the human user's ability. More ability, more impact!

This has been my experience. I’ve been working on an app (highly successfully) since February, with the help of an LLM (ChatGPT). It has not been used as an author. Rather, it’s been a “coding partner.” I’ve been the one that has submitted the work to VCS, and run the tests.

I’ve found the most utility in having it do “small stuff that I could do, myself, but it’s faster to have the LLM do it,” and in analyzing intractable bugs, like memory and threading problems. It’s really good at analyzing a bunch of code, and seeing a small typo that results in something like a strong reference.

In both these cases, my own expertise is vital. I’m asking it to act as a consultant; to give me advice and material to be integrated into a whole that I am architecting.

I guess part of it, is that I haven’t been able to completely “give in,” and wholly trust the LLM, like I hear many people do (profitably, I guess). I’m used to having my sleeves rolled up, and my hands in the dough. Catching some pretty severe mistakes, from time to time, has reinforced this perception, on my part. I wouldn’t catch these, if I didn’t know what I was doing.

I haven’t experienced this - or maybe the training data for aviation is limited. But ask a an llm for aircraft performance / flight planning data and it’s scary how bad the advice /feed back is.

LLMs are language models. So much of this can be reduced to a simple heuristic: If you can't think clearly, LLMs will not help you. If you don't know what you're asking for or how to express it precisely, it really should not be a surprise that the output is garbage. Failures of LLMs are more often failures of our own brain to consider the problem clearly. That's harder to admit than just blaming the AI.

It's important to note though, from the perspective of the LLM's objective function, that this is not a failure at all! LLMs are designed to match patterns. You give a jumbled mess of incoherent ideas, it will faithfully reproduce a token stream of incoherent ideas. It's only when the model output is subjected to the real world that it fails.

At work we call this implicit steering. To use webdev metaphor, if a non-technical person describes making a web page with a big block at the top and some things to click on and then my pictures below that, that will eventually get somewhere. Meanwhile, if you know industry jargon, you might describe a hero, with call to action buttons, and then below a 3x3 grid of images of my portfolio photos—that's likely going to generate something entirely different and likely richer. It can assume things about you (it doesn't think), it can ask you specific questions a web personal might know, it can infer domain context that is otherwise omitted with a basic conversation. Everyone wants to capitalize on corporate vibe coding but the tech literacy is hardly there, let alone more advanced topics.