Posted by robin_reala 2 days ago
If you're looking for advice in a situation where you are not sure of what the correct choices are, you will find LLM chat AI to go in circles. It says one thing. Something is dodgy about it, so you raise a tentative objection (as a non-expert). The thing does a "you are completely right, I apologize" about face and then says something different, and things have begun to slide into uncertainty.
That might not exactly be sycophancy, but it's basically the same thing: producing responses that are reflection of what is in the chat, rather than any real shit.
Pick any topic where people disagree. It could be an entirely technical topic in which engineers have settled the questions, and the only contrarians are crackpots. The problem is that the crackpots are out there writing, and this is snarfed into the training data. Crackpots use certain ways of talking about certain subjects. If you use similar vocabulary and concepts that align with some crackpot theory, the AI simply starts predicting tokens according to that, and you are now in crackpot land: what you are saying is validated using the crackpot terms. Next, write in a way that reintroduces rigidity: proper terminology and correct concepts, and, whoa, the AI is an engineer again, contradicting the previous crackpot shit.
One key point is that - “pick any topic where people disagree” - most knowledge-seeking I perform via a chat bot is to get a sense of what most people generally agree upon. A quick poll of the zeitgeist of the corpus it was trained on. What used to take me 10-15 minutes on Google now takes me 2-3 minutes with Gemini.
That holds true for Claude code too - I’m relying upon the fact that other people have already solved my problem, or have at least solved the components I need to tie together into a solution for my problem - I’m just fast forwarding through the process of digging them up myself and making sure they’re interoperable. The lion’s share of the tasks I set forth for an LLM is just ‘go find me something like this.’
How some people get from that, to weeks of AI-triggered mania, I simply do not understand.
No, the problem is the chatbot pretends to be an AI answer engine.
Recognise it is simply a zero-intelligence search engine, and you'll not be surprised at all to get crackpot writings from the web.
> write in a way that reintroduces rigidity
... and do not be surpised to get rigid crackpot writings from the web.
I think that this highlights the problem: it's not even a proper search engine. If it were, it would find existing material and link to it. The chatbot instead re-elaborates the message. This is different enough to have legal consequences (see the recent German ruling).
By squashing everything together, one loses the ability to evaluate sources, including their credibility and ulterior motives. With a search engine, I can tell if I am getting climate information from the National Snow and Ice Data Center, from an oil trading consortium, or from a random blog.
The parent poster's suggestion is trying to work around the selection of sources, but such an attempt is fighting against the questionable design of the whole thing.
I don't think it's currently solvable, as selling those models as oracles is an essential component of their business plan.
No, you wouldn't because you're a fucking machine.
"The goal of Search has always been simple: to help you ask anything on your mind"
https://blog.google/products-and-platforms/products/search/s...
And they are also perverting search results, if this is to be believed:
"The Google AI Mode recognizes search intent by understanding the meaning of a query made rather than relying entirely on keywords. The platform uses sophisticated AI models to determine what users are looking to accomplish and whether they need any information, comparisons, or final decisions. This way, Google provides more accurate search results."
https://bostoninstituteofanalytics.org/blog/latest-google-ai...
I recently built an ai chart product that attempts to add more nuance by injecting extra models to jump in when the first model misses the mark. (http://pellmell.au So far it seems like only a minority of people like it. The main feedback is that answers are confusing or too much to read
"They want to vent" :-
"they want to be heard" :-
"they want to be understood" :-
"they want to know they are understood" :-
"they want to feel they matter to and are safe with the people they value"
I believe this is a larger problem than just AI.
The internet has helped people surround themselves with only voices that agree with them and validate them, often to their detriment.
There are entire online communities urging people to cut others out of their lives over the slightest disagreement.
I get it, we can use AI agents, and they keep improving. Great, now calm down and go build stuff, and show me the stuff. I really think LLMs and agents are incredible, but I don't want to hear that over and over again for another 5 years.
HN is particularly bad for it right now. Where are all the innovations, cool technical side projects etc happening outside of LLMs?
Also sounds like Reddit where there are whole communities around what seem like part of the process - keyboards, knife sharpening, etc, rather than what you actually used your Handmade Matcha 65% MX Mechanical Collector's Edition or your Organic Kyoto 6500 diamond stone-honed knife for.
Like, yes, I get that people like to talk about their processes, and that's fine, but it seems like you can easily get sucked into making it the whole game by a sufficiently "supportive" community.
I completely agree with this sentiment. What I have seen is that these types of people never show the results of their work, they only discuss how productive they were or how fast it was to develop their project. The actual details of the project are not really mentioned.
In the rare event they do show their project, there's three categories: a worse version of something else that already exists (e.g. their version of Counter Strike that is less fun, rewriting some Python library into Rust but is now also worse), a dashboard, a visually engaging website you'll find interesting for less than 10 minutes and then never touch again.
I am not even an LLM hater, I am just not that impressed with majority of what has been built. Honestly I think back in the 90s I was far more impressed with the early websites that were totally garbage by today's standards. I remember spending so much time on the US Treasury's website as a kid learning everything about how currency works, I do not think I have ever had that level of engagement or fascination with anything that was generated by an LLM or where an LLM had significant support in the development.
Certainly this technology is nascent, however allegedly it should be making us far more productive so in the fourish years it's been around I feel like I should have witnessed something interesting by now. Perhaps I am a curmudgeon, but even in that case I think that there should at least be something else that's taking the world by storm and I am just not seeing it either.
Maybe there are some cool LLM creations, I think the speed at which it creates is also its own weakness. If anyone can create things really quickly, that includes people who are lazy and lack imagination, so then we're inundated with so much slop. Increasing accessibility does not mean that the quality will be maintained. Much like National Parks, if we make sensitive wilderness areas as accessible as possible that will degrade the quality significantly with the arrival of so many tourists, thus the parks have to consider the balance between preservation and conservation. Therefore, it was better when art et al. required some time, intention and skill before it could be shared with the world. These restrictions inherently kept the balance of quality much like how limits of accessibility in National Parks preserve the natural beauty.
> I wish there was a place I could go for tech news that just didn't talk about AI.
The obvious counterpoint is the large number of complaints about OpenAI's excessively sycophantic models back in April 2026, which led to them rolling back to a previous and less sycophantic model. I think there are those who are susceptible to model sycophancy, but it's definitely not most users. I have a vague, half-formed intuition that many of those susceptible to sycophancy are those who use AI for non-technical, non-work uses, such as inter-personal relationship questions.
As for me, as soon as I see any hint of sycophancy in a response I either stop what I'm doing or I tell the model to stop being an idiot.
Yeah. I've gotten caught in this pattern here on HN. Post something, there's some disagreement but then I see the quiet upvotes trickling in, sometimes by the dozen. That sort of signal slowly convinced me I was right about plenty of stuff.
I used AI to datamine all of my HN comments and find those patterns. Then I started asking the AI to steelman the hell out of my world view, every single argument.
AI measurably lowered the number of absolute claims I've been making. Tended to generalize and radicalize too much which would invite people to respond to the tone and the extremism instead of disproving the underlying idea. That sort of reply would actually reinforce my beliefs: if I was wrong, people would simply refute it. AI attacked my points with counterexamples until I was able to refine them into a core that's actually true.
These little distilled kernels of truth were the most valuable results. I found that many of my arguments actually have an underlying idea that holds up under scrutiny, but nobody cares because I try to prove it via extreme arguments. One such idea is anti-impossibility: laws and rules are all fine, but making it structurally impossible for people to disobey them is unacceptable. I used to try to argue that idea by defending running red lights which persuaded no one. Now I've identified the real idea and can make a much better argument for it.
Try listening to somebody and merely repeating the literal words that they say. I guarantee you, most people can't even reproduce two or tree sentences correctly. And I am not even talking about understanding the words, just the literal reproduction of them, copy-pasta. Humans just can't do it, we are not good at it.
Talking to an AI about your problems is a bit like talking to yourself, but with the added superpower of hours of google searches compressed in seconds. Of course it is dangerous and incomplete, but in a way it also beats talking to humans if you know what you are doing and are looking to and able to solve your problems yourself.
You start your comment saying the conclusions are overstated and then just drop this conclusion without any proof except anncedotal prespective.
And, it has the same deleterious effect on your mental health. I could name a bunch of billionaires that behave like sociopaths, and they're often seemingly miserable while doing it. I'm sure some of them started out as sociopaths, but they rarely behaved that way so obviously early in their rise.