Posted by lukaspetersson 6 days ago
The people who lose their prod database to AI bugs, or the lawyers getting sanctioned for relying on OpenAI to write court documents? There's also good - their stories serve as warnings to other people about the risks.
The issue is that unpaid average people are being used, or rather forced, to act as QA and Beta Testers for this mad dash into the AI space. Customer Service was already a good example of negative preception by design, and AI is just being used to make it worse.
A production database being corrupted or deleted causing a company to fail sounds good on paper. But if that database breaks a bank account, healthcare record, or something life altering for a person who has nothing to do with the decision of using it the only chance they have for making it right is probably going to be the legal system.
So unless AI advancement really does force the legal system to change the only people I see coming out ahead from the mess we are starting to see is the Lawyers who actually know what they're doing and can win cases against companies that screw up in their rush to go to AI.
Seriously, I completely agree with you.
But I really wish Anthropic would give the technology to a journalist that tries working with it productively. Most business people will try to work with AI productively because they have an incentive to save money/be efficient/etc.
Anyway, I am hoping someone at Anthropic will see this on HN, and relay this message to whatever team sets up these experiements. I for one would be fascinated to see the vending machine experiment done sincerely, with someone who wants to make it work.
The reality is that even most customers are smart enough to realize that driving a business they rely on out of business isn't in their interest. In fact, in a B2B context, I think that is often the case. Thanks.
The article being discussed here is about how AI couldn't run a real world vending machine. There was no issue in the components that would be in a standard simulation.
If it had just made stocking decisions autonomously and based changes in strategy on what products were bought most, it wouldn't have any of the issues reported.