Posted by nprateem 15 hours ago
AWS: Inaccurate Estimated Billing Data – $1.7 billion
I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
It doesn't anymore, if it ever did. Read the shareholder letters over the last 5 years like this one (https://www.aboutamazon.com/news/company-news/amazon-ceo-and...). Move fast. Embrace AI in every corner of the company. Lay off thousands of people like we're a lean startup. Oh, but make sure to have a senior engineer check any AI code you want to push to production (https://news.ycombinator.com/item?id=47323017).
Amazon has made some impressive software, but there's also a ton of junk getting shipped. Customers and "partners" have to deal with it every single day. Sometimes it's "oops"-level stuff like this, at other times it's AWS regions going dark for hours.
Browse the Amazon seller forums (https://sellercentral.amazon.com/seller-forums/discussions?s...). Every day there are stories about back-end things breaking, or AIs overriding attributes or flagging random items as pesticides. The receiving errors at Amazon warehouses are incredible, and take months and multiple support tickets to fix.
They have the best uptime of all major cloud providers.
Host your own people. Host your own.
> I think you should spin up a whole bunch more instances, and try to cause an integer overflow so they they owe you $978 Trillion.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Being able to repeat a string is fine, but it should be a str.repeat() function, not an operator overload like that.
Look at them up there, just plotting with each other! :-)
It’s probably an artifact of them all being currency multiples of 2^30
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
If someone gets access to your account they can just buy a 3-year reserved instance u7in-24tb.224xlarge and it will add almost $2m to your bill.
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.