Posted by nprateem 11 hours ago
AWS: Inaccurate Estimated Billing Data – $1.7 billion
I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.
That entails a real loss of face for them, and because they are the ones who actually run the show and actually own the company. It would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
Or in my case, actively ignore any and all recruiting from that sesspool.
The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
I got put on Focus because my "contributions were not coming through" to leadership.
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"
I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.
[edit] Testing your tests, like testing your backups, is a good idea
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
Worst case I've found was off by 15 orders of magnitude.
That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
Units and datetime will always be the bane of any professions ...
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
Fuck it, im in.
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
> In this role, you will own end-to-end bill run execution across all AWS partitions, drive the technical vision for autonomous billing operations, and build the team that ensures every customer receives an accurate cost estimated in minutes ...
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
This is a gold-mine. They need to get sued heavily for this incompetence.
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
:)
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
The employees and company have an obligation not to exploit this even if the issue is only discovered after the fact.
You don't get to export any of the responsibility to your customer. They don't prepare the bill and it's not their job to find your fuck ups
I once got a monthly water bill for ~$35,000 at a residential, single-family home. Good thing I was paying attention and looked at the bill before the auto-pay bank draft hit.
Someone had misread the meter.
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
You should charge your customer 3015000 thousand dollars.
Number felt high so I wanted to double check and I only get 301500.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
It's a shit show.
The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.
That's one way to cut staff.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
> You requested that we alert you when the actual cost associated with your Monthly AWS budget exceeds $2.00 for the current month. The month actual cost associated with this budget is $646,677,805.51.
Current usage: $1.70.
I called it a rounding error and figured it would wash out within a couple of days...
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
https://www.techtimes.com/articles/320266/20260712/anthropic...
That sounds bad.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
(Of course, they don't know they're using it, they're using a service on it)
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
None of this can happen unless you first cede control.
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.