Top
Best
New

Posted by operator-name 6 days ago

Serverless Horrors(serverlesshorrors.com)
618 points | 484 commentspage 2
general1465 5 days ago|
I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything.

That's just a problem waiting to happen while you are always running tests on production...

rurp 5 days ago||
I worked on a serverless project for several years and the lack of ability to run much of anything locally was a huge cost. Debugging cycle times were absolutely terrible. There are some tools that claim to address this but as of a few years ago they were all useless for a real project.
themafia 5 days ago|||
I use my AWS security key to run local tests. It works perfectly fine. You just need a ~/.aws/credentials file appropriately configured.

I have a makefile system which controls lambda deployments. One step of the deployment is to gather the security requirements and to build a custom IAM role for each individual lambda. Then I can just write my security requirements in a JSON file and they're automatically set and managed for me.

The real joy of AWS is that everything works through the same API system. So it's easy to programmatically create things like IAM roles like this.

nostrebored 5 days ago|||
1. Put your stuff in a stack. Deploy it to your isolated developer account. Basically free staging environment.

2. Use the officially supported docker runtime for local testing.

3. Treat it like any other code and make unit tests

4. Use one of the tools like localstack to emulate your staging env on your machine.

There are so many options that I don’t know how you could walk away with your opinion.

Nextgrid 5 days ago|||
Or you could just write conventional software. But I get it, you don't get resume points nor invites to cloud-provider conferences for that.

> Basically free staging environment. [emphasis mine]

Not really. Sure, the cost would usually be peanuts... until you have an infinite loop that recursively calls more lambdas. Then you have a huge bill (but hey that pays for your invites to their conferences, so maybe it's a blessing in disguise?). And yes, you will pretty much always get it refunded, but it's still a hassle and something that is absolutely not necessary.

Snark aside, having an opaque dev environment always constrained by bandwidth and latency that can’t be trivially backed up/duplicated is a terrible idea and why I always recommend against “serverless”, even besides the cost concerns.

Serverless is OK for small, fully self contained pieces or code that are fire and forget. But for anything complex that’s likely to require maintenance, no thanks.

rurp 5 days ago|||
Eh, I worked on a large serverless project that worked hard to follow best practices but it was still very clunky to run and test code locally. The local serverless tools simply didn't work for our project and they had so many limitations I'm skeptical they work for most non-prottypes.

Deploying a stack to your own developer environment works fine and is well worth doing, but the turnaround time is still painful compared to running a normal web framework project locally. Deploying a stack takes much much longer than restarting a local server.

Serverless isn't all bad, it has some nice advantages for scaling a project, but running and debugging a project locally is a definite weak spot.

scarface_74 5 days ago||
This is nowhere near being true.
petralithic 5 days ago||
This is some good marketing for Coolify, which the author makes as an open source platform as a service. I prefer Dokploy these days though, since it seems to be less buggy, as Coolify seems to have such bugs due to being on PHP.

https://coolify.io/

https://dokploy.com/

khromov 5 days ago|
CapRover is another good alternative, and also much more lightweight than Coolify, easily runs on even a 512MB server: https://caprover.com/
omnicognate 6 days ago||
It would help to round to the cent. With 3 digits to the right of the dot it's ambiguous whether it's a decimal point or a thousands separator, and the font and underline makes the comma vs dot distinction a bit unclear.
thedanbob 6 days ago|
A number of the titles appear to have 69 or 420 cents added to the amount that appears in the story.
trcf22 6 days ago||
After a quick check on Vercel stories, it seems all payments were discarded or mistakes in the first place.

Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?

Alifatisk 6 days ago||
> Do you need to tweet about it to be reimbursed?

This is what scares me, is social media the only way to get things sorted out nowadays? What if I don't have a large following nor an account in the first place, do I have to stomach the bill?

pelagicAustral 6 days ago|||
This is exactly what happened to me during Covid... I had a flight that got cancelled at the beginning of the pandemic since the country closed the orders (essentially). A year after, still on lock downs and et al, I wanted to enquire about a refund, for months I got not answer, until I caught wind that people using Twitter were actually getting results. Now, I don’t use social media at all, so I had to create a Twitter account, twit about my case et voila! 30 mins after I got a response and they send me a PM with a case number... Not even going to mention the airline, but it is infuriating...
wg002 6 days ago|||
I can't imagine them sending it to collections. What kind of recourse would a company like Vercel have if you don't pay it?
immibis 4 days ago||
They can sue you for the bill plus the legal costs.
interloxia 6 days ago|||
Someone at a community group I'm in messed up playing with Azure through their free for non-profits offering^. We were out about 1.2k€. Not huge but huge for us.

Encouraged by comments on HN over the years I had them ask support to kindly to wave it. After repeating the request a few times they eventually reduced their bill to <100€ but refused to wave it entirely.

So even without shaming on social media, But it probably depends. It's worth at least asking.

^The deal changed about six months ago.

cuu508 5 days ago||
It's waive, not wave
Havoc 6 days ago|||
Relying on the mercy of a support agent that may be having a bad day is a poor strategy
pjmlp 6 days ago|||
No, at least in enterprise consulting for these kind of hosting, usually there is a contact person on the support team that one can reach directly.

However these projects are measured in ways that make Oracle licenses rounding errors.

Which naturally creates market segmentation on who gets tier 1 treatment and everyone else.

viraptor 6 days ago||
Once you're in a contract + TAM territory, pricing works very differently. Also, temporary experiments and usage overruns become an interesting experience where the company may just forget to bill you a few thousands $ just because nobody looked at the setup recently. Very different situation to a retail user getting unexpected extra usage.
tonyhart7 6 days ago||
I mean if developer got charged with 100k, more often than not the bank would decline that first maybe if you didn't have that high credit limit

but what happen if this happen to corporate account and somewhere resource get leaked???

multi billions dollar company probably just shrug it off as opex and call it a day

1oooqooq 6 days ago||
last employer asked for an estimate to migrate to cloud.

it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr.

ceo told to go ahead anyway (turn out company was being sold to Apollo)

first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one.

can't imagine how companies miss that step.

chmod775 5 days ago||
These guys charge $550 for a measly terabyte of bandwidth?

If you get a dedi on a 10Gb/s guaranteed port and it works out to more than $3 / TB, you're probably getting scammed. How does "serverless" justify 150x that? Are people hosting some silly projects really dense enough to fall for that kind of pricing?

Just get a $10 VPS somewhere or throw stuff on GH pages. Your video game wiki/technical documentation/blog will be fine on there and - with some competent setup - still be ready for 10k concurrent users you'll never have.

EGreg 6 days ago||
Are there any protections these days at the cloud provider level?

Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?

JJMcJ 6 days ago||
That's why I like VPS setups. You hit the monthly maximum, and it just stops working.

I host demos, not running a business, so it's less of an issue to get interrupted. Better an interruption than a $50,000 bill for forgetting to shut off a test database from last Wednesday.

prisenco 6 days ago|||
Unless a startup has five+ nines service contracts with their customers already, a little bit of downtime once in a while is not the end of the world the cloud services want us to believe.
qcnguy 6 days ago|||
That's not comparable. With a VPS there is no monthly maximum, just a max load on a second by second basis. You can be hit with traffic of which 90% bounces because your server is down, get nowhere near your intended monthly maximum, and then the rest of the month is quiet.
JJMcJ 2 days ago|||
I got VPS and other types of shared hosting mixed up, though I have seen VPS offerings with monthly or daily maximums.
McGlockenshire 5 days ago|||
You seem to be describing this as a bad thing instead of the objectively good thing that it is.
qcnguy 5 days ago||
The ideal is obviously smoothed limits, such that you can absorb a big traffic spike if it still fits within your budget. Nobody seems to offer that.
EGreg 5 days ago||
How would you predict the smooth curve ahead of time?
bc569a80a344f9c 6 days ago|||
Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes. That means there’s some lag, and you are responsible for any costs incurred that go over budget between such reporting runs.
chuckadams 5 days ago|||
> of course AWS doesn’t check each S3 request or Lambda invocation against your budget

If it can bill them per-invocation, why can't it also check against a budget? I don't expect it to be synchronous, but a lag of minutes to respond is still better than nothing. Can you even opt-in to shutting down services from the budget tool, or is that still something you have to script by hand from Cloudwatch alarms?

bc569a80a344f9c 5 days ago||
You script it by hand.

I think figuring out how to do this faster is less trivial than it might sound. I agree that synchronous checks aren’t reasonable. But let’s take Lambdas. They can run for 15 minutes, and if you consolidate within five minutes after a resource has been billed, that gives you a twenty minute lag.

I’m not trying to make apologies for Amazon, mind you. Just saying that this isn’t exactly easy at scale, either. Sure, they bill by invocation, but that’s far from synchronous, too. In fact, getting alerts might very well be happening at the frequency of billing reconciliation, which might be an entirely reasonable thing to do. You could then argue that that process should happen more frequently, at Amazon’s cost.

McGlockenshire 5 days ago|||
> but it doesn’t natively support shutting down services [...] of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes

So, in other words, the vendor has provided substandard tooling with the explicit intent of forcing you to spend more money.

franktankbank 6 days ago||
Just set alerts that are not really timely and homeroll your own kill scripts its easy. It doesn't really work but its not really any harder than just fucking self hosting.
raw_anon_1111 5 days ago||
For everyone complaining about no free tier that blocks you from being charged

https://aws.amazon.com/free/

Experience AWS for up to 6 months without cost or commitment

Receive up to $200 USD in credits

Includes free usage of select services

No charges incurred unless you switch to the Paid Plan

Workloads scale beyond credit thresholds

Access to all AWS services and features

adamddev1 5 days ago||
I remember at the beginning of the serverless hype how they said it was great because it automatically scaled as big as you need it. Given how sudden and massive these "scaling spikes" can be, I would much rather deal with a death-hugged VPS than a $100k bill.

Plus the VPS is just so much faster in most cases.

mitjam 5 days ago|
I once found an official Microsoft example repo to deploy an LLM gateway on Azure with ALB. Glad I did the tedious work of estimating the costs before I hit the deploy button (had to go though many Biceps manifests for that). The setup would have cost me about 10k/month.
More comments...