Top
Best
New

Posted by aarondf 4 hours ago

Railway Blocked by Google Cloud(status.railway.com)
291 points | 126 commentspage 2
codegeek 3 hours ago|
This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their scale with all this VC money.
choilive 3 hours ago||
They run a decent amount of their own compute/bare metal server for customer workloads. But likely still had some critical dependencies on GCP.
cube00 15 minutes ago||
> Surprising for a company of their scale with all this VC money.

Not sure too many VCs would be cool with deep redundancy when there's more features to build to bring in more customers instead.

bilalq 53 minutes ago||
Building a startup on GCP (or even Google Workspace) is an existential risk.
padolsey 2 hours ago||
Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.
jpollock 2 hours ago||
There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with.

Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation.

Sign up for terms. It will get you payment terms!

mbreese 2 hours ago|||
Yeah, I'm not sure what to think here. We know Google is not the best at customer service and has automated account suspensions. But, what I'm curious about here is why this happened.

Railway hosts applications for customers. An uneducated guess for some possible reasons: 1) one of those customers hosted something they shouldn't have 2) railway had something spawn that took up too many resources 3) Or their account balance was too high 4) Or something...

But all of this probably culminates in someone needed to read an email that was missed.

Scaling a customer infrastructure setup like Railway is hard. This is one of the non-technical hard parts - how to make sure your account with your primary vendor is safe. But, I'm willing to wait to pass judgement here until more information is available. I'm sure the post-mortem will have lessons. I'd like to know more.

thayne 1 hour ago||||
> via their contact method, aka the email they set it up with

If it's anything like AWS, that may be just one of hundreds of emails they send every day, most of which are just noise.

scratchyone 2 hours ago|||
Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.
sammy2255 1 hour ago||
The 3-2-1 backup rule is pretty outdated in the world of cloud. You could have 3 complete copies of your data in different S3 buckets, but if they're all under the same account you've lost your blast radius protection
whalesalad 20 minutes ago||
You replicate data to different clouds.
rsync 1 hour ago||
If only there were a quick and easy way to replicate s3 buckets to an independent provider…

… on the Unix command line …

… to a cloud older than AWS…

… if only …

funtech 1 hour ago|||
Wish I could upvote this comment account more. Too many people look for something new and shiny when trusty ol tools are sitting right there. :)
lemagedurage 17 minutes ago||||
Inflated egress costs might make this prohibitively expensive, $80 per TB at GCP and AWS
oefrha 1 hour ago||||
Well having backups help, but I certainly can’t migrate my infra to rsync.net on moments’ notice (or ever since rsync.net does storage and nothing else) so my customers aren’t affected.
eclipticplane 1 hour ago|||
I don't think that technology exists. Sorry.
r_lee 3 hours ago||
seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company?

I'm exaggerating but someone said they got "auto banned"

what if that happens to a small account which hosts some really important data/services there?

xyzzy_plugh 2 hours ago||
I've managed several accounts with GCP over the years and I've always maintained a great relationship with our contacts there. Some of these accounts were quite small, on the order of <$20k/mo, and even then we were kept abreast of anything that might be cause for concern. I always maintain a standing biweekly meeting with at least someone on the other side (account exec, technical staff, whatever) and I've yet to be blindsided by anything.

Is Google's communication good? No, not particularly. The only way something like TFA happens is if the relationship is neglected (by one or both parties). I'm not saying Railway did something wrong, but there are usually many flags and opportunities to correct long before drastic actions.

I get the impression that Railway plays fast and loose with a lot of their limits and resources and that Google may not be a fan of that.

Edit: would also like to say that if you put all your resources in one GCP project you are going to have a bad time. If you organize stuff over many projects it is very unlikely that they will ever take account wide action. I've had issues with, for example, a particular tenant's behavior, but it never jeopardized the other tenants.

Avicebron 2 hours ago|||
> what if that happens to a small account which hosts some really important data/services there?

Pray to @dang that you will make the front page of HN?

throwaway85825 2 hours ago|||
Even if you are a billion dollar company you still have problems like the Australian pension did. Google is just that bad.
chi_features 2 hours ago|||
https://blog.railway.com/p/series-b

Agreed. Railway are probably not far off a billion dollar company though!

jrockway 2 hours ago|||
I don't think you can ever trust one service with critical data. Some Claude instance deletes your prod database, you have to restore from an offsite backup because it also deleted your local backups. Even at small startups we did pg_dump to AWS from GCP because ... who knows what is going to happen to GCP, and we want to continue to be in business if that happens.

I don't feel safe with any one single point of failure. "Your credit card bounced", "you thought it was dev", "you got hacked", etc. are all the same problem to me and no cloud provider solves those merely by setting up an account.

ttoinou 2 hours ago||
Railway isnt far from being a billion dollar company, no ?
tux 3 hours ago||
At this point you can’t trust Google anymore, it keeps breaking things. Imagine having Google AI do this thins automatically. Will have apocalypse in in a day.
zelon88 1 hour ago||
Wild to me that any tech sector business would want to rent an operating environment to park their entire infrastructure into. This is the equivalent to traveling shoe salesmen setting up a tent in the parking lot of a strip mall.
hnburnsy 2 hours ago||
From their founder on X...

"Absolutely. The Railway network is a mesh ring between AWS, GCP, and Metal

So: - High availability interconnects - High availability path routing between clouds - Database itself is high availability

However, Google's VPC itself is not. So we will add a shard to Metal and AWS"

hnburnsy 1 hour ago|
More here...

https://x.com/JustJake

jefborges 3 hours ago||
Railway is back, but I’m not sure if I can trust keeping my projects there, so I’m going to migrate to another company.
oofbey 1 hour ago||
After reading about how their delete database API also deletes all the backups, I concluded they are not to be trusted.
marknutter 1 hour ago||
It's not back.
shevy-java 50 minutes ago|
Do not become dependent on Google. Ever.
cindyllm 49 minutes ago|
[dead]
More comments...