Top
Best
New

Posted by mbac32768 20 hours ago

A false smoke alarm almost drowned Jane Street's only NYC datacenter(michael.bacarella.com)
518 points | 250 commentspage 4
rshnotsecure 20 hours ago|
Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.

[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane

advisedwang 20 hours ago||
Likey codes or building management wouldn't let them make the necessary modifications to safely install a Halon system
garciasn 19 hours ago||
Halon systems are banned, at the international and federal level, and have been since the 1990s; so, modern DCs don't and can't use them.
itintheory 19 hours ago|||
There are lots of 'clean agent' fire suppression systems that are not actually "Halon". I suspect people still use it as a generic term.
advisedwang 19 hours ago|||
There's tons of substitutes, the ban on Halon wouldn't be why Jane St is using water.

Many people (like me) use "halon" when they are talking about general inert gas based fire suppression as there's not really a good short word for them otherwise.

dklsf 20 hours ago|||
If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....

So fire suppression and data centers don't mix too well

shagie 20 hours ago|||
Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.

https://www.hack.org/~mc/texts/vaxen-joke.txt

Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...

           The fire alarm klaxon went off and the siren warning
           of imminent halon gas release was screaming.  We
           started to panic but the data center manager shouted
           over the din, "Don't worry, the halon system failed
           its acceptance test last week. It's disabled and
           nothing will happen."
 
           He was half right, the primary halon system indeed
           failed to discharge. But the secondary halon system
           observed that the primary had conked and instantly
           did its duty, which was to deal with Dire Disasters.
           It had twice the capacity and six times the discharge
           rate.
wolvoleo 20 hours ago||||
They don't have to blow that fast. It can be regulated.

But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.

reisse 14 hours ago|||
Yeah, like the infamous Nasdaq Nordic outage in 2018 due to errant fire suppression gas release.
mbac32768 20 hours ago|||
Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.
dklsf 20 hours ago||
Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there
hvb2 20 hours ago|||
Or CO2 but that's a life or death situation if there's any people still inside
thechao 20 hours ago||
Halon is not exactly fun to breathe, either.
wolvoleo 20 hours ago||
Especially after it's been in contact with fire, the resulting products are even worse.

But it's only used in military and aviation these days anyway.

gosub100 20 hours ago|||
Probably not safe to use on a floor that has people on it.
wolvoleo 20 hours ago||
We had it at work at a datacenter right beside offices and reception.

You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.

PS: Halon itself is of course gone but it's other less harmful gases these days.

What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).

gosub100 19 hours ago||
Just curious was this on a ground level building? Seems unsafe in a high rise because of fewer egress points.
wolvoleo 18 hours ago||
It was in a 3-level building. One DC was on the ground floor and one on the 3rd. But not exactly highrise no.
ghostly_s 14 hours ago||
> Everyone’s heard of how awesome Jane Street is.

I guess I’m the last one to hear it.

tonyhart7 9 hours ago||
should be drowning tbh, less market manipulation
lostmsu 19 hours ago||
Alarm firmware wasn't programmed in OCaml
infamouscow 18 hours ago|
They just need to hire some full stack people!
saussianglat 17 hours ago||
why don't they have a dry sprinkler system?
dokaf-ops 5 hours ago||
[flagged]
minfazhai 10 hours ago||
[flagged]
remindmi16 16 hours ago||
[flagged]
spencerawlson 11 hours ago||
[flagged]
strangeworldfil 17 hours ago|
[flagged]
More comments...