Top
Best
New

Posted by lpage 3 days ago

NTP at NIST Boulder Has Lost Power(lists.nanog.org)
488 points | 207 commentspage 2
samch 3 days ago|
Can anybody speak to the current best practices around running underground power lines? I see these types of articles about above-ground distribution systems from time-to-time, particularly in California. I feel lucky that my area has underground power, but that was installed back in the 1980s. Would it be prohibitively expensive for Boulder’s utility provider to move to underground distribution? I can’t help but think it could be worth the cost to reduce wildfire risk and offer more reliable service.
pseudonymidy 3 days ago||
Think of it like this: overhead power lines require you to dig a 5-7’ deep hole that’s 2’ in diameter every 90’. Underground power supplied through cable requires you to bury the cable minimum 3’ in the ground in rigid ductwork the entire 90’. Any time that cable runs under a roadway that ductwork needs to be encased in concrete. In urban and semi urban areas you also compete with other buried infrastructure for space - sewer, city/municipal infrastructure, gas, electrical transmission, etc.

While underground distribution systems are less prone to interruption from bad weather it depends on the circuit design. If the underground portion of the circuit is fed from overhead power lines coming from the distribution substation you will still experience interruptions from faults on the overhead. These faults can also occur on overhead transmission circuits (the lines feeding the distribution substations and/or very large industrial customers).

Underground distribution comes at a cost premium compared to overhead distribution. It’s akin to the cost of building a picket fence vs installing a geothermal heating system for your home. This is why new sub divisions will commonly have underground cable installed as the entire neighborhood is being constructed - there’s no need to retrofit underground cable into an existing area and so the costs are lower and borne upfront.

It’s more cost effective for them to turn the power off as a storm rolls through, patrol, make repairs and reenergize then to move everything underground. Lost revenue during that period is a small fraction of the cost of taking an existing grid and rebuilding it underground. This is especially true for transmission circuits that are strung between steel towers over enormous distances.

lazyc0der 3 days ago|||
Boulder County is working on utility undergrounding, https://bouldercolorado.gov/services/utility-undergrounding, and in particular they are working on undergrounding at Chautauqua, https://bouldercolorado.gov/news/undergrounding-chautauqua, which is next door to the NIST facility.
shadowpho 3 days ago|||
1/3-1/2 of the cost of the electricity we pay is distribution.

Some of it is physical infrastructure (transformers, wire, poles), but a lot of it is labor.

Labor is expensive in US. It’s a lot of labor to do, plus they’ll likely need regulatory approval, buying out land, working through easements.

At the same time you have people screaming about how expensive energy is.

Furthermore they have higher priorities, replacing ancient aging infrastructure that’s crumbling and being put on higher load every day.

tekno45 3 days ago||
Its very hard to repair and keep track of underground.

https://practical.engineering/blog/2021/9/16/repairing-under...

manosyja 3 days ago||
Germany here, never heard of any issues regarding underground power (or phone) lines. Ultra High voltage (distribution network) is above ground here, but no issued with that either.
lovich 3 days ago||
This was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails?

This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.

Maxious 3 days ago||
There's two other sites for the time.nist.gov service so it'll be okay.

Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...

lovich 3 days ago|||
What happens in the event all the sites for time.nist.gov go down? is it included in the spec?

Also thank you for that link, this is exactly the kind of esoteric knowledge that I enjoy learning about

sdrmill 3 days ago||
Most high-availability networks use pool.ntp.org or vendor-specific pools (e.g., time.cloudflare.com, time.google.com, time.windows.com). These systems would automatically switch to a surviving peer in the pool.

Many data centers and telecom hubs use local GPS/GNSS-disciplined oscillators or atomic clocks and wouldn’t be affected.

Most laptops, smartphones, tablets, etc. would be accurate enough for days before drift affected things for the most part.

Kerberos requires clocks to be typically within 5 minutes to prevent replay attacks, so they’d probably be ok.

Sysadmins would need to update hardcoded NTP configurations to point to secondary servers.

If timestamps were REALLY off, TLS certificates might fail, but that’s highly unlikely.

Databases could be corrupted due to failure of transaction ordering.

Financial exchanges are often legally required to use time traceable to a national standard like UTC(NIST). A total failure of the NIST distribution layer could potentially trigger a suspension of electronic trading to maintain audit trail integrity.

Modern power grids use Synchrophasors that require microsecond-level precision for frequency monitoring. Losing the NIST reference would degrade the grid's ability to respond to load fluctuations, increasing the risk of cascading outages.

neomantra 3 days ago||
Great list! Just double-checked the CAT timekeeping requirements [1] and the requirement is NIST sync. So a subset of all UTC.

You don’t need to actually sync to NIST. I think most people PTP/PPS to a GPS-connected Grandmaster with high quality crystals.

But one must report deviations from NIST time, so CAT Reporters must track it.

I think you are right — if there is no NIST time signal then there is no properly auditable trading and thus no trading. MFID has similar stuff but I am unfamiliar.

One of my favorite nerd possessions is my hand-signed letter from Judah Levine with my NIST Authenticated NTP key.

[1] https://www.finra.org/rules-guidance/rulebooks/finra-rules/6...

3eb7988a1663 3 days ago||||
Considering how many servers are in existence, probably the exact same procedure for starting a brand new one?
wpm 3 days ago|||
I must have one of those units oh my god
mh- 3 days ago|||
Someone needs to sell replicas (forgive the pun) of these.
wat10000 3 days ago|||
It's like a toaster oven, but it toasts time.
lambdaone 3 days ago||
There are lots of Stratum 0 servers out there; basically anything with an atomic clock will do. They all count seconds independently from one another, all slowly diverging over time, with offset intervals being measured by mutual synchronization using a number of means (how is this done is interesting all by itself). Some atomic clocks are more accurate than others, and an ensemble of these is typically regarded as 'the' master clock.

To quote the ITU: "UTC is based on about 450 atomic clocks, which are maintained in 85 national time laboratories around the world." https://www.itu.int/hub/2023/07/coordinated-universal-time-a...

Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means.

Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.

8organicbits 3 days ago||
The referenced mailing list is this Google Group (https://groups.google.com/a/list.nist.gov/g/internet-time-se...) which has some other posts about this incident.
creatonez 2 days ago|
Where are you seeing the other posts?
DamonHD 3 days ago||
So far I think I'm still seeing one of them in my peers list for my public-ish NTP server:

         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    +time-e-b.nist.g .NIST.           1 u  372 1024  377  125.260    1.314   0.280
DamonHD 3 days ago|
...and maybe it's gone:

    #time-e-b.nist.g .NIST.           1 u 1071 1024  377  125.260    1.314   0.280
sgnelson 3 days ago||
One question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.
grepfru_it 3 days ago||
Residents and some businesses of Boulder have been without power since Tuesday. There was an issue about 10 years ago which caused 1000 homes to burn down and the power company was found liable. They change their actions. Then during the next high wind event, the power company preemptively cut power and businesses sued them for loss of revenue. Now the power company is playing it safe and turning off power to residents and keeping downtown businesses powered.

Maybe their generator failing was DOGE related, but wouldn’t have happened if state level shenanigans were better handled

_se 3 days ago||
The Marshall fire was 4 years ago. Almost to the day.
cramcgrab 3 days ago||
Actually DOGE involvement at the highest level would have resulted in Tesla solar and Tesla powerwall battery backups.
throw0101c 3 days ago|||
> […] Tesla solar and Tesla powerwall battery backups.

Don't forget Solar Roof.

cramcgrab 3 days ago||||
Apparently no engineers here anymore, all armchair politicians.
hiddencost 3 days ago||
When the politicians take a hatchet to the underpinnings of society that we built, we have to get political.
cramcgrab 2 days ago||
So sad, we used to innovate and stay one step ahead of them. That’s how wealth is created.
redbush237 3 days ago||||
Lol, lmao.

Some relevant DOGE’s effects:

-time and frequency division director quit

-NIST emergency management staff at least 50% vacant

-NIST director of safety retired, and NIST safety was already understaffed when compared to DOE labs

-NOAA emergency manager on the same Boulder campus laid off

etc

the_gipsy 3 days ago|||
Any day now
stackghost 3 days ago||
Direct link, rather than a repost: https://groups.google.com/a/list.nist.gov/g/internet-time-se...
digdigdag 3 days ago||
My local GPS + PPS based NTP server finally pays off.
gilrain 3 days ago||
It’d be a good idea to protect our infrastructure from the climate we created.

It’s just a good idea, though, not a greedy one… so it won’t happen.

wolfbyte8 2 days ago|
I just got home from a holiday party where two people from NIST in Boulder work on the NIST-F1 Atomic Clock. The one scientist described the outage and the failures but reported that things are still shut down even after power was restored. She told me that she believes that the Trump administration is in the process of shutting down the program and server in Boulder. And there was low confidence in that the Washington DC superiors understood what they are doing. God I hope this isn't more Tina Peters retribution crap. Cutting off our nose to spite our face.
More comments...