Top
Best
New

Posted by pera 1 day ago

Avoid 2:00 and 3:00 am cron jobs (2013)(www.endpointdev.com)
327 points | 366 commentspage 2
PhilipRoman 1 day ago|
I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime)

    0 2 * * * run-parts /etc/periodic/daily
    0 3 * * 6 run-parts /etc/periodic/weekly
It appears that busybox cron is smart enough to handle it and only ran the jobs once.
latchkey 1 day ago|
Google's AI and ChatGPT answer disagrees.

https://www.google.com/search?q=busybox+cron+dst

"BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."

PhilipRoman 1 day ago|||
https://github.com/mirror/busybox/blob/master/miscutils/cron...
moderateclimate 20 hours ago|||
Please don't do this. AI is not a source and is frequently wrong (as it is in this case).
latchkey 19 hours ago||
You're right. It was honestly just a low effort semi-sarcastic reply that I wasn't really expecting anyone to take seriously. I should have just gone and read the source code as PhilipRoman did below.
brettgriffin 1 day ago||
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all.

Is this what people actually want, though?

I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).

Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.

Am I missing something?

waisbrot 1 day ago||
The article is mostly talking about using the `cron` scheduler and running things at 2am on Sunday because there's especially low traffic. These cron jobs are "I don't care when it runs, but it should have minimal chance of causing problems."

Your use-case is totally different: "I want this job to run at this time" so the only lesson that applies to you is that the `cron` utility might behave weird during DST switches. No idea if it underlies your cloud provider, so it may be completely irrelevant.

evil-olive 1 day ago|||
> Everybody wants the job to run four hours before the work day starts

recurring events like this - ones that take place relative to humans in a given location - are the most common snag related to "just use UTC everywhere" advice. but they're not actually in conflict at all. they just require an extra layer of indirection.

let's say this is a factory that starts early in the morning, 06:30 local time, because then "four hours before" falls into the window affected by DST.

"run at 02:30 local time" is what you want to avoid, because on the "spring forward" day, 02:30 simply does not exist. the clocks jump from 01:59:59 to 03:00:00. [0]

likewise, "run at 01:30 local time" is to be avoided, because on the "fall back", that occurs twice.

ideally, you want to think of the recurrence as being associated with a given location, and that recurrence generates individual events, which happen in UTC time.

for example, if the factory is in San Francisco, the recurrence might specify "once a day, at 02:30 SF time", which would generate a series of daily events in UTC time. during the summer those events are at 09:30Z, during the winter they're at 10:30Z.

you also want the scheduler to be smart enough to understand the "once a day" constraint - so that a "once a day, at 01:30 local time" task generates only one daily event, rather than two, even though 01:30 local time happens twice in the same day.

and then, because the individual events are all in UTC, and the system clock is as well, there's no ambiguity in the actual execution time of the event. the complexity of the recurrence is handled at a higher level.

if you want an even harder version of this problem, imagine a team in SF and a team in London, that have a Zoom meeting every Monday at 8am (SF) / 4pm (London). it's necessary for one of those locations to "own" the recurrence, because the UK does their time changes 1-3 weeks offset from the US [1], creating weeks where it isn't a straightforward 8 hour time difference.

0: https://www.timeanddate.com/time/change/usa

1: https://www.timeanddate.com/time/change/uk

jay_kyburz 1 day ago||
The more I read this thread, the more I think this is just a bug in Cron. It should really only operate on UTC, regardless of the local time set on the server.
evil-olive 1 day ago||
with systemd timers [0], explicitly specifying the timezone is an option:

    > systemd-analyze calendar '02:30'
      Original form: 02:30
    Normalized form: *-*-* 02:30:00
        Next elapse: Tue 2025-10-28 02:30:00 PDT
           (in UTC): Tue 2025-10-28 09:30:00 UTC
           From now: 10h left
    > systemd-analyze calendar '02:30 UTC'
      Original form: 02:30 UTC
    Normalized form: *-*-* 02:30:00 UTC
        Next elapse: Mon 2025-10-27 19:30:00 PDT
           (in UTC): Tue 2025-10-28 02:30:00 UTC
           From now: 3h 34min left
however, making it disregard the server timezone and use UTC by default isn't really an option. it would violate the Principle of Least Surprise, as well as being a hugely backwards-incompatible breaking change.

0: https://www.freedesktop.org/software/systemd/man/latest/syst...

monster_truck 1 day ago||
I call this "rake stomping", as long as you're running back and forth in the same places every time, you'll never notice.
loodish 1 day ago||
I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant?

If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.

hinkley 1 day ago||
2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time.

But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.

The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.

It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.

matja 1 day ago||
Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.
Zanfa 23 hours ago||
While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.
ttz 1 day ago||
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all

this is the way

bobmcnamara 1 day ago|
Use TAI
meonkeys 1 day ago||
anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron).

anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.

litoE 1 day ago||
I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem. However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.
tsoukase 1 day ago|
For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.
More comments...