Agree with the post. The job of blackbox is to turn probes into metrics. If a probe fails, that should just become a probe_success=0 metric. Blackbox did its job and should not log an error.

Why are logs usually assumed to be for human consumption only? It seems weird to me that log storage usually exists outside of the system and isn't a general purpose message bus.

It means something is wrong, yes. Now, if it's worth fixing (granted, most of the time it would), that's another story.

I make error logs fail happy path functional/integration tests for the backend applications I'm currently writing.

I agree. Error or higher should result in an alarm and indicates that some corrective action needs to be taken.

To me it's always a neat trick when you're not allowed to use print() in production code

Severity is the value and you set thresholds based on context of the error type.

Input errors do not need fixing, so no.

If my system doesn’t work - I want to be alerted. If notification was supposed to be sent but wasn’t - it’s an error regardless of whether it wasn’t sent because of a bug in my code or external service being down. It may be a warning if I’m still retrying, but if I gave up - it’s an error.

“External service down, not my problem, nothing I can do” is hardly ever the case - e.g. you may need to switch to a backup provider, initiate a support call, or at least try to figure out why it’s down and for how long.