Posted by mooreds 9 hours ago
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
> Everyone nods their head, says "that makes sense", and we all move on with our day.
Why? Why do you need to be intimidated into actually making effective change? Why is everyone walking around with their hands tied, unable or afraid to enact change? Probably because of the terrible leadership at this company.
>> "We missed it because Alice was on holiday and Bob thought the Widgets team owned it".
> Okay. How do we make ownership unambiguous when someone is unavailable?
You cannot take this step without details. That Bob-Alice story is the details. You do actually need to know that story before you can take the step of knowing that "unambiguous ownership" would be a worthwhile change to make. What exactly is this SVP doing again?
This entire post could be summarized as "executives are useless" and "do the 5-whys", both of which everyone already knows.
I do like the term "organizational folklore" though. Actually properly documenting processes & expectations is important (and rare).
Good article write-up, I didn't need to read more than one paragraph to get all the information from the whole page.
- what happened
- impact
- root causes
- what will be changed (commitments)
You can't really talk about what to change w/o talking about root causes, and you can't talk about that without talking about what happened. You also can't consider the costs of commitments to be made w/o discussing actual and potential impact because resources are limited and the cost of opportunity is real.It's fine for executives to get only a summary of impact + commitments. But the whole post-mortem process has to be followed, and some people have to be aware of all the details.
People need to tell their stories. They need to be heard and have their work and its difficulty respected.
That's what went wrong. Everything else is an abuse victim rationalizing their abuse.
I can't be the only person who thought this is a highly intelligent person being inadvertently gaslit by the SVP into thinking the exec is right to be dismissive with "I don't want the details" because introspection is common in intelligent people.
Had someone said that to me I'd have walked away after saying "fine I'll sort it'.
This illustrates that "trust" to do the job, and since they didn't want the details of the problem, they therefore don't need the specific solution description. This kind of response is also the same level of respect, it's either seen as trust in ability, or just downright disdain with plausible deniability for being rude.
1. timeline of events
2. impact
3. 5 whys — the details of how and why things happend
4. actionable items
I've never seen a post-mortem without actionable items.
Action items need an a forcing feature (like an SLO) to insure they are picked up. And someone needs to follow up when the item isn't handled timely. It should show up in team performance metrics and guide changes to processes to insure they get done.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
In any case, the actually important question "How to make sure this won’t happen again?" would presumably include the relevant parts of what "this" is as well. But in a more actionable frame.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.