Posted by yagizdegirmenci 20 hours ago
https://commoncog.com/becoming-data-driven-first-principles/
https://commoncog.com/the-amazon-weekly-business-review/
(It took that long because of a) an NDA, and b) it takes time to put the ideas to practice and understand them, and then teach them to other business operators!)
The ideas presented in this particular essay are really attributed to W. Edwards Deming, Donald Wheeler, and Brian Joiner (who created Minitab; ‘Joiner’s Rule’, the variant of Goodhart’s Law that is cited in the link above is attributed to him)
Most of these ideas were developed in manufacturing, in the post WW2 period. The Amazon-style WBR merely adapts them for the tech industry.
I hope you will enjoy these essays — and better yet, put them to practice. Multiple executives have told me the series of posts have completely changed the way they see and run their businesses.
As "Goodhart's law" is used here, in contrast, the focus is on side effects of a policy. The goal in this situation is not to make the target useless, as it is if you're doing central bank policy correctly.
Business leaders like to project success and promise growth that there is no evidence they will or can achieve, and then put it on workers to deliver that, and when there's no way to achieve the outcome other than to cheat the numbers, the workers will (and will have to).
At some point businesses stopped treating outperforming the previous year's quarter as over-delivering, and made it an expectation, regardless of what is actually doable.
Do you have enough KPIs that you can be sure that these targets also serve as useful metrics for the org as a whole? Do you randomize the assignment every quarter?
As I talk through this ... have you considered keeping some "hidden KPIs"?
can't say what the deep idea in this case is per se (haha (maybe the other commenter can shed light on that part)), but i guess if you have enough KPIs to be able to rotate them you have yourself a perpetual motion machine of the same nature as the one that some genius carried down from the mountain on stone tablets that we can sustain maximum velocity ad infinitum by splitting our work into two week chunks and calling them "sprints"... why haven't marathoners thought of this? (h/t Rich Hickey, the source of that amazing joke that i butcher here)
maybe consciousness itself is nothing more than the brain managing to optimize all of its KPIs at the same time.
They need something they can check easily so the team can get back to work. It's hard to find metrics that are both meaningful to the business and track with the work being asked of the team.
Its easy to fake one metric, it harder to consistenly paly around 100 of them.
(But then it’s no longer KPIs probably, as one looking at the data needs to recognise that details and nuance are important)
If companies knew how to make it difficult to distort the system/data, don't you think they would have done it already? This feels like telling a person learning a new language that they should try to sound more fluent.
I might be wrong but I feel like WBR treats variation (looking at the measure and saying "it has changed") as a trigger point for investigation rather than conclusion.
In that case, lets say you do something silly and measure lines of code committed. Lets also say you told everyone and it will factor into a perforance review and the company is know for stack ranking.
You introduce the LOC measure. All employees watch it like a hawk. While working they add useless blocks of code an so on.
LOC commited goes up and looks significant on XMR.
Option 1: grab champagne, pay exec bonus, congratulate yourself.
Option 2: investigate
Option 2 is better of course. But it is such a mindset shift. Option 2 lets you see if goodhart happened or not. It lets you actually learn.
(a) All processes have some natural variation, and for as long as outputs fall in the range of natural process variation, we are looking at the same process.
(b) Some processes apparently exhibit outputs outside of their natural variation. when this has happened something specific has occurred, and it is worth trying to find out what.
In the second case, there are many possible reasons for exceptional outputs:
- Measurement error,
- Failure of the process,
- Two interleaved processes masquerade as one,
- A process improvement has permanently shifted the level of the output,
- etc.
SPC tells us that we should not waste effort on investigating natural variation, and should not make blind assumptions about exceptional variation.
It says outliers are the most valuable signals we have, because they tell us we are not only looking at what we thought we were, but something ... else also.
Exactly (btw. very nice way to put it)
> stop wasting time on tracking false proxies
Some times a proxy is much cheaper. (Medical anlogy of limited depth: Instead of doing a surgery to see stuff in person, one might opt to check some ratios in the blood first)
This would not count as a false proxy however. The problem in software is, it is very hard to construct meaningful proxy metrics. Most of the time it ends up being tangential to value.
In theory you can return to the metrics later for shorter intervals.
Can't it? Amazon may be an exception, but most of the time running without numbers or quantitative goals seems to work better than having them.
This seems to pop up in a lot of areas and I find myself asking is X thing a thing I really desire or is it something that is a natural side effect of some other processes.
Once you start looking for these things that are done for their own sake (or really to gain respect in a community) you notice how pervasive they are and how different they can be for two people next to each other.
I recommend Gregory's Savage Money on the subject. My review here: https://entropicthoughts.com/book-review-savage-money
You can also ask what is life about?
This is hard to do because the conclusion may need to break moulds, leading to family estrangement and losing friends.
I suspect people who end up having a TED talk in them are people who had the ability through courage or their inherited neural makeup to go it alone despite descenting voices. Or they were raised to be encouraged to do so.