Goodhart's law for Marketing and Business - How to overcome over optimization

Goodhart’s law simply states that “When a measure becomes a target, it ceases to be a good measure”. We see this happen in everyday life far too often but may not realize its effects.

For example, a state in India decided to measure the success of its efforts toward education with the percentage of students who passed in 10th class exam. The administrations tried to improve education, but that’s a long process and can take decades to show results whereas the elections happen every 5 years. If a government wants to show improvements in results, they need more direct and quick ways. The administration figured a more manageable approach was to make the exams easier. They published small guides which would have a set of questions that were most likely to be asked in an exam and prepared exam papers from those question sets. Students now had a quick way to pass the exam with less study. The method proved effective. The passing percentages improved. But instead of improving education, the quality of education went down and students found it harder to cope with competitive exams or the job market which would value them on skills rather than exam scores. The metric you optimize for can cause the system to work against the intended goal.

It's also common in marketing. Imagine a campaign optimizing for Cost of conversions. It may increase conversions, but it will reduce the order values. Meaning a lower return on Ad spend.

If a campaign is optimizing aggressively for cost per lead, it can result in many low-quality leads. It needs a cautious and aware marketer to choose good metrics to optimize for.

Solution

No one solution can fix the paradox of the law. Whatever you choose, will have a caveat, A trojan that you would always have to be careful about.

Ideally, the best metric to optimize for is the sustainability of a business, but it's extremely hard to measure and delegate. An organization is made up of multiple functions—each with its parts to play. When you break down the relationship between smaller/quick-to-measure metrics with the larger organizational goal, make sure you put checks and balances that counteract the effect of over-optimization.

Banks do it well, with the sales and risk teams. The sales team brings the clients and the risk team evaluates those for risks to ensure that in pursuit of growth it's not leaving itself exposed to default risks.

1. Make sure you have a way for everyone to see how metrics interact. Most often, large organizations create data silos that prevent one team from having a view of how the business is doing overall.
2. Some companies have monetization models that could take months to get an idea of quality.
3. Complex business models prevent employees to get a view of how one metric affects another.

A good Analytics setup can solve the above 3 problems.

1. Connect data sources to a central system to eliminate data silos
2. Create centralized & transparent dashboards that give people an idea of the relationship between their efforts and business health
3. Machine Learning models to check the impact of marketing on business health

It's extremely difficult to break down the impact of external factors/ internal optimization on what moves the needle. It's difficult for people to claim responsibility for failure. It's far too easier to quote relationships with a confirmation bias. A centralized Analytics team that keeps everyone honest is essential for everyone to get insights on what can be course corrected and by how much.

Be cautious and Happy optimization.

Happy holiday season.