Imagine you’ve just been named the head of a bustling New York City restaurant challenged by one issue - customers complain about the customer service. A data-driven person, you search for a metric to evaluate the current customer service to validate the complaint and then track as you experiment with the restaurant’s operations. What metrics would you employ?

You might run a survey of customers at exit. You could follow with customers by telephone the day after the meal. You might ask finance to tabulate the tip amount as a percentage of the total bill and aggregate by waiter.

These are fine metrics, but they share one common problem. They are lagging indicators of customer service. By the time these metrics have been tracked, it’s too late to correct the experience customer. Just how important is latency to making operational decisions?

To explain, I’m going to share a memory from business school. There’s a game students at most business schools play called the Beer Game, which was invented at MIT Sloan School. Our marketing professor divided the class into four groups, for each of the key four parts of the beer production value chain: brewer, bottler, distributor and retailer. It’s a turn based game, and each turn the four groups chose how much to brew, bottle, distribute and stock. Our professor played the part of the customer and told the retailer how many bottles he would buy.

Latency is a key wrinkle in the game. Players pass demand and supply information through pieces of paper, but upstream demand and downstream orders are delayed by two weeks. Demand oscillates in very small amounts. The retailer, acting on slow information, overcorrects just a little bit. Then the distributor overreacts just a little bit more; the same for the bottler. And the brewer is pulling out his hair when sales boom one week and go into total freefall the next. It happens every time, and it’s because the value chain’s data suffers from latency. It’s called the Bullwhip Effect because small changes on one end of the value chain create huge disruption on the other.

The same idea holds for a startup. The four key parts of the value chain are product, marketing, sales and customer success. If there’s a lot of latency between the time a customer provides feedback and it reaches product management or marketing, there’s a potential to overcorrect and waste a lot of time and energy.

How do you solve this latency issue? With a proxy metric. Getting back to the restaurant example, a simple way of evaluating customer service is to measure how full water glasses are. Ask the hostess to walk the aisles, and tabulate the fraction of water glasses at each table that are less than half full. It’s not a perfect measurement, but it’s a great leading indicator that several waitstaff are paying attention to the table.

Proxy metrics are powerful for four reasons.

1. They are easy to measure.
2. They are correlated to (and ideally predictive of) the business’ goals
3. They are concrete and create a unifying vision for the team
4. They enable businesses to iterate faster by reducing latency

Great companies employ proxy metrics all the time. If a new Facebook user creates 7 friends in 10 days, they’ll be a long term user. The more engagement minutes a customer spent digging through their data in Looker during a trial, the greater the likelihood they convert to a paid customer. At ThredUp, the world’s largest online consignment store, buying products both on web and a mobile application usage indicates high customer lifetime value. If a user creates an expense policy on Expensify, there’s a very high chance a company will buy the product.

How do you find proxy metrics for your business? There’s no shortcut. Andrew Chen has written a blog post describing how to do it, step by step. Define the goal, explore the data, run a regression and then backtest. For more complex behaviors, like a network effect, you may need to find another statistical modeling technique.

Proxy metrics are powerful because they enable businesses to use an upstream-metric to predict a downstream outcome, which reduces latency and accelerates experimentation cycle times. Faster experiments means discovering the right growth strategies sooner, and that’s a critical competitive advantage for every startup.

Thanks to Lloyd Tabb, who inspired this post