Why Focusing On Data Points Can Hurt Your Business — And What You Should Do Instead

Zhen Liu

Why?

We use average a lot when analyzing product and business performance, but using average alone create blind spots. Because there are always variations due to different segments of the market or pure randomness, and the average value doesn’t tell you the variation of stories.

Example: How many product do our customers buy on average?

A company is trying to understand the average amount of items purchased by a customer. For New York and LA, they found that the average of the items purchased per customer is the same (45 items).

What should we do?

Find out the range around the mean by calculating variance.

Why?

Ratio metric consists of at least two metrics; for example, Click-Through Rate is Clicks divided by Views. With each metric’s variation, ratio metric’s variation is more complicated, and it doesn’t follow any common distribution.

Example

Let’s look at the table below first. You are measuring Click-Through Rate, from this table, it looks like Click-Through Rate increased from Jan to Feb. Sound great?

What should we do?

  1. Set a threshold for minimal acceptable value for denominator. As the ratio can have great variance when denominator is small, we only trust the ratio when denominator is large enough. If you have to use the ratio metric to make decisions when denominator is small, make sure you report a range that covers the fluctuation.
  2. Monitor the actual values (numerator, denominator) that we use for the ratio calculation. Understand the range of the ratio by simulating different scenarios of the numerator and denominator.
  • If your metric is ratio like Click-Through Rate, analyze different scenarios to see how the metric change as the denominator and numerator change. Be careful if your denominator is small, which means the ratio can be more sensitive to the change of data, and might not be reliable!
  • Visualize it to make sure we don’t miss any pattern in the data or outliers.