One of the ongoing challenges that we face as Product Managers is that we’re primarily charged with predicting customer and user behavior.  We’re constantly asked to come up with new ideas, new features, and new designs that we “know” will delight our users, or at the very least satisfy them.  But the fact is, predicting human behavior is incredibly difficult — there are many thousands of people who have spent hundreds of years trying to figure out why people do what they do (they’re called psychologists, sociologists, and anthropologists), and we’re still making educated guesses at best.  So, what are some of the challenges that we face?

Asking the Wrong People Leads to the Wrong Conclusions

So help me, if I hear one more company bragging about their NPS scores, I’m going to kill someone.  For those who don’t know what it is, the “Net Promoter Score” is a system through which you’re supposed to determine how satisfied your customers are with you, based on their likelihood of “promoting” your product to others.  I’d be surprised if there’s anyone here who hasn’t taken part in such an effort — knowingly or not — from the consumer side; if you’ve ever answered a survey question on an eleven-point scale (0-10) of “how likely would you be to suggest our product to others” then you’ve been NPS’d.

My problem with NPS isn’t with the theory or the science behind it — it’s actually a very well-organized, well-reasoned, and well-intentioned system.  You collect responses from your users which tell you how much they’d recommend a product or feature or your company, you collect that data, then run it through an algorithm to determine what the results are.  Bam!  You now know just what your users think of you, right!  In theory, sure…but in practice, I’ve far too often seen the following issues with NPS scoring:

• Small sample sizes make the data useless — because ratings on the lower end of the scale are given more weight than scores on the higher end (logical), if there’s a small sample size and you have one or two people rate you very poorly (possibly due to factors other than the actual product or company) then you wind up with a score that really doesn’t reflect the reality.
• Respondent bias — it’s pretty well-established that surveys are a relatively poor way to find anything other than polarized opinions.  The people most likely to respond to a survey that asks them to rate your product or your service are those who feel strongly one way or the other — and this bias in the respondents’ mindset gets reflected in the data.  If you survey 200 people, and 20 love your product, 20 hate your product, and the remaining 160 don’t care much one way or the other, you’re likely to see really bad NPS scores when they should be in the middel.
• Selection bias — The other problem that I’ve seen in the use of NPS scores is selection bias on the part of the company.  Sure, I’ve heard it justified by “We really want to know what our top accounts think,” or “We usually just get more responses from companies we have a strong relationship with,” or something of that nature.  But what it really is, is about making the NPS score into a vanity metric, by asking people who we know will only give us good scores, or “scrubbing” the data to remove “outliers” that are subjectively disqualified.

When used correctly and at the right scale, NPS can be a very useful tool for gaining insight into how our company, product, or even teams are viewed by the greater world of our customers and prospects.  When used incorrectly, NPS scores show that when we ask the wrong people we can easily be led to the wrong conclusions.

Asking People What They Will Do Doesn’t Work

People are incredibly bad at predicting what they will do in the future.  So many product managers, marketers, salespeople, and CEOs have run into this time and again that it’s astounding to me that we still rely on the age-old question, “So, would you pay $20 for this service every month?” as some reliable indicator that we’re on the right track.  This should not be news to anyone out there — even Margaret Mead (the famous anthropologist) once said, “What people do, what people say, and what people say they do are entirely different things.”  Yet, when our backs are up against the wall, we still ask people if they would use such-and-such feature in their jobs, whether they would pay money for an upgrade, whether they would change their behavior in the future based on something we’re only describing to them in the present.  It’s ridiculous, it’s faulty, and we still fucking do it. WHY?  Because it’s far easier to ask someone and trust their response than it is to admit that people suck at predicting what they’re going to do tomorrow, much less what they’re going to do a year from now.

Instead of asking people what they’ll do in the future, consider some of these other options:

• Analyze what they’ve done in the past — sometimes past behavior is the best indicator of future behavior; but beware the limits of this…
• Actually make them do the thing that you’re asking them to do — if you want them to pay, ask them to pay up-front for a pre-order or a beta or something else (cf: the video game industry, pumping out buggy, day-one patch games for decades yet still raking in hundreds of thousands of dollars in pre-orders for most AAA titles).
• Change your questions entirely — rather than asking whether they will use something, ask them how they would use it if they had it today, right now, there in front of them.

Every time we ask a user what they will do tomorrow, next week, or three months from now, a devil gets his horns.  Well, maybe not quite that bad, but we’re getting information that we should know is of questionable predictive value — and we should rely on it accordingly, not make it the fundamental basis of an entire business model.

Past Behavior Only Indicates Future Behavior If Nothing Changes

What would an article on the Clever PM’s blog be without me contradicting myself at least once?  So,  you’ll notice above that I suggested looking at past behavior to predict future behavior — and that’s true to a large extent.  Assuming that the sphere of influence and action/reaction remains relatively stable, you can use past behavior as a pretty strong predictor for future behavior.  If you have a group of users who have paid $0.99 every two months for an update to your iOS app, then it’s probably very likely that those same users will pay $0.99 every two months in perpetuity for updates of similar quality assuming all other things are equal.

And therein lies the rub — how often, over a course of time beyond a few months — are all other things really equal?  There are so few products that exist in a vacuum that it’s silly of us as Product Managers to presume that “all things are equal” over any significant time horizon.  Products change, markets change, financial conditions change, political conditions change — and all of that can have an effect on our inferences from past behavior.  How do you predict a downturn in disposable income among your target market?  How do you predict a competitor’s product suddenly undercutting you on price?  How do you predict the fallout of a news scandal involving your company or its leadership?  Sure, all of those are pretty major events that might be unlikely in your particular market or with your particular product — but you do everyone who relies on you a disservice if you willingly ignore the fact that your landscape is constantly changing.

So, while using past behavior to predict future behavior is useful in the short term, the further away you attempt to strain that connection from the moment in time in which the last action took place, the less likely you are to be right.