Take the electric razor. About a century ago, one of the first models—the Vibro-Shave— entered the US market. This razor, which was marketed to both men and women, had a vibrating handle that moved a blade from side to side. And, if you were so inclined, you could replace the top with a massage head to “smooth out” your wrinkles.
Needless to say, razors have changed over the years.
So how does product development typically happen? Companies have long relied on market research to determine how customers use their products and whether they have underlying needs that a new feature or innovation can fill. Much of this research traditionally involves interviews or focus groups with customers, who share how they use a product, what they like and don’t like. Companies then synthesize this feedback to determine whether customer needs are being met and act on that knowledge.
But interviews and focus groups are expensive and can take a huge amount of time, he says Artem Tymoshenko, assistant professor of marketing at Kellogg. “Being in the market with a new razor half a year before your competitor gives you the lead.”
So Artem Tymoshenko and his colleague John Houser of MIT Sloan wondered whether it was possible to glean similar information about customer needs from existing customer feedback—that is, user-generated content such as Amazon reviews or social media data.
They had two specific questions: First, could professional market research analysts derive useful information from these assessments? And second, could machine learning algorithms allow them to do this more efficiently?
Mining Product Reviews
To answer this first question, the researchers brought in a marketing consulting firm called Applied Marketing Science, Inc. (AMS). AMS has over twenty years of experience in market research and eliciting customer needs, and recently conducted a customer interview-based study on customer needs for oral care products.
“It was very convenient from both a business and a research point of view,” explains Timoshenko, since toothbrushes represent a fairly standard product category and one with plenty of Amazon reviews. In addition, AMS was excited about the researchers’ questions and the company was willing to cooperate.
When it comes to oral care products, many customers report needs that are pretty straightforward: the products should keep their teeth clean and white, keep their gums healthy, and not harm any previous dental work. But other clients may report less anticipated needs, such as knowing how much time they will spend in different parts of their mouths during their oral care routine. This can lead to product ideas such as toothbrushes that beep at set intervals or turn off after a certain number of minutes.
Experiential interviews conducted by AMS revealed 86 different customer needs for oral care products, a typical number for such a product category. The goal of analyzing these customer needs is to find a hidden gem: a need that is very important, but that existing products do not meet well.
To determine whether marketers can glean the same information about customer needs—and potential hidden gems—from user-generated online reviews as they can from interviews and focus groups, the researchers randomly selected a subset of Amazon reviews. for oral care products and provided that to a panel of AMS analysts. These analysts were not the ones who had collected or analyzed the customer interviews, but had similar training. Each of the subset reviews was presented to the analysts in its entirety, and together the reviews added up to 12,000 sentences—which took about the same amount of time for the analysts to review as a typical set of 20–25 experiential interview transcripts.
Going into the study, Timoshenko and Hauser felt that Amazon reviews might have some advantages over traditional customer interviews. For example, perhaps they offered access to a customer population that was unlikely to participate in a focus group.
“We could imagine that if a company was in Boston, they would be interviewing mostly Bostonians,” says Timoshenko. “But maybe people in other regions have different product experiences and usage patterns.”
Another potential advantage is that customers tend to write online reviews immediately after using something. Participants in a focus group, on the other hand, may have used the product a month or two before the interview and have already forgotten key parts of their experience.
However, the researchers also suspected that online reviews may have a significant downside. Specifically, “there’s a lot of research that suggests that online reviews are skewed toward extremely positive or extremely negative,” says Timoshenko. “So we may be missing some of the customer needs that are usually expressed in more neutral language.”
For example, the fact that a toothbrush actually cleans teeth—an important but unexciting use—might not be something a customer would bother to mention. This was a major concern, as articulating the entire set of customer needs can help product management teams identify new product opportunities, even when some of the customer needs are not surprising. after the fact.
So what did the researchers find? First, almost 97% of the customer needs identified in the interviews and focus groups were also found in the Amazon reviews.
“This immediately suggests that, at least for some categories, we are able to completely eliminate the need to conduct interviews and focus groups,” says Timoshenko. “And that’s the most time-consuming part of market research for customer needs.”
The second finding was that Amazon reviews contained eight additional customer needs (nearly 10 percent of the total) that were not mentioned during the interviews. These were not substantially different from what customers reported – they appeared to be equally important to customers and useful for future product development – suggesting that analyzing user-generated reviews could provide a more comprehensive picture of customer needs .
Tymoshenko suspects that if additional interviews and focus groups had been conducted, these needs would have emerged eventually. “But doubling the number of interviews you do is much more expensive, in money and time, than simply doubling the amount of online content we review.”
Machines that help people
The researchers then tried to see if they could use machine learning to make human analysts more efficient. Specifically, they built an algorithm to “discount” reviews, eliminating the least useful ones so that analysts can make more productive use of their time.
The researchers trained an algorithm to discount reviews in two ways: remove uninformative sentences and reduce redundant ones. Non-informative sentences, which make up nearly half of the sentences in the body, could simply say, “My son loves this product”—a perfectly legitimate sentiment, but not one that will lead to product innovation. Redundant reviews, also prevalent in the body, mention the same shortcoming or privilege over and over again.
The researchers found that preliminary screening by their algorithm allowed analysts to find the same number of customer needs in about 20 percent fewer proposals.
“That was the proof of concept,” says Timoshenko. He is confident that with more experience and engineering, efficiency will continue to increase, just as methods for traditional interview-based market research have improved over years of practice.
To that end, the researchers have made their code freely available to companies and are eager to learn how it is further developed and implemented by companies in different industries.
A company in the food industry, for example, used the researchers’ methods and found that they identify very different kinds of customer needs depending on whether they are looking for online reviews or social media data.
Tymoshenko says this highlights the fact that, as multiple sources of feedback are considered, the need for machine learning tools will increase.
“There is an even greater need to pre-process this information,” he says. “Because there are millions of Amazon reviews for a particular product — but if you want to combine that with social media data and online reviews from other sources, it just blows up the amount of content you have to process. And that makes machine learning very important.”
Unexpected benefits
In doing their research, Timoshenko and Hauser found that analyzing user-generated content has another, completely unexpected, advantage over traditional interviews and focus groups: the ability to “follow up” on an interesting comment or need client to delve deeper.
In a traditional interview setting, he explains, “you don’t have the opportunity to call the same interviewee and talk about that experience. It’s a missed opportunity.”
With user-generated content, on the other hand, you can really explore further. With an interesting prospect in mind, you can go back through the entire pool of thousands of reviews to look for additional clues. “You don’t go to the exact same customer review, but you could search for the keyword, a specific phrase, or the specific experience,” says Timoshenko.
Overall, he wants marketers to understand that machine learning can be a powerful tool—not just to replace human intelligence, but to augment it.
“One of the big breakthroughs in this research was when we agreed on the idea that machine learning can’t solve all the challenges of this process,” says Timoshenko. “Most people, when they think of machine learning, they look for fully automated solutions. It seems that humans are much better, of course, at some tasks than machines. And they will keep getting better for the foreseeable future. And shaping customer needs is one of those tasks.”
A customer might say, “I don’t like this toothbrush because it doesn’t have a 30-second timer.” But the underlying customer need wants to know how much time to spend on different parts of your dental routine.
“It’s very abstract. It is very conceptual what the customer really wants. So this step is best done by humans, who can really learn and understand the human experience of other customers.”
