Do you spot the problem here? While many of us have heard the phrase “correlation is not causation,” some of us have a harder time separating the two in practice. In this case, the relationship between app payment and spending could be causal, but it can also be boiled down to another simple fact: the people who download the app are already your best customers.
The problem can be exacerbated when graphs and other data visualizations enter the picture. This is because our powerful visual system is designed to look for patterns. In fact, according to new research from the Kellogg School, how data is visualized can significantly affect our interpretation of what we see—sometimes for the worse. Specifically, the study found that attempts to simplify graphs by grouping data into smaller numbers of “buckets” (say, two bars on a graph as opposed to ten) seemed to lead people to mistake correlation for causation.
“How you present your data really matters,” he says Cindy Xiong, who received her doctorate in psychology from Northwestern and is now at the University of Massachusetts, Amherst, and lead author of the study. “Your design decisions will trigger different initial reactions from people and ultimately different decisions from people.”
In a business context, these decisions can be costly, he says Joel Shapiroclinical associate professor of data analysis at Kellogg and co-author of the study.
“We see a lot of companies taking key business outcomes — customer retention, employee retention, patient adherence, whatever those are — and doing big data mining exercises,” he explains. But if the visualizations used to present the results of this data mining lead people to false conclusions of causation, then companies may make strategic investments that do not work out as planned.
So viewer beware: “Graphs can be powerful,” says co-author Steven Franconeri, a Northwestern psychology professor with a courtesy appointment in Kellogg’s marketing department. But it is imperative that you know how to choose the right one for the right data. “You can lead your colleagues and clients to think more rationally about these kinds of problems.”
Line Graphs and Circles and Lines (Oh My)
The research team—which included Franconeri, Shapiro and Xiong, along with Northwestern computer science professor Jessica Hallman—began by looking at four common visualization styles: bar charts, line charts, scatterplots, and plain text. They wanted to know if these styles differ in how “causal” they make the data appear to people.
They chose four associations to visualize: people who smoke more have a higher risk of developing lung cancer; Students who eat breakfast more often tend to have higher grade point averages. People who spend more on entry to sporting events tend to be physically fitter. And as more people use Internet Explorer, the homicide rate in the US tends to rise. While all of these links are correlated (yes, even that last one), they are not all causal. And, above all, they differ in manner reasonable correlated and causally seen in humans, according to an experimental pretest.
The researchers then developed four different visualizations for each scenario, similar to the ones below, to show to 136 online participants.
Each participant saw four visualizations: one representation of each scenario (smoking and cancer, breakfast and GPA, spending and fitness, and Internet Explorer and homicide) in each of the styles (text, bar graph, line graph, scatter plot). In other words, a single participant might see smoking and cancer in text, breakfast and GPA in a bar graph, spending and fitness in a line graph, and Internet Explorer and homicide in a scatter plot. The researchers randomized the order in which the scenarios and visualization styles were presented.
Participants were then asked to describe in several sentences what they inferred from each of the illustrations they saw. They then rated from 0 to 100 how much they agreed with two statements. For example, in the breakfast and GPA scenario, participants were asked: “Based on the graph, students who eat breakfast more often tend to have a higher GPA” (accurate correlation statement) and “Based on the graph, if students were to eat breakfast more often, they would have a higher GPA’ (inaccurate statement of causation).
For the association statements, the researchers found that the type of visualization did not significantly affect participants’ ratings after controlling for the relative plausibility of each scenario.
However, for causal statements, visualization style appeared to matter. Participants rated the information they saw in the bar graphs as the most causal (78 out of 100, on average) and the graphs as the least causal (67 on average).
More concentration, more assumptions of causation
What was happening?
The researchers were not convinced that an inherent property of bar charts or scatter plots was actually driving this pattern. They suspected it had something to do with the “aggregation” level of the visualizations, or how many bins the data was split into. Bar charts were highly aggregated—that is, they split the data into just two bars, compared to the scatter chart, which presented each data point as its own dot.
So the team did the same experiment again, with a few modifications. Most importantly, they developed bar charts, line charts, and scatter plots with three different levels of aggregation.
They recruited 129 new participants and repeated the same procedure as before, once again randomizing visualization style, concentration level, and the order in which items were presented. They had participants describe the visualization they had seen and rate from 0 to 100 how correlated and causal the relationship seemed.
This time the type of graph had only a small effect on participants’ ratings of causality. Far more important, however, was the effect of concentration level. Across the different visualization styles, people saw aggregated data as more causal than less aggregated data. In other words, a two-point scatterplot made a relationship between two variables look more causal than a 16-point scatterplot, and the same for bar and line graphs. Researchers hypothesize that people interpret less aggregated data as closer to raw data, where interpretation is up to the viewer.
The researchers wanted to make absolutely sure that the effects they were seeing came from the concentration and not the type of visualization. So they created an unconventional visualization that eliminated all aggregations and had a new group of participants examine them.
The result: across all visualization styles, non-centralized visualizations (those showing more data values) in Experiment 3 were rated as less causal than less centralized visualizations in Experiment 2.
How to Visualize Responsibly
Does this all point to a single better way to visualize data? Not at all, says Xiong.
“I don’t mean to imply that there is always a single correct way to display your data. I don’t think that’s the case,” he says. “But I want to say, be careful when you aggregate, because the assumption that people have can become causal. So think carefully about how you present your data and maybe iterate with your own design team or your data analysts so people don’t misinterpret what you’re trying to say.”
Careful visualization is increasingly important in an age where technology has made it easy to capture and present all kinds of data.
“There’s so much measurement, there’s so much good technology, there’s so much computational complexity,” Shapiro says. But even the most sophisticated tool rarely, by itself, determines causation. “It’s really, really good at finding trends.”
So if you see an intriguing trend, don’t automatically assume causation, Shapiro warns—instead, view it with skepticism. Is there something else that can explain why these two factors increase and decrease in parallel? If not, “you can run business experiments to say ‘x really causes y’. That’s the next step. But the first step is good critical thinking.”
