2016 Statistical Computing and Graphics Award Honored William S. Cleveland for his many contributions to data visualisation and data science. In the session at the joint statistical meetings, we did a re-run of the experiment.
Hierarchy of graphical mappings
The paper, Cleveland and McGill (1984), reported on results from an experiment on ways to map data that enables the reader to accurately return the original data values. The end result was the following hierarchy of mappings from most accurate to least:
- Position along a common scale e.g. scatter plot, bar chart
- Position on identical but nonaligned scales e.g. bars on spatial map in 3D, facetted plots, stacked bars
- Length e.g. candlestick plots, boxplots
- Angle, slope e.g. pie chart
- Area e.g. bubble chart
- Volume, curvature, shading e.g. glyphs
- Color hue e.g. heatmap, chloropleth map
Heer and Bostock (2010) ran a very similar experiment using subjects from Amazon’s Mechanical Turk, and effectively reproduced CM84’s results.
Stimuli
The audience was asked to answer two questions for a sequence of plots:
- Which is bigger
aorb? - How much bigger? (2x, 1.5x, …)
Here are some examples of plots in the survey.
Five sets of data were generated. The survey is still available at: http://bit.ly/JSM-vis16.
Results
90 people filled in the survey.
The first question is measuring whether the reader can select the larger of the two elements from the different displays.
The table shows the proportion correct by the plot type for each set of data. Plot type columns are sorted by overall proportion, so from most accurately picked the larger slice, bar, circle, or lighter color. It shows that people were more accuarate for bar charts, but almost as accurate with pie charts and bubble charts. A substantial drop in correct choice for color and unaligned bars. The same information is shown in barchart form below:
This yields a hierarchy:
- Position along a common scale
- Angle
- Area
- Color hue
- Position on identical but nonaligned scales
This is the same order reported by CM84, except for unaligned scales.
For the second question, we differenced the reported ratio with the true ratio, and plotted these against the plot type. Closer to the zero line would indicate more accurate recognition of the relative size.
The plot order matches the table. Barcharts are centered and concentrated at 0. The difference for pie charts are very close to zero, but mostly above and with more variability. It suggests people tend to over-estimate the ratio. Bubble charts have differences close to zero too, but on the lower size, suggesting that people might under-estimate the ratio. Color is similar to bubble charts, perhaps under-estimating the ratio. Differences for unaligned bars are mostly above zero, with more variability than other plot types. The results would suggest the same hierarchy as above (1) Position along a common scale (2) Angle (3) Area (4) Color hue (5) Position on identical but nonaligned scales, although color and unaligned are virtually tied.
Conclusion
Yes, we validated Cleveland and McGill’s hierarchy (almost) in the space of 20 minutes using crowd-sourcing! Thanks to all for participating!
Caveat
Accuracy is not the only metric to measure the value of data graphics. If that is all that is of interest a table would be superior. But the reader should be able to roughly return the data values, and get a larger sense of of relationships, odd sub-structures, more information.
Resources
Full code and plots and report are available at https://github.com/dicook/JSM16.