Human Perception of Statistical Charts: An Introduction to Graphical Testing Methods

class:title-slide-custom

<style>

/* colors: #EEB422, #8B0000, #191970, #00a8cc */
/* define the new color palette here! */
a, a > code {
  color: #8B0000;
  text-decoration: none;
}

.title-slide h2::after, .mline h1::after {
  content: '';
  display: block;
  border: none;
  background-color: #8B0000;
  color: #8B0000;
  height: 2px;
}

.remark-slide-content {
  background-color: #FFFFFF;
  border-top: 80px solid #8B0000;
  font-size: 20px;
  font-weight: 300;
  line-height: 1.5;
  
  background-image: url(css/UNL.svg);
	background-position: 2% 98%;
	background-size: 10%;
	border-bottom: 0;
}

.inverse {
  background-color: #8B0000;
  
  
	
  
}

.remark-slide-content > h1 {
  font-family: 'Roboto';
  font-weight: 300;
  font-size: 45px;
  margin-top: -95px;
  margin-left: -00px;
  color: #FFFFFF;
}

.title-slide {
  background-color: #FFFFFF;
  
  background-image:  url(css/UNL.svg);
	background-position: 98% 98%;
	
	background-size: 20%;
	border-bottom: 0;
	border: 10px solid 	#8B0000;
	
}

.title-slide > h1  {
  color: #111111;
  font-size: 32px;
  text-shadow: none;
  font-weight: 500;
  text-align: left;
  margin-left: 15px;
  padding-top: 80px;
}
.title-slide > h2  {
  margin-top: -25px;
  padding-bottom: -20px;
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 28px;
  text-align: left;
  margin-left: 15px;
}
.title-slide > h3  {
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 28px;
  text-align: left;
  margin-left: 15px;
  margin-bottom: -20px;
}

body {
  font-family: 'Roboto';
  font-weight: 300;
}

.remark-slide-number {
  font-size: 13pt;
  font-family: 'Roboto';
  color: #272822;
  opacity: 1;
}

.inverse .remark-slide-number {
  font-size: 13pt;
  font-family: 'Roboto';
  color: #FAFAFA;
  opacity: 1;
}

.title-slide-custom .remark-slide-number {
  display: none;
}

.title-slide-custom h3::after, .mline h1::after {
  content: '';
  display: block;
  border: none;
  background-color: #8B0000;
  color: #8B0000;
  height: 2px;
}

.title-slide-custom {
  background-color: #FFFFFF;
  
  background-image:  url(css/UNL.svg);
	background-position: 98% 98%;
	
	background-size: 20%;
	border-bottom: 0;
	border: 10px solid 	#8B0000;
	
}

.title-slide-custom > h1  {
  color: #111111;
  font-size: 40px;
  text-shadow: none;
  font-weight: 500;
  text-align: left;
  margin-left: 15px;
  padding-top: 80px;
  padding-bottom: 10px;
}
.title-slide-custom > h2  {
  margin-top: -25px;
  padding-bottom: 30px;
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 32px;
  text-align: left;
  margin-left: 15px;
}
.title-slide-custom > h3  {
  margin-top: -25px;
  padding-bottom: -25px;
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 32px;
  text-align: left;
  margin-left: 15px;
}
.title-slide-custom > h4  {
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 28px;
  text-align: left;
  margin-left: 15px;
  margin-bottom: -30px;
  padding-bottom: -25px;
}
.title-slide-custom > h5  {
  color: #111111;
  text-shadow: none;
  font-weight: 100;
  font-size: 24px;
  text-align: left;
  margin-left: 15px;
  margin-bottom: -40px;
}

</style>

<br><br>
# Human Perception of Statistical Charts: An Introduction to Graphical Testing Methods
## UNL Nerd Nite @ Saro Cider
### October 13, 2021
#### Emily A. Robinson
#### Department of Statistics, University of Nebraska - Lincoln
##### <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:black;overflow:visible;position:relative;"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg> [emily.robinson@huskers.unl.edu](emily.robinson@huskers.unl.edu)
##### <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:1em;width:1.12em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:black;overflow:visible;position:relative;"><path d="M280.37 148.26L96 300.11V464a16 16 0 0 0 16 16l112.06-.29a16 16 0 0 0 15.92-16V368a16 16 0 0 1 16-16h64a16 16 0 0 1 16 16v95.64a16 16 0 0 0 16 16.05L464 480a16 16 0 0 0 16-16V300L295.67 148.26a12.19 12.19 0 0 0-15.3 0zM571.6 251.47L488 182.56V44.05a12 12 0 0 0-12-12h-56a12 12 0 0 0-12 12v72.61L318.47 43a48 48 0 0 0-61 0L4.34 251.47a12 12 0 0 0-1.6 16.9l25.5 31A12 12 0 0 0 45.15 301l235.22-193.74a12.19 12.19 0 0 1 15.3 0L530.9 301a12 12 0 0 0 16.9-1.6l25.5-31a12 12 0 0 0-1.7-16.93z"/></svg> [www.emilyarobinson.com](https://www.emilyarobinson.com/)
##### <svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:black;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> [earobinson95](https://github.com/earobinson95)
<br>
.medium[*Slides: https://bit.ly/3msOdKX*]

???

Thank you, everyone for coming! I am a PhD candidate in the Department of Statistics at the University of Nebraska - Lincoln. I will be presenting on human perception of statistical charts and giving an overview of current graphical testing methods then introduce the current research I am conducting in graphical testing.

---
class:primary
# Introduction to Graphics

Data visualization is defined as the art of drawing **graphical charts** in order to display data (Unwin, 2020).

**What are graphics useful for?** (Lewandowsky and Spence, 1989)

📉 Data cleaning.

🔍 Exploring data structure.

🗯 Communicating information.

**Who uses graphics?**
+ Governments (Harms, 1991; Playfair, 1801; Walker, 2013).
+ Companies (Chandar, Collier, and Miranti, 2012; Yates, 1985).
+ News sources and mass media (Aisch, Cohn, Cox, et al., 2016).
+ Scientific publications (Gouretski and Koltermann, 2007).

???
To get started, we are first going to lay the foundation of graphics. Data visualization has become central tool in modern data science and statistics. Unwin 2020 defines data visualization as the art of drawing graphical charts in order to display data.

Graphics are useful for data cleaning, exploring data structure, and communicating information.

In the 18th and 19th century, governments began using graphics to understand population and economic interests. In the 20th century, we saw companies using graphics to understand the inner workings of their business and support their business decision. We often see news source displaying graphics of weather forecasts such as hurricane trajectories. Today, we see graphics everywhere from scientific journals to mass media in the newspapers, TV, and internet.

Despite the popularity of graphics, we are too accepting of them as default without asking critical questions about the graphics we create or view (Unwin, 2020). We must begin asking ourselves **How effective is this graph at communicating useful information?**

Higher quality of technology has influenced the creation, replication, and complexity of graphics. We now have an infinitely many number design choices:
+ variables displayed, type of graphic, size of graphic, aspect ratio, colors, symbols, scales, limits, ordering of categorical variables

There is a need for an established set of concepts and terminology to build their graphics from so they can actively choose which of many possible graphics to draw in order to ensure their charts are effective at communicating the intended result.

---
class:primary
# Grammar of Graphics (Wilkinson, 2013)

Graphics are viewed as a mapping **from variables** in the data set **to visual attributes** on the chart.

.center[
<img src="images/building-a-masterpiece-Allison-Horst.png" width="65%" />

Building a masterpiece, by Allison Horst
]

???

Graphics are viewed as a mapping **from variables** in the data set **to visual attributes** on the chart.

+ **aesthetics:** links between data variables and graphical features (position, color, shape, size)
+ **layers:** geometric elements (points, lines, rectangles, text, …)
+ **transformations:** transformations specify a functional link between the data and the displayed information (identity, count, bins, density, regression). Transformations act on the variables.
+ **scales:** scales map values in data space to values in the aesthetic space. Scales change the coordinate space of an aesthetic, but don’t change the underlying value (so the change is at the visual level, not the mathematical level).
+ **coordinate system:** e.g.polar or Cartesian
+ **faceting:** facets allow you to split plots by other variables to produce many sub-plots.
+ **theme:** formatting items, such as background color, fonts, margins…

Software, such as Hadley Wickham’s ggplot2, aims to implement the framework of creating charts and graphics as the grammar of graphics recommends.

---
class:primary
# Testing Statistical Graphics

Evaluate design choices and understand cognitive biases through the use of **visual tests**.

Could ask participants to:

📊 identify differences in graphs.

📖 read information off of a chart accurately.

🌎 use data to make correct real-world decisions.

✏️ predict the next few observations.

???
One way we can evaluate these design choices through the use of graphical tests.

Could ask participants to:

- identify differences in graphs.
- read information off of a chart accurately.
- use data to make correct real-world decisions.
- predict the next few observations.

All of these types of tests require different levels of use and manipulation of the information presented in the chart.

---
class:inverse
<br>
<br>
<br>
<br>
<br>
<br>
<br>
.center[
# Lineup Protocal
<font size="6">
(Buja et. al, 2009)
</font>
]
???
Efforts in the field of graphics have developed graphical testing tools and methods such as the lineup protocol to provide a framework for inferential testing.

---
class:primary
# Introduction to Visual Inference
When doing exploratory data analysis, how do we know if what we see is actually there?

???

When inspecting a plot, how do we know if what we are seeing is actually there?

---
class:primary
# Lineup Protocal (Buja et. al, 2009)

Embed a **target plot** (actual data) in a lineup of **null plots** (randomly permuted data sets).

???

One way of answering this question is to embed the true data plot (called target plot) into a set of randomly permuted data sets (called null plots). This is what we call a lineup.

This is similar to the law-enforcement procedure to line up a suspect among a set of innocents to check if a victim can identify the suspect as the perpetrator of the crime.

Here, visual evaluation of the lineup is conducted by a person. If the viewers detect the target plot, we can conclude the plots are distinguishable.

The lineup protocol is one such example of the development of tools designed for statistical graphical testing. The advancement of graphing software provides the tools necessary to develop new methods of testing graphics.

---
class:primary
# Introduction to Visual Inference

.center[
<img src="images/visual-inference-flowchart.png" width="58%" /> 
]

???
The main idea behind visual inference is that graphs are visual statistics or summaries of the data sets generated by mathematical functions. In a standard statistical analysis, a test statistic is generated from the data set and compared to the null distribution of that test statistic. Similarly, the visual statistic (target plot) is compared by a human viewer to other plots generated under the assumption of the null.

The main benefit of using visual inference is that visual tests tend to be more comprehensive. Since individuals are being asked to select one or more plots from a the lineup which are different but the difference is left unspecified the individual eye is actually spotting many differences at once. An equivalent numerical assessment may involved multiple tests using different test statistics.

---
class:primary
# Lineup Studies 📊 💹 📊

+ [Statistical inference for exploratory data analysis and model diagnostics](https://scholar.google.com/scholar_url?url=https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2009.0120%3Fcasa_token%3DebiI2xSnDhwAAAAA:kWGCok2CLw5KQZ4_cYh_ymZwPpiJcXUc5TImyvwzBBn6C0jYtFgnYEOc0dR0kPvlajIvsuOULZpTia64&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=10347521957724374176&ei=W11jYaXsKYWM6rQPo_eDyAg&scisig=AAGBfm2AEQHjQU5lNQB4Lc4o81tg3SeWkQ) (Buja, Cook, Hofmann, et al., 2009)
+ [Validation of Visual Statistical Inference, Applied to Linear Models](https://scholar.google.com/scholar_url?url=https://www.tandfonline.com/doi/pdf/10.1080/01621459.2013.808157%3Fcasa_token%3DGwA4wLBPDpsAAAAA:acfwFasv03i-MlmZ7cUcD9NYkjsk2hFroGKp4SbK1ZljNN6wVBUzJyhicqplDmhMmD4LURWeqS5qm_g&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=3201506009084463478&ei=a11jYe6GFJ2N6rQPrdWZ4A0&scisig=AAGBfm2pYvdJa0OVNuzPLcU7ceZxogmE8w) (Majumder, Hofmann, and Cook, 2013)
+ [Human Factors Influencing Visual Statistical Inference](https://scholar.google.com/scholar_url?url=https://www.tandfonline.com/doi/pdf/10.1080/01621459.2013.808157%3Fcasa_token%3Dy-x3yREurLYAAAAA:n2T18c7npd7rwui0SHzcmu2aa-zbeM66G7RWuPUgXG88nKzkqstOACzexskq0QhJ19yg8F2qky5w5NU&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=3201506009084463478&ei=gV1jYYt-gtDJBMHbsVg&scisig=AAGBfm2ZxHiHoOvYZQOJLkY9boUVKKUNvQ) (Majumder, Hofmann, and Cook, 2014)
+ [Variations of *Q*–*Q* Plots: The Power of Our Eyes!](https://scholar.google.com/scholar_url?url=https://www.tandfonline.com/doi/pdf/10.1080/00031305.2015.1077728%3Fcasa_token%3DzQIJhjnFyLkAAAAA:tKCYURplVofdEuwasvxeS3UEVUMrTYsNKDhCQ2pBkkAhyXSI-Rv9vF00OqfRi5qPYwmqZBuTKjcSdO0&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=10678752834182219121&ei=kF1jYdL3L4i5yQTj8oqgCg&scisig=AAGBfm0_2MwYhtf65JQetFCfaf8pTBN_1Q) (Loy, Follett, and Hofmann, 2016)
+ [Spatial Reasoning and Data Displays](https://scholar.google.com/scholar_url?url=https://ieeexplore.ieee.org/iel7/2945/4359476/07217849.pdf%3Fcasa_token%3DXZQihG8m_RkAAAAA:twbOtOXzriTbFGFvUvrXE8dOirkb-g8boUHl-_BqoRvAfoh4pOFNPaTpWw4N-ILRlaJG8njCPf9u&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=1511166849179793238&ei=oF1jYY7gA5LSyQT4lJigBw&scisig=AAGBfm2lpsX7rCa8i1nFQqaMCQKkWxIqhg) (VanderPlas and Hofmann, 2015)
+ [Clusters beat trend!? testing feature hierarchy in statistical graphics](https://www.tandfonline.com/doi/full/10.1080/10618600.2016.1209116?casa_token=ZTotcyTpAekAAAAA:0ZJcegDldgPo53zNw5581EKGYr5b2mtUj393V7v4OjIjhiOPN9MuiRjPdVrBvYzAoCGX3fbSpAMt5Bw) (VanderPlas and Hofmann, 2017)
+ [Statistical Significance Calculations for Scenarios in Visual Inference](https://scholar.google.com/scholar_url?url=https://onlinelibrary.wiley.com/doi/pdf/10.1002/sta4.337%3Fcasa_token%3Dl3liRM4W_tcAAAAA:RM0pP8pMkbSvUvKjtCA3aQ3IBS1PfKUKyDrp62dikvMnSWdnVvWZkjqbUKVKt-uFaeZ4CLXhCnIisLhK&hl=en&sa=T&oi=gsb-gga&ct=res&cd=0&d=18231836572357450482&ei=v11jYaauD4i5yQTj8oqgCg&scisig=AAGBfm30K-gRzHdAIegMH0DH2gyiL4DfcA) (VanderPlas, Röttger, Cook, et al., 2021)

.pull-left[
+ **Dissertation work:** Perception of log scales
].pull-right[

<img id='imglinearlineupexample' src='images/linear-lineup-example.png' alt='Linear Scale' width='40%'>

<img id='imgloglineupexample' src='images/log-lineup-example.png' alt='Log Scale' width='40%'>

<div id='modallinearlineupexample' class='modal'>
  
  <img class='modal-content' id='imgmodallinearlineupexample'>
  
  <div id='captionlinearlineupexample' class='modal-caption'></div>
</div>

<div id='modalloglineupexample' class='modal'>
  
  <img class='modal-content' id='imgmodalloglineupexample'>
  
  <div id='captionloglineupexample' class='modal-caption'></div>
</div>
]

???
**Clusters beat trend!? testing feature hierarchy in statistical graphics** (VanderPlas and Hofmann, 2017)
+ Introduces perceptual principles such as preattentive features and gestalt heuristics
+ Discusses the design and results of a factorial experiment examining the effect of plot aesthetics such as color and trend lines on participants’ assessment of ambiguous data displays.
+ Strongly suggests that plot aesthetics have a significant impact on the perception of important features in data displays.

As part of my dissertation work, I have conducted a study examining human perception of logarithmic scales. Here we see an example of the first part of the study utilizing lineups to test our ability to perceptually differentiate between two exponentially increasing trends shown on both the linear and log scale. Notice how it is much easier to pick out panel 13 as being most different when displayed on the log scale than on the linear scale.

---
class:inverse
<br>
<br>
<br>
<br>
<br>
<br>
<br>
.center[
# 'You Draw It'
]
 
---
class:primary
# Linear Regression

The principle of simple linear regression is to find the line (i.e., determine its equation) which passes as close as possible to the observations, that is, the set of points.

???
Linear regression is a statistical approach that allows to assess the linear relationship between two quantitative variables.

---
class:primary
# Linear Regression

The principle of simple linear regression is to **find the line** (i.e., determine its equation) which passes as close as possible to the observations, that is, the set of points.

???

The principle of simple linear regression is to **find the line** (i.e., determine its equation) which passes as close as possible to the observations, that is, the set of points.

---
class:primary
# Linear Regression

The principle of simple linear regression is to **find the line** (i.e., determine its equation) **which passes as close as possible to the observations**, that is, the set of points.

.center[
<img src="images/pca-plot.jpg" width="80%"/>
]

**Big Idea:** How do statistical regression results compare to intuitive, visually fitted results?

???
We are going to focus on two regression lines determined by ordinary least squares regression and regression based on the principal axis. The figure illustrates the difference between an OLS regression line which minimizes the vertical distance of points from the line and a regression line based on the principal axis (Principal Component) which minimizes the Euclidean distance of points (orthogonal) from the line. This is what we refer to as “ensemble perception” indicating the visual system can compute averages of various features in parallel across the items in a set (in this case, over the x and y-axes).

**Big Idea:** How do statistical regression results compare to intuitive, visually fitted results?

---
class:primary
# Eye Fitting Straight Lines
## Mosteller, Siegel, Trapido, et al. (1981)
.pull-left[
+ **Big Idea:** Students fitted lines by eye to four sets of points.
+ **Method:** 8.5 x 11 inch transparency with a straight line etched across the middle.
+ **Sample:** 153 graduate students and post docs in Introductory Biostatistics.
+ **Experimental Design:** Latin square.
+ **Findings:** Students tended to fit the slope of the first principal component.
].pull-right[
<img src="images/eyefitting-straight-lines-plots.png" width="95%"/>
]

???

I want to introduce a study conducted in 1981 called Eye Fitting Straight Lines by Mosteller et al. In this study:

+ Students fitted lines by eye to four sets of points.
+ 8.5 x 11 inch transparency with a straight line etched across the middle.
+ 153 graduate students and post docs in Introductory Biostatistics.
+ Latin square.
+ Students tended to fit the slope of the first principal component or major axis (the line that minimizes the sum of squares of perpendicular rather than vertical distances).

---
class:primary
# 'You Draw It' Feature 
## (New York Times, 2015)
.pull-left[
<img src="images/nyt-caraccidents-frame4.png" width="100%"/>
.center[
(Katz, 2017)
]
].pull-right[

Readers are asked to input their own assumptions about various metrics and compare how these assumptions relate to reality.

+ [Family Income affects college chances](https://www.nytimes.com/interactive/2015/05/28/upshot/you-draw-it-how-family-income-affects-childrens-college-chances.html) (Aisch, Cox, and Quealy, 2015)
+ [Just How Bad Is the Drug Overdose Epidemic?](https://www.nytimes.com/interactive/2017/04/14/upshot/drug-overdose-epidemic-you-draw-it.html) (Katz, 2017)
+ [What Got Better or Worse During Obama’s Presidency](https://www.nytimes.com/interactive/2017/01/15/us/politics/you-draw-obama-legacy.html?_r=0) (Buchanan, Park, and Pearce, 2017)
]

???

In 2015, the New York Times developed a You Draw it feature where readers are asked to input their own assumptions about various metrics and compare how these assumptions relate to reality.

The New York Times team utilizes **Data Driven Documents (D3)** that allows readers to predict these metrics through the use of drawing a line on their computer screen with their mouse.

---
class:primary
# Research Objectives

1. Validate ‘You Draw It’ as a method for graphical testing, comparing results to the less technological method utilized in Mosteller et al. (1981).

2. Extend the study with formal statistical analysis methods in order to better understand the perception of linear regression.

???

The two objectives of my current research are to:

1. Validate ‘You Draw It’ as a method for graphical testing, comparing results to the less technological method utilized in Mosteller et al. (1981).

2. Extend the study with formal statistical analysis methods in order to better understand the perception of linear regression.

---
class:primary
# 'You Draw It' Task

Study Participant Prompt: *Use your mouse to fill in the trend in the yellow box region.*

.center[
<img src="images/eyefitting_example.gif" width="60%"/>
]

???

Here we see an example of a "You Draw It" task plot used in the study. Participants are prompted to "Use your mouse to fill in the trend in the yellow box region. The yellow box region moves along as the participant draws their trend-line until the yellow region disappears."

Task plots were created using Data Driven Documents (D3), a JavaScript-based graphing framework that facilitates user interaction. We then integrate this into RShiny using the r2d3 package.

---
class:primary
# Data Generation

.pull-left[

`\(N = 30\)` points `\((x_i, y_i), i = 1,...N\)` were generated for `\(x_i \in [x_{min}, x_{max}]\)`.

Data were simulated based on linear model with additive errors:
`\begin{equation}
y_i = \beta_0 + \beta_1 x_i + e_i
\end{equation}`

where `\(e_i \sim N(0, \sigma^2).\)`

Parameters `\(\beta_0\)` and `\(\beta_1\)` were selected to reflect the four data sets used in Mosteller, Siegel, Trapido, et al. (1981).

].pull-right[

<img id='imgeyefittingexamplesimplot' src='images/eyefitting-example-simplot.png' alt=' ' width='100%'>

<div id='modaleyefittingexamplesimplot' class='modal'>
  
  <img class='modal-content' id='imgmodaleyefittingexamplesimplot'>
  
  <div id='captioneyefittingexamplesimplot' class='modal-caption'></div>
</div>
]

???

Data were generated following a linear model with additive errors.

Model equation parameters, `\(\beta_0\)` and `\(\beta_1\)`, were selected to reflect the four data sets (F, N, S, and V) used in Mosteller et al. (1981).

+ **S:** positive slope; small variance; `\(x \in [0, 20]\)`. 
+ **F:** positive slope; a large variance; `\(x \in [0, 20]\)`.  
+ **V:** steep positive slope; small variance; `\(x \in [4, 16]\)`. 
+ **N:** negative slope; large variance; `\(x \in [0, 20]\)`.

---
class:primary
# Study Design

+ Participants recruited through Twitter, Reddit, and direct email in May 2021.

+ A total of 35 individuals completed 119 unique you draw it task plots.

+ Data sets were generated randomly, independently for each participant at the start of the experiment.

+ Participants shown 2 practice plots followed by 4 task plots randomly assigned for each individual in a completely randomized design.

+ Experiment conducted and distributed through an RShiny application found [**here**](https://emily-robinson.shinyapps.io/you-draw-it-pilot-app/).

???
Participants were recruited through through Twitter, Reddit, and direct email in May 2021. The experiment was conducted and distributed through an RShiny application. Participants were first shown 2 practice plots followed by the 4 You Draw It task plots randomly assigned for each individual in a completely randomized design.

---
class:primary
# Model Data

.pull-left[
For each participant, the final data set used for analysis contains:
+ `\(x_{ijk}\)`, `\(y_{ijk,drawn}\)`, `\(\hat y_{ijk,OLS}\)`, `\(\hat y_{ijk,PCA}\)`

for 
+ parameter choice `\(i = 1,2,3,4\)`, 
+ participant j = `\(1,...N_{participant}\)` 
+ `\(x_{ijk}\)` value corresponding to increment `\(k = 1, ...,4 x_{max} + 1\)`.

**Vertical residuals** between the drawn and fitted values were calculated as: 
+ `\(e_{ijk,OLS} = y_{ijk,drawn} - \hat y_{ijk,OLS}\)`
+ `\(e_{ijk,PCA} = y_{ijk,drawn} - \hat y_{ijk,PCA}\)`.

].pull-right[

<img id='imgeyefittingtrialplot' src='images/eyefitting-trial-plot.png' alt=' ' width='100%'>

<div id='modaleyefittingtrialplot' class='modal'>
  
  <img class='modal-content' id='imgmodaleyefittingtrialplot'>
  
  <div id='captioneyefittingtrialplot' class='modal-caption'></div>
</div>
]

???
We compare the participant drawn line to two regression lines determined by ordinary least squares regression and regression based on the principal axis. The figure illustrates the difference between an OLS regression line which minimizes the vertical distance of points from the line and a regression line based on the principal axis (Principal Component) which minimizes the Euclidean distance of points (orthogonal) from the line.

Here we see an example of the feedback data from one you draw it plot. For 0.25 increments across the domain, we have the participant drawn values, the fitted values from the ordinary least squares regression, and the fitted values from the regression based on the principal axis.

We are mainly interested in the deviation of the participant drawn line from the fitted regression lines. So while it seems counter-intuitive, the residual actually becomes our response in this case.

---
class:primary
# Linear Trend Constraint
The **Linear Mixed Model** equation for each fit (OLS and PCA) residuals is given by:
`\begin{equation}
e_{ijk,fit} = \left[\gamma_0 + \alpha_i\right] + \left[\gamma_{1} x_{ijk} + \gamma_{2i} x_{ijk}\right] + p_{j} + \epsilon_{ijk}
\end{equation}`
where

+ `\(e_{ijk,fit}\)` is the residual between the drawn and fitted y-values for the `\(i^{th}\)` parameter choice, `\(j^{th}\)` participant, and `\(k^{th}\)` increment of x-value corresponding to either the OLS or PCA fit
+ `\(\gamma_0\)` is the overall intercept
+ `\(\alpha_i\)` is the effect of the `\(i^{th}\)` parameter choice (F, S, V, N) on the intercept
+ `\(\gamma_1\)` is the overall slope for `\(x\)`
+ `\(\gamma_{2i}\)` is the effect of the parameter choice on the slope
+ `\(x_{ijk}\)` is the x-value for the `\(i^{th}\)` parameter choice, `\(j^{th}\)` participant, and `\(k^{th}\)` increment
+ `\(p_{j} \sim N(0, \sigma^2_{participant})\)` is the random error due to the `\(j^{th}\)` participant's characteristics
+ `\(\epsilon_{ijk} \sim N(0, \sigma^2)\)` is the residual error.

???

Using the `lmer` function in the lme4 package, a linear mixed model (LMM) is fit separately to the OLS and PCA residuals, constraining the fit to a linear trend.

---
class:primary
# Linear Trend Constraint

.center[
<img src="images/eyefitting-lmer-plot.png" width="85%"/>
]

???
Results indicate the estimated trends of PCA residuals (orange) appear to align closer to the y = 0 horizontal (dashed) line than the OLS residuals (blue). In particular, this trend is more prominent in parameter choices with large variances (F and N). These results are consistent to those found in Mosteller
et al. (1981) indicating participants fit a trend-line closer to the estimated regression line with the slope of based on the first principal axis than the estimated OLS regression line.

---
class:primary
# Smoothing Spline Trend
The **Generalized Additive Mixed Model** equation for each fit (OLS and PCA) residuals is given by:
`\begin{equation}
e_{ijk,fit} = \alpha_i + s_{i}(x_{ijk}) + p_{j} + s_{j}(x_{ijk})
\end{equation}`
where

+ `\(e_{ijk,fit}\)` is the residual between the drawn and fitted y-values for the `\(i^{th}\)` parameter choice, `\(j^{th}\)` participant, and `\(k^{th}\)` increment of x-value corresponding to either the OLS or PCA fit
+ `\(\alpha_i\)` is the intercept for the parameter choice `\(i\)`
+ `\(s_{i}\)` is the smoothing spline for the `\(i^{th}\)` parameter choice
+ `\(x_{ijk}\)` is the x-value for the `\(i^{th}\)` parameter choice, `\(j^{th}\)` participant, and `\(k^{th}\)` increment
+ `\(p_{j} \sim N(0, \sigma^2_{participant})\)` is the error due to participant variation
+ `\(s_{j}\)` is the random smoothing spline for each participant.

???

Eliminating the linear trend constraint, the `bam` function in the mgcv package is used to fit a generalized additive mixed model (GAMM) separately to the OLS and PCA residuals to allow for estimation of smoothing splines.

---
class:primary
# Smoothing Spline Trend

.center[
<img src="images/eyefitting-gamm-plot.png" width="85%"/>
]

???
The results of the GAMM align with those in the linear constraint trend providing support that for scatter-plots with more noise (F and N), estimated trends of PCA residuals (orange) appear to align closer to the y = 0 horizontal (dashed) line than the OLS residuals (blue). However, By fitting smoothing splines, we can determine whether participants naturally fit a straight trend-line to the set of points or whether they deviate throughout the domain providing us with further insight into the curvature humans perceive in a set of points.

---
class:primary
# Conclusion

**Research Objectives:**

1. Validate ‘You Draw It’ as a method for graphical testing, comparing results to the less technological method utilized in Mosteller et al. (1981). 
2. Extend the study found in Mosteller et al. (1981) with formal statistical analysis methods for understanding the perception of linear regression.

**Results:**
+ Estimated drawn trend-lines followed closer to the regression line based on the principal axes than the OLS regression line.
+ Most prominent in data simulated with large variances.
+ Humans perform “ensemble perception” in a statistical graphic setting.

**The reproducibility of these results serve as validation of the 'You Draw It' tool and method.**

???

1. Validate ‘You Draw It’ as a method for graphical testing, comparing results to the less technological method utilized in Mosteller et al. (1981).

2. Extend the study found in Mosteller et al. (1981) with formal statistical analysis methods for understanding the perception of linear regression.

**Results:**
+ Estimated drawn trend-lines followed closer to the principal axes than the OLS regression line.
+ Most prominent in data simulated with large variances.
+ Humans perform “ensemble perception” in a statistical graphic setting as participants minimized the distance from the their regression line over both the x and y axis simultaneously

This study reinforces the differences between intuitive visual model fitting and statistical model fitting, providing information about human perception as it relates to the use of statistical graphics.

**The reproducibility of these results serve as validation of the 'You Draw It' tool and method.**

---
class:primary
# Future Work

.center[
<img src="images/loading.gif" width="50%"/>
]

✏️ Implement the 'You Draw It' method in non-linear settings.

📈 Evaluate human ability to extrapolate data from trends.

<br>

.right-col[
Gif Source: [photobucket.com](http://s280.photobucket.com/user/ariffisariff/media/animated-loading.gif.html)
]

???
✏️ Implement the 'You Draw It' method in non-linear settings.

📈 Evaluate human ability to extrapolate data from trends.

---
class:primary
# References
<font size="2">
<p><cite><a id='bib-Aisch_NYTimes_presidential_forecast'></a><a href="#cite-Aisch_NYTimes_presidential_forecast">Aisch, G., N. Cohn, A. Cox, et al.</a>
(2016).
<em>Live Presidential Forecast</em>.
URL: <a href="https://www.nytimes.com/elections/2016/forecast/president">https://www.nytimes.com/elections/2016/forecast/president</a>.</cite></p>

<p><cite><a id='bib-aisch_cox_quealy_2015'></a><a href="#cite-aisch_cox_quealy_2015">Aisch, G., A. Cox, and K. Quealy</a>
(2015).
<em>You Draw It: How Family Income Predicts Children's College Chances</em>.
URL: <a href="https://www.nytimes.com/interactive/2015/05/28/upshot/you-draw-it-how-family-income-affects-childrens-college-chances.html">https://www.nytimes.com/interactive/2015/05/28/upshot/you-draw-it-how-family-income-affects-childrens-college-chances.html</a>.</cite></p>

<p><cite><a id='bib-buchanan_park_pearce_2017'></a><a href="#cite-buchanan_park_pearce_2017">Buchanan, L., H. Park, and A. Pearce</a>
(2017).
<em>You Draw It: What Got Better or Worse During Obama's Presidency</em>.
URL: <a href="https://www.nytimes.com/interactive/2017/01/15/us/politics/you-draw-obama-legacy.html">https://www.nytimes.com/interactive/2017/01/15/us/politics/you-draw-obama-legacy.html</a>.</cite></p>

<p><cite><a id='bib-buja2009statistical'></a><a href="#cite-buja2009statistical">Buja, A., D. Cook, H. Hofmann, et al.</a>
(2009).
&ldquo;Statistical inference for exploratory data analysis and model diagnostics&rdquo;.
In: <em>Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences</em> 367.1906, pp. 4361&ndash;4383.</cite></p>

<p><cite>Carpenter, P. A. and P. Shah
(1998).
&ldquo;A model of the perceptual and conceptual processes in graph comprehension.&rdquo;
In: <em>Journal of Experimental Psychology: Applied</em> 4.2, p. 75.</cite></p>

<p><cite><a id='bib-chandar2012graph'></a><a href="#cite-chandar2012graph">Chandar, N., D. Collier, and P. Miranti</a>
(2012).
&ldquo;Graph standardization and management accounting at AT&amp;T during the 1920s&rdquo;.
In: <em>Accounting History</em> 17.1, pp. 35&ndash;62.</cite></p>

<p><cite>Chong, S. C. and A. Treisman
(2003).
&ldquo;Representation of statistical properties&rdquo;.
In: <em>Vision research</em> 43.4, pp. 393&ndash;404.</cite></p>

<p><cite>&mdash;
(2005).
&ldquo;Statistical processing: Computing the average size in perceptual groups&rdquo;.
In: <em>Vision research</em> 45.7, pp. 891&ndash;900.</cite></p>

<p><cite>Ciccione, L. and S. Dehaene
(2021).
&ldquo;Can humans perform mental regression on a graph? Accuracy and bias in the perception of scatterplots&rdquo;.
In: <em>Cognitive Psychology</em> 128, p. 101406.</cite></p>

<p><cite>Cleveland, W. S. and R. McGill
(1984).
&ldquo;Graphical perception: Theory, experimentation, and application to the development of graphical methods&rdquo;.
In: <em>Journal of the American statistical association</em> 79.387, pp. 531&ndash;554.</cite></p>
</font>

---
class:primary
# References
<font size="2">
<p><cite>Cleveland, W. S. and R. McGill
(1985).
&ldquo;Graphical perception and graphical methods for analyzing scientific data&rdquo;.
In: <em>Science</em> 229.4716, pp. 828&ndash;833.</cite></p>

<p><cite>Finney, D.
(1951).
&ldquo;Subjective judgment in statistical analysis: An experimental study&rdquo;.
In: <em>Journal of the Royal Statistical Society: Series B (Methodological)</em> 13.2, pp. 284&ndash;297.</cite></p>

<p><cite><a id='bib-gouretski2007much'></a><a href="#cite-gouretski2007much">Gouretski, V. and K. P. Koltermann</a>
(2007).
&ldquo;How much is the ocean really warming?&rdquo;
In: <em>Geophysical Research Letters</em> 34.1.</cite></p>

<p><cite>Green, T. M. and B. Fisher
(2009).
&ldquo;The personal equation of complex individual cognition during visual interface interaction&rdquo;.
In: 
<em>Workshop on Human-Computer Interaction and Visualization</em>.
Springer.
, pp. 38&ndash;57.</cite></p>

<p><cite><a id='bib-harms1991august'></a><a href="#cite-harms1991august">Harms, H.</a>
(1991).
&ldquo;August Friedrich Wilhelm Crome (1753-1833) Autor begehrter Wirtschaftskarten&rdquo;.
In: <em>Cartographica Helvetica</em> 3, pp. 33&ndash;38.</cite></p>

<p><cite>Hofmann, H., L. Follett, M. Majumder, et al.
(2012).
&ldquo;Graphical tests for power comparison of competing designs&rdquo;.
In: <em>IEEE Transactions on Visualization and Computer Graphics</em> 18.12, pp. 2441&ndash;2448.</cite></p>

<p><cite><a id='bib-katz_2017'></a><a href="#cite-katz_2017">Katz, J.</a>
(2017).
<em>You Draw It: Just How Bad Is the Drug Overdose Epidemic?</em>
URL: <a href="https://www.nytimes.com/interactive/2017/04/14/upshot/drug-overdose-epidemic-you-draw-it.html">https://www.nytimes.com/interactive/2017/04/14/upshot/drug-overdose-epidemic-you-draw-it.html</a>.</cite></p>

<p><cite><a id='bib-lewandowsky1989perception'></a><a href="#cite-lewandowsky1989perception">Lewandowsky, S. and I. Spence</a>
(1989).
&ldquo;The perception of statistical graphs&rdquo;.
In: <em>Sociological Methods &amp; Research</em> 18.2-3, pp. 200&ndash;242.</cite></p>

<p><cite><a id='bib-mosteller1981eye'></a><a href="#cite-mosteller1981eye">Mosteller, F., A. F. Siegel, E. Trapido, et al.</a>
(1981).
&ldquo;Eye fitting straight lines&rdquo;.
In: <em>The American Statistician</em> 35.3, pp. 150&ndash;152.</cite></p>

<p><cite><a id='bib-playfair1801statistical'></a><a href="#cite-playfair1801statistical">Playfair, W.</a>
(1801).
&ldquo;The statistical breviary; shewing, on a principle entirely new, the resources of every state and kingdom in Europe, Wallis, Londres&rdquo;.
In: <em>Press, Chicago</em>.</cite></p>
</font>

---
class:primary
# References
<font size="2">
<p><cite>Spence, I.
(1990).
&ldquo;Visual psychophysics of simple graphical elements.&rdquo;
In: <em>Journal of Experimental Psychology: Human Perception and Performance</em> 16.4, p. 683.</cite></p>

<p><cite><a id='bib-unwin2020data'></a><a href="#cite-unwin2020data">Unwin, A.</a>
(2020).
&ldquo;Why is data visualization important? what is important in data visualization?&rdquo;
In: <em>Harvard Data Science Review</em> 2.1.</cite></p>

<p><cite>Van Opstal, F., F. P. de
Lange, and S. Dehaene
(2011).
&ldquo;Rapid parallel semantic processing of numbers without awareness&rdquo;.
In: <em>Cognition</em> 120.1, pp. 136&ndash;147.</cite></p>

<p><cite>Vanderplas, S., D. Cook, and H. Hofmann
(2020).
&ldquo;Testing Statistical Charts: What makes a good graph?&rdquo;
In: <em>Annual Review of Statistics and Its Application</em> 7, pp. 61&ndash;88.</cite></p>

<p><cite><a id='bib-vanderplas2015spatial'></a><a href="#cite-vanderplas2015spatial">VanderPlas, S. and H. Hofmann</a>
(2015).
&ldquo;Spatial reasoning and data displays&rdquo;.
In: <em>IEEE Transactions on Visualization and Computer Graphics</em> 22.1, pp. 459&ndash;468.</cite></p>

<p><cite><a id='bib-vanderplas2017clusters'></a><a href="#cite-vanderplas2017clusters">&mdash;</a>
(2017).
&ldquo;Clusters beat trend!? testing feature hierarchy in statistical graphics&rdquo;.
In: <em>Journal of Computational and Graphical Statistics</em> 26.2, pp. 231&ndash;242.</cite></p>

<p><cite><a id='bib-walker2013statistical'></a><a href="#cite-walker2013statistical">Walker, F. A.</a>
(2013).
<em>Statistical atlas of the United States based on the results of the ninth census 1870 with contributions from many eminent men of science and several departments of the government</em>.</cite></p>

<p><cite>Wickham, H.
(2011).
&ldquo;ggplot2&rdquo;.
In: <em>Wiley Interdisciplinary Reviews: Computational Statistics</em> 3.2, pp. 180&ndash;185.</cite></p>

<p><cite><a id='bib-wilkinson2013grammar'></a><a href="#cite-wilkinson2013grammar">Wilkinson, L.</a>
(2013).
<em>The grammar of graphics</em>.
Springer Science &amp; Business Media.</cite></p>

<p><cite><a id='bib-yates1985graphs'></a><a href="#cite-yates1985graphs">Yates, J.</a>
(1985).
&ldquo;Graphs as a managerial tool: A case study of Du Pont's use of graphs in the early twentieth century&rdquo;.
In: <em>The Journal of Business Communication (1973)</em> 22.1, pp. 5&ndash;33.</cite></p>
</font>

---
class:inverse
<br>
<br>
<br>
<br>
<br>
<br>
.center[
# Thank you!
<br
<br>
<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg> **emily.robinson@huskers.unl.edu**
<svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:1em;width:1.12em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M280.37 148.26L96 300.11V464a16 16 0 0 0 16 16l112.06-.29a16 16 0 0 0 15.92-16V368a16 16 0 0 1 16-16h64a16 16 0 0 1 16 16v95.64a16 16 0 0 0 16 16.05L464 480a16 16 0 0 0 16-16V300L295.67 148.26a12.19 12.19 0 0 0-15.3 0zM571.6 251.47L488 182.56V44.05a12 12 0 0 0-12-12h-56a12 12 0 0 0-12 12v72.61L318.47 43a48 48 0 0 0-61 0L4.34 251.47a12 12 0 0 0-1.6 16.9l25.5 31A12 12 0 0 0 45.15 301l235.22-193.74a12.19 12.19 0 0 1 15.3 0L530.9 301a12 12 0 0 0 16.9-1.6l25.5-31a12 12 0 0 0-1.7-16.93z"/></svg> **www.emilyarobinson.com**
<svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> **earobinson95**
]