MTH 208 Exploratory Data Analysis

class: center, middle, inverse, title-slide

.title[
# MTH 208 Exploratory Data Analysis
]
.subtitle[
## Lesson 06: Pattern Recognition and Association Analysis
]
.author[
### <br>Ying-Ju Tessa Chen, PhD <br><br> Associate Professor <br> Department of Mathematics<br> University of Dayton <br><br> <a href="https://twitter.com/ju_tessa"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> <span class="citation">@ying-ju</span></a> <br> <a href="https://github.com/ying-ju/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> ying-ju</a> <br> <a href="mailto:ychen4@udayton.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"></path></svg> ychen4@udayton.edu</a><br>
]

---

## Learning Objectives

**Overview**

We study advanced concepts of identifying patterns, correlations, and associations within datasets in this lession. Building upon the foundation of reading scatterplots introduced earlier, we will explore techniques and methodologies to recognize underlying patterns and associations that are not immediately apparent.

**Objectives**
- Understand the difference between correlation and causation.
- Learn to identify and interpret various types of patterns in data.
- Explore methods to analyze association between categorical variables.
- Apply statistical measures to quantify relationships in data.

---

## Correlation vs. Causation

- **Correlation:** A statistical measure that expresses the extent to which two variables change together. .orange[Correlation does not imply that one variable causes the change in another.]
    - **Example:** Ice cream sales and drowning incidents are positively correlated, but one does not cause the other. Instead, they are both related to a third factor: warmer temperatures during summer months.

- **Causation (Causal Relationship):** A relationship where one variable directly affects another.
    - **Example:** A decrease in vaccination rates causes an increase in the spread of diseases that those vaccines prevent.

---
### Correlation vs. Causation (Continued)

**How to report correlation?**

-	`Bad:`  Raising salaries increases productivity.
-	`Good:`  Employees with higher salaries tend to be more productive.
-	`Bad:`  `$r = −0.99$`.  This proves that drinking more red wine lowers cholesterol.  
-	`Good:`  There is a strong negative association between red wine consumption and cholesterol levels.
-	`Bad:`  A child that has two educated parents will graduate from college.
-	`Good:`  Children with educated parents are more likely to graduate from college.

- [The vermiform appendix impacts the risk of developing Parkinson’s disease](https://www.science.org/doi/full/10.1126/scitranslmed.aar5280)
  - [Appendix Removal Lowers Parkinson's Disease Risk by up to 25%](https://www.technologynetworks.com/neuroscience/news/appendix-removal-lowers-parkinsons-disease-risk-by-up-to-25-311316)
  - [Appendix identified as a potential starting point for Parkinson's disease](https://www.sciencedaily.com/releases/2018/10/181031141606.htm)

- [PARKINSON'S DISEASE IS MORE PREVALENT IN PATIENTS WITH APPENDECTOMIES: A NATIONAL POPULATION-BASED STUDY](https://meetings.ssat.com/abstracts/2019/739.cgi)
  - [Appendix Removal Associated with Development of Parkinson's Disease](https://www.uhhospitals.org/for-clinicians/articles-and-news/articles/2019/05/appendix-removal-associated-with-development-of-parkinsons-disease)
  
---
## Pattern Recognition in Scatter Plots

Scatter plots are a fundamental tool in exploratory data analysis, offering a visual representation of the relationship between two quantitative variables. Beyond simple linear correlations, scatter plots can reveal a variety of patterns that provide deeper insights into the data:

- **Linear Relationships:** A straight-line pattern indicating a positive or negative correlation.

- **Non-linear Relationships:** Curved patterns suggest a more complex relationship that might require transformation or different analytical approaches.

- **Clusters:** Groups of points that are closely bunched together, indicating subpopulations within the dataset.

- **Outliers:** Points that fall far from the main group of data points, which may indicate anomalies or errors in the data.

---
## Association Analysis in Categorical Data

**Introduction to Chi-Square Tests**
The Chi-square test of independence is a non-parametric statistical test used to determine if there is a significant association between two categorical variables .blue[from the same population]. It's commonly applied in survey research, contingency table analysis, and various fields requiring statistical analysis of categorical data.

H0: The two variables are independent.

H1: The two variables relate to each other.

**Key Concepts**
- `Categorical Data:` Data that can be categorized into groups or categories that do not have a natural order or ranking.     - Examples include gender, race, or a yes/no response.

- `Contingency Tables:` Also known as cross-tabulation tables or two-way tables, contingency tables display the frequency distribution of variables and are a key part of conducting a Chi-square test.

- `Expected Frequencies:` The frequencies we would expect in each category if there was no association between the variables.

- `Chi-square Statistic:` A measure that tells us how far the observed frequencies are from the expected frequencies. A higher value indicates a greater discrepancy and potentially a significant association.

---
### Association Analysis in Categorical Data (Continued)

**Example: Effectiveness of a Drug Treatment**

.small[
Assume that there are 105 patients in the study and 50 of them were treated with the drug. In addition, the remaining 55 patients were in the control group. All patients' health condition was checked after a week. Here is an example using R code.

.pull-left[

```r
# Install and load necessary package
if (!require("gmodels")) install.packages("gmodels")
library(gmodels)

# Read the dataset
drug_data <- read_csv("https://goo.gl/j6lRXD")

# Print the contingency table
(drug_table <- table(drug_data[,2:3]))
```

```
##              improvement
## treatment     improved not-improved
##   not-treated       26           29
##   treated           35           15
```
]

.pull-right[

```r
# Perform the Chi-Square test
chi_square_result <- chisq.test(drug_table, 
                                correct=TRUE)

# Print the results
print(chi_square_result)
```

```
## 
## 	Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  drug_table
## X-squared = 4.6626, df = 1, p-value = 0.03083
```
]

Since p-value is  < 0.05, we reject the null hypothesis. We have sufficient evidence to conclude that the treatment and improvement are associated.

**Note:** Use the `correct=FALSE` option with reasonably large sample sizes, ie., if expected counts in any of the cells in the contingency table have more than 5 observations.
]

---
### Association Analysis in Categorical Data (Continued)

**Python code for the same example.**

.small[
.pull-left[

```python
import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency

df = pd.read_csv("https://goo.gl/j6lRXD")

# Create a contingency table
contingency_table = pd.crosstab(df['treatment'], df['improvement'])

print("Contingency Table:")
print(contingency_table)

# Perform the Chi-Square test
chi2, p, dof, expected = chi2_contingency(contingency_table)

print(f"\nChi2 Statistic: {chi2}")
print(f"\nDegrees of Freedom: {dof}")
print(f"\np-value: {p}")
print("Expected Frequencies:")
print(expected)
```
]

.pull-right[

```
## Contingency Table:
```

```
## improvement  improved  not-improved
## treatment                          
## not-treated        26            29
## treated            35            15
```

```
## 
## Chi2 Statistic: 4.6625668947297125
```

```
## 
## Degrees of Freedom: 1
```

```
## 
## p-value: 0.030827072412198585
```

```
## Expected Frequencies:
```

```
## [[31.95238095 23.04761905]
##  [29.04761905 20.95238095]]
```
]
]

---
## Quantifying Relationships

Three key statistical measures used to quantify these relationships are the .red[Pearson correlation coefficient], the .red[Spearman rank correlation coefficient], and the .red[Kendall tau rank correlation coefficient].

**Pearson Correlation Coefficient (r)**
.small[
- **Definition:** The Pearson correlation coefficient measures the linear relationship between two .blue[continuous] variables. It ranges from -1 to 1, where 1 means a perfect positive linear relationship, -1 means a perfect negative linear relationship, and 0 means no linear relationship.]

**Spearman Rank Correlation Coefficient**
.small[
- **Definition:** The Spearman correlation coefficient is a non-parametric measure of the strength and direction of the association that exists between two variables measured on at least an .blue[ordinal] scale. It assesses how well the relationship between two variables can be described using a .blue[monotonic] function.]

**Kendall tau rank correlation coefficient**
.small[
- **Definition:** The Kendall tau rank correlation coefficient, often referred to as Kendall's tau coefficient, is another non-parametric measure used to quantify the association between two measured quantities. It assesses the strength and direction of a relationship between two variables. Like Spearman's rho, it is useful for ordinal data or data that do not meet the assumptions of linearity and normal distribution required for Pearson's correlation coefficient. Kendall's tau is particularly well-suited for small datasets or datasets with a lot of tied ranks.]

**Note:** Correlation coefficients only measure linear (Pearson) or monotonic (Spearman and Kendall) relationships.

---
### Quantifying Relationships (Continued)

---
### Quantifying Relationships (Continued)

---
### Quantifying Relationships (Continued)

We calculate the Pearson, Spearman, and Kendall's tau correlation coefficients between sepal length and sepal width using R code:

```r
# Pearson correlation between sepal length and sepal width
pearson_sepal <- cor(iris$Sepal.Length, iris$Sepal.Width, method="pearson")

# Spearman correlation between sepal length and sepal width
spearman_sepal <- cor(iris$Sepal.Length, iris$Sepal.Width, method = "spearman")
cat("Pearson Correlation Coefficient (Sepal):", pearson_sepal, "\n")
```

```
## Pearson Correlation Coefficient (Sepal): -0.1175698
```

```r
cat("Spearman Correlation Coefficient (Sepal):", spearman_sepal, "\n")
```

```
## Spearman Correlation Coefficient (Sepal): -0.1667777
```

```r
cor.test(iris$Sepal.Length, iris$Sepal.Width, method="pearson")
```

```
## 
## 	Pearson's product-moment correlation
## 
## data:  iris$Sepal.Length and iris$Sepal.Width
## t = -1.4403, df = 148, p-value = 0.1519
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.27269325  0.04351158
## sample estimates:
##        cor 
## -0.1175698
```

```r
cor.test(iris$Sepal.Length, iris$Sepal.Width, method="spearman")
```

```
## 
## 	Spearman's rank correlation rho
## 
## data:  iris$Sepal.Length and iris$Sepal.Width
## S = 656283, p-value = 0.04137
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##        rho 
## -0.1667777
```

---
### Quantifying Relationships (Continued)

The same example using Python code:

```python
import seaborn as sns
import scipy.stats as stats
# Load the Iris dataset
iris = sns.load_dataset('iris')
pearson_coef, p_value = stats.pearsonr(iris['sepal_length'], iris['sepal_width'])
print(f"Pearson Correlation Coefficient (Sepal): {pearson_coef:.3f}, P-value: {p_value:.3f}")
```

```
## Pearson Correlation Coefficient (Sepal): -0.118, P-value: 0.152
```

```python
spearman_coef, p_value = stats.spearmanr(iris['sepal_length'], iris['sepal_width'])
print(f"Spearman Correlation Coefficient (Sepal): {spearman_coef:.3f}, P-value: {p_value:.3f}")
```

```
## Spearman Correlation Coefficient (Sepal): -0.167, P-value: 0.041
```

---
## Advanced Correlation Techniques

**Partial Correlation**

- `Definition:` Partial correlation measures the strength and direction of the relationship between two variables while controlling for the effect of one or more additional variables.
  - `Applicability:` Useful when you want to understand the direct relationship between two variables, independent of other variables that might affect their association.

**Autocorrelation (Serial Correlation)**

- `Definition:` Autocorrelation refers to the correlation of a variable with itself across different points in time. It's a measure of how related the values of a dataset are with its previous values.
  - `Applicability:` Particularly relevant in time-series analysis where the goal is to identify patterns or trends over time.

---
### Advanced Correlation Techniques (Continued)

Here we use R code to find the Pearson partial correlation coefficition between Sepal.Length and Sepal.Width while controlling the effect of Petal.Letngh and Petal.Width.

```r
library(ppcor)
result <- pcor.test(iris$Sepal.Length, iris$Sepal.Width,
                    iris[,c("Petal.Length", "Petal.Width")],
                    method="pearson")
print(result)
```

```
##    estimate      p.value statistic   n gp  Method
## 1 0.6285707 1.199846e-17   9.76538 150  2 pearson
```

```r
model1 <- lm(iris$Sepal.Length~iris$Petal.Length+iris$Petal.Width)
model2 <- lm(iris$Sepal.Width~iris$Petal.Length+iris$Petal.Width)

cor(model1$residuals, model2$residuals)
```

```
## [1] 0.6285707
```

---
### Advanced Correlation Techniques (Continued)

We use R code to calculate the autocorrelation in a vector by using the library .blue[tseries]. We will use a function .orange[act()] and this function has three parameters:

.pull-left[
- data, an input vector
- number of lags (we take a look at some past event from some point in time t)
- plot the auto correlation

```r
library(tseries)
mydata <- c(34, 56, 23, 45, 21, 64, 78, 90)
print(acf(mydata, pl=FALSE))
print(acf(mydata, lag=0, pl=FALSE))
print(acf(mydata, lag=1, pl=FALSE))
print(acf(mydata, lag=2, pl=FALSE))
print(acf(mydata, lag=6, pl=FALSE))
```

```
## 
## Autocorrelations of series 'mydata', by lag
## 
##      0      1      2      3      4      5      6      7 
##  1.000  0.257  0.208 -0.389 -0.093 -0.268 -0.064 -0.151
```
]

.pull-right[
.small[

```
## 
## Autocorrelations of series 'mydata', by lag
## 
## 0 
## 1
```

```
## 
## Autocorrelations of series 'mydata', by lag
## 
##     0     1 
## 1.000 0.257
```

```
## 
## Autocorrelations of series 'mydata', by lag
## 
##     0     1     2 
## 1.000 0.257 0.208
```

```
## 
## Autocorrelations of series 'mydata', by lag
## 
##      0      1      2      3      4      5      6 
##  1.000  0.257  0.208 -0.389 -0.093 -0.268 -0.064
```
]
]

---
## References

The lectures of this course are based on the ideas from the following references.

- Exploratory Data Analysis by John W. Tukey
- A Course in Exploratory Data Analysis by Jim Albert
- The Visual Display of Quantitative Information by Edward R. Tufte
- Data Science for Business: what you need to know about data mining and data-analytic thinking by Foster Provost and Tom Fawcett
- Storytelling with Data: A Data Visualization Guide for Business Professionals by Cole Nussbaumer Knaflic