MTH 209 Data Manipulation and Management

class: center, middle, inverse, title-slide

.title[
# MTH 209 Data Manipulation and Management
]
.subtitle[
## Lesson 4: Basic Data Structures (Lists, Data Frames, Factors and Tables)
]
.author[
### Ying-Ju Tessa Chen, PhD Associate Professor Department of Mathematics University of Dayton <a href="https://twitter.com/ju_tessa"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> @ying-ju</a> <a href="https://github.com/ying-ju/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> ying-ju</a> <a href="mailto:ychen4@udayton.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"></path></svg> ychen4@udayton.edu</a> 
]

---

# Learning Objectives

In this lesson, we will introduce the following data structures in R.

- Lists
- Data frames
- Factors and Tables

**Note:** This lesson is based on the book: The Art of R Programming.

.left[.footnote[Matloff, Norman. The art of R programming: A tour of statistical software design. No Starch Press, 2011]]

---
## Lists - 1

.pull-left[.footnotesize[Similar to vectors and matrices, one common operation with lists is indexing. Technically, a list is a vector. However, it is much flexible compared to the use of vectors and matrices. A list could contains elements of different types such as stings, numbers, vectors, and another list inside it. It can contain a matrix, a function, and a data frame, etc. as its elements. The following code chunk shows a simple example. ]

```r
x <- list(Name = "Tessa", 
 Programming_Skills = c("R", "Python", "Matlab"), 
 Num_Students_2023F = 84, 
 office = "SC 329E", 
 Tenured = TRUE, 
 UD_Working_Yr = 7)
```

]
.pull-right[

```r
x
```

```
## $Name
## [1] "Tessa"
## 
## $Programming_Skills
## [1] "R"      "Python" "Matlab"
## 
## $Num_Students_2023F
## [1] 84
## 
## $office
## [1] "SC 329E"
## 
## $Tenured
## [1] TRUE
## 
## $UD_Working_Yr
## [1] 7
```
]

---
## Lists - 2

.pull-left[.small[Assign a tag for each component in a list
 The component names are called *tags* in the R literature. In fact, including a name for each component is optional. We can see an example here.]

```r
list("Tessa", 
     c("R", "Python", "Matlab"), 
     84, 
     "SC 329E",
     TRUE,
     7) -> list_tessa
```
]
.pull-right[

```r
names(list_tessa) <- 
 c("Name",
 "Programming_Skills",
 "Num_Students_2023F",
 "office",
 "Tenured",
 "UD_Working_Yr")
```
]

---
## Lists - 3

The advantages of using tags in a list
 It is clearer and less error-prone to use names instead of numeric indices.

```r
list_tessa$office
```

```
## [1] "SC 329E"
```

```r
list_tessa[[2]]
```

```
## [1] "R"      "Python" "Matlab"
```

```r
list_tessa[[2]][3]
```

```
## [1] "Matlab"
```

---
## Lists - 4

.pull-left[

```r
A <- matrix(sample(1:100, 9), ncol=3)
new_list <- list(list_tessa, A)
new_list[[1]]
```

```r
new_list[[2]]
```

```
##      [,1] [,2] [,3]
## [1,]   22   40   30
## [2,]   32   48   42
## [3,]    6   26   16
```

.small[**Question:** How to extract the value in the 2nd row and 1st column in the second component in the list?]]

---
## Lists - 5

.pull-left[Since lists are vectors, they can be created via vector().

```r
y <- vector(mode="list") # create a vector with size 0
z <- vector("list", 10) # create a vector with size 10
y[["Method"]] <- "Decision Tree" 
y[["Sensitivity"]] <- 0.85
y[["Specificity"]] <- 0.89
y
```

```
## $Method
## [1] "Decision Tree"
## 
## $Sensitivity
## [1] 0.85
## 
## $Specificity
## [1] 0.89
```

]
.pull-right[
Another way to create a list.

```r
w <- rep(list(NA), 2)
w
```

```
## [[1]]
## [1] NA
## 
## [[2]]
## [1] NA
```

]

---
## Lists - 6

List Indexing
 There are three ways to access an individual component of a list and return it in the data type of this component. Each of these is useful in different contexts.

```r
y$Specificity
```

```
## [1] 0.89
```

```r
y[["Specificity"]]
```

```
## [1] 0.89
```

```r
y[[2]]
```

```
## [1] 0.85
```

---
## Lists - 7

An alternative to the 2nd and 3rd techniques listed is to use single brackets rather than double brackets.

```r
y["Specificity"]
```

```
## $Specificity
## [1] 0.89
```

```r
y[2] # the 2 is the index of "Sensitivity" within the list y
```

```
## $Sensitivity
## [1] 0.85
```

---
## Lists - 8

.pull-left[

```r
new_list[2]
```

```
## [[1]]
##      [,1] [,2] [,3]
## [1,]   22   40   30
## [2,]   32   48   42
## [3,]    6   26   16
```
]
.pull-right[

```r
new_list[[2]]
```

```
##      [,1] [,2] [,3]
## [1,]   22   40   30
## [2,]   32   48   42
## [3,]    6   26   16
```

```r
list_tessa[2]
```

```
## $Programming_Skills
## [1] "R"      "Python" "Matlab"
```
]

**Question:** How to extract the third element in the 2nd component of the list *list_tessa*?

---
## Lists - 9

We should pay attention to the use of indices when subsetting values in a list.

```r
class(new_list[[1]])
```

```
## [1] "list"
```

```r
class(new_list[[2]])
```

```
## [1] "matrix" "array"
```

**Question:** Try the following commands and discuss the difference between them.

1. *new_list[[1:2]]* and *new_list[1:2]*

2. *new_list[[1]][1:2]* and *new_list[[2]][1:2]*

3. *new_list[[2]][1:2,]* and *new_list[[2]][,1:2]*

---
## Data Frames - 1

A data frame is like a matrix. It has a two-dimensional data structure. But it differs from a matrix since the data type of each column could be different. Technically, a data frame is a list, with the components that list being equal-length vectors.

.pull-left[Creating Data Frames

```r
kids <- c("Vicky", "Patrick", "Jasmine")
ages <- c(14, 10, 8)
# create the data frame using kids and ages
df <- data.frame(kids, ages,
 stringsAsFactors = F)
df
```

```
##      kids ages
## 1   Vicky   14
## 2 Patrick   10
## 3 Jasmine    8
```

]

.pull-right[We can include columns directly in the data frame when creating it as well.

```r
data.frame(kids = c("Vicky", 
                    "Patrick", 
                    "Jasmine"), 
           ages = c(14, 10, 8), 
           stringsAsFactors = F)
```

```
##      kids ages
## 1   Vicky   14
## 2 Patrick   10
## 3 Jasmine    8
```

]

---
## Data Frames - 2

Accessing Data Frames
 There are multiple ways to access a data frame.

.pull-left[

```r
df[[1]] # list-like
```

```
## [1] "Vicky"   "Patrick" "Jasmine"
```

```r
df$kids # component name 
```

```
## [1] "Vicky"   "Patrick" "Jasmine"
```
]
.pull-right[

```r
df[,1]  # matrix-like
```

```
## [1] "Vicky"   "Patrick" "Jasmine"
```

```r
df[, "kids"] # matri-like with the column name
```

```
## [1] "Vicky"   "Patrick" "Jasmine"
```
]

**Question:** consider four ways to access the first column of the data frame above, which way(s) would generally considered to be clear and safer than others?

---
## Data Frames - 3

.pull-left[
Checking Data Structures
 We can use str() function to display the structure of an arbitrary R object.

```r
str(df)
```

```
## 'data.frame':	3 obs. of 2 variables:
## $ kids: chr "Vicky" "Patrick" "Jasmine"
## $ ages: num 14 10 8
```
]
.pull-right[
Other Matrix-Like Operations
 Many matrix operations apply to data frames as well. We will use a R built-in dataset ToothGrowth. First, we use the head() function to see the first six rows of the data.

```r
head(ToothGrowth) 
```

```
##    len supp dose
## 1  4.2   VC  0.5
## 2 11.5   VC  0.5
## 3  7.3   VC  0.5
## 4  5.8   VC  0.5
## 5  6.4   VC  0.5
## 6 10.0   VC  0.5
```
]

---
## Data Frames - 4

Use ToothGrowth data to answer the following questions.

1. Find a data frame that contains observations from the 3rd row to 7th row in ToothGrowth.

2. Find a data frame that contains values from 3rd row to 7th row for the 3rd column in ToothGrowth.

3. Find a data frame that only contains observations such that their tooth length (*len*) are smaller than the mean tooth length of guinea pigs in ToothGrowth.

---
## Factors & Tables - 1

.pull-left[.small[Since we have introduced factors in Lesson 2, we will focus the concepts about tables. Before we talk about how we could work with tables, let's see an example that demonstrates how we can rearrange the levels in a factor.]

```r
set.seed(2022)
seasons <- factor(
 sample(c("Spring", "Summer", "Fall"), 
 1000, replace = T)
 )
levels(seasons)
```

```
## [1] "Fall"   "Spring" "Summer"
```
]
.pull-right[.small[Here, we can see that the order of levels in *seasons* is not *Spring*, *Summer*, *Fall*, which didn't reflect the time order.  ]

```r
barplot(table(seasons), ylim=c(0, 350))
```

<img src="data:image/png;base64,#Lesson04_Data_Structures_2_files/figure-html/bar-1.png" style="display: block; margin: auto;" />
]

---
## Factors & Tables - 2

.pull-left[.small[Continue from the previous slide, we want to change the order of levels in *seasons*.]

```r
seasons <- factor(seasons, 
 labels = c("Spring", "Summer", "Fall"))
barplot(table(seasons), ylim=c(0, 350))
```

<img src="data:image/png;base64,#Lesson04_Data_Structures_2_files/figure-html/change_order-1.png" style="display: block; margin: auto;" />
]
.pull-right[.small[Now, if we want to add another category *Winter* in *seasons*, an intuitive way is to combine the previous vector with the new vector. But quickly we find that the data type of the new object is changed to "character". ]

```r
winters <- rep("Winter", 10)
all_seasons <- c(seasons, winters)
class(all_seasons)
```

```
## [1] "character"
```
]

---
## Factors & Tables - 3

.pull-left[.small[If we look into this further by checking the frequency distribution of the data, we find that the first three categories were changed to the characters of 1, 2, and 3.]

```r
table(all_seasons)
```

```
## all_seasons
##      1      2      3 Winter 
##    342    320    338     10
```

**Question:** How to deal with this issue?
]
.pull-right[.small[We should change the data type of *winters* to factor.]

```r
winters <- as.factor(winters)
all_seasons <- c(seasons, winters)
class(all_seasons)
```

```
## [1] "factor"
```

.small[We can add a new level to a factor object if needed.]

```r
levels(seasons) <- 
 c(levels(seasons), "Winter")
```
]

---
## Factors & Tables - 4

.small[On the previous slides, we used the table() function to find frequency distribution of a categorical variable. In fact, this function can be used to create a contingency table of two categorical variables. ]

```r
set.seed(2022)
Grade <- sample(c(LETTERS[1:4], "F"), 100, replace=T)
Sex <- sample(c("Male", "Female"), 100, replace=T, prob=c(0.4, 0.6)) 
table(Grade, Sex)
```

```
##      Sex
## Grade Female Male
##     A     12    7
##     B     10   11
##     C     11    9
##     D     11    9
##     F     16    4
```

.small[**Note:** Similar to data frames, most matrix/array operations can be used on tables.]

---
# Summary of Main Points

By now, you should know

- Basic Data Structures: Lists, Data Frames, Factors and Tables

- Recycle elements from a data object

- How to extract elements from a data object

---
# Supplementary Materials

Here are some useful supplementary materials for self-learning.

.pull-left[
.center[[<img src="https://d33wubrfki0l68.cloudfront.net/565916198b0be51bf88b36f94b80c7ea67cafe7c/7f70b/cover.png" height="250px">](https://adv-r.hadley.nz)]
.small[
* [Names and values](https://adv-r.hadley.nz/names-values.html)
* [Matrices and arrays](http://adv-r.had.co.nz/Data-structures.html#matrices-and-arrays)
* [Data frames](http://adv-r.had.co.nz/Data-structures.html#data-frames)
]
]
.pull-right[
.center[[<img src="https://d33wubrfki0l68.cloudfront.net/b88ef926a004b0fce72b2526b0b5c4413666a4cb/24a30/cover.png" height="250px">](https://r4ds.hadley.nz)]
.small[
* [Hierarchical data](https://r4ds.hadley.nz/rectangling.html)
]
]