Dayton Data Challenge

Working with Larger Data in R

UD Data Analytics Club

2026-09-09

Tonight’s Plan

Goal: Learn how to start working with a dataset that feels larger than usual — without making the workflow overwhelming.

15–20 min
Dayton Data Challenge overview

20–25 min
R demo with real-world data

5–10 min
Try it yourself

Remaining time
Questions + discussion

Dayton Data Challenge

What is it?

An undergraduate data competition where teams:

  • work with real-world data
  • choose their own questions
  • use tools of their choice
  • communicate insights to judges

Quick Facts

  • Nov. 2–7, 2026
  • Teams of 2–4
  • Free to participate
  • Register by Oct. 16

Interested in the Dayton Data Challenge?

Scan to learn more and register

Today’s R Mini-Workshop

When does a dataset become large?

A dataset is “large” when its size starts affecting how you work with it.

Too many rows

Too many columns

Slow or memory-heavy

Today we focus on a practical workflow — not on memorizing lots of packages.

Workflow for Larger Data

1
Inspect

→

2
Ask

→

3
Reduce

→

4
Explore

→

5
Communicate

Do not begin by loading everything and asking, “Now what?”

Load and Inspect the Data

We will use lower-level U.S. COVID-19 data to practice a larger-data workflow.

pacman::p_load(tidyverse, COVID19)

covid <- covid19(
  country = "US",
  level = 3
)

dim(covid)
glimpse(covid)
object.size(covid)

Start with a Question

What does the population distribution look like across Ohio counties?

This question tells us what we need:

  • Ohio
  • county
  • population

It also tells us what we do not need:

  • every state
  • every date
  • every available column

Reduce Early

ohio <- covid |>
  filter(
    administrative_area_level_2 == "Ohio"
  ) |>
  select(
    county = administrative_area_level_3,
    population
  ) |>
  filter(!is.na(population)) |>
  distinct()

dim(ohio)
object.size(ohio)

Large dataset → working dataset for our question

A Little EDA

summary(ohio$population)

ohio |>
  arrange(desc(population)) |>
  slice_head(n = 10)

What do you notice about the population sizes of Ohio counties?

Visualize One Useful Result

ggplot(ohio, aes(x = population)) +
  geom_histogram(bins = 30) +
  labs(
    x = "Population",
    y = "Number of counties",
    title = "Population Distribution Across Ohio Counties"
  ) +
  theme_minimal()

A Real Larger File: Chess Games

Before loading a large file, ask:

  • How large is the file on disk?
  • How much memory does my computer have?
  • Do I really need the entire file?

Faster ≠ unlimited: Hardware and memory still matter.

Read the Chess File with fread()

pacman::p_load(data.table)

chess_file <-
  "ficsgamesdb_2017_standard_movetimes_26046.pgn"

file.info(chess_file)$size

system.time({
  chess_raw <- as.vector(
    fread(
      chess_file,
      header = FALSE,
      sep = "\n",
      stringsAsFactors = FALSE
    )
  )
})

What Did We Actually Load?

length(chess_raw)

format(
  object.size(chess_raw),
  units = "MB"
)

head(chess_raw, 20)

Reading data and structuring data are two different problems.

Try It Yourself

Choose one U.S. state other than Ohio.

  1. Keep only the county and population variables.
  2. Remove missing and repeated population records.
  3. Make one numerical summary.
  4. Create one visualization.

Bonus: How much smaller is your working dataset than the original?

Takeaways

  • Begin with a question, not with every variable.
  • Use glimpse() to understand the structure.
  • Keep the rows and columns you need.
  • Do a small amount of EDA before jumping to complex methods.
  • If the file is too large, use a faster reader or a different workflow.

Question → Data → EDA → Insight

Want to Try a Larger Dataset?

Use the FICS Chess Data Worksheet to explore a larger real-world data file on your own.

You will:

  • check your computer’s memory and system information
  • choose an appropriate FICS chess dataset
  • inspect part of the file before loading everything
  • compare file size, reading time, and memory usage
  • think about what to do when data exceed your computer’s limits

Didn’t get to it today? Try it later at your own pace.

Workshop Materials

Find it on GitHub.

Thank you for participating!

Questions or want to continue the discussion?

Tessa Chen
Department of Mathematics
University of Dayton

ychen4@udayton.edu