class: center, middle, inverse, title-slide .title[ # MTH 208 Exploratory Data Analysis ] .subtitle[ ## Lesson 01: Introduction to EDA ] .author[ ### <br>Ying-Ju Tessa Chen, PhD <br><br> Associate Professor <br> Department of Mathematics<br> University of Dayton <br><br> <a href="https://twitter.com/ju_tessa"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> <span class="citation">@ying-ju</span></a> <br> <a href="https://github.com/ying-ju/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> ying-ju</a> <br> <a href="mailto:ychen4@udayton.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"></path></svg> ychen4@udayton.edu</a><br> ] --- ## Learning Objectives - What is EDA, and why is it important? - Overview of the data analysis pipeline - Differences between EDA, confirmatory analysis, and predictive modeling --- ## What is EDA? Exploratory Data Analysis (EDA) is a foundational approach in the field of data science, essential for .red[analyzing], .red[summarizing], and .red[understanding] data sets. At its core, EDA is about uncovering the underlying .red[structure], .red[patterns], .red[anomalies], and .red[insights] in data, often before formal modeling commences. It is instrumental in identifying the most appropriate statistical techniques and methods for further analysis. --- ## Why is EDA important? EDA combines a variety of statistical techniques with visual methods to facilitate a deeper understanding of the data. This approach allows data scientists and analysts to: - **Discover Patterns**: Uncover underlying patterns and relationships hidden within the data.This is essential for understanding the data set before applying complex models or statistical techniques. - **Spot Anomalies**: Identify outliers or unusual data points that might indicate errors in data collection or potential areas of interest for further investigation. - **Test Hypotheses**: Formulate and test hypotheses about the data, paving the way for more in-depth analysis. - **Check Assumptions**: Verify the assumptions underlying - statistical analyses, ensuring their validity and appropriateness. --- ## Why is EDA important (Continue)? .pull-left-2[Exploratory Data Analysis (EDA) is distinguished by its flexible and exploratory nature. Unlike rigid, hypothesis-driven methods, EDA encourages open-ended exploration and intuitive data understanding, making it an essential first step in data analysis and subsequent statistical testing and modeling. The concept and practice of EDA were significantly shaped and popularized by American mathematician **John Tukey** in the 1970s. Tukey's work in this area was groundbreaking; he brought together a range of methods and an overarching philosophy that emphasized the exploratory aspect of analyzing data. It's important to note, however, that while Tukey played a pivotal role in organizing and advocating for these methods, the practice of exploratory analysis of data has been a part of statistics since its inception. Today, EDA remains a critical tool in the data discovery process. Its techniques, continually evolving with advances in computing and data visualization technologies, are fundamental to data science, enabling practitioners to make informed, data-driven decisions. ] .pull-right-2[ .center[[<img src="https://upload.wikimedia.org/wikipedia/en/e/e9/John_Tukey.jpg" width="250px">](https://en.wikipedia.org/wiki/John_Tukey) <div>John Wilder Tukey<br>(June 16, 1915 – July 26, 2000) </div>] ] --- ## Four principles of EDA We will introduce various methods for exploring data, each embodying the core characteristics that define the philosophy of EDA. Tukey established key principles that guide the exploratory process. These principles are fundamental to understanding data and are integral to the EDA methodology. - **Revelation**: This principle is about uncovering the underlying structure of the data. The idea is to reveal hidden patterns, trends, and relationships that are not immediately obvious. This often involves visualizing the data in various ways to uncover these insights. - **Resistance**: This refers to the robustness of the analysis methods. The techniques used in EDA should be resistant to outliers and not be overly influenced by a few data points. This ensures that the analysis gives a true representation of the data as a whole. - **Reexpression**: This principle involves transforming or reexpressing data to make it easier to understand or reveal hidden aspects. This can include applying logarithmic transformations, scaling, or other mathematical manipulations to make patterns more evident or data more amenable to analysis. - **Residuals**: The focus here is on examining the residuals - the differences between observed and predicted values in a model. Analyzing residuals can reveal whether the chosen model is appropriate or if there are patterns in the data that the model isn’t capturing. .center[ `\(\mbox{Residual} = \mbox{Observed Value (Data)} - \mbox{Predicted Values (Fitted by a model)}\)`] --- ### Case Study We will study data .green[obesity_cost_full] from R package .orange[obcost]. This data set gives the name of each state in the United States, a Year variable from 1996 to 2018, and some variables related to medical costs on obesity. We will focus on the average total economic cost of obesity by state from 1996 to 2018. The following table shows a few rows of the data with the variables of interest. .pull-left[ |State | TC_it| |:----------|-----------:| |Alabama | 3723234610| |Alaska | 654101733| |Arizona | 4206639898| |Arkansas | 2137036028| |California | 26992318816| ] .pull-right[ Here are some questions that might interest us. - What states have contributed to the most average medical cost on obesity? - What is a typical number of medical cost on obesity? - Is Ohio contributing more or less medical cost than Michigan? - Are southeastern states contributing less medical cost than western states? - In relation to population size, which state contributes the most average medical costs related to obesity? ] --- ### Case Study (Continue) .pull-left[ <img src="data:image/png;base64,#Lesson01_files/figure-html/unnamed-chunk-2-1.png" style="display: block; margin: auto;" /> <br> We can see that the majority of the average total prices are between 0 and 10 billions, with only a few states having substantial average expenses. This data is heavily right-skewed. Consider the variable `\(\log_{10}\mbox{(TC_it)}\)`. We can improve the presentation of the data by reexpressing the average cost by taking logs. ] .pull-right[ <img src="data:image/png;base64,#Lesson01_files/figure-html/unnamed-chunk-3-1.png" style="display: block; margin: auto;" /> <br> This graphical display is more appropriate for viewing the data. The log average costs are evenly spread out between 8.5 and 10 and we can see more interesting structure in the data. ] --- ### Case Study (Continue) .pull-left[ A typical log average total cost can be seen to be about 9.46 (.red[How did I estimate a typical value of the data here?]). We can summarize the data by the typical value of 9.46. Then, the residuals can be computed by `$$\mbox{Residual} = \log (\mbox{TC_it}) - 9.46.$$` <img src="data:image/png;base64,#Lesson01_files/figure-html/unnamed-chunk-4-1.png" style="display: block; margin: auto;" /> ] .pull-right[ On the log scale, two states have average total costs that are 0.75 less than the average of 9.46, while another two states exceed the average by the same margin. The residual plot shows the variation of these log average costs from the overall average. ] --- ### Case Study (Continue) .pull-left[ <div id="htmlwidget-dd0ae014f9d5bf0fcc72" style="width:400px;height:300px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-dd0ae014f9d5bf0fcc72">{"x":{"data":[{"x":[0.10919539133782012,-0.64585288911064931,0.16265927208654968,-0.13197062026073184,0.96986403725901538,0.046127814672191819,-0.015536707551168313,-0.55529394939741739,0.65432475157916958,0.40598281835795608,-0.53140292238497011,-0.43519155766606227,0.54562536430379549,0.25649643390675786,-0.064298039217759714,-0.11589568734318781,0.069334224416384288,0.11976865187884714,-0.49071732664624329,0.20590714540664098,0.24462386943615702,0.44073022805381706,0.15332634393957859,-0.10164473820390954,0.21345813625205171,-0.63770208106495851,-0.28138924441164942,-0.20384241495238414,-0.47620402703093284,0.36289064645327457,-0.32537214823442046,0.72898554008311722,0.39581765476302866,-0.68402502973031865,0.51077036780961338,-0.0010534422004474238,-0.014566131655568779,0.54290961463725773,-0.59026664442470611,0.075736766031471989,-0.64509747642750126,0.22900203982596259,0.84469726021035629,-0.18707091325609071,-0.83440047530660522,0.33937135344505531,0.26994212878790358,-0.30270534875358984,0.19710908335108551,-0.821521238070967],"y":[-0.16573766134679319,0.30184243433177471,-0.28682778961956501,-0.29123234041035179,0.30105680543929336,-0.067125656083226182,0.058450165390968312,0.33794873133301739,-0.10299405846744775,0.31507782414555552,-0.25760374832898381,0.093756957910954952,-0.058966006338596333,0.35655671581625936,0.37688659634441135,0.07775655426084993,0.18638012614101174,-0.070820617489516735,-0.10748210120946172,-0.028670980222523235,-0.22489786930382252,-0.11004431433975698,-0.17385207600891592,0.14148075729608534,0.071072029694914829,0.20380721595138318,0.15454086102545261,0.029740257933735859,-0.23615141808986664,0.37255379036068914,-0.13806863240897654,0.15655611865222452,-0.23012574017047882,0.26296095699071886,0.0015303485095500835,0.36798794567584991,-0.2763977920636535,-0.031345594115555264,0.19640724509954455,-0.027192140743136417,-0.3708170767873526,-0.34202098064124586,0.092449405044317257,-0.02951488345861436,0.21243608258664615,0.15903313215821979,-0.16968464571982622,-0.086795711517334007,-0.14312279168516401,-0.12936862483620643],"text":["Alabama:<br />0.1094","Alaska:<br />-0.6459","Arizona:<br />0.1624","Arkansas:<br />-0.1317","California:<br />0.9697","Colorado:<br />0.046","Connecticut:<br />-0.0158","Delaware:<br />-0.5551","Florida:<br />0.6542","Georgia:<br />0.4058","Hawaii:<br />-0.5312","Idaho:<br />-0.4353","Illinois:<br />0.5454","Indiana:<br />0.2564","Iowa:<br />-0.0641","Kansas:<br />-0.116","Kentucky:<br />0.0693","Louisiana:<br />0.1198","Maine:<br />-0.4906","Maryland:<br />0.2059","Massachusetts:<br />0.2449","Michigan:<br />0.4407","Minnesota:<br />0.1532","Mississippi:<br />-0.1018","Missouri:<br />0.2133","Montana:<br />-0.6379","Nebraska:<br />-0.2812","Nevada:<br />-0.2038","New Hampshire:<br />-0.4762","New Jersey:<br />0.3628","New Mexico:<br />-0.3255","New York:<br />0.7289","North Carolina:<br />0.3958","North Dakota:<br />-0.6838","Ohio:<br />0.5108","Oklahoma:<br />-0.001","Oregon:<br />-0.0144","Pennsylvania:<br />0.5428","Rhode Island:<br />-0.5906","South Carolina:<br />0.0757","South Dakota:<br />-0.6452","Tennessee:<br />0.229","Texas:<br />0.8445","Utah:<br />-0.1868","Vermont:<br />-0.8344","Virginia:<br />0.3395","Washington:<br />0.2697","West Virginia:<br />-0.3029","Wisconsin:<br />0.197","Wyoming:<br />-0.8214"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(0,0,0,1)","opacity":1,"size":5.6692913385826778,"symbol":"circle","line":{"width":1.8897637795275593,"color":"rgba(0,0,0,1)"}},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":25.741248097412488,"r":7.3059360730593621,"b":51.650754116507542,"l":10.958904109589042},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":21.253632212536321},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-0.92461370093488626,1.0600772628872963],"tickmode":"array","ticktext":["-0.5","0.0","0.5","1.0"],"tickvals":[-0.5,0,0.5,1],"categoryorder":"array","categoryarray":["-0.5","0.0","0.5","1.0"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.6529680365296811,"tickwidth":0.66417600664176002,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":17.002905770029059},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176002,"zeroline":false,"anchor":"y","title":{"text":"Residual","font":{"color":"rgba(0,0,0,1)","family":"","size":21.253632212536321}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-1.1000000000000001,1.1000000000000001],"tickmode":"array","ticktext":["-1.0","-0.5","0.0","0.5","1.0"],"tickvals":[-1,-0.5,0,0.5,1],"categoryorder":"array","categoryarray":["-1.0","-0.5","0.0","0.5","1.0"],"nticks":null,"ticks":"","tickcolor":null,"ticklen":3.6529680365296811,"tickwidth":0,"showticklabels":false,"tickfont":{"color":null,"family":null,"size":0},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176002,"zeroline":false,"anchor":"x","title":{"text":"","font":{"color":null,"family":null,"size":0}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.8897637795275593,"font":{"color":"rgba(0,0,0,1)","family":"","size":17.002905770029059}},"hovermode":"closest","width":400,"height":300,"barmode":"relative"},"config":{"doubleClick":"reset","modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"source":"A","attrs":{"658479ab6373":{"x":{},"y":{},"text":{},"type":"scatter"}},"cur_data":"658479ab6373","visdat":{"658479ab6373":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.20000000000000001,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> ] .pull-right[ Recall the questions we might be interested: - What states have contributed to the most average medical cost on obesity? - What is a typical number of medical cost on obesity? - Is Ohio contributing more or less medical cost than New York? - Are southeastern states contributing less medical cost than western states? - In relation to population size, which state contributes the most average medical costs related to obesity? ] --- ## Overview of the data analysis pipeline <img src="data:image/png;base64,#figures/pipeline.png" width="90%" style="display: block; margin: auto;" /> --- ## Differences between EDA, confirmatory analysis, and predictive modeling To understand the various facets of data analysis, it is crucial to know the differences between Exploratory Data Analysis (EDA), Confirmatory Data Analysis, and Predictive Modeling. Each has its own objectives, methodologies, and implications. - **Exploratory Data Analysis** - .blue[Objective]: To uncover patterns, anomalies, relationships, and trends in data without any prior hypotheses. - .blue[Approach]: Primarily visual and intuitive; involves summarizing the main characteristics of a dataset through visualization and descriptive statistics. - .blue[Outcome]: Generates hypotheses, insights, and a deeper understanding of data's underlying structure and characteristics. - .blue[Key Characteristics]: Flexible, open-ended, often the first step in data analysis. --- ## Differences between EDA, confirmatory analysis, and predictive modeling (Continue) - **Confirmatory Data Analysis** - .blue[Objective]: To test hypotheses or models that were formulated prior to data analysis. - .blue[Approach]: Uses statistical tools to verify or refute predefined hypotheses. This analysis is usually more structured than EDA and follows a more rigid methodology. - .blue[Outcome]: Provides evidence to support or reject hypotheses, often with a focus on statistical significance. - .blue[Key Characteristics]: Hypothesis-driven, relies on statistical inference, more formal and structured. --- ## Differences between EDA, confirmatory analysis, and predictive modeling (Continue) - **Predictive Modeling** - .blue[Objective]: To create models that can predict future outcomes based on historical data. - .blue[Approach]: Involves selecting and training models (like regression, classification, or machine learning algorithms) on existing data to forecast unknown or future values. - .blue[Outcome]: Produces a model that can be used for forecasting, along with metrics to gauge its accuracy and reliability. - .blue[Key Characteristics]: Forward-looking, uses machine learning and statistical techniques, focused on prediction accuracy. --- ## Differences between EDA, confirmatory analysis, and predictive modeling (Continue) - **Scope and Focus**: EDA is about exploring and understanding data without specific aims; confirmatory analysis tests specific theories or hypotheses; predictive modeling forecasts future events based on historical patterns. - **Methodology**: EDA is more about visualization and intuition, confirmatory analysis relies on statistical tests, and predictive modeling focuses on algorithmic and statistical methods for forecasting. - **Sequence in Data Analysis**: Often, EDA precedes confirmatory analysis and predictive modeling. Insights gained from EDA can inform the development of hypotheses for confirmatory analysis or feature selection in predictive modeling. - **Nature of Conclusions**: EDA provides insights and raises questions, confirmatory analysis offers statistical evidence, and predictive modeling delivers actionable predictions. It is important to note that these methods are not mutually exclusive but are often used in conjunction as part of a comprehensive data analysis strategy. Understanding their differences helps in selecting the appropriate approach based on the objectives of a specific data analysis task. --- ## References The lectures of this course are based on the ideas from the following references. - Exploratory Data Analysis by John W. Tukey - A Course in Exploratory Data Analysis by Jim Albert - The Visual Display of Quantitative Information by Edward R. Tufte - Data Science for Business: what you need to know about data mining and data-analytic thinking by Foster Provost and Tom Fawcett - Storytelling with Data: A Data Visualization Guide for Business Professionals by Cole Nussbaumer Knaflic