class: center, middle, inverse, title-slide .title[ # MTH 411 Probability & Statistics I ] .subtitle[ ## Harnessing Distribution Fitting: Transforming Industry Data into Actionable Insights ] .author[ ### <br>Ying-Ju Tessa Chen, PhD <br><br> Associate Professor <br> Department of Mathematics<br> University of Dayton <br><br> <a href="https://twitter.com/ju_tessa"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> <span class="citation">@ying-ju</span></a> <br> <a href="https://github.com/ying-ju/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> ying-ju</a> <br> <a href="mailto:ychen4@udayton.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"></path></svg> ychen4@udayton.edu</a><br> ] --- ## Introduction - What is Distribution Fitting? - .blue[Definition]: Distribution fitting is the process of selecting a statistical distribution that best fits a dataset. - .blue[Purpose]: By understanding the underlying distribution of the data, we can make more informed decisions, predictions, and analyses. --- ## Introduction (Continue) - Why is it Important? - .blue[Predictive Power]: Understanding data distributions allows for better predictions. For example, understanding the distribution of machine breakdowns can help in predictive maintenance. - .blue[Decision Making]: With a good fit, businesses can make decisions based on the likelihood of future events. For example, a retailer can determine the probability of stockouts. --- ## Introduction (Continue) - Real-life Analogy - .blue[Weather Forecasting]: Just as meteorologists use patterns and data to predict the weather, industries use distribution fitting to forecast business-related outcomes. Not always perfect, but better than guessing. - Importance in Industry - .blue[Tailored Solutions]: Different industries have unique challenges. By fitting distributions to their specific data, industries can tailor solutions to their unique challenges. - .blue[Risk Management]: For sectors like finance, understanding distributions is crucial for risk assessment and management. --- ## Basics of Distribution Fitting - Common Statistical Distributions - **Normal Distribution**: Often seen in natural phenomena. Used in quality control, stock market returns, etc. - **Exponential Distribution**: Represents time between events in a Poisson process. Useful for modeling time-to-failure data. - **Weibull Distribution**: Commonly used in reliability engineering. Can describe various shapes of data from exponential to bell-shaped. - **Poisson Distribution**: Represents the number of events in fixed intervals of time or space. Useful for modeling rare events. --- ## Basics of Distribution Fitting (Continue) - Methods of Fitting Distributions - **Method of Moments**: Matching sample moments (like mean and variance) with theoretical moments of a distribution. - **Maximum Likelihood Estimation (MLE)**: Finds the parameters that maximize the likelihood of the observed data given a distribution. - **Visual Aid**: Show a graph where the likelihood function is maximized. - **Least Squares**: Minimizes the sum of the squared differences between observed and estimated values. --- ## Basics of Distribution Fitting (Continue) - Goodness of Fit Tests - **Purpose**: After fitting, how do we know the chosen distribution is a good fit? - **Kolmogorov-Smirnov Test**: Compares the empirical distribution function of the sample data with the cumulative distribution function of the chosen distribution. - **Visual Aid**: Show a graph comparing the two functions. - **Anderson-Darling Test**: A variation of K-S test that gives more weight to the tails. - **Chi-Squared Test**: Compares the observed frequencies in certain intervals to the expected frequencies based on the chosen distribution. --- ## Basics of Distribution Fitting (Continue) - The Importance of Visualization - **Histograms and Probability Plots**: Before fitting any distribution, visualizing data can give a good initial idea about its shape and potential distributions. .pull-left[ <img src="data:image/png;base64,#MTH411_distribution_files/figure-html/unnamed-chunk-1-1.png" width="75%" style="display: block; margin: auto;" /> ] .pull-right[ <img src="data:image/png;base64,#MTH411_distribution_files/figure-html/unnamed-chunk-2-1.png" width="75%" style="display: block; margin: auto;" /> ] --- ## Industry Applications - Quality Control and Manufacturing - .red[Process Monitoring] - **Normal Distribution**: Many manufacturing processes produce data that follows a normal distribution. Control charts based on the normal distribution are used to monitor process stability. - .green[Example]: Production of a particular component, where the objective is to maintain a specific diameter. Variations can be monitored using control charts. - .red[Reliability Engineering] - **Weibull Distribution**: Commonly used to model the life of industrial products and to understand failure mechanisms. - .green[Example]: Predicting the lifespan of a batch of light bulbs. Understanding when most failures occur can guide warranty decisions and maintenance schedules. --- ## Industry Applications (Continue) - Finance and Economics - .red[Stock Returns Modeling] - **Normal and Log-normal Distributions**: While stock returns do not strictly follow a normal distribution, it's often used as a first approximation. Log-normal is used for modeling stock prices. - .green[Example]: Predicting the likelihood of a stock's return falling within a certain range in the next month. - .red[Risk Management] - **Value at Risk (VaR)**: A measure used to assess the risk of investments. Requires understanding the distribution of returns. - .green[Example]: A portfolio manager needs to know the maximum potential loss over a given time period at a certain confidence level. --- ## Industry Applications (Continue) - Telecommunications - .red[Call Arrivals] - **Poisson Distribution**: Often used to model the number of call arrivals in a fixed period. - .green[Example]: A call center planning its staffing based on the expected number of calls during peak hours. - .red[Call Center Staffing] - **Erlang Distribution**: Useful for modeling the time a call center agent spends on calls. - .green[Example]: Determining the number of agents needed during different shifts to ensure customer wait times are minimized. --- ## Industry Applications (Continue) - Retail and E-commerce - .red[Demand Prediction] - **Normal Distribution**: Sometimes used to forecast demand, especially for products with consistent sales patterns. - .green[Example]: A retailer forecasting the demand for a popular product during the holiday season. - .red[Customer Lifetime Value Estimation] - Various distributions can be used based on purchasing patterns to predict how much a customer will spend over time. - .green[Example]: An e-commerce platform predicting the total revenue from a new user over the next two years. --- ## Case Study: Real Data Analysis .red[Insurance Pricing & Risk Analysis] An insurance company has been witnessing variability in the charges it offers to its clients. The company aims to understand the underlying distribution of these charges to: - **Segment Customers**: Identify distinct customer segments based on their charges. - **Risk Assessment**: Understand potential outliers or high-risk clients who might claim more than the average. - **Product Tailoring**: Design insurance packages tailored to different charge brackets. - **Predictive Modeling**: Use the distribution as a foundation for predictive models that forecast future claims. --- ## Case Study: Real Data Analysis (Continue) To achieve these objectives, the company needs to: - Visualize the distribution of charges. - Fit appropriate statistical distributions to the data. - Evaluate the goodness of fit. --- ## Case Study: Real Data Analysis (Continue) - Fitting Distributions: We will fit the Log-normal, Gamma, and Weibull distribution to the `charges` data. <img src="data:image/png;base64,#MTH411_distribution_files/figure-html/unnamed-chunk-3-1.png" width="40%" style="display: block; margin: auto;" /> --- ## Case Study: Real Data Analysis (Continue) - Visual Assessment: We'll overlay the fitted distributions over a histogram to visually assess the fit. <img src="data:image/png;base64,#MTH411_distribution_files/figure-html/unnamed-chunk-4-1.png" width="40%" style="display: block; margin: auto;" /> --- ## Case Study: Real Data Analysis (Continue) - Goodness-of-Fit Test: We'll use the Kolmogorov-Smirnov (KS) test to quantitatively assess the goodness of fit for each distribution. `\(H_0:\)` The sample is from the reference probability distribution `\(H_a:\)` The sample is not from the reference probability distribution | | Lognormal| Gamma| Weibull| |:-------|---------:|-----:|-------:| |p-value | 0.0556606| 0| 2e-07| The Kolmogorov-Smirnov test shows that we don't have evidence to conclude the insurance charges are not from a lognormal model with a 0.05 significance level. We have sufficient evidence to conclude that the insurance charges are not from a Gamma or Weibull distribution. --- ## Challenges and Limitations - Assumptions of Distributions - **Underlying Assumptions**: Every distribution comes with its own set of assumptions. For instance, the normal distribution assumes that the data is symmetric around the mean. - **Misfit**: If the data does not adhere to these assumptions, fitting such a distribution can lead to incorrect analyses and decisions. - Overfitting - **Fitting Noise**: In the quest to get a perfect fit, one might end up fitting the noise in the data rather than the underlying pattern. - **Predictive Performance**: An overfitted distribution might perform poorly in predicting future observations. --- ## Challenges and Limitations (Continue) - Limitations of Goodness-of-Fit Tests - **P-values**: A large dataset might result in significant p-values even for trivial differences between the observed and expected frequencies. - **Subjectivity**: Deciding which test to use and interpreting its results can sometimes be subjective. - Real-world Complexities - **Multiple Influencing Factors**: Real-world data, especially in industries like finance and healthcare, might be influenced by a plethora of factors, making it challenging to fit a single distribution. - **Time-dependence**: In some scenarios, the underlying distribution might change over time, complicating the fitting process. --- class: middle, center .Large[Questions?]