{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Red Wine Quality Prediction \n", "\n", "by Nicole Bidwell, Ruocong Sun, Alysen Townsley, Hongyang Zhang" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove-input" ] }, "outputs": [], "source": [ "import pandas as pd \n", "from myst_nb import glue\n", "import pickle" ] }, { "cell_type": "code", "execution_count": 109, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove-input" ] }, "outputs": [ { "data": { "application/papermill.record/text/plain": "62.3" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "test_set_score" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
fixed acidityvolatile aciditycitric acidresidual sugarchloridesfree sulfur dioxidetotal sulfur dioxidedensitypHsulphatesalcoholquality
07.40.700.001.90.07611.034.00.99783.510.569.45
17.80.880.002.60.09825.067.00.99683.200.689.85
27.80.760.042.30.09215.054.00.99703.260.659.85
311.20.280.561.90.07517.060.00.99803.160.589.86
47.40.700.001.90.07611.034.00.99783.510.569.45
\n
", "application/papermill.record/text/plain": " fixed acidity volatile acidity citric acid residual sugar chlorides \\\n0 7.4 0.70 0.00 1.9 0.076 \n1 7.8 0.88 0.00 2.6 0.098 \n2 7.8 0.76 0.04 2.3 0.092 \n3 11.2 0.28 0.56 1.9 0.075 \n4 7.4 0.70 0.00 1.9 0.076 \n\n free sulfur dioxide total sulfur dioxide density pH sulphates \\\n0 11.0 34.0 0.9978 3.51 0.56 \n1 25.0 67.0 0.9968 3.20 0.68 \n2 15.0 54.0 0.9970 3.26 0.65 \n3 17.0 60.0 0.9980 3.16 0.58 \n4 11.0 34.0 0.9978 3.51 0.56 \n\n alcohol quality \n0 9.4 5 \n1 9.8 5 \n2 9.8 5 \n3 9.8 6 \n4 9.4 5 " }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "wine_quality_df" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "1599" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "wine_quality_df_nrows" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "11" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "wine_quality_df_nfeatures" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "3" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "min_wine_quality" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "8" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "max_wine_quality" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
fit_timescore_timetest_scoretrain_score
00.0010.0000.4420.440
10.0000.0010.4380.441
20.0000.0010.4380.441
30.0000.0010.4420.440
40.0000.0010.4440.440
\n
", "application/papermill.record/text/plain": " fit_time score_time test_score train_score\n0 0.001 0.000 0.442 0.440\n1 0.000 0.001 0.438 0.441\n2 0.000 0.001 0.438 0.441\n3 0.000 0.001 0.442 0.440\n4 0.000 0.001 0.444 0.440" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "dummy_df" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "44.08" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "dummy_valid_score" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
mean_test_scoremean_train_scoremean_fit_timeCclass_weightcriterionmax_depthn_neighborsgamma
model_name
svc0.6130.7680.2261000.0No Class WeightNaNNaNNaN0.01
knn0.5981.0000.008NaNNaNNaNNaN1.0NaN
decision_tree0.5930.9940.017NaNNo Class Weightgini16.0NaNNaN
logistic0.5860.5920.0350.1No Class WeightNaNNaNNaNNaN
\n
", "application/papermill.record/text/plain": " mean_test_score mean_train_score mean_fit_time C \\\nmodel_name \nsvc 0.613 0.768 0.226 1000.0 \nknn 0.598 1.000 0.008 NaN \ndecision_tree 0.593 0.994 0.017 NaN \nlogistic 0.586 0.592 0.035 0.1 \n\n class_weight criterion max_depth n_neighbors gamma \nmodel_name \nsvc No Class Weight NaN NaN NaN 0.01 \nknn NaN NaN NaN 1.0 NaN \ndecision_tree No Class Weight gini 16.0 NaN NaN \nlogistic No Class Weight NaN NaN NaN NaN " }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "comparison_df" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "58.6" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "logistic_gs_score" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "59.3" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "decision_tree_gs_score" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "59.8" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "knn_gs_score" } }, "output_type": "display_data" }, { "data": { "application/papermill.record/text/plain": "61.3" }, "metadata": { "scrapbook": { "mime_prefix": "application/papermill.record/", "name": "svc_gs_score" } }, "output_type": "display_data" } ], "source": [ "comparison_df = pd.read_csv(\"../results/tables/comparison_df.csv\", index_col=0).round(3)\n", "wine_quality_df = pd.read_csv('../data/winequality-red.csv', sep = ';')\n", "dummy_df = pd.read_csv('../results/tables/cv_results.csv').round(3)\n", "\n", "glue('test_set_score', (pd.read_csv(\"../results/tables/test_set_score.csv\", index_col=0).round(3)).loc[0, 'test_set_score'] * 100, \n", " display=False)\n", "glue(\"wine_quality_df\", wine_quality_df.head(), display=False)\n", "glue('wine_quality_df_nrows', wine_quality_df.shape[0], display=False)\n", "glue('wine_quality_df_nfeatures', wine_quality_df.shape[1] - 1, display=False)\n", "glue('min_wine_quality', wine_quality_df['quality'].min(), display=False)\n", "glue('max_wine_quality', wine_quality_df['quality'].max(), display=False)\n", "glue('dummy_df', dummy_df, display=False)\n", "glue('dummy_valid_score', (dummy_df['test_score'].mean() * 100).round(3), display=False)\n", "glue('comparison_df', comparison_df, display=False)\n", "glue('logistic_gs_score', (comparison_df.loc['logistic', 'mean_test_score'] * 100).round(3), display=False)\n", "glue('decision_tree_gs_score', (comparison_df.loc['decision_tree', 'mean_test_score'] * 100).round(3), display=False)\n", "glue('knn_gs_score', (comparison_df.loc['knn', 'mean_test_score'] * 100).round(3), display=False)\n", "glue('svc_gs_score', (comparison_df.loc['svc', 'mean_test_score'] * 100).round(3), display=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this project our group seeks to use machine learning algorithms to predict wine quality (scale of 0 to 10) using physiochemical properties of the liquid. We use a train-test split and cross-validation to simulate the model encountering unseen data. We use and tune the parameters of several classification models: logistic regression, decision tree, kNN, and SVM RBF to see which one has the highest accuracy, and then deploy the winner onto the test set. The final test set accuracy is around {glue:text}`test_set_score` percent. Depending on the standard, this can be decent or poor. However, a more important note is that for the really extreme quality ones (below 5 or above 6), the model was unable to identify quite a few of them correctly, suggesting that it is not very robust to outliers. We include a final discussion section on some of the potential causes for this performance as well as proposed solutions for any future analysis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Red wines have a long history that can be traced all the way back to the ancient Greeks. Today, they are more accessible to an average person than ever and the entire industry is estimated to be worth around 109.5 billion USD {cite}`market_trends`. Despite its ubiquity, most people can barely tell the difference between a good and a bad wine, to the point where we need trained professionals (sommeliers) to understand the difference. In this project, we seek to use machine learning algorithms to predict the quality of the wine based on the physiochemical properties of the liquid. This model, if effective, could allow manufactures and suppliers to have a more robust understanding of the wine quality based on measurable properties." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Methods & Results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### EDA\n", "#### Dataset Description\n", "The dataset is the \"winequality-red.csv\" file from the UC Irvine Machine Learning Repository {cite}`misc_wine_quality_186`, which was originally referenced from Decision Support Systems, Elsevier {cite}`cortez2009modeling`. The dataset contains physiochemical proprties (features) of red vinho verde wine samples from the north of Portugal, along with an associated wine quality score from 0 (worst) to 10 (best). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{glue:figure} wine_quality_df\n", ":width: 400px\n", ":height: 400px\n", ":name: \"wine_quality_df\"\n", ":align: left\n", "\n", "First five rows of the red wine dataframe.\n", "```" ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "There are {glue:text}`wine_quality_df_nfeatures` feature columns representing physiochemical characteristics of the wines, such as fixed acidity, residual sugar, chlorides, density, etc. There are {glue:text}`wine_quality_df_nrows` rows or observations in the dataset, with no missing values. The target is the quality column which is listed as a set of ordinal values from {glue:text}`min_wine_quality` to {glue:text}`max_wine_quality`, although they could go as low as 0 or as high as 10 (this data set does not contain observations across the entire range). Most observations have an \"average\" quality of 5 or 6, with fewer below a score of 5 or above a score of 6." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Columns\n", "- `fixed acidity`: grams of tartaric acid per cubic decimeter.\n", "- `volatile acidity`: grams of acetic acid per cubic decimeter.\n", "- `citric acid`: grams of citric acid per cubic decimeter.\n", "- `residual sugar`: grams of residual sugar per cubic decimeter.\n", "- `chlorides`: grams of sodium chloride per cubic decimeter.\n", "- `free sulfur dioxide`: grams of unreacted sulfur dioxide per cubic decimeter. \n", "- `total sulfur dioxide`: grams of total sulfur dioxide per cubic decimeter. \n", "- `density`: density of the wine in grams per cubic decimeter.\n", "- `pH`: pH value of the wine\n", "- `sulphates`: grams of potassium sulphate per cubic decimeter\n", "- `alcohol` : percentage volume of alcohol content. \n", "- `quality` : integer range from 0 (representing low-quality) to 10 (representing high-quality)." ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Visualization" ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We first observe the distribution of the features using their statistical summaries and a histogram. We can see that the majority of features have a skewed distribution, with many containing outliers. Volatile acidity, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide, and sulphates all have very extreme outliers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{figure} ../results/figures/repeating_hists_plot.png\n", "---\n", "width: 1000px\n", "name: repeating_hists_plot\n", "---\n", "Histograms showing the distrbution of each feature in the red wine dataframe.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Model Training\n", "#### Model Selection and Hyperparameter Tuning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our method for model selection involves using 5-fold cross-validation and hyperparameter tuning on several models: logistic regression, decision tree, kNN and SVM RBF. We use validation accuracy as our metric. Below we first use a dummy classifier to establish the baseline." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{glue:figure} dummy_df\n", ":figwidth: 400px\n", ":name: \"dummy_df\"\n", "\n", "Cross valdidation results for the Dummy Classifier baseline model.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, the baseline obtains an accuracy of around {glue:text}`dummy_valid_score` percent. We now use cross cross validation paired with hyperparameter tuning to identify a model that performs the best. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{glue:figure} comparison_df\n", ":width: 400px\n", ":name: \"comparison_df\"\n", "\n", "Grid search results for the four models: Logistic Regression, Decision Tree, kNN, and SVC.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that logistic regression has a best validation score of {glue:text}`logistic_gs_score` percent; decision tree is {glue:text}`decision_tree_gs_score` percent; kNN is {glue:text}`knn_gs_score` percent, and RBF SVM is {glue:text}`svc_gs_score` percent. As a result, we will use the tuned RBF SVM as our model on the test set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Test Set Deployment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The best model's score on the test set is around {glue:text}`test_set_score`, which shows a slight improvement compared with the validation score. We want to further probe into its performance by looking at the confusion matrix." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{figure} ../results/figures/confusion_matrix_plot.png\n", "---\n", "width: 600px\n", "name: confusion_matrix_plot\n", "---\n", "Confusion matrix of the SVC model performance on the test data.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the really mediocre wines (5 and 6), the model can predict most of them correctly, but the model fails to predict a large proportion of extreme ones correctly, suggesting that the model is not too robust against outliers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Discussion " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this project, we built several machine learning classification models seeking to predict the wine quality based on the physiochemical properties of the liquid. By trying out different models with different hyperparameters, we have found that for our data set, the best performing model is RBF SVM. However, despite being the best, the accuracy is only around {glue:text}`test_set_score` percent. Depending on the situation this can be poor or decent. More importantly, the algorithm seems to not be able to identify the outliers precisely, and in the case where people want to be able to find really good or bad wines, this model's performance would not be able to meet people's expectations. Our group's discussion has concluded that there might be several factors leading to this phenomenon:\n", "\n", "### High correlations:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{figure} ../results/figures/correlation_matrix_plot.png\n", "---\n", "width: 1000px\n", "name: correlation_matrix_plot\n", "---\n", "Correlation matrix for all red wine physiochemical features in the dataframe.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Several variables in the data set appear to have a substantial amount of correlation (in the range of 0.6) and this collinearity could have potentially caused problems with some of our models. Given this and the high dimensionality, we could have implemented a dimensionality reduction algorithm (such as PCA) to reduce the number of features and therefore eliminate some of the collinearity.\n", "\n", "### Potential Interactions:\n", "In our logistic regression model we did not take any of the potential interaction into the account. With this many qualities it is possible that some of the features affect the effect of others {cite}`log_regression_PCA` {cite}`deciphering_interactions`.\n", "\n", "### Problem Formulation:\n", "The response variable could be treated as a number instead and an approach of regression question could have better captured the nature of our problem and produced a better model. Additionally, due to the limited scope of our data set (no observation below {glue:text}`min_wine_quality` or above {glue:text}`max_wine_quality`), a classification model trained on this data set would not be able to identify any observation outside of the scope correctly. A regression algorithm is more immune to this kind of problem. \n", "\n", "### Infeasibility of the Problem\n", "Despite the potential improvements we have identified (or not) for our project, there still exists a possibility that even with all these improvements, the accuracy would not improve that much. And that is not due to the incorrect setup for the analyses, but rather the fact that some of the underlying uncontrollable factors in the process of wine making simply makes it impossible to detect patterns for really good or bad wines, and their qualities can only determined by actually tasting rather than prediction using numerical representations of some of its properties. However, among all the possible problems we have identified, this is the only one where we have zero proposed solutions for." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Software Attributions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To complete this analysis, visualize data, and build the machine learning model, Python {cite}`10.5555/1593511` and associated libraries, including Pandas {cite}`mckinney2010data`, NumPy {cite}`numpy`, scikit-learn {cite}`scikit-learn`, Altair {cite}`vanderplas2018altair`, Seaborn {cite}`Waskom2021`, and Matplotlib {cite}`Hunter:2007` were used. \n", "\n", "We acknowledge the contributions of the open-source community and developers behind these tools, which significantly facilitated our analysis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## References " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{bibliography}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "dsci522_group20_env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 4 }