Lab 6. How to Assess Performance ============================ Overview This lab will introduce you to model evaluation, where you evaluate or assess the performance of each model that you train before you decide to put it into production. By the end of this lab, you will be able to create an evaluation dataset. You will be equipped to assess the performance of linear regression models using **mean absolute error** (**MAE**) and **mean squared error** (**MSE**). You will also be able to evaluate the performance of logistic regression models using accuracy, precision, recall, and F1 score. Exercise 6.01: Importing and Splitting Data ------------------------------------------- In this exercise, you will import data from a repository and split it into a training and an evaluation set to train a model. Splitting your data is required so that you can evaluate the model later. This exercise will get you familiar with the process of splitting data; this is something you will be doing frequently. **Note:** The Car dataset that you will be using in this lab was taken from the UCI Machine Learning Repository. The following steps will help you complete the exercise: 1. Open a new Jupyter notebook. 2. Import the required libraries: ``` import pandas as pd from sklearn.model_selection import train_test_split ``` 3. Create a Python list: ``` # data doesn't have headers, so let's create headers _headers = ['buying', 'maint', 'doors', 'persons', \ 'lug_boot', 'safety', 'car'] ``` The data that you are reading in is stored as a CSV file. The browser will download the file to your computer. You can open the file using a text editor. If you do, you will see something similar to the following:  `CSV` files normally have the name of each column written in the first row of the data. For instance, have a look at this dataset\'s CSV file, which you used in *Lab 3, Binary Classification*:  4. Read the data: ``` df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/car.data', \ names=_headers, index_col=None) ``` 5. Print out the top five records: ``` df.head() ``` The code in this step is used to print the top five rows of the DataFrame. The output from that operation is shown in the following screenshot:  Caption: The top five rows of the DataFrame 6. Create a training and an evaluation DataFrame: ``` training, evaluation = train_test_split(df, test_size=0.3, \ random_state=0) ``` Note The third parameter random\_state is set to 0 to ensure reproducibility of results. 7. Create a validation and test dataset: ``` validation, test = train_test_split(evaluation, test_size=0.5, \ random_state=0) ``` This code is similar to the code in *Step 6*. In this step, the code splits our evaluation data into two equal parts because we specified `0.5`, which means `50%`. Data Structures -- Vectors and Matrices --------------------------------------- In this section, we will look at different data structures, as follows. ### Scalars You assign them to variables, such as in the following expression: ``` temperature = 23 ``` If you had to store the temperature for 5 days, you would need to store the values in 5 different values, such as in the following code snippet: ``` temp_1 = 23 temp_2 = 24 temp_3 = 23 temp_4 = 22 temp_5 = 22 ``` ### Vectors Consider the following code snippet for creating a Python list: ``` temps_list = [23, 24, 23, 22, 22] ``` You can create a vector from the list using the `.array()` method from `numpy` by first importing `numpy` and then using the following snippet: ``` import numpy as np temps_ndarray = np.array(temps_list) ``` You can proceed to verify the data type using the following code snippet: ``` print(type(temps_ndarray)) ``` The code snippet will cause the compiler to print out the following:  Caption: The temps\_ndarray vector data type You may inspect the contents of the vector using the following code snippet: ``` print(temps_ndarray) ``` This generates the following output:  Caption: The temps\_ndarray vector ``` print(temps_list) ``` The code snippet yields the following output:  Caption: List of elements in temps\_list Vectors have a shape and a dimension. Both of these can be determined by using the following code snippet: ``` print(temps_ndarray.shape) ``` The output is a Python data structure called a **tuple** and looks like this:  Caption: Shape of the temps\_ndarray vector ### Matrices To convert `temps_ndarray` into a matrix with five rows and one column, you would use the following snippet: ``` temps_matrix = temps_ndarray.reshape(-1, 1) ``` To see the new shape, use the following snippet: ``` print(temps_matrix.shape) ``` You will get the following output:  Caption: Shape of the matrix You can print out the value of the matrix using the following snippet: ``` print(temps_matrix) ``` The output of the code is as follows:  Caption: Elements of the matrix You may reshape the matrix to contain `1` row and `5` columns and print out the value using the following code snippet: ``` print(temps_matrix.reshape(1,5)) ``` The output will be as follows:  Caption: Reshaping the matrix Finally, you can convert the matrix back into a vector by dropping the column using the following snippet: ``` vector = temps_matrix.reshape(-1) ``` You can print out the value of the vector to confirm that you get the following:  Exercise 6.02: Computing the R[2] Score of a Linear Regression Model ---------------------------------------------------------------------------------- As mentioned in the preceding sections, R[2] score is an important factor in evaluating the performance of a model. Thus, in this exercise, we will be creating a linear regression model and then calculating the R[2] score for it. The following attributes are useful for our task: - CIC0: information indices - SM1\_Dz(Z): 2D matrix-based descriptors - GATS1i: 2D autocorrelations - NdsCH: Pimephales promelas - NdssC: atom-type counts - MLOGP: molecular properties - Quantitative response, LC50 \[-LOG(mol/L)\]: This attribute represents the concentration that causes death in 50% of test fish over a test duration of 96 hours. The following steps will help you to complete the exercise: 1. Open a new Jupyter notebook to write and execute your code. 2. Next, import the libraries mentioned in the following code snippet: ``` # import libraries import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression ``` In this step, you import `pandas`, which you will use to read your data. You also import `train_test_split()`, which you will use to split your data into training and validation sets, and you import `LinearRegression`, which you will use to train your model. 3. Now, read the data from the dataset: ``` # column headers _headers = ['CIC0', 'SM1', 'GATS1i', 'NdsCH', 'Ndssc', \ 'MLOGP', 'response'] # read in data df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/'\ 'qsar_fish_toxicity.csv', \ names=_headers, sep=';') ``` In this step, you create a Python list to hold the names of the columns in your data. You do this because the CSV file containing the data does not have a first row that contains the column headers. You proceed to read in the file and store it in a variable called `df` using the `read_csv()` method in pandas. You specify the list containing column headers by passing it into the `names` parameter. This CSV uses semi-colons as column separators, so you specify that using the `sep` parameter. You can use `df.head()` to see what the DataFrame looks like:  Caption: The first five rows of the DataFrame 4. Split the data into features and labels and into training and evaluation datasets: ``` # Let's split our data features = df.drop('response', axis=1).values labels = df[['response']].values X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.2, \ random_state=0) X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ random_state=0) ``` In this step, you create two `numpy` arrays called `features` and `labels`. You then proceed to split them twice. The first split produces a `training` set and an `evaluation` set. The second split creates a `validation` set and a `test` set. 5. Create a linear regression model: ``` model = LinearRegression() ``` In this step, you create an instance of `LinearRegression` and store it in a variable called `model`. You will make use of this to train on the training dataset. 6. Train the model: ``` model.fit(X_train, y_train) ``` You should get an output similar to the following:  Caption: Training the model 7. Make a prediction, as shown in the following code snippet: ``` y_pred = model.predict(X_val) ``` In this step, you make use of the validation dataset to make a prediction. This is stored in `y_pred`. 8. Compute the R[2] score: ``` r2 = model.score(X_val, y_val) print('R^2 score: {}'.format(r2)) ``` In this step, you compute `r2`, which is the R[2] score of the model. The R[2] score is computed using the `score()` method of the model. The next line causes the interpreter to print out the R[2] score. The output is similar to the following:  Caption: R2 score 9. You see that the R[2] score we achieved is `0.56238`, which is not close to 1. In the next step, we will be making comparisons. 10. Compare the predictions to the actual ground truth: ``` _ys = pd.DataFrame(dict(actuals=y_val.reshape(-1), \ predicted=y_pred.reshape(-1))) _ys.head() ``` The output looks similar to the following:  Exercise 6.03: Computing the MAE of a Model ------------------------------------------- The goal of this exercise is to find the score and loss of a model using the same dataset as *Exercise 6.02*, *Computing the R2 Score of a Linear Regression Model*. In this exercise, we will be calculating the MAE of a model. The following steps will help you with this exercise: 1. Open a new Jupyter notebook file. 2. Import the necessary libraries: ``` # Import libraries import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_absolute_error ``` In this step, you import the function called `mean_absolute_error` from `sklearn.metrics`. 3. Import the data: ``` # column headers _headers = ['CIC0', 'SM1', 'GATS1i', 'NdsCH', 'Ndssc', \ 'MLOGP', 'response'] # read in data df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/'\ 'qsar_fish_toxicity.csv', \ names=_headers, sep=';') ``` In the preceding code, you read in your data. This data is hosted online and contains some information about fish toxicity. The data is stored as a CSV but does not contain any headers. Also, the columns in this file are not separated by a comma, but rather by a semi-colon. The Python list called `_headers` contains the names of the column headers. 4. Split the data into `features` and `labels` and into training and evaluation sets: ``` # Let's split our data features = df.drop('response', axis=1).values labels = df[['response']].values X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.2, \ random_state=0) X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ random_state=0) ``` 5. Create a simple linear regression model and train it: ``` # create a simple Linear Regression model model = LinearRegression() # train the model model.fit(X_train, y_train) ``` In this step, you make use of your training data to train a model. In the first line, you create an instance of `LinearRegression`, which you call `model`. In the second line, you train the model using `X_train` and `y_train`. `X_train` contains the `features`, while `y_train` contains the `labels`. 6. Now predict the values of our validation dataset: ``` # let's use our model to predict on our validation dataset y_pred = model.predict(X_val) ``` 7. Compute the MAE: ``` # Let's compute our MEAN ABSOLUTE ERROR mae = mean_absolute_error(y_val, y_pred) print('MAE: {}'.format(mae)) ``` In this step, you compute the MAE of the model by using the `mean_absolute_error` function and passing in `y_val` and `y_pred`. `y_val` is the label that was provided with your training data, and `y_pred `is the prediction from the model. The preceding code should give you an MAE value of \~ 0.72434:  Figure 6.17 MAE score 8. Compute the R[2] score of the model: ``` # Let's get the R2 score r2 = model.score(X_val, y_val) print('R^2 score: {}'.format(r2)) ``` You should get an output similar to the following:  In this exercise, we have calculated the MAE, which is a significant parameter when it comes to evaluating models. You will now train a second model and compare its R[2] score and MAE to the first model to evaluate which is a better performing model. Exercise 6.04: Computing the Mean Absolute Error of a Second Model ------------------------------------------------------------------ In this exercise, we will be engineering new features and finding the score and loss of a new model. The following steps will help you with this exercise: 1. Open a new Jupyter notebook file. 2. Import the required libraries: ``` # Import libraries import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_absolute_error # pipeline from sklearn.pipeline import Pipeline # preprocessing from sklearn.preprocessing import MinMaxScaler from sklearn.preprocessing import StandardScaler from sklearn.preprocessing import PolynomialFeatures ``` 3. Read in the data from the dataset: ``` # column headers _headers = ['CIC0', 'SM1', 'GATS1i', 'NdsCH', 'Ndssc', \ 'MLOGP', 'response'] # read in data df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/'\ 'qsar_fish_toxicity.csv', \ names=_headers, sep=';') ``` 4. Split the data into training and evaluation sets: ``` # Let's split our data features = df.drop('response', axis=1).values labels = df[['response']].values X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.2, \ random_state=0) X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ random_state=0) ``` In this step, you begin by splitting the DataFrame called `df` into two. The first DataFrame is called `features` and contains all of the independent variables that you will use to make your predictions. The second is called `labels` and contains the values that you are trying to predict. In the third line, you split `features` and `labels` into four sets using `train_test_split`. `X_train` and `y_train` contain 80% of the data and are used for training your model. `X_eval` and `y_eval` contain the remaining 20%. In the fourth line, you split `X_eval` and `y_eval` into two additional sets. `X_val` and `y_val` contain 75% of the data because you did not specify a ratio or size. `X_test` and `y_test` contain the remaining 25%. 5. Create a pipeline: ``` # create a pipeline and engineer quadratic features steps = [('scaler', MinMaxScaler()),\ ('poly', PolynomialFeatures(2)),\ ('model', LinearRegression())] ``` 6. Create a pipeline: ``` # create a simple Linear Regression model with a pipeline model = Pipeline(steps) ``` 7. Train the model: ``` # train the model model.fit(X_train, y_train) ``` On the next line, you call the `fit` method and provide `X_train` and `y_train` as parameters. Because the model is a pipeline, three operations will happen. First, `X_train` will be scaled. Next, additional features will be engineered. Finally, training will happen using the `LinearRegression` model. The output from this step is similar to the following:  Caption: Training the model 8. Predict using the validation dataset: ``` # let's use our model to predict on our validation dataset y_pred = model.predict(X_val) ``` 9. Compute the MAE of the model: ``` # Let's compute our MEAN ABSOLUTE ERROR mae = mean_absolute_error(y_val, y_pred) print('MAE: {}'.format(mae)) ``` In the first line, you make use of `mean_absolute_error` to compute the mean absolute error. You supply `y_val` and `y_pred`, and the result is stored in the `mae` variable. In the following line, you print out `mae`:  Caption: MAE score The loss that you compute at this step is called a validation loss because you make use of the validation dataset. This is different from a training loss that is computed using the training dataset. This distinction is important to note as you study other documentation or books, which might refer to both. 10. Compute the R[2] score: ``` # Let's get the R2 score r2 = model.score(X_val, y_val) print('R^2 score: {}'.format(r2)) ``` In the final two lines, you compute the R[2] score and also display it, as shown in the following screenshot:  Exercise 6.05: Creating a Classification Model for Computing Evaluation Metrics ------------------------------------------------------------------------------- In this exercise, you will create a classification model that you will make use of later on for model assessment. You will make use of the cars dataset from the UCI Machine Learning Repository. You will use this dataset to classify cars as either acceptable or unacceptable based on the following categorical features: - `buying`: the purchase price of the car - `maint`: the maintenance cost of the car - `doors`: the number of doors on the car - `persons`: the carrying capacity of the vehicle - `lug_boot`: the size of the luggage boot - `safety`: the estimated safety of the car The following steps will help you achieve the task: 1. Open a new Jupyter notebook. 2. Import the libraries you will need: ``` # import libraries import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression ``` 3. Import your data: ``` # data doesn't have headers, so let's create headers _headers = ['buying', 'maint', 'doors', 'persons', \ 'lug_boot', 'safety', 'car'] # read in cars dataset df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/car.data', \ names=_headers, index_col=None) df.head() ``` You should get an output similar to the following:  Caption: Inspecting the DataFrame 4. Encode categorical variables as shown in the following code snippet: ``` # encode categorical variables _df = pd.get_dummies(df, columns=['buying', 'maint', 'doors',\ 'persons', 'lug_boot', \ 'safety']) _df.head() ``` The output should now resemble the following screenshot:  Caption: Encoding categorical variables 5. Split the data into training and validation sets: ``` # split data into training and evaluation datasets features = _df.drop('car', axis=1).values labels = _df['car'].values X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.3, \ random_state=0) X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ test_size=0.5, \ random_state=0) ``` 6. Train a logistic regression model: ``` # train a Logistic Regression model model = LogisticRegression() model.fit(X_train, y_train) ``` In this step, you create an instance of `LogisticRegression` and train the model on your training data by passing in `X_train` and `y_train` to the `fit` method. You should get an output that looks similar to the following:  Caption: Training a logistic regression model 7. Make a prediction: ``` # make predictions for the validation set y_pred = model.predict(X_val) ``` In this step, you make a prediction on the validation dataset, `X_val`, and store the result in `y_pred`. A look at the first 10 predictions (by executing `y_pred[0:9]`) should provide an output similar to the following:  Caption: Prediction for the validation set Exercise 6.06: Generating a Confusion Matrix for the Classification Model ------------------------------------------------------------------------- The goal of this exercise is to create a confusion matrix for the classification model you trained in *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*. Note You should continue this exercise in the same notebook as that used in *Exercise 6.05, Creating a Classification Model for Computing Evaluation Metrics.* If you wish to use a new notebook, make sure you copy and run the entire code from *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*, and then begin with the execution of the code of this exercise. The following steps will help you achieve the task: 1. Open a new Jupyter notebook file. 2. Import `confusion_matrix`: ``` from sklearn.metrics import confusion_matrix ``` In this step, you import `confusion_matrix` from `sklearn.metrics`. This function will let you generate a confusion matrix. 3. Generate a confusion matrix: ``` confusion_matrix(y_val, y_pred) ``` In this step, you generate a confusion matrix by supplying `y_val`, the actual classes, and `y_pred`, the predicted classes. The output should look similar to the following:  Exercise 6.07: Computing Precision for the Classification Model --------------------------------------------------------------- In this exercise, you will be computing the precision for the classification model you trained in *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*. The following steps will help you achieve the task: 1. Import the required libraries: ``` from sklearn.metrics import precision_score ``` In this step, you import `precision_score` from `sklearn.metrics`. 2. Next, compute the precision score as shown in the following code snippet: ``` precision_score(y_val, y_pred, average='macro') ``` In this step, you compute the precision score using `precision_score`. The output is a floating-point number between 0 and 1. It might look like this:  Exercise 6.08: Computing Recall for the Classification Model ------------------------------------------------------------ The goal of this exercise is to compute the recall for the classification model you trained in *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*. The following steps will help you accomplish the task: 1. Open a new Jupyter notebook file. 2. Now, import the required libraries: ``` from sklearn.metrics import recall_score ``` In this step, you import `recall_score` from `sklearn.metrics`. This is the function that you will make use of in the second step. 3. Compute the recall: ``` recall_score(y_val, y_pred, average='macro') ``` You should get an output that looks like the following:  Exercise 6.09: Computing the F1 Score for the Classification Model ------------------------------------------------------------------ In this exercise, you will compute the F1 score for the classification model you trained in *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*. Note You should continue this exercise in the same notebook as that used in *Exercise 6.05, Creating a Classification Model for Computing Evaluation Metrics.* If you wish to use a new notebook, make sure you copy and run the entire code from *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*, and then begin with the execution of the code of this exercise. The following steps will help you accomplish the task: 1. Open a new Jupyter notebook file. 2. Import the necessary modules: ``` from sklearn.metrics import f1_score ``` In this step, you import the `f1_score` method from `sklearn.metrics`. This score will let you compute evaluation metrics. 3. Compute the F1 score: ``` f1_score(y_val, y_pred, average='macro') ``` In this step, you compute the F1 score by passing in `y_val` and `y_pred`. You also specify `average='macro'` because this is not binary classification. You should get an output similar to the following:  Exercise 6.10: Computing Model Accuracy for the Classification Model -------------------------------------------------------------------- The goal of this exercise is to compute the accuracy score of the model trained in *Exercise 6.04*, *Computing the Mean Absolute Error of a Second Model*. The following steps will help you accomplish the task: 1. Continue from where the code for *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*, ends in your notebook. 2. Import `accuracy_score()`: ``` from sklearn.metrics import accuracy_score ``` In this step, you import `accuracy_score()`, which you will use to compute the model accuracy. 3. Compute the accuracy: ``` _accuracy = accuracy_score(y_val, y_pred) print(_accuracy) ``` In this step, you compute the model accuracy by passing in `y_val` and `y_pred` as parameters to `accuracy_score()`. The interpreter assigns the result to a variable called `c`. The `print()` method causes the interpreter to render the value of `_accuracy`. The result is similar to the following:  Exercise 6.11: Computing the Log Loss for the Classification Model ------------------------------------------------------------------ The goal of this exercise is to predict the log loss of the model trained in *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*. Note You should continue this exercise in the same notebook as that used in *Exercise 6.05, Creating a Classification Model for Computing Evaluation Metrics.* If you wish to use a new notebook, make sure you copy and run the entire code from *Exercise 6.05* and then begin with the execution of the code of this exercise. The following steps will help you accomplish the task: 1. Open your Jupyter notebook and continue from where *Exercise 6.05*, *Creating a Classification Model for Computing Evaluation Metrics*, stopped. 2. Import the required libraries: ``` from sklearn.metrics import log_loss ``` In this step, you import `log_loss()` from `sklearn.metrics`. 3. Compute the log loss: ``` _loss = log_loss(y_val, model.predict_proba(X_val)) print(_loss) ``` Exercise 6.12: Computing and Plotting ROC Curve for a Binary Classification Problem ----------------------------------------------------------------------------------- The goal of this exercise is to plot the ROC curve for a binary classification problem. The data for this problem is used to predict whether or not a mother will require a caesarian section to give birth. From the UCI Machine Learning Repository, the abstract for this dataset follows: \"This dataset contains information about caesarian section results of 80 pregnant women with the most important characteristics of delivery problems in the medical field.\" The attributes of interest are age, delivery number, delivery time, blood pressure, and heart status. The following steps will help you accomplish this task: 1. Open a Jupyter notebook file. 2. Import the required libraries: ``` # import libraries import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import roc_curve from sklearn.metrics import auc ``` In this step, you import `pandas`, which you will use to read in data. You also import `train_test_split` for creating training and validation datasets, and `LogisticRegression` for creating a model. 3. Read in the data: ``` # data doesn't have headers, so let's create headers _headers = ['Age', 'Delivery_Nbr', 'Delivery_Time', \ 'Blood_Pressure', 'Heart_Problem', 'Caesarian'] # read in cars dataset df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/caesarian.csv.arff',\ names=_headers, index_col=None, skiprows=15) df.head() # target column is 'Caesarian' ```  The `head()` method will print out the top five rows and should look similar to the following:  Caption: The top five rows of the DataFrame 4. Split the data: ``` # target column is 'Caesarian' features = df.drop(['Caesarian'], axis=1).values labels = df[['Caesarian']].values # split 80% for training and 20% into an evaluation set X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.2, \ random_state=0) """ further split the evaluation set into validation and test sets of 10% each """ X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ test_size=0.5, \ random_state=0) ``` In this step, you begin by creating two `numpy` arrays, which you call `features` and `labels`. You then split these arrays into a `training` and an `evaluation` dataset. You further split the `evaluation` dataset into `validation` and `test` datasets. 5. Now, train and fit a logistic regression model: ``` model = LogisticRegression() model.fit(X_train, y_train) ``` In this step, you begin by creating an instance of a logistic regression model. You then proceed to train or fit the model on the training dataset. The output should be similar to the following:  Caption: Training a logistic regression model 6. Predict the probabilities, as shown in the following code snippet: ``` y_proba = model.predict_proba(X_val) ``` In this step, the model predicts the probabilities for each entry in the validation dataset. It stores the results in `y_proba`. 7. Compute the true positive rate, the false positive rate, and the thresholds: ``` _false_positive, _true_positive, _thresholds = roc_curve\ (y_val, \ y_proba[:, 0]) ``` In this step, you make a call to `roc_curve()` and specify the ground truth and the first column of the predicted probabilities. The result is a tuple of false positive rate, true positive rate, and thresholds. 8. Explore the false positive rates: ``` print(_false_positive) ``` In this step, you instruct the interpreter to print out the false positive rate. The output should be similar to the following:  Caption: False positive rates Note The false positive rates can vary, depending on the data. 9. Explore the true positive rates: ``` print(_true_positive) ``` In this step, you instruct the interpreter to print out the true positive rates. This should be similar to the following:  Caption: True positive rates 10. Explore the thresholds: ``` print(_thresholds) ``` In this step, you instruct the interpreter to display the thresholds. The output should be similar to the following:  Caption: Thresholds 11. Now, plot the ROC curve: ``` # Plot the RoC import matplotlib.pyplot as plt %matplotlib inline plt.plot(_false_positive, _true_positive, lw=2, \ label='Receiver Operating Characteristic') plt.xlim(0.0, 1.2) plt.ylim(0.0, 1.2) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('Receiver Operating Characteristic') plt.show() ``` The output should look similar to the following:  Caption: ROC curve Exercise 6.13: Computing the ROC AUC for the Caesarian Dataset -------------------------------------------------------------- The goal of this exercise is to compute the ROC AUC for the binary classification model that you trained in *Exercise 6.12*, *Computing and Plotting ROC Curve for a Binary Classification Problem*. Note You should continue this exercise in the same notebook as that used in *Exercise 6.12, Computing and Plotting ROC Curve for a Binary Classification Problem.* If you wish to use a new notebook, make sure you copy and run the entire code from *Exercise 6.12* and then begin with the execution of the code of this exercise. The following steps will help you accomplish the task: 1. Open a Jupyter notebook to the code for *Exercise 6.12*, *Computing and Plotting ROC Curve for a Binary Classification Problem,* and continue writing your code. 2. Predict the probabilities: ``` y_proba = model.predict_proba(X_val) ``` In this step, you compute the probabilities of the classes in the validation dataset. You store the result in `y_proba`. 3. Compute the ROC AUC: ``` from sklearn.metrics import roc_auc_score _auc = roc_auc_score(y_val, y_proba[:, 0]) print(_auc) ``` In this step, you compute the ROC AUC and store the result in `_auc`. You then proceed to print this value out. The result should look similar to the following:  Caption: Computing the ROC AUC Note The AUC can be different, depending on the data. Saving and Loading Models ========================= You will eventually need to transfer some of the models you have trained to a different computer so they can be put into production. There are various utilities for doing this, but the one we will discuss is called `joblib`. `joblib` supports saving and loading models, and it saves the models in a format that is supported by other machine learning architectures, such as `ONNX`. `joblib` is found in the `sklearn.externals` module. Exercise 6.14: Saving and Loading a Model ----------------------------------------- In this exercise, you will train a simple model and use it for prediction. You will then proceed to save the model and then load it back in. You will use the loaded model for a second prediction, and then compare the predictions from the first model to those from the second model. You will make use of the car dataset for this exercise. The following steps will guide you toward the goal: 1. Open a Jupyter notebook. 2. Import the required libraries: ``` import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression ``` 3. Read in the data: ``` _headers = ['CIC0', 'SM1', 'GATS1i', 'NdsCH', 'Ndssc', \ 'MLOGP', 'response'] # read in data df = pd.read_csv('https://raw.githubusercontent.com/'\ 'fenago/data-science/'\ 'master/Lab06/Dataset/'\ 'qsar_fish_toxicity.csv', \ names=_headers, sep=';') ``` 4. Inspect the data: ``` df.head() ``` The output should be similar to the following:  Caption: Inspecting the first five rows of the DataFrame 5. Split the data into `features` and `labels`, and into training and validation sets: ``` features = df.drop('response', axis=1).values labels = df[['response']].values X_train, X_eval, y_train, y_eval = train_test_split\ (features, labels, \ test_size=0.2, \ random_state=0) X_val, X_test, y_val, y_test = train_test_split(X_eval, y_eval,\ random_state=0) ``` 6. Create a linear regression model: ``` model = LinearRegression() print(model) ``` The output will be as follows:  Caption: Training a linear regression model 7. Fit the training data to the model: ``` model.fit(X_train, y_train) ``` 8. Use the model for prediction: ``` y_pred = model.predict(X_val) ``` 9. Import `joblib`: ``` from sklearn.externals import joblib ``` 10. Save the model: ``` joblib.dump(model, './model.joblib') ``` The output should be similar to the following:  Caption: Saving the model 11. Load it as a new model: ``` m2 = joblib.load('./model.joblib') ``` 12. Use the new model for predictions: ``` m2_preds = m2.predict(X_val) ``` 13. Compare the predictions: ``` ys = pd.DataFrame(dict(predicted=y_pred.reshape(-1), \ m2=m2_preds.reshape(-1))) ys.head() ``` The output should be similar to the following:  Caption: Comparing predictions Activity 6.01: Train Three Different Models and Use Evaluation Metrics to Pick the Best Performing Model -------------------------------------------------------------------------------------------------------- You work as a data scientist at a bank. The bank would like to implement a model that predicts the likelihood of a customer purchasing a term deposit. The bank provides you with a dataset, which is the same as the one in *Lab 3*, *Binary Classification*. You have previously learned how to train a logistic regression model for binary classification. You have also heard about other non-parametric modeling techniques and would like to try out a decision tree as well as a random forest to see how well they perform against the logistic regression models you have been training. In this activity, you will train a logistic regression model and compute a classification report. You will then proceed to train a decision tree classifier and compute a classification report. You will compare the models using the classification reports. Finally, you will train a random forest classifier and generate the classification report. You will then compare the logistic regression model with the random forest using the classification reports to determine which model you should put into production. The steps to accomplish this task are: 1. Open a Jupyter notebook. 2. Load the necessary libraries. 3. Read in the data. 4. Explore the data. 5. Convert categorical variables using `pandas.get_dummies()`. 6. Prepare the `X` and `y` variables. 7. Split the data into training and evaluation sets. 8. Create an instance of `LogisticRegression`. 9. Fit the training data to the `LogisticRegression` model. 10. Use the evaluation set to make a prediction. 11. Use the prediction from the `LogisticRegression` model to compute the classification report. 12. Create an instance of `DecisionTreeClassifier`: ``` dt_model = DecisionTreeClassifier(max_depth= 6) ``` 13. Fit the training data to the `DecisionTreeClassifier` model: ``` dt_model.fit(train_X, train_y) ``` 14. Using the `DecisionTreeClassifier` model, make a prediction on the evaluation dataset: ``` dt_preds = dt_model.predict(val_X) ``` 15. Use the prediction from the `DecisionTreeClassifier` model to compute the classification report: ``` dt_report = classification_report(val_y, dt_preds) print(dt_report) ``` Note We will be studying decision trees in detail in *Lab 7, The Generalization of Machine Learning Models*. 16. Compare the classification report from the linear regression model and the classification report from the decision tree classifier to determine which is the better model. 17. Create an instance of `RandomForestClassifier`. 18. Fit the training data to the `RandomForestClassifier` model. 19. Using the `RandomForestClassifier` model, make a prediction on the evaluation dataset. 20. Using the prediction from the random forest classifier, compute the classification report. 21. Compare the classification report from the linear regression model with the classification report from the random forest classifier to decide which model to keep or improve upon. 22. Compare the R[2] scores of all three models. The output should be similar to the following:  Summary ======= In this lab we observed that some of the evaluation metrics for classification models require a binary classification model. We saw that when we worked with more than two classes, we were required to use the one-versus-all approach. The one-versus-all approach builds one model for each class and tries to predict the probability that the input belongs to a specific class. We saw that once this was done, we then predicted that the input belongs to the class where the model has the highest prediction probability. We also split our evaluation dataset into two, it\'s because `X_test` and `y_test` are used once for a final evaluation of the model\'s performance. You can make use of them before putting your model into production to see how the model would perform in a production environment.