lee/mlessentials

Fork 0

mirror of https://github.com/fenago/data-science.git synced 2026-05-04 16:41:05 +00:00

Files

T

fenago 39139f2b0e added

2021-02-09 03:17:43 +05:00

28 KiB

Raw Permalink Blame History

Lab 15. Ensemble Learning

Overview

By the end of this lab, you will be able to describe ensemble learning and apply different ensemble learning techniques to your dataset. You will also be able to fit a dataset on a model and analyze the results after ensemble learning.

In this lab, we will be using the credit card application dataset, where we will try to predict whether a credit card application will be approved.

Exercise 15.01: Loading, Exploring, and Cleaning the Data

In this exercise, we will download the credit card dataset, load it into our Jupyter notebook, and perform a few basic explorations. In addition, we will also clean the dataset to remove unwanted characters.

The following steps will help you to complete this exercise:

Open a new Jupyter notebook file.
Now, import pandas into your Jupyter notebook:
```
import pandas as pd
```

Next, set the path of the GitHub repository where crx.data is uploaded, as mentioned in the following code snippet:

#Load data from the GitHub repository
filename = 'https://raw.githubusercontent.com'\
           '/fenago/data-science'\
           '/master/Lab15/Dataset/crx.data'

Read the file using the pd.read_csv() function from the pandas data frame:
```
credData = pd.read_csv(filename,sep= ",",\
                       header = None,\
                       na_values =  "?")
credData.head()
```
After reading the file, print the data frame using the .head() function. You should get the following output:

Caption: Loading data into the Jupyter notebook

Change the classes to 1 and 0.

If you notice in the dataset, the classes represented in column 15 are special characters: + for approved and - for not approved. You need to change this to numerical values of 1 for approved and 0 for not approved, as shown in the following code snippet:
```
# Changing the Classes to 1 & 0
credData.loc[credData[15] == '+' , 15] = 1
credData.loc[credData[15] == '-' , 15] = 0
credData.head()
```
You should get the following output:

Caption: Data frame after replacing special characters

Find the number of null values in the dataset.

We'll now find the number of null values in each of the features using the .isnull() function. The .sum() function sums up all such null values across each of the columns in the dataset.

This is represented in the following code snippet:
```
# Finding number of null values in the data set
credData.isnull().sum()
```
You should get the following output:

Caption: Summarizing null values in the dataset

Now, print the shape and data types of each column:

# Printing Shape and data types
print('Shape of raw data set',credData.shape)
print('Data types of data set',credData.dtypes)

You should get the following output:

Caption: Shape and data types of each column

Remove the rows with na values.

In order to clean the dataset, let's remove all the rows with na values using the .dropna() function with the following code snippet:
```
# Dropping all the rows with na values
newcred = credData.dropna(axis = 0)
newcred.shape
```
You should get the following output:
```
(653, 16)
```
Verify that no null values exist:
```
# Verifying no null values exist
newcred.isnull().sum()
```
You should get the following output:

Caption: Verifying that no null values are present

Next, make dummy values from the categorical variables.

As you can see from the data types, there are many variables with categorical values. These have to be converted to dummy values using the pd.get_dummies() function. This is done using the following code snippet:
```
"""
Separating the categorical variables to 
make dummy variables
"""
credCat = pd.get_dummies(newcred[[0,3,4,5,6,8,9,11,12]])
```
Separate the numerical variables.

We will also be separating the numerical variables from the original dataset to concatenate them with the dummy variables. This step is done as follows:
```
# Separating the numerical variables
credNum = newcred[[1,2,7,10,13,14]]
```
Note

You can view these new DataFrames by running the commands credCat and credNum.
Create the X and y variables. The dummy variables and the numerical variables will now be concatenated to form the X variable. The y variable will be created separately by taking the labels of the dataset. Let's see these steps in action in the following code snippet:
```
"""
Making the X variable which is a concatenation 
of categorical and numerical data
"""
X = pd.concat([credCat,credNum],axis = 1)
print(X.shape)
# Separating the label as y variable
y = pd.Series(newcred[15], dtype="int")
print(y.shape)
```
You should get the following output:
```
(653, 46)
(653,)
```

Normalize the dataset using the MinMaxScaler() function:

# Normalizing the data sets
# Import library function
from sklearn import preprocessing
# Creating the scaling function
minmaxScaler = preprocessing.MinMaxScaler()
# Transforming with the scaler function
X_tran = pd.DataFrame(minmaxScaler.fit_transform(X))

Split the dataset into training and test sets.

As the final step of data preparation, we will now split the dataset into training and test sets using the train_test_split() function:

from sklearn.model_selection import train_test_split
# Splitting the data into train and test sets
X_train, X_test, y_train, y_test = train_test_split\
                                   (X_tran, y, test_size=0.3,\
                                    random_state=123)

We now have the required dataset ready for further actions. As always, let's start off by fitting a benchmark model using logistic regression on the cleaned dataset. This will be achieved in the next activity.

Activity 15.01: Fitting a Logistic Regression Model on Credit Card Data

You have just cleaned the dataset that you received to predict the creditworthiness of your customers. Before applying ensemble learning methods, you want to fit a benchmark model on the dataset.

Perform the following steps to complete this activity:

Open a new Jupyter notebook.
Implement all the appropriate steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, until you have split the dataset into training and test sets.
Fit a logistic regression model on the training set.
Get the predictions on the test set.
Print the confusion matrix and classification report for the benchmark model.

You should get an output similar to the following after fitting the logistic regression model on the dataset:

Caption: Expected output after fitting the logistic regression model

Exercise 15.02: Ensemble Model Using the Averaging Technique

In this exercise, we will implement an ensemble model using the averaging technique. The base models that we will use for this exercise are the logistic regression model, which we used as our benchmark model, and the KNN and random forest models, which were introduced in Lab 4, Multiclass Classification with RandomForest, and Lab 8, Hyperparameter Tuning:

Open a new Jupyter notebook.
Execute all the appropriate steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, to split the dataset into train and test sets.

Let's define the three base models. Import the selected classifiers, which we will use as base models:

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
model1 = LogisticRegression(random_state=123)
model2 = KNeighborsClassifier(n_neighbors=5)
model3 = RandomForestClassifier(n_estimators=500)

Fit all three models on the training set:

# Fitting all three models on the training data
model1.fit(X_train,y_train)
model2.fit(X_train,y_train)
model3.fit(X_train,y_train)

We will now predict the probabilities of each model using the predict_proba() function:

"""
Predicting probabilities of each model 
on the test set
"""
pred1=model1.predict_proba(X_test)
pred2=model2.predict_proba(X_test)
pred3=model3.predict_proba(X_test)

Average the predictions generated from all of the three models:

"""
Calculating the ensemble prediction by 
averaging three base model predictions
"""
ensemblepred=(pred1+pred2+pred3)/3

Display the first four rows of the ensemble prediction array:
```
# Displaying first 4 rows of the ensemble predictions
ensemblepred[0:4,:]
```
You should get an output similar to the following:

Caption: First four rows of ensemble predictions

As you can see from the preceding output, we have two probabilities
for each example corresponding to each class.

Print the order of each class from the prediction output. As you can see from Step 6, the prediction output has two columns corresponding to each class. In order to find the order of the class prediction, we use a method called .classes_. This is implemented in the following code snippet:
```
# Printing the order of classes for each model
print(model1.classes_)
print(model2.classes_)
print(model3.classes_)
```
You should get the following output:

Caption: Order of classes

We now have to get the final predictions for each example from the output probabilities. The final prediction will be the class with the highest probability. To get the class with the highest probability, we use the numpy function, .argmax(). This is executed as follows:
```
import numpy as np
pred = np.argmax(ensemblepred,axis = 1)
pred
```
From the preceding code, axis = 1 means that we need to find the index of the maximum value across the columns.

You should get the following output:

Caption: Array output

Generate the confusion matrix for the predictions:

# Generating confusion matrix
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)

You should get an output similar to the following:

Caption: Confusion matrix

Generate a classification report:

# Generating classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get an output similar to the following:

Exercise 15.03: Ensemble Model Using the Weighted Averaging Technique

In this exercise, we will implement an ensemble model using the weighted averaging technique. We will use the same base models, logistic regression, KNN, and random forest, which were used in Exercise 15.02, Ensemble Model Using the Averaging Technique:

Open a new Jupyter notebook.
Execute all the steps from Exercise 15.02, Ensemble Model Using the Averaging Technique, up until predicting the probabilities of the three models.
Take the weighted average of the predictions. In the weighted averaging method, weights are assigned arbitrarily based on our judgment of each of the predictions. This is done as follows:
```
"""
Calculating the ensemble prediction by applying 
weights for each prediction
"""
ensemblepred=(pred1 *0.60 + pred2 * 0.20 + pred3 * 0.20)
```
Please note that the weights are assigned in such a way that the sum of all weights becomes 1 (0.6 + 0.2 + 0.2 = 1).
Display the first four rows of the ensemble prediction array:
```
# Displaying first 4 rows of the ensemble predictions
ensemblepred[0:4,:]
```
You should get an output similar to the following:

Caption: Array output for ensemble prediction

As you can see from the output, we have two probabilities for each
example corresponding to each class.

Print the order of each class from the prediction output:

# Printing the order of classes for each model
print(model1.classes_)
print(model2.classes_)
print(model3.classes_)

You should get the following output:

Caption: Order of class from prediction output

Calculate the final predictions from the probabilities.

We now have to get the final predictions for each example from the output probabilities using the np.argmax() function:
```
import numpy as np
pred = np.argmax(ensemblepred,axis = 1)
pred
```
You should get an output similar to the following:

Caption: Array for final predictions

Generate the confusion matrix for the predictions:

# Generating confusion matrix
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)

You should get an output similar to the following:

Caption: Confusion matrix

Generating a classification report:

# Generating classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get an output similar to the following:

Iteration 2 with Different Weights

Take the weighted average of the predictions.

In this iteration, we increase the weight of logistic regression prediction from 0.6 to 0.7 and decrease the other two from 0.2 to 0.15:
```
"""
Calculating the ensemble prediction by applying 
weights for each prediction
"""
ensemblepred=(pred1 *0.70+pred2 * 0.15+pred3 * 0.15)
```
Calculate the final predictions from the probabilities.

We now have to get the final predictions for each example from the output probabilities using the np.argmax() function:
```
# Generating predictions from probabilities
import numpy as np
pred = np.argmax(ensemblepred,axis = 1)
```

Generate the confusion matrix for the predictions:

# Generating confusion matrix
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)

You should get an output similar to the following:

Caption: Confusion matrix

Generate a classification report:

# Generating classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get an output similar to the following:

Exercise 15.04: Ensemble Model Using Max Voting

In this exercise, we will implement an ensemble model using the max voting technique. The individual learners we will select are similar to the ones that we chose in the previous exercises, that is, logistic regression, KNN, and random forest:

Open a new Jupyter notebook.
Execute all the steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, up until the splitting of the dataset into train and test sets.

We will now import the selected classifiers, which we will use as the individual learners:

"""
Defining the voting classifier and three 
individual learners
"""
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
# Defining the models
model1 = LogisticRegression(random_state=123)
model2 = KNeighborsClassifier(n_neighbors=5)
model3 = RandomForestClassifier(n_estimators=500)

Having defined the individual learners, we can now construct the ensemble model using the VotingClassifier() function. This is implemented by the following code snippet:
```
# Defining the ensemble model using VotingClassifier
model = VotingClassifier(estimators=[('lr', model1),\
                        ('knn', model2),('rf', model3)],\
                         voting= 'hard')
```
As you can see from the code snippet, the individual learners are given as input using the estimators argument. Estimators take each of the defined individual learners along with the string value to denote which model it is. For example, lr denotes logistic regression. Also, note that the voting is "hard, " which means that the output will be class labels and not probabilities.

Fit the training set on the ensemble model:

# Fitting the model on the training set
model.fit(X_train,y_train)

Print the accuracy scores after training:

"""
Predicting accuracy on the test set using 
.score() function
"""
model.score(X_test,y_test)

You should get an output similar to the following:

0.9081632653061225

Generate the predictions from the ensemble model on the test set:

# Generating the predictions on the test set
preds = model.predict(X_test)

Generate the confusion matrix for the predictions:

# Printing the confusion matrix
from sklearn.metrics import confusion_matrix
# Confusion matrix for the test set
print(confusion_matrix(y_test, preds))

You should get an output similar to the following:

Caption: Confusion matrix

Generate the classification report:

# Printing the classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, preds))

You should get an output similar to the following:

Exercise 15.05: Ensemble Learning Using Bagging

In this exercise, we will implement an ensemble model using bagging. The individual learner we will select will be random forest:

Open a new Jupyter notebook.
Execute all the steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, up until the splitting of the dataset into train and test sets.

Define the base learner, which will be a random forest classifier:

# Defining the base learner
from sklearn.ensemble import RandomForestClassifier
bl1 = RandomForestClassifier(random_state=123)

Having defined the individual learner, we can now construct the ensemble model using the BaggingClassifier() function. This is implemented by the following code snippet:
```
# Creating the bagging meta learner
from sklearn.ensemble import BaggingClassifier
baggingLearner = \
BaggingClassifier(base_estimator=bl1, n_estimators=10, \
                  max_samples=0.8, max_features=0.7)
```
The arguments that we have given are arbitrary values. The optimal values have to be identified using experimentation.

Fit the training set on the ensemble model:

# Fitting the model using the meta learner
model = baggingLearner.fit(X_train, y_train)

Generate the predictions from the ensemble model on the test set:

# Predicting on the test set using the model
pred = model.predict(X_test)

Generate a confusion matrix for the predictions:

# Printing the confusion matrix
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, pred))

You should get an output similar to the following:

Caption: Confusion matrix

Generate the classification report:

# Printing the classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get an output similar to the following:

Exercise 15.06: Ensemble Learning Using Boosting

In this exercise, we will implement an ensemble model using boosting. The individual learner we will select will be the logistic regression model. The steps for implementing this algorithm are very similar to the bagging algorithm:

Open a new Jupyter notebook file.
Execute all of the steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, up until the splitting of the dataset into train and test sets.

Define the base learner, which will be a logistic regression classifier:

# Defining the base learner
from sklearn.linear_model import LogisticRegression
bl1 = LogisticRegression(random_state=123)

Having defined the individual learner, we can now construct the ensemble model using the AdaBoostClassifier() function. This is implemented by the following code snippet:
```
# Define the boosting meta learner
from sklearn.ensemble import AdaBoostClassifier
boosting = AdaBoostClassifier(base_estimator=bl1, \
                              n_estimators=200)
```
The arguments that we have given are arbitrary values. The optimal values have to be identified using experimentation.

Fit the training set on the ensemble model:

# Fitting the model on the training set
model = boosting.fit(X_train, y_train)

Generate the predictions from the ensemble model on the test set:

# Getting the predictions from the boosting model
pred = model.predict(X_test)

Generate a confusion matrix for the predictions:

# Printing the confusion matrix
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, pred))

You should get a similar output to the following:

Caption: Confusion matrix

Generate the classification report:

# Printing the classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get an output similar to the following:

Exercise 15.07: Ensemble Learning Using Stacking

In this exercise, we will implement an ensemble model using stacking. The individual learners we will use are KNN and random forest. Our meta learner will be logistic regression:

Open a new Jupyter notebook.
Execute all of the steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, up until the splitting of the dataset into train and test sets.

Import the base learners and the meta learner. In this implementation, we will be using two base learners (KNN and random forest). The meta learner will be logistic regression:

# Importing the meta learner and base learners
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
bl1 = KNeighborsClassifier(n_neighbors=5)
bl2 = RandomForestClassifier(random_state=123)
ml = LogisticRegression(random_state=123)

Once the base learners and meta learner are defined, we will proceed to create the stacking classifier:

# Creating the stacking classifier
from mlxtend.classifier import StackingClassifier
stackclf = StackingClassifier(classifiers=[bl1, bl2],\
                              meta_classifier=ml)

The arguments that we have been given are the two base learners and the meta learner.

Fit the training set on the ensemble model:

# Fitting the model on the training set
model = stackclf.fit(X_train, y_train)

Generate the predictions from the ensemble model on the test set:

# Generating predictions on test set
pred = model.predict(X_test)

Generate the classification report:

# Printing the classification report
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get a similar output to the following:

Caption: Classification report

Generate a confusion matrix for the predictions:

# Printing the confusion matrix
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, pred))

You should get a similar output to the following:

Caption: Confusion matrix

Activity 15.02: Comparison of Advanced Ensemble Techniques

Scenario: You have tried the benchmark model on the credit card dataset and have got some benchmark metrics. Having learned some advanced ensemble techniques, you want to determine which technique to use for the credit card approval dataset.

In this activity, you will use all three advanced techniques and compare the results before selecting your final technique.

The steps are as follows:

Open a new Jupyter notebook.
Implement all steps from Exercise 15.01, Loading, Exploring, and Cleaning the Data, up until the splitting of the dataset into train and test sets.
Implement the bagging technique with the base learner as the logistic regression model. In the bagging classifier, define n_estimators = 15, max_samples = 0.7, and max_features = 0.8. Fit the model on the training set, generate the predictions, and print the confusion matrix and the classification report.
Implement boosting with random forest as the base learner. In the AdaBoostClassifier, define n_estimators = 300. Fit the model on the training set, generate the predictions, and print the confusion matrix and classification report.
Implement the stacking technique. Make the KNN and logistic regression models base learners and random forest a meta learner. Fit the model on the training set, generate the predictions, and print the confusion matrix and classification report.
Compare the results across all three techniques and select the best technique.
Output: You should get an output similar to the following for all three methods. Please note you will not get exact values as output due to variability in the prediction process.

The output for bagging would be as follows:

Caption: Output for bagging

The output for boosting would be as follows:

Caption: Output for boosting

The output for stacking would be as follows:

Summary

In this lab, we learned about various techniques of ensemble learning. Let's summarize our learning in this lab.

At the beginning of the lab, we were introduced to the concepts of variance and bias and we learned that ensemble learning is a technique that aims to combine individual models to create a superior model, thereby reducing variance and bias and improving performance. To further explore different techniques of ensemble learning, we downloaded the credit card approval dataset. We also fitted a benchmark model using logistic regression.

28 KiB Raw Permalink Blame History

Lab 15. Ensemble Learning

Exercise 15.01: Loading, Exploring, and Cleaning the Data

Activity 15.01: Fitting a Logistic Regression Model on Credit Card Data

Exercise 15.02: Ensemble Model Using the Averaging Technique

Exercise 15.03: Ensemble Model Using the Weighted Averaging Technique

Iteration 2 with Different Weights

Exercise 15.04: Ensemble Model Using Max Voting

Exercise 15.05: Ensemble Learning Using Bagging

Exercise 15.06: Ensemble Learning Using Boosting

Exercise 15.07: Ensemble Learning Using Stacking

Activity 15.02: Comparison of Advanced Ensemble Techniques

Summary

28 KiB

Raw Permalink Blame History