lee/mlessentials

Fork 0

mirror of https://github.com/fenago/data-science.git synced 2026-05-06 01:22:41 +00:00

Files

T

fenago 39139f2b0e added

2021-02-09 03:17:43 +05:00

29 KiB

Raw Permalink Blame History

Lab 13. Imbalanced Datasets

In this lab, you will be dealing with imbalanced datasets, which are very prevalent in real-life scenarios. You will be using techniques such as SMOTE, MSMOTE, and random undersampling to address imbalanced datasets.

Exercise 13.01: Benchmarking the Logistic Regression Model on the Dataset

In this exercise, we will be analyzing the problem of predicting whether a customer will buy a term deposit. For this, you will be fitting a logistic regression model, as you did in Lab 3, Binary Classification, and you will look closely at the metrics:

Open a new notebook in Jupyter.

Next, import pandas and load the data from the GitHub repository:

import pandas as pd
filename = 'https://raw.githubusercontent.com/fenago'\
           '/data-science/master/'\
           'Lab13/Dataset/bank-full.csv'

Now, load the data using pandas

#Loading the data using pandas
bankData = pd.read_csv(filename,sep=";")
bankData.head()

Your output would be as follows:

Caption: The first 5 rows of bankData

Now, to break the dataset down further, let\'s perform some
feature-engineering steps.

Normalize the numerical features (age, balance, and duration) through scaling, which was covered in Lab 3, Binary Classification. Enter the following code:
```
from sklearn.preprocessing import RobustScaler
rob_scaler = RobustScaler()
```

After scaling the numerical data, we convert each of the columns to a scaled version, as in the following code snippet:

# Converting each of the columns to scaled version
bankData['ageScaled'] = rob_scaler.fit_transform\
                        (bankData['age'].values.reshape(-1,1))
bankData['balScaled'] = rob_scaler.fit_transform\
                        (bankData['balance']\
                         .values.reshape(-1,1))
bankData['durScaled'] = rob_scaler.fit_transform\
                        (bankData['duration']\
                         .values.reshape(-1,1))

Now, drop the original features after we introduce the scaled features using the .drop() function:

# Dropping the original columns
bankData.drop(['age','balance','duration'], \
              axis=1, inplace=True)

Display the first five columns using the .head() function:
```
bankData.head()
```
The output is as follows:

Caption: bankData with scaled features

Convert all the categorical variables to dummy variables using the .get_dummies() function:

bankCat = pd.get_dummies(bankData[['job','marital','education',\
                                   'default','housing','loan',\
                                   'contact','month',\
                                   'poutcome']])

Separate the numerical data and observe the shape:

bankNum = bankData[['ageScaled','balScaled','day',\
                    'durScaled','campaign','pdays','previous']]
bankNum.shape

The output would be as follows:

(45211, 7)

Create the independent variables, X, and dependent variables, Y, from the combined dataset for modeling, as in the following code snippet:

# Merging with the original data frame
# Preparing the X variables
X = pd.concat([bankCat, bankNum], axis=1)
print(X.shape)
# Preparing the Y variable
Y = bankData['y']
print(Y.shape)
X.head()

The output is as follows:

Now, import the necessary functions of train_test_split() and LogisticRegression from sklearn:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

Split the data into train and test sets at a 70:30 ratio using the test_size = 0.3 variable in the splitting function. We also set random_state for the reproducibility of the code:
```
X_train, X_test, y_train, y_test = train_test_split\
                                   (X, Y, test_size=0.3, \
                                    random_state=123)
```

Now, fit the model using .fit on the training data:

# Defining the LogisticRegression function
bankModel = LogisticRegression()
bankModel.fit(X_train, y_train)

Your output should be as follows:

Next, find the prediction on the test set and print the accuracy scores:

pred = bankModel.predict(X_test)
print('Accuracy of Logistic regression model prediction on '\
      'test set: {:.2f}'\
      .format(bankModel.score(X_test, y_test)))

You should get the following output:

Accuracy of Logistic regression model prediction on test set: 0.90

Now, use both the confusion_matrix() and classification_report() functions to generate the metrics for further analysis, which we will cover in the Analysis of the Result section:

# Confusion Matrix for the model
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get the following output:

To understand more about the reasons behind the skewed results, we will analyze these metrics in detail in the following section.

Analysis of the Result

To analyze the results obtained in the previous section, let's expand the confusion matrix in the form:

We enter the values 11707, 291, 1060, and 506 from the output we got from the previous exercise. We then place these values as shown in the diagram. We will represent the propensity to take a term deposit (No) as the positive class and the other as the negative class. So, from the confusion matrix, we can calculate the accuracy measures, which were covered in Lab 3, Binary Classification. The accuracy of the model is given by:

Caption: Accuracy of a model

In our case, it will be (11707 + 506) / (11707 + 1060 + 291 + 506), or 90%.

The precision value of any class is given by:

The recall value for any class can be represented as follows:

The following code will generate the percentages of the classes in the training data:

print('Percentage of negative class :',\
      (y_train[y_train=='yes'].value_counts()\
       /len(y_train) ) * 100)
print('Percentage of positive class :',\
      (y_train[y_train=='no'].value_counts()\
       /len(y_train) ) * 100)

You should get the following output:

Percentage of negative class: yes    11.764148
Name: y, dtype: float64
Percentage of positive class: no    88.235852
Name: y, dtype: float64

From this, we can see that the majority of the training set (88%) is made up of the positive class. This imbalance is one of the major reasons behind the poor metrics that we have had with the logistic regression classifier we have selected.

Now, let's look at the challenges of imbalanced datasets.

Challenges of Imbalanced Datasets

As seen from the classifier example, one of the biggest challenges with imbalanced datasets is the bias toward the majority class, which ended up being 88% in the previous example. This will result in suboptimal results. However, what makes such cases even more challenging is the deceptive nature of results if the right metric is not used.

Let's take, for example, a dataset where the negative class is around 99% and the positive class is 1% (as in a use case where a rare disease has to be detected, for instance).

Have a look at the following code snippet:

Data set Size: 10,000 examples
Negative class : 9910
Positive Class : 90

Suppose we had a poor classifier that was capable of only predicting the negative class; we would get the following confusion matrix:

Caption: Confusion matrix of the poor classifier

From the confusion matrix, let's calculate the accuracy measures. Have a look at the following code snippet:

# Classifier biased to only negative class
Accuracy = (TP + TN ) / ( TP + FP + FN + TN)
 = (0 + 9900) / ( 0 + 0 + 90 + 9900) = 9900/10000
 = 99%

It is important to identify cases with imbalanced datasets and equally important to pick the right metric for analyzing such datasets. The right metric in this example would have been to look at the recall values for both the classes:

Recall Positive class  = TP / ( TP + FN ) = 0 / ( 0 + 90)
 = 0
Recall Negative Class = TN / ( TN + FP) = 9900 / ( 9900 + 0)
= 100%

From the recall values, we could have identified the bias of the classifier toward the majority class, prompting us to look at strategies for mitigating such biases, which is the next topic we will focus on.

Strategies for Dealing with Imbalanced Datasets

Now that we have identified the challenges of imbalanced datasets, let's look at strategies for combatting imbalanced datasets:

Exercise 13.02: Implementing Random Undersampling and Classification on Our Banking Dataset to Find the Optimal Result

In this exercise, you will undersample the majority class (propensity 'No') and then make the dataset balanced. On the new balanced dataset, you will fit a logistic regression model and then analyze the results:

Open a new Jupyter notebook for this exercise.
Perform the initial 12 steps of Exercise 13.01, Benchmarking the Logistic Regression Model on the Dataset, such that the dataset is split into training and testing sets.
Now, join the X and y variables for the training set before resampling:
```
"""
Let us first join the train_x and train_y for ease of operation
"""
trainData = pd.concat([X_train,y_train],axis=1)
```
In this step, we concatenated the X_train and y_train datasets to one single dataset. This is done to make the resampling process in the subsequent steps easier. To concatenate the two datasets, we use the .concat() function from pandas. In the code, we use axis = 1 to indicate that the concatenation is done horizontally, which is along the columns.
Now, display the new data with the .head() function:
```
trainData.head()
```
You should get the following output

Next, find the indexes of the sample dataset where the propensity is yes:
```
ind = trainData[trainData['y']=='yes'].index
print(len(ind))
```
You should get the following output:
```
3723
```
Separate by the minority class as in the following code snippet:
```
minData = trainData.loc[ind]
print(minData.shape)
```
You should get the following output:
```
(3723, 52)
```
Now, find the indexes of the majority class:
```
ind1 = trainData[trainData['y']=='no'].index
print(len(ind1))
```
You should get the following output:
```
27924
```
Separate by the majority class as in the following code snippet:
```
majData = trainData.loc[ind1]
print(majData.shape)
majData.head()
```
You should get the following output:

Caption: Output after separating the majority classes

Once the majority class is separated, we can proceed with sampling
from the majority class. Once the sampling is done, the shape of the
majority class dataset and its head are printed.

Take a random sample equal to the length of the minority class to
make the dataset balanced.

Extract the samples using the .sample() function:
```
majSample = majData.sample(n=len(ind),random_state = 123)
```
The number of examples that are sampled is equal to the number of examples in the minority class. This is implemented with the parameters (n=len(ind)).
Now that sampling is done, the shape of the majority class dataset and its head is printed:
```
print(majSample.shape)
majSample.head()
```
You should get the following output:

Caption: Output showing the shape of the majority class dataset

Now, we move onto preparing the new training data

After preparing the individual dataset, we can now concatenate them together using the pd.concat() function:
```
"""
Concatenating both data sets and then shuffling the data set
"""
balData = pd.concat([minData,majSample],axis = 0)
```
Note

In this case, we are concatenating in the vertical direction and, therefore, axis = 0 is used.
Now, shuffle the dataset so that both the minority and majority classes are evenly distributed using the shuffle() function:
```
# Shuffling the data set
from sklearn.utils import shuffle
balData = shuffle(balData)
balData.head()
```
You should get the following output:

Caption: Output after shuffling the dataset

Now, separate the shuffled dataset into the independent variables, X_trainNew, and dependent variables, y_trainNew. The separation is to be done using the index features 0 to 51 for the dependent variables using the .iloc() function in pandas. The dependent variables are separated by sub-setting with the column name 'y':
```
# Making the new X_train and y_train
X_trainNew = balData.iloc[:,0:51]
print(X_trainNew.head())
y_trainNew = balData['y']
print(y_trainNew.head())
```
You should get the following output:

Caption: Shuffling the dataset into independent variables

Now, fit the model on the new data and generate the confusion matrix
and classification report for our analysis.

First, define the LogisticRegression function with the following code snippet:

from sklearn.linear_model import LogisticRegression
bankModel1 = LogisticRegression()
bankModel1.fit(X_trainNew, y_trainNew)

You should get the following output:

Caption: Fitting the model

Next, perform the prediction on the test with the following code snippet:
```
pred = bankModel1.predict(X_test)
print('Accuracy of Logistic regression model prediction on '\
      'test set for balanced data set: {:.2f}'\
      .format(bankModel1.score(X_test, y_test)))
```
You should get the following output:
```
Accuracy of Logistic regression model prediction on test set for balanced data set:0.83
```
{:.2f}'.format is used to print the string values along with the accuracy score, which is output from bankModel1.score(X_test, y_test). In this, 2f means a numerical score with two decimals.

Now, generate the confusion matrix for the model and print the results:

from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get the following output:

Implementation of SMOTE and MSMOTE

SMOTE and MSMOTE can be implemented from a package called smote-variants in Python. Let's now implement both these methods and analyze the results.

Exercise 13.03: Implementing SMOTE on Our Banking Dataset to Find the Optimal Result

In this exercise, we will generate synthetic samples of the minority class using SMOTE and then make the dataset balanced. Then, on the new balanced dataset, we will fit a logistic regression model and analyze the results:

Implement all the steps of Exercise 13.01, Benchmarking the Logistic Regression Model on the Dataset, until the splitting of the train and test sets (Step 12).

Now, print the count of both the classes before we oversample:

# Shape before oversampling
print("Before OverSampling count of yes: {}"\
      .format(sum(y_train=='yes')))
print("Before OverSampling count of no: {} \n"\
      .format(sum(y_train=='no')))

You should get the following output:

Before OverSampling count of yes: 3694
Before OverSampling count of no: 27953

Next, we will be oversampling the training set using SMOTE.

Begin by importing sv and numpy:

!pip install smote-variants
import smote_variants as sv
import numpy as np

Now, instantiate the SMOTE library to a variable called oversampler using the sv.SMOTE() function:
```
# Instantiating the SMOTE class
oversampler= sv.SMOTE()
```
This is a common way of instantiating any of the variants of SMOTE from the smote_variants library.
Now, sample the process using the .sample() function of oversampler:
```
# Creating new training set
X_train_os, y_train_os = oversampler.sample\
                         (np.array(X_train), np.array(y_train))
```
Note

Both the X and y variables are converted to numpy arrays before applying the .sample() function.

Now, print the shapes of the new X and y variables and the counts of the classes. You will note that the size of the overall dataset has increased from the earlier count of around 31,647 (3694 + 27953) to 55,906. The increase in size can be attributed to the fact that the minority class has been oversampled from 3,694 to 27,953:

# Shape after oversampling
print('After OverSampling, the shape of train_X: {}'\
      .format(X_train_os.shape))
print('After OverSampling, the shape of train_y: {} \n'\
      .format(y_train_os.shape))
print("After OverSampling, counts of label 'Yes': {}"\
      .format(sum(y_train_os=='yes')))
print("After OverSampling, counts of label 'no': {}"\
      .format(sum(y_train_os=='no')))

You should get the following output:

After OverSampling, the shape of train_X: (55906, 51)
After OverSampling, the shape of train_y: (55906,) 
After OverSampling, counts of label 'Yes': 27953
After OverSampling, counts of label 'no': 27953

Define the LogisticRegression function:

# Training the model with Logistic regression model
from sklearn.linear_model import LogisticRegression
bankModel2 = LogisticRegression()
bankModel2.fit(X_train_os, y_train_os)

Now, predict using .predict on the test set, as mentioned in the following code snippet:
```
pred = bankModel2.predict(X_test)
```

Next, print the accuracy values:

print('Accuracy of Logistic regression model prediction on '\
      'test set for Smote balanced data set: {:.2f}'\
      .format(bankModel2.score(X_test, y_test)))

Your output should be as follows:

Accuracy of Logistic regression model prediction on test set for Smote balanced data set: 0.83

Then, generate ConfusionMatrix for the model:

from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)

The matrix is as follows:

[[10042  1956]
 [  306  1260]]

Generate Classification_report for the model:

from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get the following output:

Caption: Classification report for the model

In the next exercise, we will be implementing MSMOTE.

Exercise 13.04: Implementing MSMOTE on Our Banking Dataset to Find the Optimal Result

In this exercise, we will generate synthetic samples of the minority class using MSMOTE and then make the dataset balanced. Then, on the new balanced dataset, we will fit a logistic regression model and analyze the results. This exercise will be very similar to the previous one.

Implement all the steps of Exercise 13.01, Benchmarking the Logistic Regression Model on the Dataset, until the splitting of the train and test sets (Step 12).

Now, print the count of both the classes before we oversample:

# Shape before oversampling
print("Before OverSampling count of yes: {}"\
      .format(sum(y_train=='yes')))
print("Before OverSampling count of no: {} \n"\
      .format(sum(y_train=='no')))

You should get the following output:

Before OverSampling count of yes: 3723
Before OverSampling count of no: 27924

Note

The counts mentioned in this output can vary because of variability in the sampling process.

Next, we will be oversampling the training set using MSMOTE.

Begin by importing the sv and numpy:
```
!pip install smote-variants
import smote_variants as sv
import numpy as np
```
The library files that are required for oversampling the training set include the smote_variants library, which we installed earlier. This is imported as sv. The other library that is required is numpy, as the training set will have to be given a numpy array for the smote_variants library.
Now, instantiate the MSMOTE library to a variable called oversampler using the sv.MSMOTE() function:
```
# Instantiating the MSMOTE class
oversampler= sv.MSMOTE()
```

Now, sample the process using the .sample() function of oversampler:

# Creating new training set
X_train_os, y_train_os = oversampler.sample\
                         (np.array(X_train), np.array(y_train))

Note

Both the X and y variables are converted to numpy arrays before applying the .sample() function.

Now, print the shapes of the new X and y variables and also the counts of the classes:

# Shape after oversampling
print('After OverSampling, the shape of train_X: {}'\
      .format(X_train_os.shape))
print('After OverSampling, the shape of train_y: {} \n'\
      .format(y_train_os.shape))
print("After OverSampling, counts of label 'Yes': {}"\
      .format(sum(y_train_os=='yes')))
print("After OverSampling, counts of label 'no': {}"\
      .format(sum(y_train_os=='no')))

You should get the following output:

After OverSampling, the shape of train_X: (55848, 51)
After OverSampling, the shape of train_y: (55848,) 
After OverSampling, counts of label 'Yes': 27924
After OverSampling, counts of label 'no': 27924

Now that we have generated synthetic points using MSMOTE and balanced the dataset, let's fit a logistic regression model on the new sample and analyze the results using a confusion matrix and a classification report.

Define the LogisticRegression function:

# Training the model with Logistic regression model
from sklearn.linear_model import LogisticRegression
# Defining the LogisticRegression function
bankModel3 = LogisticRegression()
bankModel3.fit(X_train_os, y_train_os)

Now, predict using .predict on the test set as in the following code snippet:
```
pred = bankModel3.predict(X_test)
```

Next, print the accuracy values:

print('Accuracy of Logistic regression model prediction on '\
      'test set for MSMOTE balanced data set: {:.2f}'\
      .format(bankModel3.score(X_test, y_test)))

You should get the following output:

Accuracy of Logistic regression model prediction on test set for MSMOTE balanced data set: 0.84

Generate the ConfusionMatrix for the model:

from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix(y_test, pred)
print(confusionMatrix)

The matrix should be as follows:

[[10167  1831]
 [  314  1252]]

Generate the Classification_report for the model:

from sklearn.metrics import classification_report
print(classification_report(y_test, pred))

You should get the following output:

Activity 13.01: Finding the Best Balancing Technique by Fitting a Classifier on the Telecom Churn Dataset

You are working as a data scientist for a telecom company. You have encountered a dataset that is highly imbalanced, and you want to correct the class imbalance before fitting the classifier to analyze the churn. You know different methods for correcting the imbalance in datasets and you want to compare them to find the best method before fitting the model.

In this activity, you need to implement all of the three methods that you have come across so far and compare the results.

Note

You will be using the telecom churn dataset that you used in Lab 10, Analyzing a Dataset.

Use the MinMaxscaler function to scale the dataset instead of the robust scaler function you have been using so far. Compare the methods based on the results you get by fitting a logistic regression model on the dataset.

The steps are as follows:

Implement all the initial steps, which include installing smote-variants and loading the data using pandas.
Normalize the numerical raw data using the MinMaxScaler() function we learned about in Lab 3, Binary Classification.
Create dummy data for the categorical variables using the pd.get_dummies() function.
Separate the numerical data from the original data frame.
Concatenate numerical data and dummy categorical data using the pd.concat() function.
Split the earlier dataset into train and test sets using the train_test_split() function.

Since the dataset is imbalanced, you need to perform the various techniques mentioned in the following steps.
For the undersampling method, find the index of the minority class using the .index() function and separate the minority class. After that, sample the majority class and make the majority dataset equal to the minority class using the .sample() function. Concatenate both the minority and under-sampled majority class to form a new dataset. Shuffle the dataset and separate the X and Y variables.
Fit a logistic regression model on the under-sampled dataset and name it churnModel1.
For the SMOTE method, create the oversampler using the sv.SMOTE() function and create the new X and Y training sets.
Fit a logistic regression model using SMOTE and name it churnModel2.
Import the smote-variant library and instantiate the MSMOTE algorithm using the sv.MSMOTE() function.
Create the oversampled data using the oversampler. Please note that the X and y variables have to be converted to a numpy array before oversampling
Fit the logistic regression model using the MSMOTE dataset and name the model churnModel3.
Generate the three separate predictions for each model.
Generate separate accuracy metrics, classification reports, and confusion matrices for each of the predictions.
Analyze the results and select the best method.

Expected Output:

The final metrics that you can expect will be similar to what you see here.

Undersampling Output

Caption: Undersampling output report

SMOTE Output

Caption: SMOTE output report

MSMOTE Output

Caption: MSMOTE output report

Summary

In this lab, we learned about imbalanced datasets and strategies for addressing imbalanced datasets. We introduced the use cases where imbalanced datasets would be encountered. We looked at the challenges posed by imbalanced datasets and we were introduced to the metrics that should be used in the case of an imbalanced dataset. We formulated strategies for dealing with imbalanced datasets and implemented different strategies, such as random undersampling and oversampling, for balancing datasets. We then fit different models after balancing the datasets and analyzed the results.

29 KiB Raw Permalink Blame History

Lab 13. Imbalanced Datasets

Exercise 13.01: Benchmarking the Logistic Regression Model on the Dataset

Analysis of the Result

Challenges of Imbalanced Datasets

Strategies for Dealing with Imbalanced Datasets

Exercise 13.02: Implementing Random Undersampling and Classification on Our Banking Dataset to Find the Optimal Result

Implementation of SMOTE and MSMOTE

Exercise 13.03: Implementing SMOTE on Our Banking Dataset to Find the Optimal Result

Exercise 13.04: Implementing MSMOTE on Our Banking Dataset to Find the Optimal Result

Activity 13.01: Finding the Best Balancing Technique by Fitting a Classifier on the Telecom Churn Dataset

Summary

29 KiB

Raw Permalink Blame History