The Art of Feature Selection in Python: Removing Highly Correlative Features

Redundant features are those that provide little or no additional information to the model and can even decrease its performance. Removing such features can improve the accuracy and efficiency of the model. In this tutorial, we will learn how to remove highly correlative redundant features in Python.

Step 1: Calculate the correlation matrix

The first step in removing highly correlative redundant features is to calculate the correlation between all the features. We can do this using the corr method in Pandas. This method returns a correlation matrix that shows the correlation between each pair of features.

1
import pandas as pd
2

3
# Load the dataset
4
df = pd.read_csv("data.csv")
5

6
# Calculate the correlation matrix
7
corr_matrix = df.corr()

Step 2: Identify the highly correlated features

Next, we need to identify the features that have a high correlation (above a certain threshold). We can choose the threshold based on our application and the acceptable level of correlation. For example, we can consider a threshold of 0.8 and consider all the pairs of features with a correlation greater than 0.8 as highly correlated.

1
# Identify the features with a correlation greater than 0.8
2
written = []
3
for column in corr_matrix:
4
    for index in corr_matrix.index[corr_matrix[column] > 0.8]:
5
        if index not in written:
6
            print(f"{column} and {index} have a correlation of {corr_matrix[column][index]:.2f}")
7
            written.append(column)

Step 3: Remove one of the highly correlated features

Finally, we can remove one of the highly correlated features. It doesn’t matter which one we remove because both of them provide similar information.

1
# Remove one of the highly correlated features
2
df = df.drop(columns=["feature_to_remove"])

Step 4: Use the modified dataframe to train your model

We can now use the modified dataframe to train our model.

1
# Use the modified dataframe to train your model
2
X = df.drop(columns=["target"])
3
y = df["target"]

Conclusion

In this tutorial, we learned how to remove highly correlative redundant features in Python. By calculating the correlation matrix and identifying the highly correlated features, we were able to remove one of the redundant features and improve the performance of the model.