A Beginner’s Guide to Cross-Validation in Python’s scikit-learn

Cross-validation is a technique that allows us to evaluate the performance of a machine learning model by training it on different subsets of the dataset and averaging the results. This is especially useful when we have limited data, as it allows us to use all of the available data for training and evaluation.

In this tutorial, we will learn how to perform cross-validation in Python using the scikit-learn library.

1. Importing libraries and dataset

First, we need to import the necessary libraries and the dataset that we will be using for this tutorial. We will be using the scikit-learn library for cross-validation, and the pandas library for reading and manipulating the dataset.

1
import pandas as pd
2
from sklearn.model_selection import cross_val_score

Next, we will read the dataset into a Pandas DataFrame using the read_csv function.

1
df = pd.read_csv('my_dataset.csv')

2. Performing cross-validation

To perform cross-validation in scikit-learn, we can use the cross_val_score function. This function takes as input the model, the features (predictors), and the target variable, and returns an array of scores for each fold of the cross-validation.

1
# Separate the features and target variable
2
X = df.drop('target', axis=1)
3
y = df['target']
4

5
# Create a model
6
model = RandomForestClassifier(random_state=123)
7

8
# Perform 5-fold cross-validation
9
scores = cross_val_score(model, X, y, cv=5)
10

11
# Print the mean and standard deviation of the scores
12
print(f'Mean score: {scores.mean():.3f}')
13
print(f'Standard deviation: {scores.std():.3f}')

By default, the cross_val_score function uses stratified k-fold cross-validation, which means that the folds are stratified, i.e., they have the same class distribution as the original dataset.

3. Customizing the cross-validation strategy

We can customize the cross-validation strategy by using different cross-validation generators, such as KFold or ShuffleSplit.

For example, to use the KFold generator and perform 5-fold cross-validation with a random seed, we can do the following:

1
from sklearn.model_selection import KFold
2

3
# Create a KFold generator with 5 folds and a random seed
4
kfold = KFold(n_splits=5, random_state=123)
5

6
# Perform cross-validation with the KFold generator
7
scores = cross_val_score(model, X, y, cv=kfold)
8

9
# Print the mean and standard deviation of the scores
10
print(f'Mean score: {scores.mean():.3f}')
11
print(f'Standard deviation: {scores.std():.3f}')

4. Scoring metric

By default, the cross_val_score function uses the accuracy score as the metric to evaluate the model’s performance. However, we can use any other scoring metric by passing it to the scoring parameter.

For example, to use the F1 score as the scoring metric, we can do the following:

1
from sklearn.metrics import f1_score
2

3
# Use the F1 score as the scoring metric
4
scores = cross_val_score(model, X, y, cv=kfold, scoring='f1')
5

6
# Print the mean and standard deviation of the scores
7
print(f'Mean score: {scores.mean():.3f}')
8
print(f'Standard deviation: {scores.std():.3f}')

5. Wrapping it up

In this tutorial, we learned how to perform cross-validation in Python using the scikit-learn library. We saw how to use the cross_val_score function with different cross-validation generators and scoring metrics. Cross-validation is a useful technique for evaluating the performance of a machine learning model and for choosing the best model for a given dataset.