Semi-Supervised Learning with Scikit-learn’s SelfTrainingClassifier: A Visual Guide

In many real-world scenarios, we often find ourselves with a lot of unlabeled data and a small portion of labeled data. Manually labeling this data can be expensive and time-consuming. This is where semi-supervised learning shines. It leverages both labeled and unlabeled data to improve the model’s performance.

Scikit-learn offers the SelfTrainingClassifier, a semi-supervised learning method that iteratively pseudo-labels the unlabeled data and refines the model. In this tutorial, we’ll understand how the SelfTrainingClassifier works and visualize its pseudo-labeling process using an animation.

Setting up the Environment

Before we dive in, ensure you have the necessary packages installed:

1
pip install numpy matplotlib scikit-learn

Generating Synthetic Data

Let’s start by creating a synthetic dataset with two clusters:

1
from sklearn.datasets import make_blobs
2

3
X, y = make_blobs(n_samples=400, centers=2, cluster_std=5, random_state=42)
4
y[123:] = -1  # Unlabeled samples

We’ve deliberately left a large portion of the samples as unlabeled (denoted by -1).

Initializing the Base Classifier

For this tutorial, we’ll use a logistic regression model with an elastic net penalty:

1
from sklearn.linear_model import LogisticRegression
2

3
base_classifier = LogisticRegression(penalty='elasticnet',
4
                                    solver='saga',
5
                                    max_iter=10000,
6
                                    class_weight='balanced',
7
                                    l1_ratio=0.1,
8
                                    C=0.001)

The SelfTrainingClassifier

The SelfTrainingClassifier in Scikit-learn takes in the base classifier and iteratively pseudo-labels the unlabeled data:

1
from sklearn.semi_supervised import SelfTrainingClassifier
2

3
self_training_clf = SelfTrainingClassifier(base_classifier, criterion="k_best", k_best=2, max_iter=10)

Here, the k_best criterion means that in each iteration, the two most confidently predicted samples will be pseudo-labeled.

Training and Collecting Iteration Data

We’ll manually run the classifier for 10 iterations, capturing the state of the labels at each step:

1
iterations_data = []
2

3
def train_one_iteration():
4
    prev_labels = y.copy()
5
    self_training_clf.fit(X, y)
6
    new_labels = self_training_clf.transduction_
7
    y[:] = new_labels
8
    pseudo_labeled_indices = np.where((prev_labels == -1) & (new_labels != -1))
9
    return pseudo_labeled_indices
10

11
for i in range(10):
12
    indices = train_one_iteration()
13
    iterations_data.append((y.copy(), indices))

Visualizing the Process

Using Matplotlib, we can animate the pseudo-labeling process:

1
import matplotlib.pyplot as plt
2
from matplotlib.animation import FuncAnimation
3

4
fig, ax = plt.subplots()
5
scatter = ax.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired, edgecolors='k')
6
highlighted, = ax.plot([], [], 'ro', markersize=5)
7

8
def init():
9
    return scatter, highlighted
10

11
def update(frame):
12
    scatter.set_array(iterations_data[frame][0])
13
    highlighted.set_data(X[iterations_data[frame][1], 0], X[iterations_data[frame][1], 1])
14
    ax.set_title(f"SelfTrainingClassifier Iteration: {frame+1}")
15
    return scatter, highlighted
16

17
ani = FuncAnimation(fig, update, frames=len(iterations_data), init_func=init, blit=False)
18
ani.save('self_training_animation.gif', writer='imagemagick', fps=1)

This will save the animation as a GIF. The red dots in the animation represent the samples that have been pseudo-labeled in each iteration.

Conclusion

The SelfTrainingClassifier is a powerful tool for semi-supervised learning, especially when you have a large amount of unlabeled data. By visualizing its pseudo-labeling process, we gain insights into how the classifier is gradually confident about labeling the unlabeled samples. This can be especially useful for understanding and debugging your semi-supervised learning models.