Self- and Semi-Supervised Learning for Tabular Data

Deep learning has transformed images, text, and audio, yet gradient-boosted trees still win most competitions on tabular data. Tables are the everyday format of biology and medicine: gene expression matrices, clinical variables, lab measurements, and drug-screening readouts. Understanding why neural networks underperform here, and how to close the gap, is a practical question for anyone modeling biomedical data.

Why tables are hard for neural networks

Images have strong spatial structure and text has sequential structure, and convolutional or attention-based architectures exploit that structure directly. Tabular data has no such prior. Columns are heterogeneous, often mixing continuous, categorical, and ordinal features on wildly different scales. There is no natural notion of locality between adjacent columns, so the inductive biases that make CNNs and transformers powerful simply do not apply.

On top of that, biomedical tables are usually small and expensive to label. A drug-response study might profile thousands of features across only a few hundred samples, and labeling requires costly assays or expert annotation. This is exactly the regime where tree ensembles shine and where naive deep networks overfit.

Self-supervision: learning without labels

Self-supervised learning sidesteps the label bottleneck by inventing a task the model can solve using the data alone. The idea that works well for tables comes from VIME (Value Imputation and Mask Estimation), introduced at NeurIPS 2020.

VIME corrupts each record by randomly masking a subset of its features and replacing them with values sampled from the empirical distribution of each column. The network is then trained on two complementary pretext tasks:

• Mask estimation asks the model to recover the binary mask, that is, to identify which features were corrupted.
• Value imputation asks the model to reconstruct the original, uncorrupted values.

To succeed at both, the encoder must learn the correlation structure among features. It cannot simply memorize; it has to understand that certain columns move together, which is precisely the representation we want.

Semi-supervision: using the labels you do have

Once you have a pretrained encoder, you can fine-tune it on whatever labels exist. VIME adds a semi-supervised objective that encourages consistency: multiple corrupted versions of the same unlabeled record should yield similar predictions. This consistency loss pushes the decision boundary into low-density regions of the input space, a classic semi-supervised principle, and lets the labeled and unlabeled data reinforce each other.

Why this matters for biomedical data

The self- then semi-supervised recipe fits the shape of real biomedical problems. You often have far more unlabeled molecular profiles than annotated outcomes, and the features are genuinely correlated because biology is structured. Learning that structure first, then grounding it with a handful of labels, is more sample-efficient than training end to end from scratch.

I reimplemented VIME in PyTorch to make the method easier to adapt to genomics workflows; the code is available in my projects. If you work with tables, it is worth measuring against a strong tree baseline before assuming deep learning cannot compete: the gap narrows considerably once you exploit unlabeled data.

Takeaways

Tabular deep learning is not a solved problem, but self-supervision changes the calculus when labels are scarce. Mask estimation and value imputation give a network a reason to learn feature correlations, and a consistency objective lets unlabeled records improve a supervised model. For biomedical tables, where unlabeled data is abundant and labels are precious, that combination is a pragmatic default worth trying.