Can Zero-Shot Models Beat XGBoost and LightGBM?

I kept hearing about the performance of zero-shot models. At first, the idea that a model could outperform one fitted to a dataset without updating its own weights sounded like science fiction.

Curiosity got the better of me. I benchmarked Mitra v2 and TabPFN-3.5-Fast against XGBoost and LightGBM across three public datasets. Would the pretrained models come out ahead? Would feature engineering help? Could I run everything on a laptop GPU?

Each experiment has a notebook, saved predictions, and a report generated from the measured results.

Three Datasets, Three Prediction Tasks

I chose three UCI datasets covering different kinds of supervised learning:

The primary metrics were ROC-AUC for Online Shoppers, RMSE for Seoul Bike, and macro-F1 for Dry Bean, with supporting metrics recorded in the notebooks.

Keeping the Comparison Consistent

Every model used the same 60% training, 20% validation, and 20% test split, with seed 42. Classification splits preserved class proportions, and all source rows were retained.

1. Training data fitted the classical models, supplied context for foundation models, and fitted learned preprocessing.
2. Validation data selected classical hyperparameters.
3. Test data evaluated each predeclared model and feature setting after selection.

Each classical search tested four configurations with 80 trees, varying learning rate and either tree depth or leaf count. This kept reproduction practical, though broader tuning could improve the baselines. A dummy model provided a reference score.

What “Zero-Shot” Means Here

Here, zero-shot means frozen pretrained weights with no task-specific fine-tuning. The foundation models still received labeled training examples as context and used native preprocessing. Their inference settings were not tuned on validation scores.

I evaluated each foundation model twice:

• Raw: original features with the formatting required for the model to run.
• Engineered: original features plus documented, handcrafted features.

Including both tree models and the dummy baseline gave seven variants per dataset: 21 overall.

Does Feature Engineering Still Help?

Both classical models used engineered features. Foundation models were evaluated with and without these additions:

These features use only information within each row. For example, cyclical hour features place 11 p.m. and midnight close together, capturing a relationship that an ordinary hour column leaves implicit.

The Results

All 21 variants completed on the full datasets. These are their primary test scores:

Higher ROC-AUC and macro-F1 are better; lower RMSE is better. Compare scores within each column.

Online Shoppers: Close Scores, Different Tradeoffs

Mitra v2 with raw features led ROC-AUC at 0.9366, versus 0.9321 for LightGBM. TabPFN led average precision with raw features and F1 with engineered features, showing why the operating goal and decision threshold matter.

Feature engineering did not improve either foundation model's ROC-AUC on this split.

Seoul Bike: The Largest Performance Differences

TabPFN with engineered features achieved an RMSE of 104.7439, versus 179.4589 for XGBoost. Calendar features also helped Mitra, reducing its RMSE from 205.1263 to 125.4689.

This used a random split with observed weather, so nearby hours could fall in different partitions. The task was retrospective prediction. Forecasting would require a chronological split and inputs available at forecast time.

Dry Bean: Small Differences at the Top

TabPFN with raw features led macro-F1 at 0.9390; other foundation settings scored 0.9370–0.9381, and tree models scored around 0.9335. Feature engineering had mixed effects, and these small gaps come from a single split.

What It Took to Run

The reference runs used Python 3.12 and an NVIDIA RTX 3050 Laptop GPU with 6 GB of memory. Classical models ran on CPU, while foundation models ran on GPU.

Recorded wall times per variant were roughly:

• XGBoost and LightGBM: 3–5 seconds, including the small validation search.
• TabPFN: 12–21 seconds.
• Mitra v2: 38–230 seconds, depending on the dataset and feature setting.

Timings include startup, loading, prediction, and classical tuning. They reflect one run with different hardware for each model family.

All experiments fit on the laptop GPU. Full training partitions exceed TabPFN's default CPU row limit, so small CPU setup checks do not establish full-data feasibility.

How to Run the Project

Clone the benchmark repository and create a Python 3.12 environment in Linux or WSL:

git clone https://github.com/debabrot/tabular-foundation-model-benchmark.git
cd tabular-foundation-model-benchmark
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m pip check

Follow the runtime guide for PyTorch setup and model access. If required, accept TabPFN's terms and supply an authorized HF_TOKEN.

Download the pinned weights and check them on small synthetic examples:

.venv/bin/python scripts/smoke_models.py --models tabpfn mitra --device cpu --download --timeout 300

Then run the three notebooks sequentially using a CUDA GPU:

.venv/bin/python scripts/execute_notebook.py notebooks/01_binary_classification.ipynb --device cuda
.venv/bin/python scripts/execute_notebook.py notebooks/02_regression.ipynb --device cuda
.venv/bin/python scripts/execute_notebook.py notebooks/03_multiclass_classification.ipynb --device cuda

Datasets download automatically. Each run saves notebook outputs, metrics, predictions, splits, and plots.

Finally, rebuild the report and run the verification checks:

.venv/bin/python scripts/build_report.py
.venv/bin/python scripts/build_report.py --check
.venv/bin/python -m unittest discover -s tests -v

The report builder verifies metrics from saved predictions without refitting. Tests requiring missing dataset or weight caches skip.

What I Take Away from This Benchmark

The answer to the title's question is yes, on these recorded splits: a foundation-model setting led each primary metric. Feature engineering still mattered, especially for Seoul Bike.

That does not establish a universal winner. Each dataset used one split, without significance tests. The classification datasets contain duplicates, further grouping information is unavailable, and possible pretraining overlap has not been audited. Online Shoppers also uses session-derived information whose availability at a live prediction time needs checking.

Repeated splits, broader classical tuning, and a separate chronological bike-demand experiment would make useful follow-ups. The full results report includes supporting metrics and limitations; the notebooks trace each experiment from loading data to evaluation.