Trends in tabular foundation research
Based on 151 papers from ICML workshop "Foundation Models for Structured Data"
151 papers on tabular and time series foundation models were presented at “Foundation Models for Structured Data”, a workshop that took place for the second time at ICML this month in Seoul.
I didn’t attend, but still marked the event in my calendar to check on current trends in tabular foundation research. Going through the titles and some abstracts was very informative, and I want to share my observations about the state of tabular foundation research.
If you are new to tabular foundation models, start with this series:
The rise of tabular foundation models
Tree-based boosting algorithms have sat on the tabular throne for many years now. Many times, the deep learners have attempted to dethrone XGBoost, CatBoost, and other tree-based algorithms, but without success.
Here are the trends and observations, based on this list of papers:
Tabular foundation models are expanding beyond just regression and classification, as I expected. There are papers on survival analysis, Bayesian inference, causal identification, and more.
Time series was the biggest ML task/modality (besides tabular) with around 50 out of 151 papers. I attribute it to the workshop framing (“tabular and time-series”). By the way, I just published a book chapter on time series forecasting with tabular foundation models.
Tabular foundation models are moving away from being just models toward becoming more of a system, with research on external infrastructure such as context management, input-space adapters, distillation, and integration into agents.
Benchmarking and evaluation were a huge topic. Also, some critique about the current, narrow way of benchmarking. I personally think this is improving thanks to new benchmarks like ScoringBench and BeyondArena.
Some researchers examined prior designs such as causal DAGs and graph priors.
An emerging topic is mechanistic interpretability for foundation models.
If you are curious about the application of tabular foundation models, you’ll find some applied papers in the list as well, in domains like healthcare.
Some papers explored multi-modal foundation models, like tabular+text and tabular+image.
Some topics were surprisingly underrepresented:
The focus was not as much on scaling foundation models, at least compared to industry labs, where scaling tabular foundation models to larger data and speeding up inference is top priority.
There was a bit on relational foundation modeling (extending foundation models to a relational multi-table setup), but I expected more here.
Papers that caught my interest: Beyond Accuracy: Toward Trustworthy Tabular Foundation Models in Industrial Applications and Exploring Differences Between Tabular Enterprise Data and Public Benchmarks.
Is there some sort of a ''original'' paper which introduced foundational models? think of "Attention is all you need" type of papers.