Very interesting for me - I found TabPFN very useful where data size is limited (~100 samples or less). Also the stats functions/properties, e.g. generating synthetic data, are exciting but need better understanding.
Thank you very much for the article. Indeed tabular ML is going to get "weird-er", but I think it's good to see transformer advancements in tabular data, not just text/images.
Thanks for the intro, very interesting! I also find super surprising that being pre-trained mostly on synthetic data they are effective at predicting on real data. Something worth exploring in more depth.
This is a great intro into this. I'm really looking forward to more content and trying it out. I guess, my only question at the moment is how can I validate or explain the results where Local and Global explanations are needed ? Is it possible?
You can apply the entire toolbox of model-agnostic interpretability tools, such as Shapley values, partial dependence plots, and permutation feature importance. Since they require lots of predictions, they may be slow to compute though
To me this sounds a lot more promising than trying to use NNs for tabular data and what is most interesting to me is the fact that oftentimes causal graphs are used to generate the synthetic data. I'm wondering if this could help to learn detect causal structures in your real life data w/o having to generate a causal graph for your real life data. So far all the papers I found do not shed light on this question but stick to using prediction style KPIs such has accuracy or RSME to meaure peformance. I'd be interested to understand how good this approach is in providing a better feature importance ranking (where better means: rank causal correlations over spurious ones and not rank the ones high the give you the best accuracy/RSME) compared to tree methods...
P.S.: It might be worth also looking into LimiX (they claim to outperform tabPVN_v2 substantially, there is a lot of movement in this area currently).
I think one of the key takeaways is that as inference truly becomes commoditized (and thus model implementation & tuning are deemphasized,) data representation becomes the singular most important thing.
Agreed with this, there is now a ton of art in the data representation and nomalization. I also would like to understand more on this topic, there are some cool recent biological models like Stack (https://arcinstitute.org/news/foundation-model-stack) where the input data representation is very intuitive, and I'd like to be able to use them better.
This seems interesting! I would be curious to see how these models are performing in the Time Series domain, as well as explainability or how conformal prediction would work. Very cool topic, looking forward to reading more from you!
Very interesting for me - I found TabPFN very useful where data size is limited (~100 samples or less). Also the stats functions/properties, e.g. generating synthetic data, are exciting but need better understanding.
I am very much interested - ran into TabPFN from the time series angle recently.
Tabular data may very well undergoing the same disruption transformer did before to NLP and CV.
3 areas critical for improvement:
1. Handling large dataset - this is improving rapidly.
2. Interoperability - tough one to crack IMO.
3. Efficient and fast inference , even for single instances.
Number 2 & 3 will need to be improved to increase adoption in many domains.
At any rate, it is a very useful offline benchmark to have for your current datasets.
Thank you very much for the article. Indeed tabular ML is going to get "weird-er", but I think it's good to see transformer advancements in tabular data, not just text/images.
Thanks for the intro, very interesting! I also find super surprising that being pre-trained mostly on synthetic data they are effective at predicting on real data. Something worth exploring in more depth.
This is a great intro into this. I'm really looking forward to more content and trying it out. I guess, my only question at the moment is how can I validate or explain the results where Local and Global explanations are needed ? Is it possible?
You can apply the entire toolbox of model-agnostic interpretability tools, such as Shapley values, partial dependence plots, and permutation feature importance. Since they require lots of predictions, they may be slow to compute though
Nice writing. I like it, keep going.
I noticed few typos:
- In the last subsection, replace "xgboot, catboos" with "xgboost, catboost".
- which often turned out ... -> which have often turned out ...
Fixed. Thanks!
Please, write more about it. Its difficilt to dedicate the needed time to better understand them, and you help a lot with your material
These are my list of things to dive deeper into this year for sure. Would love to see more writing
Very much interested and appreciate your taking time and digging deeper with an open mind. Thank you Christoph.
Very interesting! Please deep dive!
To me this sounds a lot more promising than trying to use NNs for tabular data and what is most interesting to me is the fact that oftentimes causal graphs are used to generate the synthetic data. I'm wondering if this could help to learn detect causal structures in your real life data w/o having to generate a causal graph for your real life data. So far all the papers I found do not shed light on this question but stick to using prediction style KPIs such has accuracy or RSME to meaure peformance. I'd be interested to understand how good this approach is in providing a better feature importance ranking (where better means: rank causal correlations over spurious ones and not rank the ones high the give you the best accuracy/RSME) compared to tree methods...
P.S.: It might be worth also looking into LimiX (they claim to outperform tabPVN_v2 substantially, there is a lot of movement in this area currently).
"https://mindfulmodeler.substack.com/p/tabular-ml-is-about-to-get-weird"
Wow, this is very thought provoking and could have big implications for model explainability.
I think one of the key takeaways is that as inference truly becomes commoditized (and thus model implementation & tuning are deemphasized,) data representation becomes the singular most important thing.
Agreed with this, there is now a ton of art in the data representation and nomalization. I also would like to understand more on this topic, there are some cool recent biological models like Stack (https://arcinstitute.org/news/foundation-model-stack) where the input data representation is very intuitive, and I'd like to be able to use them better.
This seems interesting! I would be curious to see how these models are performing in the Time Series domain, as well as explainability or how conformal prediction would work. Very cool topic, looking forward to reading more from you!
You can check the paper "TabPFN-TS" which adapts TabPFN-v2 to time series forecasting. Works really well !
Interpretability in this kind of models could be a very interesting topic!!!
Very interesting. Very interested in the interpretability anlge, especially wrt to trees. Possibly another reason they are hard to dethrone(?)