Abstract
Recommender systems have emerged as a new weapon to help online firms to realize many of their strategic goals (e.g., to improve sales, revenue, customer experience etc.). How- ever, many existing techniques commonly approach these goals by seeking to recover preference (e.g., estimating rat- ings) in a matrix completion framework. This paper aims to bridge this significant gap between the clearly-defined strate- gic objectives and the not-so-well-justified proxy.
We show it is advantageous to think of a recommender system as an analogy to a monopoly economic market with the system as the sole seller, users as the buyers and items as the goods. This new perspective motivates a game-theoretic formulation for recommendation that enables us to identify the optimal recommendation policy by explicit optimizing certain strategic goals.
In This Paper, We Revisit And Ex-
tend our prior work, the Collaborative-Competitive Filtering preference model , towards a game-theoretic framework. The proposed framework consists of two components. First, a conditional preference model that characterizes how a user would respond to a recommendation action; Second, know- ing in advance how the user would respond, how a recom- mender system should act (i.e., recommend) strategically to maximize its goals.
We Show How Objectives Such As
click-through rate, sales revenue and consumption diversity can be optimized explicitly in this framework. Experiments are conducted on a commercial recommender system and demonstrate promising results.
Categories And Subject Descriptors
H.5.3 [Information Systems]: Web-based Interaction; H.3.3 [Information Search and Retrieval]: Information fil-
Algorithms, Performance
Keywords: Recommendation optimization, Collaborative
Games, Econometric Model, Expected Utility Theory
1.
Introduction
Recommender systems have become a core component for today’s online businesses. With the abilities of connecting Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
1st submitted: Monday Feb. 6 2012; revised: Tuesday Jul. 31 2012. merchant supply (i.e., items of various types such as retailing products, movies, articles, ads, experts, etc.) to market de- mands (i.e., potentially interested consumers), recommender systems are helping online firms (e.g. Amazon, Netflix, Ya- hoo!) to realize many of their hard-to-attain business goals (e.g., to boost sales, improve revenue, enhance customer ex- periences) [5, 6, 9, 30].
Compared To An Offline Market,
online recommender system has the unbeatable convenience in control, intervention, monitoring and measurement of the market, and consequently the appealing opportunity to ad- just its operational actions (i.e., recommendation policy) to optimize certain strategic objectives. Surprisingly, despite the fact that many of these goals are clearly defined, they are not optimized in today’s recommender systems in a well- justified way. Instead, research on recommendation has been focused almost exclusively on learning preference (e.g., esti- mating a user’s rating to a movie) in a matrix completion formulation [27, 22, 15, 8, 31].
It Is Rather Unclear How
preference learning, as a proxy, approximates these goals, or how a strategic intervention should be designed to achieve certain goals.
In this paper, we seek to bridge this significant gap. We show it is advantageous to look at the user-system interac- tions and think of a recommender system as an analogy to a monopoly economic market (i.e., system as the sole seller, users as buyers and items as goods)1, rather than user-item interactions as in the conventional matrix completion for- mulation.
This New Perspective Motivates A Novel Game-
theoretic formulation, upon which recommendation policy can be optimized strategically with respect to business ob- jectives such as click-through rate, sales revenue and con- sumption diversity.
User-System Interactions
Recommender systems are commonly designed by analyz- ing the dyadic user-item interactions as can be recorded by a matrix, for example, users assigning ratings to movies.
Research has thus been focused exclusively on estimating preference or equivalently completing the matrix of “who- like-what” [4, 27, 22, 31, 2, 1, 15, 8]. This matrix-completion formulation of recommendation has been extensively inves- tigated and become especially popular thanks to the Netflix Prize Competition. Nonetheless, as we show in this paper, the formulation of recommendation as user-item interaction or matrix completion is inherently flawed — recommenda- tion is not solely about what you know (i.e., knowledge about
Interchangeably “System” And
“seller”, “user” and “buyer”, “item” and “good”. the user), but more importantly about how you act (i.e., how to recommend items to serve the user or persuade the user to consume).
Instead, It Is Advantageous To Think Of Rec-
ommendation as an interaction between the system and the users and formulate it as an interdependent decision-making process (aka games) .
In a typical interaction, the system acts by providing a set of personalized recommendations, and user reacts by mak- ing choices, i.e., by choosing to consume some of the recom- mended items (e.g., click a link, rent a movie, view a News article, purchase a product). This process in many aspects resembles what happens in a monopoly market where the recommender system, as the sole seller, has absolute market power to manipulate the market, yet the utility it receives depends on the reaction of the buyers (i.e., users), e.g., the success of an advertising system is directly related to how users react (i.e., whether they click the ads or not). Clearly, the action of the seller and the reaction of the buyer are in- terdependent – the two players (i.e., seller and buyer) each has its own objective (i.e., utility) to achieve, yet how and to what extend they can achieve their own objectives de- pends also on the decision of the other player. Conventional matrix completion formulation for recommendation is inher- ently flawed as it is incapable to capture such interdependent decision making interactions. As a results, although many business objectives in e-commerce are clearly defined, how a recommender can be designed to optimize these goals hasn’t yet been explored.
Recommendation As Collaborative Games
In this paper, we present a game-theoretic formulation for recommendation, where the user-system interactions are modeled as a collection of coupled games with each game played between the seller and one buyer (i.e., between the system and one user). For the sake of statistical inference, it is nonetheless important not to model these games as mu- tually independent. We therefore bring forward the notion of “collaborative games” that similar games are expected to yield similar outcomes, which enables us to pool the sparse data across games to obtain reliable statistical estimation.
We extend our prior work on “Collaborative-Competitive Filtering”(CCF) preference model towards a game-theoretic framework. The framework consists of two components: (1) a conditional model pu(R|A) that characterizes the reaction R of a buyer u in the context of any given action A of the seller; this model enables us to predict in advance what the outcome of a game would be (e.g., how the buyer would respond); and (2) given pu(R|A) for every buyer u, a for- mulation for optimizing the seller’s action policy A w.r.t. a predefined payoff(e.g., a strategic goal).
To effectively model pu(R|A), we revisit and extend the CCF preference model that integrates latent factor mod- els in collaborative filtering with choice models in economet- rics. By using latent factor based utility parametrization, the model encodes the “collaboration effects” among games [7, 23] to advocate the notion of “collaborative games”. As the policy spaces are prohibitively large yet the observations are extremely sparse, this formulation is essential for reli- able statistical inference because it enables the sparse data to be pooled across games. It also remarkably reduces the parametric complexity of pu(R|A) significantly from a pro- hibitive high-order polynomial scale down to a linear scale.
The knowledge about users’ reaction behavior, as charac- terized by pu(R|A), enables us to predict“future”(i.e., user’s reaction) with uncertainty and further to optimize the ac- tion (i.e., the recommendation policy) of the recommender system strategically . For any input action A, the possi- ble outcomes of the games occur with probabilities defined upon pu(R|A). Given a payoff(i.e., a function of the out- come) that is von Neumann-Morgenstern rational, the ex- pected utility theory asserts that the best action is the one that maximizes the expected payoff. We show how busi- ness objectives such as click-trough rate, sales revenue and consumption diversity can be formulated explicitly as ex- pected utilities and used in turn to optimize a recommender system’s action policy.
We also show that the CCF model is sequentially ratio- nal and thus approximates the perfect Nash equilibrium . Experiments on a real-world commercial system demonstrate that the proposed CCF model not only outperforms CF models in both offline and online tests but is also highly effective in achieving satisfactory strategic goals.
Outline: The rest of the paper is structured as follows. We first briefly review the current matrix completion for- mulation and collaborative filtering in Section 2. We then present our new game-theoretic formulation in Section 3 and the CCF model in Section 4. Experiments are presented in Section 5, followed by summary and conclusion in Section 6.
2.
Collaborative Filtering
Many existing approaches generally think of recommen- dation as user-item interactions and therefore aim to re- cover/estimate the preference of each individual user to the
I ∈I := {1, 2, . . . , M},
this is naturally formulated as a matrix completion prob- lem, where we are given observations of dyadic responses {(u, i, yui)} with each yui being an observed response in- dicating user’s preference (e.g. user’s rating to an item, or indication of whether user u likes item i), the goal is to
(U, I) →Yui Where U ∈U, I ∈I
which constitutes a large matrix Y ∈Y|U|×|I|. Assume each item can be consumed multiple times, recommendations are usually done by a simple preference-based ranking according to Y , (i.e., recommending the items with highest yui scores to user u). This formulation include both of the two major categories of approaches to recommendation, i.e., content- based filtering [4, 8] and collaborative filtering [27, 22, 1, 15], among which we briefly review the latter.
It is worth noting that the observed responses are often extremely sparse in realistic systems, i.e., while we might have millions of users and items, only a tiny proportion (con- siderably less than 1%) of the entries of the matrix Y are observable.
This “Data Sparseness” Issue Has Been Widely
recognized as one of the key challenges of recommender sys- tem [22, 1, 15]. To this end, collaborative filtering (CF) ex- plores the notion of “collaboration effects”, i.e., similar users have similar preferences to similar items. By encoding col- laboration, CF pools the sparse observations in such a way that for predicting ˆy(u, i) it also borrows observations from other (similar) users/items. Generally speaking, existing CF methods fall into either of the following two categories.
Neighborhood models. A popular class of approaches to CF is based on propagating the observations of responses among items or users that are considered as neighbors. The model first defines a similarity measure between items / users. Then, an unseen response between user u and item i is approximated based on the responses of neighboring users or items [27, 22], for example, by simply averaging the neigh- boring responses with similarities as weights.
Latent factor models. This class of methods learns pre- dictive latent factors to estimate the missing dyadic responses. The basic idea is to associate latent factors2, φu ∈Rk for each user u and ψi ∈Rk for each item i, and assume a
U Ψi; Θ),
where Θ denotes the set of hyper-parameters. This way the factors could explain past responses and in turn make pre- diction for future ones. This model implicitly encodes the Aldous-Hoover theorem for exchangeable matrices – yui are independent of each other given φu and ψi. In essence, it amounts to a low-rank approximation of the matrix Y that naturally embeds both users and items into a vector space in which the inner-products directly reflect the semantic re- latedness.
To design a concrete model [2, 1, 15, 24, 28], one needs to specify a distribution for the dependence. Afterwards, the model boils down to an optimization problem. For example
Two Commonly-Used Formulations Are:
- ℓ2 regression The most popular learning formulation is to minimize the ℓ2 loss within an empirical risk mini-
||Ψi||2,
where O denotes the set of (u, i) dyads for which the responses yui are observed, λU and λI are regulariza- tion weights.
- Logistic Another popular formulation [24, 1] is to use logistic regression by optimizing the cross-entropy:
||Ψi||2
3.
Collaborative Games
Based on the perspective of user-item interactions, the matrix completion formulation for recommendation has led to numerous algorithms which excel at a number of data sets, including the prize-winning work of and many other successful collaborative filtering algorithms [27, 22, 26, 1, 15, 31, 17]. However, as we discussed, this formulation is inherently flawed; instead, it is advantageous to model the user-system interactions so as to capture the interdependent 2We assume each latent factor φ contains a constant compo- nent so as to absorb user/item-specific offset into the latent factor φ and ψ.
Table 1: An Example Trace Of User-System Interac-
tions in recommendation. decision-making process between the system and the users. This motivates a novel game-theoretic formulation for rec- ommendation and opens up a promising direction that en- able us to optimize recommendation policy strategically in respect of important business objectives, which cannot be achieved otherwise with the conventional matrix completion formulation.
Consider a typical scenario of user-system interaction in a recommender system: we have N users u ∈U := {1, 2, . . . , N} and M items i ∈I := {1, 2, . . . , M}; when a user u visits the site, the system recommends a set of items A = {i1, . . . , il} and u in turn chooses a (possibly empty) subset R ⊆A for consumption (e.g. buys some of the recommended products).
From now on, we refer to A as action, and R as reaction. For simplicity, we assume each action is fixed-size with a given length, |A| = l, and that each reaction is either empty or contains exactly one choice, |R| = 1 or 0. Therefore, we have A ∈A = Il and R ∈R ⊂˜I = I ∪{∅}. Table 1 shows an example trace of such interactions.
The behavior of the recommender system and that of the users are interdependent.
On The One Hand, Since People
make different decisions when facing different contexts, a user’s decision R depends crucially on the action of the sys- tem, A, (i.e., what was provided to him). For instance, an item i would not have been chosen by u if it were not pre- sented to him at the first place; likewise, user u could choose another item if the context A changes such that a better item were recommended to him. On the other hand, how a rec- ommender system acts also depend on user’s behavior (i.e., response), because the success of recommendation (i.e., in terms of click-through, revenue, etc.) is defined directly on how users react to it (e.g., purchase a product, click an ad, rent a movie). It is therefore nature to formulate recom- mendation based on game theory, as analogy to a monopoly market where the recommender as the sole seller, a user as a buyer and the items as the goods.
Formally, the user-system interactions in a recommender system can be formulated as a set of N non-cooperative games G = {Gn = (Pn, Zn, Un), n = 1, 2, . . . , N}. For each game Gn, the player set Pn = {S, un} consists of two players, i.e., the system (i.e., seller) S and a user (i.e., buyer) un; the policy space Zn = A × R ⊂Il × ˜I is the set of all possible action-reaction pairs Zn = (An, Rn), where Z is called an outcome and Z the outcome space; and the utility (i.e., payoff) function Un = {US(Zn), Uu(Zn)} consists of the system’s payoffUS and the user’s payoffUu.
At An
interaction t, a user ut visits the system and the game Gut is played with outcome Zt = (At, Rt) and utility output U(At, Rt). Since the users’ behavior is not in our control, our goal in designing a recommender system is to generate a system action (recommendations) A˜t for an incoming visit ˜t of user u˜t so as to maximize the system’s payoffUs(Z˜t).
It is important to emphasize that the games in G should not be modeled as independent games. Particularly, since the outcome space can be very large, yet observations are typically sparse, it is practically important to still be able to leverage the collaboration effect such that similar games are expected to yield similar outcomes. This way it enables us to pool the sparse evidences across different but similar games and in turn obtain reliable statistical inference. For this reason, we term the formulation “collaborative games” with a slight abuse of terminology.
This game-theoretic formulation provides a novel perspec- tive for recommendation. Particularly, since the strategies of the buyer and the seller are interdependent, to optimize the seller’s action, we have to (1) for each candidate action A, predict the buyer’s reaction R in advance; and then (2) find the best action A by maximizing the achievable payoff Us(A, R).
4.
Filtering
Our recent work established the first principled model for learning preference from user-system interactions in rec- ommendation system. Unlike conventional preference learn- ing models which are trained on the who-like-what matrix, our CCF preference model is trained on user-system interac- tions where the system action A is used as a context in which a user’s reaction (e.g., “like”) R is made; in other word, CCF model doesn’t only capture who-like-what, but it also con- siders what are the options available to the user when the “like” decision is made. As demonstrated in our experiments and many other successful applications (e.g., online test on Yahoo! and Netflix), the CCF preference model signifi- cantly improves recommendation performance on a variety of data sets. However, like many existing recommendation algorithms, our prior CCF model is still within the conven- tional matrix completion framework. To be precise, all these models only care about, and are only capable to model, the behavior of the user (i.e., what a user likes). These tech- niques are lacking as they largely ignore the interdependent or game-theoretic nature of the user-system interactions in recommendation, and consequently, none of them is able to optimize the recommendation policy explicitly in respect of a prescribed objective (although many strategic objectives for a recommender system are clearly defined).
In this paper, we extend our prior work and present a game-theoretic framework for recommendation. We would like to keep the name “Collaborative-Competitive Filtering” or CCF since the preference model we established in our prior work is revised and used as one essential component of this framework. The CCF framework consists of two compo- nents: (1) a model Pu(R|A) that predicts in advance (with uncertainty) a buyer’s reaction R to a given action A; and (2) a formulation for finding the best action strategy (i.e., recommendation policy) for the seller.
Conditional User Reaction Modeling
The first part of the framework is to predict a buyer’s choice R in the context of any given action A of the seller’s. In a decision environment with imperfect information, this means to quantify the conditional distribution pu(R|A). The full parametrized version of this distribution requires O(NM l+1) free parameters, statistical estimation of which is practically prohibitive since the observations are typically available only at a scale far less than O(NM) (e.g., in matrix comple- tion, usually less than 1% entries are observed). In this sec- tion, we revisit and extend our CCF preference model by presenting a conditional reaction model with complexity
Behavioral Axioms Of Choice Process
We first present an axiomatic view of the choice process. We assume a good (i.e., item) i has a potential utility rui to a buyer u. Moreover, we assume a buyer u is a rational de- cision maker: he knows that his choice of a good i will be at the expense of other available alternatives i′ ∈A, therefore he compares among all the alternatives before making his choice. In other words, for each decision, u considers both revenue and opportunity cost, and decides which good to buy based on the potential profit of each good in A. Specifically, the opportunity cost cui is the potential loss of u from buy- ing a good i that excludes him to buy other alternatives: cui = max{rui′ : i′ ∈A \ i}; the profit πui = rui −cui is the net gain of an decision. Based on the rational decision theory , we have the following axiom about the buyer’s choice reaction.
Axiom 1 [Local optimality of choice]: A rational deci- sion is a decision maximizing the profit: i∗= arg maxi∈A πui. This axiom implies a local competitive effect: the buyer u turns to chooses the good that is locally the best in the context of the available alternatives in At. Unfortunately, the axiom restricts the utility function only up to an arbi- trary order-preserving transformation (e.g. a monotonically increasing function), and hence cannot yield a unique solu- tion . Another issue is that it is deterministic, less useful since we don’t have perfect information about how users re- act. To this end, we draw an stochastic counterpart of this
Axiom From The Random Utility Theory [18, 21]:
Axiom 2 [Independence of Irrelevant Alternatives]: For any given context set A, the relative odds of a user u’s selecting an item i ∈A over another item j ∈A should be independent of the presence or absence of any irrelevant
(1)
Note that this axiom brings the parametric complexity of pu(R|A) significantly down from O(NM l+1) to O(NM 2).
User Utility Parametrization
In the spirit of the random utility theory [18, 21], we decompose the buyer’s utility function into two parts, i.e., Uu(i) = rui + eui, where: (1) rui is a deterministic compo- nent characterizing the intrinsic interest of the buyer u to the good i; (2) the second part eui is a stochastic unobserved error term reflecting the uncertainty, richness and complex- ity of the choice process.
Under Very Mild Conditions, It
has been shown that the error terms eui are independently and identically distributed with the Weibull (extreme point)
Distribution :
P(eui ⩽ǫ) = e−e−ǫ.
(2)
Furthermore, to encode the collaborative effect such that the observed evidences could be pooled across similar games, we parametrize the deterministic utilities, rui, with the multi-
(3)
where φu ∈Rk and ψi ∈Rk are low-rank latent profiles for user u and item i respectively, just as in the collaborative filtering models we described in Section 2.
The Multinomial Logit Factor Model
The behavioral axiom and the low-rank parametrization together lead to the following theorem. Theorem 1: Suppose the utility function Uu(i) = rui + ǫui, where ǫ are i.i.d. Weibull variables, then the distribu- tion of selecting one item that satisfies Axiom 2 is given by
Pu(I|A) = Erui/P
j∈A eruj for any i ∈A. Proof. c.f. .
✷
The above model is well-known as the multinomial logit model, which has been extensively used for modeling con- ventional offline consumer choice behavior (e.g., choose of occupation, brand, housing) in econometrics [21, 19], socio- metrics and marketing science [10, 12]. We adapt it for modeling online game-theoretic interactions in recommender systems. In contrast to the traditional choice models, where the deterministic part of the utility rui is a linear mapping w⊤xui of observed features xui (i.e., measured user and item features), here we employ the multiplicative latent factor parametrization.
The Formulation Proposed Hereby Seam-
lessly integrate two distinct methodologies — choice models in econometrics and factorization models in collaborative fil- tering. This integration is significant because it enables us to model the seller-buyer games collaboratively, rather than independently as in conventional choice models. That is, it enables us to pool data across games such that the inter- actions engaging similar users, similar actions and similar reactions are dealt with in a similar way.
Moreover, in conventional choice models, it is assumed that in each interaction t, the buyer will take at least one item i∗∈At. This assumption is, however, not true in our case since user’s visit to a recommender system does not always yields a response. For example, users frequently visit online e-commerce website without making any purchase, or browse a news portal without clicking on any ad. Actually, such nonresponded visits may account for a vast majority of the traffics that an recommender system receives. More interestingly, different users may have different propensities for giving a response. It is important to reflect this in the model as well. To this end, we add a scalar latent factor, θu, for each user u to capture the response propensity of the buyer u.
At An Interaction T, We Assume Buyer Ut Makes
an effective purchase only if he feels that the overall quality of the offered goods At are good enough. In other words, there is a certain reserve utility that needs to be exceeded for a user to respond. In keeping with the multinomial logit model and the latent factor parametrization, we have the
U Ψj) Otherwise (5)
which we refer to as multinomial logit factor or MLF model. Note that this new formulation reduces the parametric com- plexity of pu(R|A) significantly to linear scale, i.e., O(k(N + M) + N) ≈O(N + M), where k is the dimensionality of the latent factor φ ∈Rk and ψ ∈Rk, which is generally a small number (usually up to a few hundreds).
Position Bias
An important factor that was overlooked by the MLF model yet is important in practice is the position bias. In particular, the choice of a buyer depends not only on the utilities of the available alternatives but also on how they are placed (i.e., the positions), e.g., users usually pay atten- tions only to a few top-ranked goods and totally disregard the others. Such position bias is evident in many online de- cision making scenarios, e.g., Web search, recommendation, advertising. We extend the MLF model by adding a set of position-specific latent factors {βp ∈Rk, p = 1, . . . , l} via:
(6)
where p(i) denotes the position of item i, ⟨φ, ψ, β⟩= 1⊤(φ ◦
I=1 Φ[I]Ψ[I]Β[I] Is A Three-Way Inner Product, ◦
denotes Hadamard (aka element-wise) product.
Conditional Maximum Likelihood Estimation
Given a collection of training interactions {(ut, At, Rt)}, the latent factors, φ and ψ, can be estimated using penalized
P=1
||βp||2. where δ∅,t = 1 if Rt = ∅, or 0 otherwise.
Distributed Stochastic Optimization
Due to the use of bilinear multiplications, although the conditional likelihood is convex w.r.t.
Rui As Each Of The
objective terms is strongly concave, it is nontheless non- convex w.r.t. the latent factors φ and ψ. Moreover since the interactions evolve over time, it is desirable to have al- gorithms that are sufficiently efficient and preferably capa- ble to update dynamically so as to reflect upcoming data streams, therefore excluding offline learning algorithms such as classical SVD-based factorization algorithms or spec- tral eigenvalue decomposition methods . Here, we use a distributed stochastic gradient variant based on the Hadoop MapReduce framework. The infrastructure is analogous to what was proposed in . The basic module is a stochastic gradient descent algorithm, which loops over all the observa- tions and updates the parameters by moving in the direction defined by negative gradient. For example, for a given re- sponded session (u, A, i∗), we can carry out the following to update the latent factors on each machine separately:
#
.
Ai Adaptive Learning
This project focuses on ai adaptive learning using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.
We propose a novel high-performance and interpretable canon-
addition, unlike tree learning, DNNs enable gradient descent- ical deep tabular data learning architecture, TabNet. TabNet based end-to-end learning for tabular data which can have a uses sequential attention to choose which features to reason multitude of benefits: (i) efficiently encoding multiple data from at each decision step, enabling interpretability and more types like images along with tabular data; (ii) alleviating the efficient learning as the learning capacity is used for the most need for feature engineering, which is currently a key aspect
salient features. We demonstrate that TabNet outperforms in tree-based tabular data learning methods; (iii) learning other variants on a wide range of non-performance-saturated from streaming data and perhaps most importantly (iv) end- tabular datasets and yields interpretable feature attributions to-end models allow representation learning which enables plus insights into its global behavior. Finally, we demonstrate many valuable application scenarios including data-efficient
self-supervised learning for tabular data, significantly improv- domain adaptation (Goodfellow, Bengio, and Courville 2016), ing performance when unlabeled data is abundant. generative modeling (Radford, Metz, and Chintala 2015) and
Introduction We propose a new canonical DNN architecture for tabular
Deep neural networks (DNNs) have shown notable success data, TabNet. The main contributions are summarized as: efficiently encode the raw data into meaningful representa- enabling flexible integration into end-to-end learning. tions, fuel the rapid progress. One data type that has yet to 2. TabNet uses sequential attention to choose which fea- see such success with a canonical architecture is tabular data. tures to reason from at each decision step, enabling in-
Despite being the most common data type in real-world AI terpretability and better learning as the learning capacity (as it is comprised of any categorical and numerical features), is used for the most salient features (see Fig. 1). This under-explored, with variants of ensemble decision trees for each input, and unlike other instance-wise feature se- Why? First, because DT-based approaches have certain bene- and van der Schaar 2019), TabNet employs a single deep
fits: (i) they are representionally efficient for decision mani- learning architecture for feature selection and reasoning. folds with approximately hyperplane boundaries which are 3. Above design choices lead to two valuable properties: (i) common in tabular data; and (ii) they are highly interpretable TabNet outperforms or is on par with other tabular learn- in their basic form (e.g. by tracking decision nodes) and there ing models on various datasets for classification and re-
are popular post-hoc explainability methods for their ensem- gression problems from different domains; and (ii) TabNet ble form, e.g. (Lundberg, Erion, and Lee 2018) – this is an enables two kinds of interpretability: local interpretability important concern in many real-world applications; (iii) they that visualizes the importance of features and how they are fast to train. Second, because previously-proposed DNN are combined, and global interpretability which quantifies
architectures are not well-suited for tabular data: e.g. stacked the contribution of each feature to the trained model. convolutional layers or multi-layer perceptrons (MLPs) are 4. Finally, for the first time for tabular data, we show signif- vastly overparametrized – the lack of appropriate inductive icant performance improvements by using unsupervised bias often causes them to fail to find optimal solutions for tab- pre-training to predict masked features (see Fig. 2).
ular decision manifolds (Goodfellow, Bengio, and Courville
Why is deep learning worth exploring for tabular data?
One obvious motivation is expected performance improve- Feature selection: Feature selection broadly refers to judi- Copyright © 2021, Association for the Advancement of Artificial ciously picking a subset of features based on their useful-
Professional occupation related Investment related
Feedback from Feedback to
Feature selection Input processing Feature selection Input processing
previous step next step … …
Predicted output (whether the income level >$50k)
selection enables interpretability and better learning as the capacity is used for the most salient features. TabNet employs multiple decision blocks that focus on processing a subset of input features for reasoning. Two decision blocks shown as examples process features that are related to professional occupation and investments, respectively, in order to predict the income level.
Unsupervised pre-training Supervised fine-tuning
Age Cap. gain Education Occupation Gender Relationship Age Cap. gain Education Occupation Gender Relationship 5 2000 ? Exec-managerial F Wife 6 2000 Bachelors Exec-managerial M Husband 1 0 ? Farming-fishing M ? 2 0 High-school Farming-fishing M Unmarried
? 50 Doctorate Prof-specialty M Husband 4 50 Doctorate Prof-specialty M Husband 2 ? ? Handlers-cleaners F Wife 2 0 High-school Handlers-cleaners F Wife 5 3000 Bachelors ? ? Husband 5 3000 Bachelors Exec-managerial M Husband
3 0 Bachelors ? F ? 3 100 Bachelors Prof-specialty F Wife ? 0 High-school Armed-Forces ? Husband 2 0 High-school Armed-Forces M Husband
TabNet decoder Decision making
Age Cap. gain Education Occupation Gender Relationship Income > $50k
3 M False
level can be guessed from the occupation, or the gender can be guessed from the relationship. Unsupervised representation learning by masked self-supervised learning results in an improved encoder model for the supervised learning task.
ward selection and Lasso regularization (Guyon and Elisseeff performance with compact representations. 2003) attribute feature importance based on the entire training Tree-based learning: DTs are commonly-used for tabular data, and are referred as global methods. Instance-wise fea- data learning. Their prominent strength is efficient picking ture selection refers to picking features individually for each of global features with the most statistical information gain
to maximize the mutual information between the selected mance of standard DTs, one common approach is ensembling features and the response variable, and in (Yoon, Jordon, and to reduce variance. Among ensembling methods, random van der Schaar 2019) by using an actor-critic framework to forests (Ho 1998) use random subsets of data with randomly mimic a baseline while optimizing the selection. Unlike these, selected features to grow many trees. XGBoost (Chen and
sity in end-to-end learning – a single model jointly performs recent ensemble DT approaches that dominate most of the feature selection and output mapping, resulting in superior recent data science competitions. Our experimental results
!# + Softmax !" < % !" > % !# > & !# > &
ReLU ReLU &
$" !" − $" % −1 −$" !" + $" % −1 −1 $# !# − $# & % −1 −$# !# + $# & !"
FC FC
W: [$" , - $" , 0, 0] W: [0, 0, $# , - $# ] !" < % b: [-a $" , a $" , -1, -1] b: [-1, -1, -d $# , d $# ] !# < & !" > % !# < & [!" ] [!# ]
M: [1, 0] M: [0, 1]
(right). Relevant features are selected by using multiplicative sparse masks on inputs. The selected features are linearly transformed, and after a bias addition (to represent boundaries) ReLU performs region selection by zeroing the regions. Aggregation of multiple regions is based on addition. As C and C get larger, the decision boundary gets sharper.
for various datasets show that tree-based models can be out- constructs a sequential multi-step architecture, where each performed when the representation capacity is improved with step contributes to a portion of the decision based on the deep learning while retaining their feature selecting property. selected features; (iii) improves the learning capacity via non- Integration of DNNs into DTs: Representing DTs with linear processing of the selected features; and (iv) mimics
DNN building blocks as in (Humbird, Peterson, and McClar- ensembling via higher dimensions and more steps. ren 2018) yields redundancy in representation and ineffi- cient learning. Soft (neural) DTs (Wang, Aggarwal, and Liu Fig. 4 shows the TabNet architecture for encoding tabu- functions, instead of non-differentiable axis-aligned splits. mapping of categorical features with trainable embeddings.
However, losing automatic feature selection often degrades We do not consider any global feature normalization, but performance. In (Yang, Morillo, and Hospedales 2018), a soft merely apply batch normalization (BN). We pass the same D- binning function is proposed to simulate DTs in DNNs, by dimensional features f ∈ <B×D to each decision step, where 2019) proposes a DNN architecture by explicitly leveraging multi-step processing with Nsteps decision steps. The ith
expressive feature combinations, however, learning is based step inputs the processed information from the (i − 1)th step on transferring knowledge from gradient-boosted DT. (Tanno to decide which features to use and outputs the processed ing from primitive blocks while representation learning into sion. The idea of top-down attention in the sequential form edges, routing functions and leaf nodes. TabNet differs from is inspired by its applications in processing visual and text
these as it embeds soft feature selection with controllable data (Hudson and Manning 2018) and reinforcement learn- Self-supervised learning: Unsupervised representation relevant information in high dimensional input. learning improves supervised learning especially in small Feature selection: We employ a learnable mask M[i] ∈ has shown significant advances – driven by the judicious capacity of a decision step is not wasted on irrelevant
choice of the unsupervised learning objective (masked input ones, and thus the model becomes more parameter effi- prediction) and attention-based deep learning. cient. The masking is multiplicative, M[i] · f . We use an attentive transformer (see Fig. 4) to obtain the masks us- TabNet for Tabular Learning ing the processed features from the preceding step, a[i − 1]:
M[i] = sparsemax(P[i − 1] · hi (a[i − 1])). Sparsemax nor-
DTs are successful for learning from real-world tabular malization (Martins and Astudillo 2016) encourages sparsity datasets. With a specific design, conventional DNN building by mapping the Euclidean projection onto the probabilistic blocks can be used to implement DT-like output manifold, simplex, which is observed to be superior in performance and e.g. see Fig. 3). In such a design, individual feature selec- aligned with the goal of sparse feature selection for explain-
tion is key to obtain decision boundaries in hyperplane form, PD which can be generalized to a linear combination of features ability. Note that j=1 M[i]b,j = 1. hi is a trainable func- where coefficients determine the proportion of each feature. tion, shown in Fig. 4 using a FC layer, followed by BN. P[i] TabNet is based on such functionality and it outperforms DTs is the prior scale term, denoting how much a particular feature
Qi while reaping their benefits by careful design which: (i) uses has been used previously: P[i] = j=1 (γ − M[j]), where γ sparse instance-wise feature selection learned from data; (ii) is a relaxation parameter – when γ = 1, a feature is enforced
+ Softmax
Feature Feature …
transformer transformer
x Nsteps Features
+ Softmax
Feature …
transformer transformer Feature Feature Feature Feature transformer
Encoded representation
transformer transformer Attentive transformer … Mask transformer …
Step 2 Decision step dependent
transformer transformer
BN Feature Feature
FC BN transformer transformer
+ 0.5 0.5 0.5 Agg. Agg. Features Features FC FC + +
Reconstructed + … Feature attributes + … features
(a) TabNet encoder architecture (b) TabNet decoder architecture Feature transformer Feature Attentive transformer Shared across decision steps Decision step dependent transformer GLU
Decision step dependent Prior scales
+ 0.5 0.5 0.5
0.5 0.5 0.5
+ Attentive transformer (c) (d)
Prior scales
divides the processed representation to be used by the attentive transformer of the subsequent step as well as for the overall Attentive BN FC
output. For each step, the feature selection mask provides interpretable information about the model’s functionality, and the +
masks can be aggregated to obtain global feature transformer important attribution. (b) TabNet decoder, composed of a feature transformer block at each step. (c) A feature transformer block example – 4-layer network is shown, where 2 are shared across all decision
Prior scales
steps and 2 are decision step-dependent. Each layer is composed of a fully-connected (FC) layer, BN and GLU nonlinearity. (d) +
An attentive transformer block example – a single layer mapping is modulated with a prior scale information which aggregates Sparsemax
how much each feature has been used before the current decision step. sparsemax (Martins and Astudillo 2016) is used for BN FC
normalization of the coefficients, resulting in sparse selection of the salient features. +
to be used only at one decision step and as γ increases, more propose the aggregate.feature importance mask, Magg−b,j = flexibility is provided to use a feature at multiple decision PNsteps ηb [i]Mb,j [i]
PD PNsteps
ηb [i]Mb,j [i].2 i=1 i=1 steps. P is initialized as all ones, 1B×D , without any prior j=1
on the masked features. If some features are unused (as in self- Tabular self-supervised learning: We propose a decoder supervised learning), corresponding P entries are made 0 architecture to reconstruct tabular features from the Tab- to help model’s learning. To further control the sparsity of the Net encoded representations. The decoder is composed of selected features, we propose sparsity regularization in the feature transformers, followed by FC layers at each deci-
form of entropy (Grandvalet and Bengio 2004), Lsparse = sion step. The outputs are summed to obtain the recon-
PNsteps PB PD −Mb,j [i] log(Mb,j [i]+)
i=1 b=1 j=1 Nsteps ·B , where is a structed features. We propose the task of prediction of miss- small number for numerical stability. We add the sparsity reg- ing feature columns from the others. Consider a binary mask ularization to the overall loss, with a coefficient λsparse . Spar- S ∈ {0, 1}B×D . The TabNet encoder inputs (1 − S) · f̂ sity provides a favorable inductive bias for datasets where and the decoder outputs the reconstructed features, S · f̂ . We
most features are redundant. initialize P = (1 − S) in encoder so that the model em- Feature processing: We process the filtered features using phasizes merely on the known features, and the decoder’s last a feature transformer (see Fig. 4) and then split for the FC layer is multiplied with S to output the unknown features. decision step output and information for the subsequent We consider the reconstruction loss in self-supervised phase:
step, [d[i], a[i]] = fi (M[i] · f ), where d[i] ∈ <B×Nd and 2
PB PD (f̂b,j −fb,j )·Sb,j
a[i] ∈ <B×Na . For parameter-efficient and robust learning b=1 j=1
√ PB PB 2
. Normalization b=1 (fb,j −1/B b=1 fb,j ) with high capacity, a feature transformer should comprise layers that are shared across all decision steps (as the same with the population standard deviation of the ground truth features are input across different decision steps), as well as is beneficial, as the features may have different ranges. We decision step-dependent layers. Fig. 4 shows the implementa- sample Sb,j independently from a Bernoulli distribution with
tion as concatenation of two shared layers and two decision parameter ps , at each iteration. step-dependent layers. Each FC layer is followed by BN and eventually connected to a normalized residual √ connection We study TabNet in wide range of problems, that contain with normalization. Normalization with 0.5 helps to sta- regression or classification tasks, particularly with published bilize learning by ensuring that the variance throughout the benchmarks. For all datasets, categorical inputs are mapped
For faster training, we use large batch sizes with BN. Thus, bedding and numerical columns are input without and pre- except the one applied to the input features, we use ghost BN processing.4 We use standard classification (softmax cross (Hoffer, Hubara, and Soudry 2017) form, using a virtual batch entropy) and regression (mean squared error) loss functions size BV and momentum mB . For the input features, we ob- and we train until convergence. Hyperparameters of the Tab-
serve the benefit of low-variance averaging and hence avoid Net models are optimized on a validation set and listed in ghost BN. Finally, inspired by decision-tree like aggregation Appendix. TabNet performance is not very sensitive to most as in Fig. 3, we construct the overall decision embedding hyperparameters as shown with ablation studies in Appendix. as dout = i=1 PNsteps ReLU(d[i]). We apply a linear mapping In Appendix, we also present ablation studies on various de-
Wfinal dout to get the output mapping.1 sign and guidelines on selection of the key hyperparameters. Interpretability: TabNet’s feature selection masks can shed For all experiments we cite, we use the same training, val- light on the selected features at each step. If Mb,j [i] = 0, idation and testing data split with the original work. Adam optimization algorithm (Kingma and Ba 2014) and Glorot then j th feature of the bth sample should have no contribution uniform initialization are used for training of all models.5
to the decision. If fi were a linear function, the coefficient
Mb,j [i] would correspond to the feature importance of fb,j . Instance-wise feature selection
Although each decision step employs non-linear processing, their outputs are combined later in a linear way. We aim Selection of the salient features is crucial for high perfor- to quantify an aggregate feature importance in addition to mance, especially for small datasets. We consider 6 tabular requires a coefficient that can weigh the relative importance samples). The datasets are constructed in such a way that of each step in the decision. We simply propose ηb [i] = only a subset of the features determine the output. For Syn1-
PNd Syn3, salient features are same for all instances (e.g., the
c=1 ReLU(db,c [i]) to denote the aggregate decision con- tribution at ith decision step for the bth sample. Intuitively, if 2
Normalization is used to ensure D
P j=1 Magg−b,j = 1. db,c [i] < 0, then all features at ith decision step should have 3
0 contribution to the overall decision. As its value increases, prove the performance, but interpretation of individual dimensions
it plays a higher role in the overall linear combination. Scal- may become challenging. ing the decision mask at each decision step with ηb [i], we Specially-designed feature engineering, e.g. logarithmic trans- formation of variables highly-skewed distributions, may further
For discrete outputs, we additionally employ softmax during
training (and argmax during inference). An open-source implementation will be released.
Global: using only globally-salient features, Tree Ensembles (Geurts, Ernst, and Wehenkel 2006), Lasso-regularized model, L2X
Syn Syn Syn Syn Syn Syn
No selection .5 ± .0 .7 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .6 ± .0 Tree .5 ± .1 .8 ± .0 .8 ± .0 .6 ± .0 .7 ± .0 .7 ± .0 Lasso-regularized .4 ± .0 .5 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .7 ± .0
INVASE .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0
Global .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0 TabNet .6 ± .0 .8 ± .0 .8 ± .0 .7 ± .0 .7 ± .0 .8 ± .0
output of Syn depends on features X -X ), and global fea- Table 3: Performance for Poker Hand induction dataset. ture selection, as if the salient features were known, would give high performance. For Syn4-Syn6, salient features are Model Test accuracy (%) instance dependent (e.g., for Syn4, the output depends on ei- DT 50.0 ther X -X or X -X depending on the value of X ), which MLP 50.0
makes global feature selection suboptimal. Table 1 shows that Deep neural DT 65.1
TabNet outperforms others (Tree Ensembles (Geurts, Ernst, XGBoost 71.1
and Wehenkel 2006), LASSO regularization, L2X (Chen LightGBM 70.0 van der Schaar 2019). For Syn1-Syn3, TabNet performance TabNet 99.2 is close to global feature selection - it can figure out what Rule-based 100.0 features are globally important. For Syn4-Syn6, eliminating instance-wise redundant features, TabNet improves global feature selection. All other methods utilize a predictive model Poker Hand (Dua and Graff 2017): The task is classifica-
with 43k parameters, and the total number of parameters is tion of the poker hand from the raw suit and rank attributes of 101k for INVASE due to the two other models in the actor- the cards. The input-output relationship is deterministic and critic framework. TabNet is a single architecture, and its size hand-crafted rules can get 100% accuracy. Yet, conventional is 26k for Syn1-Syn and 31k for Syn4-Syn6. The compact DNNs, DTs, and even their hybrid variant of deep neural DTs
representation is one of TabNet’s valuable properties. (Yang, Morillo, and Hospedales 2018) severely suffer from the imbalanced data and cannot learn the required sorting and Performance on real-world datasets ranking operations (Yang, Morillo, and Hospedales 2018).
Tuned XGBoost, CatBoost, and LightGBM show very slight
as it can perform highly-nonlinear processing with its depth, Model Test accuracy (%) without overfitting thanks to instance-wise feature selection.
CatBoost 85.1 Table 4: Performance on Sarcos dataset. Three TabNet mod-
AutoML Tables 94.9 els of different sizes are considered.
Forest Cover Type (Dua and Graff 2017): The task is clas- MLP 2.1 0.14M
sification of forest cover type from cartographic variables. Adaptive neural tree 1.2 0.60M approaches that are known to achieve solid performance (AutoML 2019), an automated search framework based on TabNet-M 0.2 0.59M ensemble of models including DNN, gradient boosted DT, TabNet-L 0.1 1.75M with very thorough hyperparameter search. A single TabNet without fine-grained hyperparameter search outperforms it. Sarcos (Vijayakumar and Schaal 2000): The task is re-
gressing inverse dynamics of an anthropomorphic robot arm.
very small model is possible with a random forest. In the very and TabNet merely focuses on the relevant ones. For Syn4, small model size regime, TabNet’s performance is on par the output depends on either X -X or X -X depending parameters. When the model size is not constrained, TabNet feature selection – it allocates a mask to focus on the indi- achieves almost an order of magnitude lower test MSE. cator X , and assigns almost all-zero weights to irrelevant
features (the ones other than two feature groups). models are denoted with -S and -M. Real-world datasets: We first consider the simple task of mushroom edibility prediction (Dua and Graff 2017). Tab- Model Test acc. (%) Model size Net achieves 100% test accuracy on this dataset. It is indeed Sparse evolutionary MLP 78.4 81K known (Dua and Graff 2017) that “Odor” is the most discrim-
What is this project about?
This project covers practical implementation and research aspects of the topic using AI/ML techniques.