Enquire Now
Medical Computer Vision · Clinical Diagnostics · PyTorch / TensorFlow · GPU Optimized · 2026

Stock Market Price Prediction Lstm Capstone

Tensor Pipeline · Custom Loss Formulations · Model Quantization · Accelerated Inference — A rigorous deep learning engineering project focused on automated pathological lesion segmentation and radiological disease classification. Architected for thesis defense viva presentations, IEEE reproduction, and high-throughput production deployment.

PyTorch
Core Framework
AMP FP16
Mixed Precision
TensorRT
Quantized Serving

Contextual Quantum Neural Networks for Stock Price Prediction

(Dated: February 17, 2026)

In this paper, we apply quantum machine learning (QML) to predict the distribution of stock prices of multiple assets using a contextual quantum neural network. Our approach captures re- cent trends to predict future stock price distributions, moving beyond traditional models that fo- cus on entire historical data.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Utilizing the principles of quantum superposition, we introduce a new training technique called the quantum batch gradient update (QBGU), which accelerates the standard stochastic gradient descent (SGD) in quantum applications and improves convergence.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Consequently, we propose a quantum multi-task learning (QMTL) architecture, specifically, the share-and-specify ansatz, that integrates task-specific operators controlled by quantum labels, en- abling the simultaneous and efficient training of multiple assets on the same quantum circuit as well as enabling efficient portfolio representation with logarithmic overhead in the number of qubits.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Through extensive experimentation on S&P 500 data for Apple, Google, Microsoft, and Amazon stocks, we demonstrate that our approach outperforms quantum single-task learning (QSTL) models by effectively capturing inter-asset correlations. Our findings highlight the transformative potential of QML in financial applications, paving the way for more advanced, resource-efficient quantum algorithms in stock price prediction and other complex financial modeling tasks.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Keywords: Quantum Machine Learning, Quantum Neural Networks, Quantum Finance, Quantum Multi-

Introduction

Quantum computing is a computational paradigm that transcends the limitations of classical computing by har- nessing the principles of quantum superposition and en- tanglement. These unique features enable quantum com- puters to tackle complex problems faster than classical systems can[1, 2, 4, 5]. Owing to the potential exponen- tial scaling of computational power with the number of qubits, quantum computing is expected to revolutionize diverse sectors, including medicine, engineering, energy, and finance [6, 7]. Despite its immense potential, build- ing a fully functional quantum computer is a monumental challenge that could take years, if not decades, to achieve a clear computational advantage over classical comput- ers [8, 9]. However, the near-term applications of quan- tum computing are particularly promising in fields like finance, where its ability to process vast amounts of com- plex data with fewer resources is transformative. Even with today’s noisy intermediate scale quantum (NISQ) devices - limited by a small number of qubits and short coherence times - quantum methods can provide ap- proximate solutions to certain financial problems, making them highly relevant in the immediate future .

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Machine learning has become essential for financial tasks like, asset management , risk analysis , crash detection , and portfolio optimization . The abil- ity of machine learning algorithms to analyze massive datasets, recognize patterns, and make fast predictions provides a significant competitive edge.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Quantum Ma-

chine learning (QML) emerges at the intersection of these fields, combining quantum computing’s ability to process and represent complex states efficiently with the powerful predictive tools of machine learning . With the exponential growth of financial data, current machine learning systems are quickly reaching the boundaries of classical computational models.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In This Context, Quan-

tum algorithms present a promising alternative by offer- ing faster or higher quality solutions for specific classes of problems. Additionally, breakthroughs in quantum learn- ing theory suggest that, under certain conditions, there is a provable distinction between classical and quantum learnability . This implies that problems deemed chal- lenging for classical systems could see significant improve- ments through the adoption of QML approaches.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Quantum machine learning can be broadly categorized into two main components: parametric quantum circuit (PQC) optimization and classical-to-quantum informa- tion encoding . These categories represent two core components of QML algorithms and workflows, each ad-

2

dressing a different aspect of how classical data inter- acts with quantum systems and how quantum models are trained. PQCs are quantum circuits that contain tunable parameters, interpretable as weights of a quantum neu- ral network.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

These Parameters Are Adjusted Iteratively

to minimize or maximize a cost function, similar to how classical machine learning algorithms optimize parame- ters during training. PQCs can further be classified into two categories: variational quantum circuits (VQC) and quantum neural networks (QNN). VQCs involve a hybrid architecture where a classical computer works alongside a quantum computer in a loop, while QNNs consist of circuit architectures tailored to specific problems.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Recently, there have been a surge of research interest in these areas. For instance, hybrid architectures using parameterized quantum circuits in combination with clas- sical optimization loops have been successfully deployed for classification tasks , while support vector machines (SVM) have been utilized for data classification . In another study, quantum state space was used as the fea- ture space to improve the learnability of QML and achieve quantum advantage in classification tasks . Quantum versions of machine learning models, such as Boltzmann

Machines , Recurrent Neural Networks , Genera-

tive adversarial neural networks , reinforcement learn-

Ing , And Reservoir Computing , Have Been Exten-

sively studied. In finance, quantum machine learning has been applied to various tasks, including options pricing , time-series forecasting , and stock price prediction . A notable example is a hybrid architecture developed for financial predictions , while quantum Wasserstein generative adversarial neural networks were employed for time-series predictions on the S&P 500 .

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Unsupervised Quan-

tum machine learning has also been explored for cluster- ing and fraud detection .

On The Other Hand, Sig-

nificant progress has also been made in loading classical information onto quantum states.

For Example, Quan-

tum adversarial neural networks have been utilized to load random distributions onto quantum circuits using

Feature Maps , And Quantum Wasserstein Gans Have

achieved similar tasks with a gradient penalty, improving performance over previous approaches .

While Many Financial Computational Problems Like

portfolio management and risk analysis require training across multiple assets, only few studies have addressed this need.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

For Instance, Joint Learning Of Two Distribu-

tions was achieved in , though its direct application to financial problems like stock price prediction remains limited due to computational expense and the reliance on feature maps, which will be further explained in this paper.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

A Related Study Applied Quantum Reservoir

computing and multi-task learning , which is hindered by the complexity of quantum reservoir systems, lacking generalization.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In this paper, we aim to address the challenge of train- ing a parameterized quantum circuit over multiple assets on a single quantum device by utilizing minimal resources and optimizing various components of the training pro- cess. The contributions of this work are as follows: 1. We adopted fidelity loss over quantum represen-

Coded Context Distribution And Empirical Context-

continuation distribution to accelerate convergence and improve convergence quality compared to train-

Ing By Stochastic Gradient Descent (Sgd) Through

expensive reconstruction of classical distributions.

Tion Supported) Backpropagation, Across Different

loss functions.

(Qmtl) Architecture - Share-And-Specify Ansatz -

for predicting the stock price distribution over a portfolio of assets. By achieving logarithmic over-

Head In The Number Of Assets In The Portfolio And

controllable scaling in the size of the context, we can load highly nontrivial non-static distributions

Gorithms For Speed-Up Like Quantum Linear System

solving at inference time.

Section Ii Gives

a brief background of the required terminology used throughout this paper. Section III gives an overview of loading classical data onto quantum registers.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Section

IV and V introduces quantum single-task and multi-task learning respectively followed by numerical simulations in Section VI.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Background

In this section, we introduce all the background infor- mation needed to understand this paper including the time-series prediction model and contextual Quantum Neural Networks (QNN) for modeling asset futures.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

3

Figure 1. Quantum Neural Networks for Contextual Sequence Generation. Given historical data of context and continuations, a Quantum Neural Network is trained to produce quantum distributions over future prices, enabling utilization of quantum advantageous algorithms, such as quantum risk analysis , downstream for tasks (labeled M in the rightmost diagram at a particular future) over all sequences in superposition.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Time-Series Prediction

A financial asset price prediction model is a time-series forecasting model designed to predict future asset prices by leveraging historical numeric price data and additional contextual information . In this section, we outline the mathematical framework and notations that will be used subsequently to develop a quantum machine learn- ing (QML) model for multi-asset price prediction.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

For

any given financial asset, the associated contextual string represents a sequence of numeric values corresponding to the asset over a specific time frame. Typically, this se- quence is formed by considering a series of consecutive past asset prices within the designated time window. In this work, we utilize asset price data derived from the S&P 500 index, which provides historical stock prices for corporations listed on the index. Stock prices are inher- ently volatile, influenced by market activity, news, and other external factors. These short-term fluctuations of- ten introduce noise that can adversely affect model per- formance.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Therefore, In This Paper, We Preprocess The

stock prices by computing the finite difference between consecutive stock prices to capture the moving difference (returns). This is followed by performing a moving aver- age, smoothing out these short-term fluctuations, reveal- ing the underlying trends.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

The value or movement of an asset at a time, denoted by t, is represented by a random variable Xt and we de- note Xt = xt when it attains a value xt. Thus, a con- text string of T ≥1 numeric values is represented by a random vector X(T ) = (X1, X2, . . . , XT ).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Then The

task of a prediction model f is to predict the value(s) of X(T +τ)\X(T ) := (XT +1, XT +2, . . . , XT +τ) as a proba- bility distribution f(X(T +τ)\(T ); X(T )) for some values of the future time τ ≥0. To be specific, given X(T ) = x(T ), the task of the prediction model is to assign probabilities to future states x(T +τ)\x(T ) := (xT +1, xT +2, . . . , xT +τ), the possible asset-price at the future times between T + 1 and T + τ, as f(x(T +τ)\T ; x(T )) and the effi- ciency of f is determined by the closeness of a dis- tance between f(x(T +τ)\T ; x(T )) and the observed dis- tribution over x(T +τ)\x(T ) to zero. Thus the design of a prediction model f(X(T +τ)\(T ); X(T ), θ) with param- eters θ is concerned with modeling f such that the loss function L(f(x(T +τ)\(T ); x(T )), x(T +τ)\x(T )), which esti- mates a notion of closeness between the probability dis- tribution f(X(T +τ)\(T ); X(T )) and observed frequencies X(T +τ)\X(T ) is minimized for all or a collection of as- sets in a financial market. In most occasions dealing with time-series data models, the loss function is considered as the mean squared error (MSE), binary cross entropy or any other custom function specifically designed for the task.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In our proposal of developing a QML model for multi- asset price prediction, it is customary to encode the con- text string data x(T ) into a quantum state which will be evolved under a unitary transformation. We define the

(1)

to encode x(T ) after encoding the context data point xt into its corresponding qudit |xt⟩for 1 ≤t ≤T. Here, ⊗ denotes the Kronecker product (also called tensor prod- uct) of two vectors. In this framework, the state of the qudit, |xt⟩, represents the quantized return of an asset at

4

time t. The mapping of returns to the orthogonal states |0⟩, |1⟩, . . . , |d −1⟩is determined by dividing the range of possible returns into d discrete intervals. Each inter- val corresponds to one of the orthogonal basis states of the qudit. For instance, let the minimum and maximum returns between 1 ≤t ≤T be xmin and xmax, then the price range is divided into d equal intervals of length

D −1

. For any price xt, its corresponding qudit state |xt⟩is de-

,

it ∈{0, 1, . . . , d −1}.

(2)

This mapping ensures that each basis state |it⟩rep- resents a specific quantized interval of prices.

Over

time, as the asset price evolves, the state of the qu-

∈

Cd transitions between the basis states.

Θ

designed to capture the stochastic behavior of returns and generate an approximation to the quantum state

Such That Direct Measurement Samples From The Un-

derlying distribution.

Is A T + Τ-Qudit

basis state with ϕt ∈{0, . . . , d −1} and c(ϕ) ∈C such

That P

ϕ |c(ϕ)|2 = 1.

Now, The Prediction Is A Density

operator after taking the partial trace over the context:

Y(T +Τ)Ed

y(T +τ) .

(4)

Note that the prediction of future returns constitutes only the last τ qudits. However, in our situation, the QNN

),

making it not suitable for reuse for repeated predictions. To circumvent that, we require the whole input and out-

) To Be Involved In The

loss function to make sure that the context qudits are un- changed. With this, we can approximate the prediction

X(T )E Φt +1

. . .

Φt +1

. . .

, Which

is a superposition of all possible outcomes with different

Φt +T

→xmin + ϕT +t · ∆x.

(6)

These mapped values can then be used to calculate the expected value or movement of the stock prices by com- bining with the probability (|c(ϕ)|2) of each possible out- come. Measuring the last τ qudits over the computation basis states yields a sample from the underlying probabil-

Ity Distribution Over Futures By The Model:

f(ϕT +1, . . . , ϕT +τ; x(T ), θ) = |c(ϕ)|2.

(7)

Given M measurement samples, we define the resulting distribution fM(X(T +τ)\(T ); x(T )) based on the frequency of observing specific continuation strings x(T +τ)\(T ); in the high sample limit fM ≈f. We can also estimate the most probable future outcomes, which would be useful in repeated predictions. If we wish to predict a future state, τR + T, we can use R repeated application of the underlying QNN to generate a superposition over those outcomes. Fig. 1 shows how a contextual QNN can be utilized for such time-series based predictions.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

,

In this paper, we are particularly interested in τ = 1 sce- nario along with binary quantization (d = 2), for which the qudits are reduced to qubits such that |0⟩and |1⟩ correspond to the negative and positive stock price move- ment respectively. Hereafter, we stick to the binary quan- tization d = 2 throughout the paper unless otherwise stated. This choice provides a simple and interpretable

Mapping Between Price Movements And Quantum Mea-

surement outcomes, where each qubit directly represents an upward or downward movement in returns. Using bi- nary encoding allows us to focus on the model’s ability to learn and reproduce the underlying conditional proba- bility distributions, without additional complexity intro- duced by higher-dimensional encoding schemes.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

More-

over, binary quantization keeps the circuit depth and the

5

number of qubits manageable, which is particularly im- portant when studying scalability across multiple assets. By reducing representational overhead, we can isolate and analyze the learning dynamics of QMTL itself, establish- ing a clear baseline before extending the framework to multi-level quantization in later experiments.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In Addi-

tion, during training, we also consider the statistics of the contextual data including the contextual probabil- ity distribution P(X(T )), the target probability distri- bution P(X(T +1)), the conditional probability distribu- tion P(XT +1|X(T )), and the total probability distribu- tion P(XT +1, X(T )). In particular, we train θ such that f(XT +1; X(T ), θ) ≈P(XT +1|X(T )).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Now We Recall That A Qnn Architecture On An N-

qubit register represents a parametrized unitary matrix ˆU(θ) of dimension 2n, which is defined by a sequence of parametrized quantum gates that produces an n-qubit output state ˆU(θ) |Ψ⟩for any input state |Ψ⟩∈C2n, where θ is the set of (real) parameters in the QNN and |Ψ⟩encodes a classical input data for the problem. The parameters in θ can be learned and trained to produce a desired output state which is approximated by perform- ing several quantum measurements to all or a subset of the qubits. Deciding the parametrized quantum circuit (PQC), also known as ansatz which represents ˆU(θ) in a QNN model is one of the fundamental problems in QML applications. In this section, we will introduce the impor- tant components of a QNN.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

1.

Loading Classical Data

The first step in a QNN involves encoding classical data into quantum states. This is typically done using

Quantum Feature Maps , Where Classical Input Data

Ψ ∈Cγ, 1 ≤γ ≤2n is encoded into a quantum state |Ψ⟩, where 2n is its dimension. This state can be prepared by

(8)

where ˆUF (Ψ) is a quantum feature map circuit that de- pends on the classical data Ψ, and |0⟩⊗n represents the initial quantum state. A feature map consists of a set of controlled rotation gates, parameterized by the contents in the classical register Ψ.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Different Feature Maps Can

be employed depending on the application to project the classical data on the quantum state space. Some of the common feature maps used are the first and second-order Pauli-Z evolution circuits .

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

2.

Parametric Quantum Circuits

Once the classical data is encoded into a quantum state, it is processed by a PQC. The goal of the training process is to optimize the parameters θ such that the PQC out- puts a quantum state that corresponds to accurate pre- dictions for the learning task. In this paper, inspired from the traditional layered neural networks, we focus on PQCs with L repeated layers, composed on fixed and parameter- ized gates such that the unitary represented by the circuit

J Glij(Θlij) As Shown In

Fig. 2, where V l is a fixed unitary at layer l (which could be identity or a sequence of CNOT gates) and Glij(θlij) denote a single-qubit rotation gate acting at layer l, at po- sition (i, j). Here, i ∈[1, n] and j ∈[1, c], with i and j cor- responding to the row and column indices of the quantum circuit, respectively as shown in Fig. 2. In this notation,

Θ = Ll

l=1 θl, where θl = {θl11, θl12, . . . , θlnc} such that m(= Lnc) is the total number of parameters in the circuit, n(= m/Lc) is the number of of qubits and c(= m/Ln) is the number of sub-layers in each layer.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

This Represen-

tation will be explored in detail in later sections. Each parameter θlij ∈θ can correspond to the angles of single qubit rotation gates such as RX(θlij), RY (θlij), RZ(θlij).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In the proposed QML model in this paper, along with the

, We Need Ancilla Qubits To

extract information from the output state of a QNN for the learning task. The ancilla qubit states control the ap- plication of certain quantum gates on the main quantum register to obtain a desired output state. Assuming that there are τ ancilla qubits, the QNN represents a unitary matrix ˆU(θ) with dimension 2T +τ ×2T +τ, resulting in the number of qubits n = T + τ. Setting the initial state of the ancilla register as |0⟩⊗τ , the output state is given by

L ˆU L(Θl) Is The Unitary Operator Corre-

sponding to the parametric quantum circuit. In addition to rotation gates, our circuit may include fixed gates, such as CNOT gates, that help spread entanglement in the sys- tem.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

3.

Training

Classical neural networks are primarily trained by back- propagation, which is not directly possible in QNNs.

6

Figure 2. Parameterized Quantum Circuit. Block dia- gram of the layered parametric quantum circuit showing vari- ous blocks in the lth layer such as fixed unitary and parametric rotation gates.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Quantum states collapse upon measurement, meaning the quantum information is destroyed. Backpropagation re- lies on preserving intermediate computations (like activa- tions) during the forward pass for use in the backward pass. In quantum systems, measurements required to ex- tract information disrupt the state, preventing reuse. In addition, the no-cloning theorem prohibits copying quan- tum states for reuse, further complicating backpropaga- tion. Consequently, QNNs have adopted to other tech- niques for gradient computation such as parameter shift and simultaneous perturbation stochastic approxima- tion (SPSA) . Unlike backpropagation, which requires explicit differentiation through each layer, these methods compute gradients by evaluating the quantum circuit at slightly shifted parameter values. It works based on the fact that the output of quantum circuits often depends on the parameters through trigonometric functions (such as sine and cosine), enabling exact gradient computation.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

=

x(T +τ) ˆU †(θ) ˆB ˆU(θ)|x(T +τ)⟩.

(11)

where δ is hyperparameter, typically between 0 and 1 and α is a stochastic perturbation vector with dimen- sions same as θ. The vector α is sampled from a zero Figure 3. SWAP Test. A diagram of the SWAP test, which measures the fidelity loss between two wavefunction states

Y(T +Τ)E

. mean distribution such that αlij ∈{−1, 1}. The primary difference between the two approaches is in parameter up- dates: parameter-shift updates one parameter at a time, requiring 2m evaluations for m parameters, while SPSA updates all parameters simultaneously with just two eval- uations per iteration.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

4.

Loss Functions

Different loss functions are employed depending on the nature of the QNN and the gradient update rule.

For

instance, mean squared error (MSE) is commonly used in regression tasks due to its smooth gradients and sim- plicity in optimization. However, for classification tasks, cross-entropy loss might be preferred as it better captures the probabilistic nature of the output. Therefore, we in- troduce different loss functions that are relevant to our paper.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Mean squared error loss is the most common loss func- tion used for optimization tasks and in our context, it is

Defined As:

Lm(θ, x(T +τ)) = |fM(x(T +τ)\(T ); x(T )) −δ(x(T +τ))|2.

(12)

Note that for the MSE loss, full state measurement is required i.e., all the qubits must be measured and the returns have to retrieved from the output state vector as per Eq. (6). To circumvent this, we use the fidelity loss or

2 (|0⟩+ |1⟩), Whose Logical State Is

then used to control SWAP the wavefunction (e.g. qubit by qubit). Thereafter, a Hadamard gate is applied on the ancilla, leading to a phase kick back, and then the ancilla is measured in the computational (Pauli-Z) basis.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

7

Figure 4. Loading a distribution onto a hardware ef- ficient ansatz. The circuit inside the blue box ( ˆO(α, β)) is applied sequentially for a sufficient number of iterations fol- lowed by an MSE loss with the SPSA update rule.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

of the ancilla qubit is proportional to the fidelity

Loading Classical Data

Contextual data x(T ) can be encoded onto a quantum state using feature maps , as described in Eq. (8). However, feature maps are computationally intensive and slow to train due to the convoluted nature of ˆUf. More- over, training requires encoding a different contextual in- put during each iteration, further increasing the complex- ity.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

To Address This Challenge, We Propose Loading The

entire contextual distribution onto the quantum state us- ing a hardware-efficient ansatz . By encoding the data only once, our approach achieves significantly faster train- ing. In this section, we discuss the procedure for encoding the contextual probability distribution P(X(T )), where X(T ) represents the contextual time-series data of an as- set, onto a quantum state using the hardware efficient ansatz architecture.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Once the data has been preprocessed, we construct the

(2) To Obtain A Basis For The

T-qubit representation of the returns, where the quan- tization levels for the returns are also chosen. Now, we need to represent the stochastic behavior of the asset in the space defined by the basis vectors such that the in- formation can be used to train the QNN model and make reasonable predictions.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

To This End, We Normalize The

preprocessed data and compute its histogram to obtain the contextual probability distribution of the quantized returns as P(X(T ) = (i1, . . . , iT )), where it denotes the quantized return at time t. Thereafter, we load the con- textual distribution onto the T-qubit state as follows

(15)

where P(i1, . . . , iT ) is the value of the quantized distribu-

It

. Note that, using the definition of x(T ) and equations (1)

Eq. (16) Can Be Achieved Using The Grover-Rudolph

technique for state preparation . However, due to its higher computational complexity, we adopt a simpler ma- chine learning-based approach for state preparation. This method utilizes a hardware-efficient ansatz combined with a mean squared error (MSE) loss function (Eq. (12)), where the parameters of the rotation gates are iteratively updated in a loop using the SPSA rule to minimize the loss function, as illustrated in Fig.4. This circuit is chosen due to its hardware-efficient architecture , which con- stitutes a repeated layers of RY , RZ, and CNOT gates to perform rotations and entanglement. Although several loss functions exist for comparing distributions, we choose MSE due to its smooth gradients and computational ef- ficiency, enabling faster optimization. For each layer, the

(17)

where CNOTj,(j+1)%T acts on qubits j (control) and (j + 1)%T (target) and RY (αi), RZ(βi) are 2 × 2 unitary matrices acting on the jth qubit. To enhance the accuracy of loading the distribution, multiple such transformations are applied iteratively, as illustrated in the figure. This approach improves the learnability of the circuit, thereby increasing the fidelity of the loaded distribution.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Note

that the parameters in the circuit {{αi} ∪{βi}} are opti- mized through the training process to accurately load the contextual distribution.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

8

Figure 5. Quantum Batch Learning. A diagram showing the learning procedure of our proposed quantum batch gradi- ent update for a context of T. The top most qubit is an ancilla qubit. For a batch, a distribution over inputs is loaded suc- ceeding qubits (the input qubits) and the following qubit(s) is for the output. The joint distribution of the inputs and out- puts for the batch is loaded on the subsequent qubits. The circuit inside the Grey box ( ˆU(θ)) is applied sequentially for a required number of iterations, is a contextual quantum neural network to prepare an approximate of the loaded joint distri- bution.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

A SWAP test is then used to take the fidelity loss between the two distributions.

Quantum Single-Task Learning

Now that we have encoded the contextual distribution into the quantum states, we can move on to discussing predictions based on the given context. Before diving into quantum multi-task learning for predicting multiple asset prices, we’ll first present a complete case of using quantum circuits for predicting the price of a single stock, which we will refer to as quantum single-task learning (QSTL).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Unlike previous approaches [35, 37], which rely only on using the entire historical data to make predictions, we in- corporate contextual information as well to forecast future outcomes, as shown in Fig. 5. This method offers the ad- vantage of adapting to the constantly changing stock mar- ket, whereas past methods may become less relevant due to outdated data. In our approach, the contextual distri-

, Which,

along with a prediction qubits, forms the input of the

⊗|0⟩⊗Τ. Hereafter, We Will Stick

to τ = 1 and binary quantization unless otherwise stated.

To The Last

T + 1 qubits, which serves as an input to the SWAP test. The layered PQC forms the bulk of the circuit followed by the SWAP test between the predicted distribution and target distribution (as shown in Fig.5). Finally, measure- ment is done on the ancillary qubit, obtaining the fidelity loss, which guides the training process through SPSA gra- dient update rule.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

A unitary transformation ˆU(θ), where θ is the trainable parameters, enables the circuit to learn the conditional probability distribution represented by P(XT +1|x(T ), θ).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

This probability distribution serves as the desired output state of the PQC at the end of the training process, allow- ing for the prediction of future returns via measurement.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Learning conditional probability distributions is crucial for time-series prediction as they capture the dependency between past context and future outcomes. In contrast, marginal distributions are unsuitable for time-series pre- diction because they lack the ability to incorporate tem- poral dependencies and contextual information.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Quantum Data Processing Has Been Explored For Ma-

chine Learning due to the inherent memory compres- sion associated. For example Ref. utilizes access in superposition to models for quantum speedup in tasks like k-means clustering, while Ref. achieves quantum speedup for Support Vector Machines by learning over a superposition of data points.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In The Context Of Qnns,

Ref. recently considered a construction for parallel quantum batches in which a reduced density matrix is constructed over the ancilla, output, and label space. In comparison, our approach loads the entire contextual dis- tribution at once according to Eq. (16), resulting in a straight forward loss function that leads to higher quality gradients. Then our QNN generates an approximate rep- resentation of the continuation for each context in super- position. This leverages the linearity of quantum circuits, allowing a superposition of all possible inputs to train the circuit without breaking the correspondence between the respective inputs and outputs. This correspondence reduces the multi-step stochastic gradient descent to a single step.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

For example, consider Eq. (9), when the input of the



as shown in Fig. 5. Let the gradient update for θ, obtained from SPSA, be denoted as g(θ, x(T +1)), with

, (18)

where the last step is obtained by using Eq. (11) with

(19)

where β is the learning rate. Similarly, if the input to the PQC is a distribution of basis states (superposition of all

Is The Total Distribution Learned By

the circuit ˆU(θ) over the contextual distribution

Ψ(T )E

. Given that the gradient update corresponding to the out-

X(T )∈{0,1}T

P(x(T )) g(θ, x(T +1)).

(20)

This expression (Eq. 20) effectively corresponds to ap- plying stochastic gradient descent (SGD) across all input context samples from the dataset, achieving a single gra- dient update after processing the entire dataset. Notably, this result is obtained in one step due to the inherent linearity of quantum mechanics, facilitating the quantum circuit to train on large batches. Using the QBGU train- ing process, the model ˆU(θ) in Fig. 5 is trained to learn Figure 6.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Share-and-specify Ansatz.

A Diagram Of Our

Quantum Multi-Task Learning Architecture showing various components for the QNN. A single asset state is loaded by setting the label |k⟩and the inference-time context x(T ) is loaded over qubits |0⟩⊗T +τ. The input can then be processed through the parameterized circuit, composed of L layers of

That Can

be utilized for a downstream task. For example, as depicted in the figure, measurement can be used to sample possible continuations xτ.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

the conditional probability distribution P(xT +1|x(T ), θ),

Y(T +1)

at the time of inference.

As

shown in Fig. 5, we preload both the contextual distri-

, And

the SWAP test then measures the distance between the estimated and the original target distributions, guiding the training process.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Multi-Task Learning (Mtl) Is A Machine Learning

approach where multiple tasks are learned at the same time, allowing the model to share information between them. It uses shared parameters to capture common pat- terns across all tasks, while task-specific parameters fo- cus on unique aspects of each task. In financial time se- ries prediction, MTL can be used to predict the prices or trends of multiple assets together.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Shared Parame-

ters can represent factors that affect the entire market, such as economic indicators, while task-specific parame- ters account for unique characteristics of each asset, like individual volatility or trading patterns. This helps the model make better predictions by learning both shared and asset-specific information. In this section, we extend these ideas to QNNs by introducing the quantum multi- task learning (QMTL).

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

In Order To Incorporate Mtl In A Qnn Framework,

we introduce the share-and-specify ansatz (Fig. 6) which

10

Figure 7. Quantum Multi-Task Learning Architecture for T = 3 and K = 2. The circuits inside the Grey ( ˆUs(θs)) and

2 (Θ1

2)) are applied sequentially for a required number of iterations followed by the SWAP test. breaks each layer of the PQC into a block of universal gates (shared ansatz), followed by a block of asset spe- cific label-controlled gates acting based on the state of the label registers (specify ansatz). The share ansatz for each layer can be embodied by the same gates used in the single-asset task, leading to identity operations applied to

(21)

where 1K is the identity operator with dimension K ×K. The label qubits, collectively represented by the qudit |k⟩= |k1⟩⊗|k2⟩· · · ⊗|klog K⟩with kj ∈{0, 1}, are used to distinguish the assets. Each asset is assigned a unique label k ∈[1, K], ensuring that the quantum operations are applied selectively to the corresponding asset based on its label. These label qubits are then used to form the

(22)

resulting in the specify ansatz (with control) for asset k

1K −|K⟩⟨K|

is a projection onto the orthogonal space of |k⟩⟨k|,

K ) Is Only Applied When The La-

bel qudit is |k⟩.

Is A Control Rotation Based On

label |k⟩. We can then define the specify ansatz for one

(24)

where the dimension of ˆU(θ) is 2T +τK × 2T +τK. Note

, Undergoes

the transformation determined by the label qudit and the

⊗|0⟩⊗Τ 

⊗|k⟩.

S Helps Facilitate Learn-

ing across assets while the specify ansatz layer ˆU l

K Al-

lows focus on potential peculiarities of an individual asset. This entire architecture is demonstrated in Fig. 6. 1.

stock market price prediction lstm capstone Diagram
Figure: System Model & Simulation Flow for Stock Market Price Prediction Lstm Capstone

Two-Assets Case

Extending the previous framework to handle two as- sets, the forward pass equations for a set of inputs

Ai Adaptive Learning

This project focuses on ai adaptive learning using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

We propose a novel high-performance and interpretable canon-

addition, unlike tree learning, DNNs enable gradient descent- ical deep tabular data learning architecture, TabNet. TabNet based end-to-end learning for tabular data which can have a uses sequential attention to choose which features to reason multitude of benefits: (i) efficiently encoding multiple data from at each decision step, enabling interpretability and more types like images along with tabular data; (ii) alleviating the efficient learning as the learning capacity is used for the most need for feature engineering, which is currently a key aspect

salient features. We demonstrate that TabNet outperforms in tree-based tabular data learning methods; (iii) learning other variants on a wide range of non-performance-saturated from streaming data and perhaps most importantly (iv) end- tabular datasets and yields interpretable feature attributions to-end models allow representation learning which enables plus insights into its global behavior. Finally, we demonstrate many valuable application scenarios including data-efficient

self-supervised learning for tabular data, significantly improv- domain adaptation (Goodfellow, Bengio, and Courville 2016), ing performance when unlabeled data is abundant. generative modeling (Radford, Metz, and Chintala 2015) and

Introduction We propose a new canonical DNN architecture for tabular

Deep neural networks (DNNs) have shown notable success data, TabNet. The main contributions are summarized as: efficiently encode the raw data into meaningful representa- enabling flexible integration into end-to-end learning. tions, fuel the rapid progress. One data type that has yet to 2. TabNet uses sequential attention to choose which fea- see such success with a canonical architecture is tabular data. tures to reason from at each decision step, enabling in-

Despite being the most common data type in real-world AI terpretability and better learning as the learning capacity (as it is comprised of any categorical and numerical features), is used for the most salient features (see Fig. 1). This under-explored, with variants of ensemble decision trees for each input, and unlike other instance-wise feature se- Why? First, because DT-based approaches have certain bene- and van der Schaar 2019), TabNet employs a single deep

fits: (i) they are representionally efficient for decision mani- learning architecture for feature selection and reasoning. folds with approximately hyperplane boundaries which are 3. Above design choices lead to two valuable properties: (i) common in tabular data; and (ii) they are highly interpretable TabNet outperforms or is on par with other tabular learn- in their basic form (e.g. by tracking decision nodes) and there ing models on various datasets for classification and re-

are popular post-hoc explainability methods for their ensem- gression problems from different domains; and (ii) TabNet ble form, e.g. (Lundberg, Erion, and Lee 2018) – this is an enables two kinds of interpretability: local interpretability important concern in many real-world applications; (iii) they that visualizes the importance of features and how they are fast to train. Second, because previously-proposed DNN are combined, and global interpretability which quantifies

architectures are not well-suited for tabular data: e.g. stacked the contribution of each feature to the trained model. convolutional layers or multi-layer perceptrons (MLPs) are 4. Finally, for the first time for tabular data, we show signif- vastly overparametrized – the lack of appropriate inductive icant performance improvements by using unsupervised bias often causes them to fail to find optimal solutions for tab- pre-training to predict masked features (see Fig. 2).

ular decision manifolds (Goodfellow, Bengio, and Courville

Why is deep learning worth exploring for tabular data?

One obvious motivation is expected performance improve- Feature selection: Feature selection broadly refers to judi- Copyright © 2021, Association for the Advancement of Artificial ciously picking a subset of features based on their useful-

Professional occupation related Investment related

Feedback from Feedback to

Feature selection Input processing Feature selection Input processing

previous step next step … …

Predicted output (whether the income level >$50k)

selection enables interpretability and better learning as the capacity is used for the most salient features. TabNet employs multiple decision blocks that focus on processing a subset of input features for reasoning. Two decision blocks shown as examples process features that are related to professional occupation and investments, respectively, in order to predict the income level.

Unsupervised pre-training Supervised fine-tuning

Age Cap. gain Education Occupation Gender Relationship Age Cap. gain Education Occupation Gender Relationship 5 2000 ? Exec-managerial F Wife 6 2000 Bachelors Exec-managerial M Husband 1 0 ? Farming-fishing M ? 2 0 High-school Farming-fishing M Unmarried

? 50 Doctorate Prof-specialty M Husband 4 50 Doctorate Prof-specialty M Husband 2 ? ? Handlers-cleaners F Wife 2 0 High-school Handlers-cleaners F Wife 5 3000 Bachelors ? ? Husband 5 3000 Bachelors Exec-managerial M Husband

3 0 Bachelors ? F ? 3 100 Bachelors Prof-specialty F Wife ? 0 High-school Armed-Forces ? Husband 2 0 High-school Armed-Forces M Husband

TabNet decoder Decision making

Age Cap. gain Education Occupation Gender Relationship Income > $50k

3 M False

level can be guessed from the occupation, or the gender can be guessed from the relationship. Unsupervised representation learning by masked self-supervised learning results in an improved encoder model for the supervised learning task.

ward selection and Lasso regularization (Guyon and Elisseeff performance with compact representations. 2003) attribute feature importance based on the entire training Tree-based learning: DTs are commonly-used for tabular data, and are referred as global methods. Instance-wise fea- data learning. Their prominent strength is efficient picking ture selection refers to picking features individually for each of global features with the most statistical information gain

to maximize the mutual information between the selected mance of standard DTs, one common approach is ensembling features and the response variable, and in (Yoon, Jordon, and to reduce variance. Among ensembling methods, random van der Schaar 2019) by using an actor-critic framework to forests (Ho 1998) use random subsets of data with randomly mimic a baseline while optimizing the selection. Unlike these, selected features to grow many trees. XGBoost (Chen and

sity in end-to-end learning – a single model jointly performs recent ensemble DT approaches that dominate most of the feature selection and output mapping, resulting in superior recent data science competitions. Our experimental results

!# + Softmax !" < % !" > % !# > & !# > &

ReLU ReLU &

$" !" − $" % −1 −$" !" + $" % −1 −1 $# !# − $# & % −1 −$# !# + $# & !"

FC FC

W: [$" , - $" , 0, 0] W: [0, 0, $# , - $# ] !" < % b: [-a $" , a $" , -1, -1] b: [-1, -1, -d $# , d $# ] !# < & !" > % !# < & [!" ] [!# ]

M: [1, 0] M: [0, 1]

(right). Relevant features are selected by using multiplicative sparse masks on inputs. The selected features are linearly transformed, and after a bias addition (to represent boundaries) ReLU performs region selection by zeroing the regions. Aggregation of multiple regions is based on addition. As C and C get larger, the decision boundary gets sharper.

for various datasets show that tree-based models can be out- constructs a sequential multi-step architecture, where each performed when the representation capacity is improved with step contributes to a portion of the decision based on the deep learning while retaining their feature selecting property. selected features; (iii) improves the learning capacity via non- Integration of DNNs into DTs: Representing DTs with linear processing of the selected features; and (iv) mimics

DNN building blocks as in (Humbird, Peterson, and McClar- ensembling via higher dimensions and more steps. ren 2018) yields redundancy in representation and ineffi- cient learning. Soft (neural) DTs (Wang, Aggarwal, and Liu Fig. 4 shows the TabNet architecture for encoding tabu- functions, instead of non-differentiable axis-aligned splits. mapping of categorical features with trainable embeddings.

However, losing automatic feature selection often degrades We do not consider any global feature normalization, but performance. In (Yang, Morillo, and Hospedales 2018), a soft merely apply batch normalization (BN). We pass the same D- binning function is proposed to simulate DTs in DNNs, by dimensional features f ∈ <B×D to each decision step, where 2019) proposes a DNN architecture by explicitly leveraging multi-step processing with Nsteps decision steps. The ith

expressive feature combinations, however, learning is based step inputs the processed information from the (i − 1)th step on transferring knowledge from gradient-boosted DT. (Tanno to decide which features to use and outputs the processed ing from primitive blocks while representation learning into sion. The idea of top-down attention in the sequential form edges, routing functions and leaf nodes. TabNet differs from is inspired by its applications in processing visual and text

these as it embeds soft feature selection with controllable data (Hudson and Manning 2018) and reinforcement learn- Self-supervised learning: Unsupervised representation relevant information in high dimensional input. learning improves supervised learning especially in small Feature selection: We employ a learnable mask M[i] ∈ has shown significant advances – driven by the judicious capacity of a decision step is not wasted on irrelevant

choice of the unsupervised learning objective (masked input ones, and thus the model becomes more parameter effi- prediction) and attention-based deep learning. cient. The masking is multiplicative, M[i] · f . We use an attentive transformer (see Fig. 4) to obtain the masks us- TabNet for Tabular Learning ing the processed features from the preceding step, a[i − 1]:

M[i] = sparsemax(P[i − 1] · hi (a[i − 1])). Sparsemax nor-

DTs are successful for learning from real-world tabular malization (Martins and Astudillo 2016) encourages sparsity datasets. With a specific design, conventional DNN building by mapping the Euclidean projection onto the probabilistic blocks can be used to implement DT-like output manifold, simplex, which is observed to be superior in performance and e.g. see Fig. 3). In such a design, individual feature selec- aligned with the goal of sparse feature selection for explain-

tion is key to obtain decision boundaries in hyperplane form, PD which can be generalized to a linear combination of features ability. Note that j=1 M[i]b,j = 1. hi is a trainable func- where coefficients determine the proportion of each feature. tion, shown in Fig. 4 using a FC layer, followed by BN. P[i] TabNet is based on such functionality and it outperforms DTs is the prior scale term, denoting how much a particular feature

Qi while reaping their benefits by careful design which: (i) uses has been used previously: P[i] = j=1 (γ − M[j]), where γ sparse instance-wise feature selection learned from data; (ii) is a relaxation parameter – when γ = 1, a feature is enforced

+ Softmax

Feature Feature …

transformer transformer

x Nsteps Features

+ Softmax

Feature …

transformer transformer Feature Feature Feature Feature transformer

Encoded representation

transformer transformer Attentive transformer … Mask transformer …

Step 2 Decision step dependent

transformer transformer

BN Feature Feature

FC BN transformer transformer

+ 0.5 0.5 0.5 Agg. Agg. Features Features FC FC + +

Reconstructed + … Feature attributes + … features

(a) TabNet encoder architecture (b) TabNet decoder architecture Feature transformer Feature Attentive transformer Shared across decision steps Decision step dependent transformer GLU

Decision step dependent Prior scales

+ 0.5 0.5 0.5

0.5 0.5 0.5

+ Attentive transformer (c) (d)

Prior scales

divides the processed representation to be used by the attentive transformer of the subsequent step as well as for the overall Attentive BN FC

output. For each step, the feature selection mask provides interpretable information about the model’s functionality, and the +

masks can be aggregated to obtain global feature transformer important attribution. (b) TabNet decoder, composed of a feature transformer block at each step. (c) A feature transformer block example – 4-layer network is shown, where 2 are shared across all decision

Prior scales

steps and 2 are decision step-dependent. Each layer is composed of a fully-connected (FC) layer, BN and GLU nonlinearity. (d) +

An attentive transformer block example – a single layer mapping is modulated with a prior scale information which aggregates Sparsemax

how much each feature has been used before the current decision step. sparsemax (Martins and Astudillo 2016) is used for BN FC

normalization of the coefficients, resulting in sparse selection of the salient features. +

to be used only at one decision step and as γ increases, more propose the aggregate.feature importance mask, Magg−b,j = flexibility is provided to use a feature at multiple decision PNsteps ηb [i]Mb,j [i]

PD PNsteps

ηb [i]Mb,j [i].2 i=1 i=1 steps. P is initialized as all ones, 1B×D , without any prior j=1

on the masked features. If some features are unused (as in self- Tabular self-supervised learning: We propose a decoder supervised learning), corresponding P entries are made 0 architecture to reconstruct tabular features from the Tab- to help model’s learning. To further control the sparsity of the Net encoded representations. The decoder is composed of selected features, we propose sparsity regularization in the feature transformers, followed by FC layers at each deci-

form of entropy (Grandvalet and Bengio 2004), Lsparse = sion step. The outputs are summed to obtain the recon-

PNsteps PB PD −Mb,j [i] log(Mb,j [i]+)

i=1 b=1 j=1 Nsteps ·B , where  is a structed features. We propose the task of prediction of miss- small number for numerical stability. We add the sparsity reg- ing feature columns from the others. Consider a binary mask ularization to the overall loss, with a coefficient λsparse . Spar- S ∈ {0, 1}B×D . The TabNet encoder inputs (1 − S) · f̂ sity provides a favorable inductive bias for datasets where and the decoder outputs the reconstructed features, S · f̂ . We

most features are redundant. initialize P = (1 − S) in encoder so that the model em- Feature processing: We process the filtered features using phasizes merely on the known features, and the decoder’s last a feature transformer (see Fig. 4) and then split for the FC layer is multiplied with S to output the unknown features. decision step output and information for the subsequent We consider the reconstruction loss in self-supervised phase:

step, [d[i], a[i]] = fi (M[i] · f ), where d[i] ∈ <B×Nd and 2

PB PD (f̂b,j −fb,j )·Sb,j

a[i] ∈ <B×Na . For parameter-efficient and robust learning b=1 j=1

√ PB PB 2

. Normalization b=1 (fb,j −1/B b=1 fb,j ) with high capacity, a feature transformer should comprise layers that are shared across all decision steps (as the same with the population standard deviation of the ground truth features are input across different decision steps), as well as is beneficial, as the features may have different ranges. We decision step-dependent layers. Fig. 4 shows the implementa- sample Sb,j independently from a Bernoulli distribution with

tion as concatenation of two shared layers and two decision parameter ps , at each iteration. step-dependent layers. Each FC layer is followed by BN and eventually connected to a normalized residual √ connection We study TabNet in wide range of problems, that contain with normalization. Normalization with 0.5 helps to sta- regression or classification tasks, particularly with published bilize learning by ensuring that the variance throughout the benchmarks. For all datasets, categorical inputs are mapped

For faster training, we use large batch sizes with BN. Thus, bedding and numerical columns are input without and pre- except the one applied to the input features, we use ghost BN processing.4 We use standard classification (softmax cross (Hoffer, Hubara, and Soudry 2017) form, using a virtual batch entropy) and regression (mean squared error) loss functions size BV and momentum mB . For the input features, we ob- and we train until convergence. Hyperparameters of the Tab-

serve the benefit of low-variance averaging and hence avoid Net models are optimized on a validation set and listed in ghost BN. Finally, inspired by decision-tree like aggregation Appendix. TabNet performance is not very sensitive to most as in Fig. 3, we construct the overall decision embedding hyperparameters as shown with ablation studies in Appendix. as dout = i=1 PNsteps ReLU(d[i]). We apply a linear mapping In Appendix, we also present ablation studies on various de-

Wfinal dout to get the output mapping.1 sign and guidelines on selection of the key hyperparameters. Interpretability: TabNet’s feature selection masks can shed For all experiments we cite, we use the same training, val- light on the selected features at each step. If Mb,j [i] = 0, idation and testing data split with the original work. Adam optimization algorithm (Kingma and Ba 2014) and Glorot then j th feature of the bth sample should have no contribution uniform initialization are used for training of all models.5

to the decision. If fi were a linear function, the coefficient

Mb,j [i] would correspond to the feature importance of fb,j . Instance-wise feature selection

Although each decision step employs non-linear processing, their outputs are combined later in a linear way. We aim Selection of the salient features is crucial for high perfor- to quantify an aggregate feature importance in addition to mance, especially for small datasets. We consider 6 tabular requires a coefficient that can weigh the relative importance samples). The datasets are constructed in such a way that of each step in the decision. We simply propose ηb [i] = only a subset of the features determine the output. For Syn1-

PNd Syn3, salient features are same for all instances (e.g., the

c=1 ReLU(db,c [i]) to denote the aggregate decision con- tribution at ith decision step for the bth sample. Intuitively, if 2

Normalization is used to ensure D

P j=1 Magg−b,j = 1. db,c [i] < 0, then all features at ith decision step should have 3

0 contribution to the overall decision. As its value increases, prove the performance, but interpretation of individual dimensions

it plays a higher role in the overall linear combination. Scal- may become challenging. ing the decision mask at each decision step with ηb [i], we Specially-designed feature engineering, e.g. logarithmic trans- formation of variables highly-skewed distributions, may further

For discrete outputs, we additionally employ softmax during

training (and argmax during inference). An open-source implementation will be released.

Global: using only globally-salient features, Tree Ensembles (Geurts, Ernst, and Wehenkel 2006), Lasso-regularized model, L2X

Syn Syn Syn Syn Syn Syn

No selection .5 ± .0 .7 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .6 ± .0 Tree .5 ± .1 .8 ± .0 .8 ± .0 .6 ± .0 .7 ± .0 .7 ± .0 Lasso-regularized .4 ± .0 .5 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .7 ± .0

INVASE .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0

Global .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0 TabNet .6 ± .0 .8 ± .0 .8 ± .0 .7 ± .0 .7 ± .0 .8 ± .0

output of Syn depends on features X -X ), and global fea- Table 3: Performance for Poker Hand induction dataset. ture selection, as if the salient features were known, would give high performance. For Syn4-Syn6, salient features are Model Test accuracy (%) instance dependent (e.g., for Syn4, the output depends on ei- DT 50.0 ther X -X or X -X depending on the value of X ), which MLP 50.0

makes global feature selection suboptimal. Table 1 shows that Deep neural DT 65.1

TabNet outperforms others (Tree Ensembles (Geurts, Ernst, XGBoost 71.1

and Wehenkel 2006), LASSO regularization, L2X (Chen LightGBM 70.0 van der Schaar 2019). For Syn1-Syn3, TabNet performance TabNet 99.2 is close to global feature selection - it can figure out what Rule-based 100.0 features are globally important. For Syn4-Syn6, eliminating instance-wise redundant features, TabNet improves global feature selection. All other methods utilize a predictive model Poker Hand (Dua and Graff 2017): The task is classifica-

with 43k parameters, and the total number of parameters is tion of the poker hand from the raw suit and rank attributes of 101k for INVASE due to the two other models in the actor- the cards. The input-output relationship is deterministic and critic framework. TabNet is a single architecture, and its size hand-crafted rules can get 100% accuracy. Yet, conventional is 26k for Syn1-Syn and 31k for Syn4-Syn6. The compact DNNs, DTs, and even their hybrid variant of deep neural DTs

representation is one of TabNet’s valuable properties. (Yang, Morillo, and Hospedales 2018) severely suffer from the imbalanced data and cannot learn the required sorting and Performance on real-world datasets ranking operations (Yang, Morillo, and Hospedales 2018).

Tuned XGBoost, CatBoost, and LightGBM show very slight

as it can perform highly-nonlinear processing with its depth, Model Test accuracy (%) without overfitting thanks to instance-wise feature selection.

CatBoost 85.1 Table 4: Performance on Sarcos dataset. Three TabNet mod-

AutoML Tables 94.9 els of different sizes are considered.

Forest Cover Type (Dua and Graff 2017): The task is clas- MLP 2.1 0.14M

sification of forest cover type from cartographic variables. Adaptive neural tree 1.2 0.60M approaches that are known to achieve solid performance (AutoML 2019), an automated search framework based on TabNet-M 0.2 0.59M ensemble of models including DNN, gradient boosted DT, TabNet-L 0.1 1.75M with very thorough hyperparameter search. A single TabNet without fine-grained hyperparameter search outperforms it. Sarcos (Vijayakumar and Schaal 2000): The task is re-

gressing inverse dynamics of an anthropomorphic robot arm.

very small model is possible with a random forest. In the very and TabNet merely focuses on the relevant ones. For Syn4, small model size regime, TabNet’s performance is on par the output depends on either X -X or X -X depending parameters. When the model size is not constrained, TabNet feature selection – it allocates a mask to focus on the indi- achieves almost an order of magnitude lower test MSE. cator X , and assigns almost all-zero weights to irrelevant

features (the ones other than two feature groups). models are denoted with -S and -M. Real-world datasets: We first consider the simple task of mushroom edibility prediction (Dua and Graff 2017). Tab- Model Test acc. (%) Model size Net achieves 100% test accuracy on this dataset. It is indeed Sparse evolutionary MLP 78.4 81K known (Dua and Graff 2017) that “Odor” is the most discrim-

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.