Ai Energy Forecasting

This project focuses on ai energy forecasting using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

Connecting the Dots: Multivariate Time Series Forecasting with

Graph Neural Networks

Zonghan Wu Shirui Pan∗ Guodong Long

Jing Jiang Xiaojun Chang Chengqi Zhang

ABSTRACT ACM Reference Format:

Modeling multivariate time series has long been a subject that has Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi

Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting

attracted researchers from a diverse range of fields including eco- with Graph Neural Networks. In 26th ACM SIGKDD Conference on Knowl- nomics, finance, and traffic. A basic assumption behind multivariate edge Discovery and Data Mining (KDD ’20), August 23–27, 2020, Virtual Event, time series forecasting is that its variables depend on one another USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/XXXXXX. but, upon looking closely, it’s fair to say that existing methods fail to XXXXXX

fully exploit latent spatial dependencies between pairs of variables.

In recent years, meanwhile, graph neural networks (GNNs) have

shown high capability in handling relational dependencies. GNNs 1 INTRODUCTION require well-defined graph structures for information propagation Modern societies have benefited from a wide range of sensors which means they cannot be applied directly for multivariate time to record changes in temperature, price, traffic speed, electricity series where the dependencies are not known in advance. In this usage, and many other forms of data. Recorded time series from

paper, we propose a general graph neural network framework de- different sensors can form multivariate time series data and can signed specifically for multivariate time series data. Our approach be interlinked. For example, the rise in daily temperature may automatically extracts the uni-directed relations among variables cause an increase in electricity usage. To capture systematic trends through a graph learning module, into which external knowledge over a group of dynamically changing variables, the problem of

like variable attributes can be easily integrated. A novel mix-hop multivariate time series forecasting has been studied for at least propagation layer and a dilated inception layer are further proposed sixty years. It has seen tremendous applications in the domains of to capture the spatial and temporal dependencies within the time economics, finance, bioinformatics, and traffic. series. The graph learning, graph convolution, and temporal con- Multivariate time series forecasting methods inherently assume

volution modules are jointly learned in an end-to-end framework. interdependencies among variables. In other words, each variable Experimental results show that our proposed model outperforms depends not only on its historical values but also on other variables. the state-of-the-art baseline methods on 3 of 4 benchmark datasets However, existing methods do not exploit latent interdependencies

and achieves on-par performance with other approaches on two among variables efficiently and effectively. Statistical methods, such traffic datasets which provide extra structural information. as vector auto-regressive model (VAR) and Gaussian process model (GP), assume a linear dependency among variables. The model CCS CONCEPTS complexity of statistical methods grows quadratically with the

• Computing methodologies → Neural networks; Artificial number of variables. They face the problem of overfitting with a intelligence. large number of variables. Recently developed deep-learning-based methods, including LSTNet and TPA-LSTM , are powerful KEYWORDS to capture non-linear patterns. LSTNet encodes short-term local

Graph neural networks, graph structure learning, multivariate time information into low dimensional vectors using 1D convolutional series forecasting, spatial-temporal graphs neural networks and decodes the vectors through a recurrent neural network. TPA-LSTM processes the inputs by a recurrent neural ∗ Corresponding Author. network and employs a convolutional neural network to calculate the attention score across multiple steps. LSTNet and TPA-LSTM do

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed not model the pair-wise dependencies among variables explicitly, for profit or commercial advantage and that copies bear this notice and the full citation which weakens model interpretability. on the first page. Copyrights for components of this work owned by others than ACM Graphs are a special form of data which describes the relation-

must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a ships between different entities. Recently, graph neural networks KDD ’20, August 23–27, 2020, Virtual Event, USA permutation-invariance, local connectivity, and compositionality. © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7998-4/20/08. . . $15.00 By propagating information through structures, graph neural net-

https://doi.org/10.1145/XXXXXX.XXXXXX works allow each node in a graph to be aware of its neighborhood

context. Multivariate time series forecasting can be viewed natu- rally from a graph perspective. Variables from multivariate time Graph Structure Graph series can be considered as nodes in a graph, and they are inter- Learning Convolution linked through their hidden dependency relationships. It follows that modeling multivariate time series data using graph neural net-

Time series Temporal Forecasting

works can be a promising way to preserve their temporal trajectory (data points) Convolution Results while exploiting the interdependency among time series.

The most suitable type of graph neural networks for multivari-

ate time series is spatial-temporal graph neural networks. Spatial- temporal graph neural networks take multivariate time series and Figure 1: A concept map of our proposed framework. an external graph structure as inputs, and they aim to predict fu- ture values or labels of multivariate time series. Spatial-temporal graph neural networks have achieved significant improvements • To the best of our knowledge, this is the first study on mul- compared to methods that do not utilize structural information. tivariate time series data generally from a graph-based per-

However, these approaches still fall short for modeling multivariate spective with graph neural networks. time series due to the following challenges: • We propose a novel graph learning module to learn hidden spatial dependencies among variables. Our method opens a • Challenge 1: Unknown Graph Structure. Existing GNN ap- new door for GNN models to handle data without explicit proaches rely heavily on a pre-defined graph structure in graph structure.

order to perform time series forecasting. In most cases, multi- • We present a joint framework for modeling multivariate time variate time series does not have an explicit graph structure. series data and learning graph structures. Our framework The relationships among variables has to be discovered from is more generic than any existing spatial-temporal graph data rather than being provided as ground truth knowledge. neural network as it can handle multivariate time series

• Challenge 2: Graph Learning & GNN Learning. Even though with or without a pre-defined graph structure. a graph structure is available, most GNN approaches focus • Experimental results show that our method outperforms the only on message passing (GNN Learning) and overlook the state-of-the-art methods on 3 of 4 benchmark datasets and fact that the graph structure is not optimal and should be achieves on-par performance with other GNNs on two traffic

updated during training. The question then is how to simul- datasets which provide extra structural information. taneously learn the graph structure and the GNN for time series in an end-to-end framework. 2 BACKGROUNDS In this paper, we propose a novel approach to overcome these 2.1 Multivariate Time Series Forecasting challenges. As demonstrated by Figure 1, our framework consists Time series forecasting has been studied for a long time. The ma-

of three core components - the graph learning layer, the graph jority of existing methods follow a statistical approach. The auto- convolution module, and the temporal convolution module. For regressive integrated moving average (ARIMA) generalizes a Challenge 1, we propose a novel graph learning layer, which ex- family of a linear model, including auto-regressive (AR), moving tracts a sparse graph adjacency matrix adaptively based on data. average (MA), and auto-regressive moving average (ARMA). The

Furthermore, we develop a graph convolution module to address the vector auto-regressive model (VAR) extends the AR model to cap- spatial dependencies among variables, given the adjacency matrix ture the linear interdependencies among multiple time series. Simi- computed by the graph learning layer. This is designed specifically larly, the vector auto-regressive moving average model (VARMA) is for directed graphs and avoids the over-smoothing problem that proposed as a multivariate version of the ARMA model. Gaussian

frequently occurs in graph convolutional networks. Finally, we pro- process (GP), as a Bayesian approach, models the distribution of a pose a temporal convolution module to capture temporal patterns multivariate variable over functions. GP can be applied naturally to by modified 1D convolutions. It can both discover temporal patterns model multivariate time series data . Although statistical models with multiple frequencies and process very long sequences. are widely used in time series forecasting due to their simplicity

As all parameters are learnable through gradient descent, the and interpretability, they make strong assumptions with respect proposed framework is able to model multivariate time series data to a stationary process and they do not scale well to multivariate and learn the internal graph structure simultaneously in an end-to- time series data. Deep-learning-based approaches are free from end manner (for Challenge 2). To reduce the difficulty of solving a stationary assumptions and they are effective methods to capture

occupation in processing large graphs, we propose a learning algo- deep-learning-based models designed for multivariate time series rithm that uses a curriculum learning strategy to find a better local forecasting. They employ convolutional neural networks to capture optimum and splits multivariate time series into subgroups during local dependencies among variables and recurrent neural networks training. The advantages here are that our proposed framework is to preserve long-term temporal dependencies. Convolutional neu-

generally applicable to both small and large graphs, short and ral networks encapsulate interactions among variables into a global long time series, with and without externally defined graph hidden state. Therefore, they cannot fully exploit latent dependen- structures. In summary, our main contributions are as follows: cies between pairs of variables.

2.2 Graph Neural Networks among nodes using the graph adjacency matrix. The graph adja-

Graph neural networks have enjoyed great success in handling cency matrix is not given by the multivariate time series data in spatial dependencies among entities in a network. Graph neural most cases and will be learned by our model. networks assume that the state of a node depends on the states of its neighbors. To capture this type of spatial dependency, various kinds 4 FRAMEWORK OF MTGNN of graph neural networks have been developed through message 4.1 Model Architecture

passing , information propagation , and graph convolution

We first elaborate on the general framework of our model. As illus-

. Sharing similar roles, they essentially capture a node’s high- trated in Figure 2, MTGNN on the highest level consists of a graph level representation by passing information from a node’s neighbors learning layer, m graph convolution modules, m temporal convolution to the node itself. Most recently, we have seen the emergence of a modules, and an output module. To discover hidden associations type of graph neural networks known as spatial-temporal graph among nodes, a graph learning layer computes a graph adjacency

neural networks. This form of neural networks is proposed initially matrix, which is later used as an input to all graph convolution to solve the problem of traffic prediction [3, 13, 21, 23, 26] and modules. Graph convolution modules are interleaved with temporal skeleton-based action recognition [18, 22]. The inputs to spatial- convolution modules to capture spatial and temporal dependencies temporal graph neural networks are multivariate time series with an respectively. Figure 3 gives a demonstration of how a temporal con-

external graph structure which describes the relationships among volution module and a graph convolution module collaborate with variables in multivariate time series. For spatial-temporal graph each other. To avoid the problem of gradient vanishing, residual neural networks, spatial dependencies among nodes are captured by connections are added from the inputs of a temporal convolution graph convolutions, while temporal dependencies among historical module to the outputs of a graph convolution module. Skip connec-

states are preserved by recurrent neural networks [13, 17] or 1D tions are added after each temporal convolution module. To get the convolutions [22, 23]. Although existing spatial-temporal graph final outputs, the output module projects the hidden features to the neural networks have achieved significant improvements compared desired output dimension. In more detail, the core components of to methods without using a graph structure, they are incapable of our model are illustrated in the following:

handling pure multivariate time series data effectively due to the absence of a pre-defined graph and lack of a general framework.

4.2 Graph Learning Layer

3 PROBLEM FORMULATION The graph learning layer learns a graph adjacency matrix adap- In this paper, we focus on the task of multivariate time series fore- tively to capture the hidden relationships among time series data. casting. Let zt ∈ R N denote the value of a multivariate variable of To construct a graph, existing studies measure the similarity be- dimension N at time step t, where zt [i] ∈ R denote the value of the tween pairs of nodes by a distance metric, such as dot product and

i th variable at time step t. Given a sequence of historical P time steps Euclidean distance . This leads inevitably to the problem of high of observations on a multivariate variable, X = {zt1 , zt2 , · · · , zt P }, time and space complexity with O(N 2 ). It means the computation our goal is to predict the Q-step-away value of Y = {zt P +Q }, or and memory cost grows quadratically with the increase of graph a sequence of future values Y = {zt P +1 , zt P +2 , · · · , zt P +Q }. More size. This restricts the model’s capability of handling larger graphs.

generally, the input signals can be coupled with other auxiliary To address this limitation, we adopt a sampling approach, which features such as time of the day, day of the week, and day of the only calculates pair-wise relationships among a subset of nodes. season. Concatenating the input signals with auxiliary features, This cuts off the bottleneck of computation and memory in each we assume the inputs instead are X = {St1 , St2 , · · · , St P } where minibatch. More details will be provided in Section 4.6.

Another problem is that existing distance metrics are often sym-

Sti ∈ R N ×D , D is the feature dimension, the first column of Sti metric or bi-directional. In multivariate time series forecasting, we equals to zti , and the rest are auxiliary features. We aim to build a expect that the change of a node’s condition causes the change of mapping f (·) from X to Y by minimizing the absolute loss with l2 another node’s condition such as traffic flow. Therefore the learned regularization. relation is supposed to be uni-directional. Our proposed graph

Graphs describe the relationships among entities in a network. learning layer is specifically designed to extract uni-directional We give a formal definition of graph-related concepts below. relationships, illustrated as follows:

Definition 3.1 (Graph). A graph is formulated as G = (V , E) where

V is the set of nodes, and E is the set of edges. We use N to denote M1 = tanh(αE1 Θ1 ) (1) the number of nodes in a graph. M2 = tanh(αE2 Θ2 ) (2) Definition 3.2 (Node Neighborhood). Let v ∈ V to denote a node A = ReLU (tanh(α(M1 MT2 − M2 MT1 ))) (3) and e = (v, u) ∈ E to denote an edge pointing from u to v. The

f or i = 1, 2, · · · , N (4) neighborhood of a node v is defined as N (v) = {u ∈ V |(v, u) ∈ E}. idx = arдtopk(A[i, :]) (5)

Definition 3.3 (Adjacency Matrix). The adjacency matrix is a

mathematical representation of a graph, denoted as A ∈ R N ×N A[i, −idx] = 0, (6) with Ai j = c > 0 if (vi , v j ) ∈ E and Ai j = 0 if (vi , v j ) < E. where E1 , E2 represents randomly initialized node embeddings, From a graph-based perspective, we consider variables in multi- which are learnable during training, Θ1 , Θ2 are model parameters, variate time series as nodes in graphs. We describe the relationships α is a hyper-parameter for controlling the saturation rate of the

Node embeddings

Graph Learning Layer

Residual Connections Residual Connections Residual Connections

A A A

1×1 TC Module GC TC Module GC … TC Module GC

Conv (d=𝑞 " ) Module (d=𝑞# ) Module (d=𝑞 $ ) Module

Inputs: 𝜒 ∈ 𝑅 *+,×-×. Skip Connections Skip Connections Skip Connections

Output Module 0 ∈ 𝑅 *123×-

poral convolution modules and graph convolution modules are interleaved with each other to capture temporal and spatial dependencies respectively. The hyper-parameter, dilation factor d, which controls the receptive field size of a temporal con- volution module, is increased at an exponential rate of q. The graph learning layer learns the hidden graph adjacency matrix, which is used by graph convolution modules. Residual connections and skip connections are added to the model to avoid the

problem of gradient vanishing. The output module projects hidden features to the desired dimension to get the final results.

𝐻*+,

+ +

N N A Mix-hop Mix-hop AT

Propagation Propagation MLP0 MLP1 MLPK

Layer Layer

A A … A

D D’ 𝐻 (() 𝐻 (#) 𝐻 (')

T T’ 𝐻%& 𝐻%& 𝐻%&

(a) GC module (b) Mix-hop propagation layer and a graph convolution module collaborate with each other. A tem- poral convolution module filters the inputs by sliding a 1D window over the time and node axes, as denoted by the red. A graph convo- lution module filters the inputs at each step, denoted by the blue. advantage of our approach is that we can learn stable and inter- pretable node relationships over the period of the training dataset.

Once the model is trained in an on-line learning version, our graph

activation function, and arдtopk(·) returns the index of the top-k adjacency matrix is also adaptable to change as new training data largest values of a vector. The asymmetric property of our proposed updates the model parameters. graph adjacency matrix is achieved by Equation 3. The subtraction term and the ReLU activation function regularize the adjacency 4.3 Graph Convolution Module matrix so that if Avu is positive, its diagonal counterpart Auv will The graph convolution module aims to fuse a node’s information

be zero. Equation 5-6 is a strategy to make the adjacency matrix with its neighbors’ information to handle spatial dependencies sparse while reducing the computation cost of the following graph in a graph. The graph convolution module consists of two mix- convolution. For each node, we select its top-k closest nodes as its hop propagation layers to process inflow and outflow information neighbors. While retaining the weights for connected nodes, we passed through each node separately. The net inflow information

set the weights of non-connected nodes as zero. is obtained by adding the outputs of the two mix-hop propagation Incorporate External Data. The inputs to the graph learning layer layers. Figure 4 shows the architecture of the graph convolution are not limited to node embeddings. In case that external knowl- module and the mix-hop propagation layer. edge about the attributes of each node is given, we can also set E1 = E2 = Z, where Z is a static node feature matrix. Some works Mix-hop Propagation Layer. Given a graph adjacency matrix, we

have considered capturing dynamic spatial dependencies [8, 18]. In propose the mix-hop propagation layer to handle information flow other words, they dynamically adjust the weight of two connected over spatially dependent nodes. The proposed mix-hop propaga- nodes based on temporal inputs. However, assuming dynamic spa- tion layer consists of two steps - the information propagation step tial dependencies makes the model extremely hard to converge and the information selection step. We first give the mathematical

when we need to learn the graph structure at the same time. The form of these two steps and then illustrate our motivations. The

information propagation step is defined as follows: tanh sigmoid × (k ) (k −1) Concatenate

H = βHin + (1 − β)ÃH , (7) Dilated Dilated

Inception Inception

1×2 1×3 1×6 1×7 where β is a hyper parameter, which controls the ratio of retaining Layer Layer

the root node’s original states. The information selection step is defined as follows

K (a) TC module (b) Dilated inception layer

Hout = H(k ) W(k ) , (8)

i=0 Figure 5: The temporal convolution and dilated inception layer.

where K is the depth of propagation, Hin represents the input hidden states outputted by the previous layer, Hout represents 4.4 Temporal Convolution Module the output hidden states of the current layer, H(0) = Hin , Ã = The temporal convolution module applies a set of standard dilated D̃−1 (A + I), and D̃ii = 1 + j Ai j . In Figure 4b, we demonstrate the Í 1D convolution filters to extract high-level temporal features. This information propagation step and information selection step in the module consists of two dilated inception layers. One dilated incep-

proposed mix-hop propagation layer. It first propagates information tion layer is followed by a tangent hyperbolic activation function horizontally and selects information vertically. and works as a filter. The other layer is followed by a sigmoid acti- The information propagation step propagates node information vation function and functions as a gate to control the amount of along with the given graph structure recursively. A severe limitation information that the filter can pass to the next module. Figure 5

of graph convolutional networks is that node hidden states converge shows the architecture of the temporal convolution module and to a single point as the number of graph convolution layers goes to the dilated inception layer. infinity. This is because the graph convolutional network with many

Dilated Inception Layer. The temporal convolution module cap-

layers reaches the random walk’s limit distribution regardless of the tures sequential patterns of time series data through 1D convo- initial node states. To address this problem, motivated by Klicpera lutional filters. To come up with a temporal convolution module that is able to both discover temporal patterns with various ranges the propagation process so that the propagated node states can and handle very long sequences, we propose the dilated inception both preserve locality and explore a deep neighborhood. However,

layer which combines two widely applied strategies from convo- if we only apply Equation 7, some node information will be lost. lutional neural networks, i.e., using filters with multiple sizes Under the extreme circumstance that no spatial dependencies exist, and applying dilated convolution . aggregating neighborhood information simply adds useless noises

First, choosing the right kernel size is a challenging problem for

to each node. Therefore, the information selection step is introduced convolutional networks. The filter size can be too large to represent to filter out important information produced at each hop. According short-term signal patterns subtly, or too small to discover long-term to Equation 8, the parameter matrix W(k ) functions as a feature signal patterns sufficiently. In image processing, a widely employed selector. When the given graph structure does not entail spatial strategy is called inception, which concatenates the outputs of 2D

dependencies, Equation 8 is still able to preserve the original node- convolution filters with three different kernel sizes, 1 × 1, 3 × 3, and self information by adjusting W(k ) to 0 for all k > 0. 5 × 5. Moving from 2D images to 1D time series, the set of 1 × 1, Connection to existing works. The idea of mix-hop has been ex- 1 × 3, and 1 × 5 filter sizes do not suit the nature of temporal signals. nism to weight information among different hops. They both apply 1 × 3, and 1 × 5 cannot well encompass those periods. Alternatively,

GCN for information propagation. However, as GCN faces the over- we propose a temporal inception layer consisting of four filter sizes, smoothing problem, information from higher hops may not or viz. 1 × 2, 1 × 3, 1 × 6, and 1 × 7. The aforementioned periods can negatively contribute to the overall performance. To avoid this, our all be covered by the combination of these filter sizes. For example, approach keeps a balance between local and neighborhood infor- to represent the period 12, a model can pass the inputs through a

model with two mix-hop layers has the capability to represent the 1 × 6 filter from the second temporal inception layer. delta difference between two consecutive hops. Our approach can Second, the receptive field size of a convolutional network grows achieve the same effect with only one mix-hop propagation layer. in a linear progression with the depth of the network and the kernel Suppose K = 2, W(0) = 0, W(1) = −1, and W(2) = 1, then size of the filter. Consider a convolutional network with m 1D

convolution layers of kernel size c, the receptive field size of the Hout = ∆(H(2) , H(1) ) = H2 − H1 . (9) convolutional network is,

R = m(c − 1) + 1. (10)

From this perspective, using summation is more efficient to rep-

resent all linear interactions of different hops compared with the To process very long sequences, it requires either a very deep net- concatenation method. work or very large filters. We adopt dilated convolution to reduce

model complexity. Dilated convolution operates a standard convo- Algorithm 1 The learning algorithm of MTGNN. lution filter on down-sampled inputs with a certain frequency. For 1: Input: The dataset O , node set V , the initialized MTGNN model f (·) example, where the dilation factor is 2, it applies standard convo- with Θ, learning rate γ , batch size b, step size s, split size m (default=1). lution on inputs sampled every two steps. Following , we let 2: set it er = 1, r = 1

the dilation factor for each layer increase exponentially at a rate of 3: repeat ′ q (q > 1). Suppose the initial dilation factor is 1, the receptive field 4: sample a batch (X ∈ R b×T ×N ×D , Y ∈ R b×T ×N ) from O . size of a m layer dilated convolutional network with kernel size c is 5: random split the node set V into m groups, ∪m i =1Vi = V . 6: if it er %s == 0 and r <= T ′ then R = 1 + (c − 1)(qm − 1)/(q − 1). (11) 7: r =r +1

This indicates that the receptive field size of the network also grows 8: end if 9: for i in 1:m do exponentially with an increase in the number of hidden layers at 10: compute Ŷ = f (X[:, :, id(Vi ), :]; Θ) the rate of q. Therefore, using this dilation strategy can capture 11: compute L = loss( Ŷ[:, : r, :], Y[:, : r, id (Vi )]) much longer sequences than proceeding without it. 12: compute the stochastic gradient of Θ according to L.

Formally, combining inception and dilation, we propose the di-

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings presented in the source material. The practical implications and implementation considerations are further elaborated to support complete understanding of the subject matter across different application domains and use cases.

Speaker Verification Xvector