Machine Learning For Energy Consumption Prediction

Machine Learning For Energy Consumption Prediction

Energy & Smart Grid · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: prediction and forecasting using meter, load, generation and time-series energy data.

GAN Q-learning

Thang Doan∗ Bogdan Mazoure∗

Clare Lyle

School of Computer Science

Abstract

Distributional reinforcement learning (distributional RL) has seen empirical success

in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose

GAN Q-learning, a novel distributional RL method based on generative adversarial

networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.

1 Introduction

Reinforcement learning (RL) has recently had great success in solving complex tasks with continuous

control [21, 22]. However, as these methods often have high variance results while dealing with unstable environments, distributional perspectives on the state-value function in RL have begun to gain popularity . Note that the distributional perspective is distinct from Bayesian approaches to the RL problem as the former models the inherent variability of the returns from a state and not the agent’s confidence in its prediction of the average return. Up to now, deep learning methods in RL used multiple function approximators (typically a network

with shared hidden layers) to fit a state value or state-action value distribution. For instance, used k-heads on the state-action value function Q for every available action and used it to model a distribution. In , a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) instead of the critic, hence allowing for a closed-form derivation of update rules. More recently, introduced a distributional algorithm C which aimed to solve the RL problem by

learning a categorical probability vector over returns Q. Unlike which uses a generative network to learn the underlying transition model of the environment, we utilize a generative network to model the distribution approximation of the Bellman updates. In this work, we build on top of the aforementioned distributional RL methods and introduce a novel way to learn the state-value distribution. Inspired by the analogy between the actor-critic architecture and generative adversarial networks (GANs) , we leverage the later in order to implicitly represent

the distribution of the Bellman target update through a generator/discriminator architecture. We show that, although sometimes volatile, our proposed algorithm is a viable alternative to now considered classical deep Q networks (DQN). We aim to provide a unifying view on distributional RL through ∗ These authors contributed equally.

the minimization of the Earth-Mover distance and without explicitly using the distributional Bellman projection operator on the support of the Q values.

2 Related Work

2.1 Background

Multiple tasks in machine learning require finding an optimal behaviour in a given setting, i.e. solving the reinforcement learning problem. We proceed to formulate the task as follows. Let (S,A,R,P ,s ,γ) be a 5-tuple representing a Markov decision process (MDP) where S is the set of states, A the set of allowed actions, R : S × A → R is the (deterministic or stochastic) reward function, P are the environment transition probabilities and s ⊆ S is the set of initial states. At a

given time step t = 1, 2, 3, ... an agent acts according to a policy π(a|s) = P [at |st ]. The environment is characterized by its set of initial states in which the agent starts, as well as the transition model which encodes the mechanics of moving from one state to another. In order to compare states, we introduce the state value function P V : S → R which gives the expected sum of discounted rewards in ∞ that state. That is, V (s) = Eπ,P [ t=1 γ t Rt |S = s]. The reinforcement learning problem is two-fold: (1) given a fixed policy π we would like to obtain

the correct state value function Vπ (s), ∀s ∈ S and (2) we wish to find the optimal policy π ∗ (a) which yields the highest V (s) for all states of the MDP or, equivalently, π ∗ = argmaxπ Vπ (s). The first task is known in the reinforcement learning literature as prediction and the second as control. In order to find the value function for each state, we need to solve the Bellman equations : Vπ (s) = Eπ,P [R(st , at ) + γVπ (st+1 )|st = s], (1)

for Vπ = {Vπ (s ), ..., Vπ (sn )} and R = {R(s , a ), ..., R(sP n , an )}. If we define a state-action ∞ value function Qπ (s, a) : S × A → R as Qπ (s, a) = Eπ,P [ t=0 γ t R(St , At )|S = s, A = a], then we can rewrite Eq.1 as:

Qπ (s, a) = Eπ,P [R(s, a) + γQπ (s , a )] or

(2) Qπ = R + γPπ Qπ , for Qπ = {Qπ (s , a ), ..., Qπ (sn , an )}. While both Eq.1 and Eq.2 have the following direct solution obtained by matrix inversion: Vπ = (I − γP )−1 R. However it should be noted that this is only well-defined for finite-state MDPs, and further as this computation has O(n2.4 ) complexity , it is only computationally feasible for small

MDPs. Therefore, sample-based algorithms such as Temporal Difference (TD) for prediction

and SARSA or Q-learning [18, 25] for control are preferred for more general classes of environments.

2.2 Distributional Reinforcement Learning

In the setting of distributional reinforcement learning , we seek to learn the distribution of returns from a state, rather than the mean of the returns. We translate the Bellman operator on points to an operator on distributions. The vector of mean rewards therefore becomes a function of reward distributions R(s, a). We can thus represent Qπ (s, a) as Zπ (s, a), a random variable whose law is the returns following a state. Both expected and distributional distributional quantities are linked through

the following relation: Qπ (s, a) = E[Zπ (s, a)]. We finally arrive to the distributional Bellman equations: d T π Zπ (s, a) ≡ R(s, a) + γZπ (S 0 , A ) with fixed point: (3) d Zπ (s, a) = T π Zπ (s, a), ∀s ∈ S ∀a ∈ A where T π denotes an analogue to the well-known Bellman operator now defined over distributions. Eq.3 is the distributional counterpart of Eq.2, where equality holds for sequences of random variables.

In traditional reinforcement learning algorithms, we use experience in the MDP to improve an

estimated state value function in order to minimize the expected distance between the value function’s output for a state and the actual returns from that state. In distributional reinforcement learning, we still aim to minimize the distance between our output and the true distribution, but now we have more

freedom in how we choose our definition of "distance", as there are many metrics on probability distributions which have subtly different properties.

The p−Wasserstein metric between two real-valued random variables X and Y with cumulative

distribution functions FX and FY is given by  1  p inf E |X − Y |p 

Wp (FX , FY ) = (4)

(X,Y ) (X,Y )

For p = 1 we recover the widely known Earth-Mover distance. More generally, for any 1 ≤ p ≤ q ≤ ∞, Wp (FX , FY ) ≤ Wq (FX , FY ) holds and is useful in contraction arguments. The maximal p−Wasserstein metric, W p = sup(s,a) Wp (Z(s, a), Z 0 (s, a)) is defined over state-action tuples (s, a) ∈ S × A for any two value distributions Z, Z 0 . It has been shown that while T π is a contraction under the maximal p−Wasserstein metric , the Bellman optimality operator T is not necessarily a contraction. This result implies that the control

setting requires a treatment different from prediction, which is done in through the C algorithm in order to guarantee proper convergence.

2.3 Generative Models

Generative models such as hidden Markov models (HMMs), restricted Boltzmann machines (RBMs)

, variational auto-encoders (VAEs) and generative adversarial networks (GANs) learn the distribution of data for all classes. Moreover, they provide a mechanism which allows to sample new observations from the learned distribution. In this work, we make use of generative adversarial networks which consist of two neural networks playing a zero-sum game against each other. The generator network G : Z → X is a mapping from a high-dimensional noise space Z = Rdz onto the input space X on which a target distribution fX

is defined. The generator’s task consists in fitting the underlying distribution of observed data fX as closely as possible. The discriminator network D : X → R ∩ [0, 1] scores each input as the probability of coming from the real data distribution fX or from the generator G. Both networks are gradually improved through alternating or simultaneous gradient descent updates.

The classical GAN algorithm minimizes the Jensen-Shannon divergence (JS) between the real and

generated distributions. Recently, suggested to replace the JS metric by the Wasserstein-1 or

Earth-Mover divergence. We make use of an improved version of this algorithm, Wasserstein GAN

with Gradient Penalty . It’s objective is given below: min max E [D(x)] + E [−D(x)] + p(λ), (5)

G D x∼fX (x) x∼G(z)

where p(λ) = λ(||∇x̃ D(x̃)|| − 1)2 , x̃ = εx + (1 − ε)G(Z), ε ∼ Uniform(0, 1), and Z ∼ fZ (z). Setting λ = 0 recovers the original WGAN objective.

3 GAN Q-learning

3.1 Motivation

We borrow the two-player game analogy from the original GAN paper : the generator network’s purpose is to produce realistic samples of the optimal state-action value distribution (estimate of Zπ∗ (s, a)). On the other hand, the discriminator network aims to distinguish real samples of T Z(s, a) from the samples Z(s, a) outputted by G. The generator network improves its performance through the signal from the discriminator, which is reminiscent of the actor-critic architecture .

3.2 Algorithm

At each timestep, G receives stochastic noise z ∼ N (0, 1) and a state s as input and returns a sample G(z|(s, a)) for every action a from the current estimate of Z(s, a). We then select the action a∗ = max G(z|(s, a)). The agent then applies the chosen action a∗ , receives a reward r and a transitions to state s . The tuple (s, a, r, s ) is saved into a replay buffer B as done in . Each

update step consists in sampling a tuple (s, a, r, s ) uniformly from the buffer B and proceed to update the generator and discriminator according to Eq 5. Values obtained from the Bellman backup operator x ≡ x(s, a, r, s ) = r + γmaxG(z|(s , a)) are a considered as coming from the real distribution. The discriminator’s goal is to differentiate between the Bellman target x = r + γmaxG(z|(s , a)) and the output produced by Z(s, a). We obtain the a following updates for G and D, respectively:

  E [DwD (x|(s, a))] − E [DwD (X|(s, a))] + p(λ) , (s,a)∼B (s,a,r,s

 )∼B

X∼Gw (z|(s,a))

L(wD , wG ) = G

(6)  E [−D wD (X|(s, a))] , (s,a)∼B    z∼N (0,1) 

X∼Gw (z|(s,a))

where wG , wD are weights of the generator and discriminator networks with respect to which the gradient updates w(t+1) ← w(t) − αt ∇w(t) L(w(t) ) are taken. Note that to further stabilize the training process, one can use a target network G updated every k epochs as in . Due to the nature of GANs, we advise training the model in a batch setting using experience replay and a second (target) network for a more stable training process.

Algorithm 1 GAN Q-learning

Input: MDP M , discriminator and generator networks D, G, learning rate α,ndisc the number of updates of the discriminator, ngen the number of updates of the generator, gradient penalty coefficient λ, batch size m. Initialize replay buffer B to capacity N, D and G with random weights, Z, s , a . t←0 for episode = 1, ..., M do for t = 1, ..., Tmax do sample a batch z ∼ N (0, 1) at ← maxG(z|(st , a)) a sample st+1 ∼ P (·|st , at )

Store transition (st ,at ,rt ,st+1 ) in B

{Updating Discriminator} for n = 1, ..., ndisc do

Sample minibatch {s, a, r, s }m i=1 from B

m sample batch ( {z}i=1 ∼ N (0, 1) ri , s terminal

Set yi = r + γmax G(z |(s , a )), otherwise

i i i i ai sample a batch {}m i=1 ∼ N (0, 1) x̃i ← i yi + (1 − i )max G(zi |(s0i , ai )) ai

L(i) ← D(G(zi |(si , a∗i )|(sP ∗ ∗ ∗

i , ai )) − D(yi |(si , ai )) + λ(|∇x̃ D(x̃i |(si , ai ))| − 1)

1 m

wD ← RMSProp(∇wD m i=1 L(i) , α ) end for {Updating Generator} for n = 1, ..., ngen do sample a batch of {z (i) }m i=1 P ∼ N (0, 1)

1 m (i)

wG ← RMSProp(−∇wG m i=1 L , α ) end for end for end for

Here, the objective function L is identical to Eq.5, where fX (x) is taken to be r(s, a) + γmax G(z|(s , a)). a

Using a generative model to represent the state-action value distribution allows for an alternative

to explicit exploration strategies. Indeed, at the beginning of the training process, taking an action

ment. Here, π denotes the greedy policy.

is analogous to using a decaying exploration strategy (since G has not clearly sepa

Continued Discussion and Extended Analysis (Part 2)

GAN Q-learning

Thang Doan∗ Bogdan Mazoure∗

Clare Lyle

School of Computer Science

Abstract

Distributional reinforcement learning (distributional RL) has seen empirical success

in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose

GAN Q-learning, a novel distributional RL method based on generative adversarial

networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.

1 Introduction

Reinforcement learning (RL) has recently had great success in solving complex tasks with continuous

control [21, 22]. However, as these methods often have high variance results while dealing with unstable environments, distributional perspectives on the state-value function in RL have begun to gain popularity . Note that the distributional perspective is distinct from Bayesian approaches to the RL problem as the former models the inherent variability of the returns from a state and not the agent’s confidence in its prediction of the average return. Up to now, deep learning methods in RL used multiple function approximators (typically a network

with shared hidden layers) to fit a state value or state-action value distribution. For instance, used k-heads on the state-action value function Q for every available action and used it to model a distribution. In , a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) instead of the critic, hence allowing for a closed-form derivation of update rules. More recently, introduced a distributional algorithm C which aimed to solve the RL problem by

learning a categorical probability vector over returns Q. Unlike which uses a generative network to learn the underlying transition model of the environment, we utilize a generative network to model the distribution approximation of the Bellman updates. In this work, we build on top of the aforementioned distributional RL methods and introduce a novel way to learn the state-value distribution. Inspired by the analogy between the actor-critic architecture and generative adversarial networks (GANs) , we leverage the later in order to implicitly represent

the distribution of the Bellman target update through a generator/discriminator architecture. We show that, although sometimes volatile, our proposed algorithm is a viable alternative to now considered classical deep Q networks (DQN). We aim to provide a unifying view on distributional RL through ∗ These authors contributed equally.

the minimization of the Earth-Mover distance and without explicitly using the distributional Bellman projection operator on the support of the Q values.

2 Related Work

2.1 Background

Multiple tasks in machine learning require finding an optimal behaviour in a given setting, i.e. solving the reinforcement learning problem. We proceed to formulate the task as follows. Let (S,A,R,P ,s ,γ) be a 5-tuple representing a Markov decision process (MDP) where S is the set of states, A the set of allowed actions, R : S × A → R is the (deterministic or stochastic) reward function, P are the environment transition probabilities and s ⊆ S is the set of initial states. At a

given time step t = 1, 2, 3, ... an agent acts according to a policy π(a|s) = P [at |st ]. The environment is characterized by its set of initial states in which the agent starts, as well as the transition model which encodes the mechanics of moving from one state to another. In order to compare states, we introduce the state value function P V : S → R which gives the expected sum of discounted rewards in ∞ that state. That is, V (s) = Eπ,P [ t=1 γ t Rt |S = s]. The reinforcement learning problem is two-fold: (1) given a fixed policy π we would like to obtain

the correct state value function Vπ (s), ∀s ∈ S and (2) we wish to find the optimal policy π ∗ (a) which yields the highest V (s) for all states of the MDP or, equivalently, π ∗ = argmaxπ Vπ (s). The first task is known in the reinforcement learning literature as prediction and the second as control. In order to find the value function for each state, we need to solve the Bellman equations : Vπ (s) = Eπ,P [R(st , at ) + γVπ (st+1 )|st = s], (1)

for Vπ = {Vπ (s ), ..., Vπ (sn )} and R = {R(s , a ), ..., R(sP n , an )}. If we define a state-action ∞ value function Qπ (s, a) : S × A → R as Qπ (s, a) = Eπ,P [ t=0 γ t R(St , At )|S = s, A = a], then we can rewrite Eq.1 as:

Qπ (s, a) = Eπ,P [R(s, a) + γQπ (s , a )] or

(2) Qπ = R + γPπ Qπ , for Qπ = {Qπ (s , a ), ..., Qπ (sn , an )}. While both Eq.1 and Eq.2 have the following direct solution obtained by matrix inversion: Vπ = (I − γP )−1 R. However it should be noted that this is only well-defined for finite-state MDPs, and further as this computation has O(n2.4 ) complexity , it is only computationally feasible for small

MDPs. Therefore, sample-based algorithms such as Temporal Difference (TD) for prediction

and SARSA or Q-learning [18, 25] for control are preferred for more general classes of environments.

2.2 Distributional Reinforcement Learning

In the setting of distributional reinforcement learning , we seek to learn the distribution of returns from a state, rather than the mean of the returns. We translate the Bellman operator on points to an operator on distributions. The vector of mean rewards therefore becomes a function of reward distributions R(s, a). We can thus represent Qπ (s, a) as Zπ (s, a), a random variable whose law is the returns following a state. Both expected and distributional distributional quantities are linked through

the following relation: Qπ (s, a) = E[Zπ (s, a)]. We finally arrive to the distributional Bellman equations: d T π Zπ (s, a) ≡ R(s, a) + γZπ (S 0 , A ) with fixed point: (3) d Zπ (s, a) = T π Zπ (s, a), ∀s ∈ S ∀a ∈ A where T π denotes an analogue to the well-known Bellman operator now defined over distributions. Eq.3 is the distributional counterpart of Eq.2, where equality holds for sequences of random variables.

In traditional reinforcement learning algorithms, we use experience in the MDP to improve an

estimated state value function in order to minimize the expected distance between the value function’s output for a state and the actual returns from that state. In distributional reinforcement learning, we still aim to minimize the distance between our output and the true distribution, but now we have more

freedom in how we choose our definition of "distance", as there are many metrics on probability distributions which have subtly different properties.

The p−Wasserstein metric between two real-valued random variables X and Y with cumulative

distribution functions FX and FY is given by  1  p inf E |X − Y |p 

Wp (FX , FY ) = (4)

(X,Y ) (X,Y )

For p = 1 we recover the widely known Earth-Mover distance. More generally, for any 1 ≤ p ≤ q ≤ ∞, Wp (FX , FY ) ≤ Wq (FX , FY ) holds and is useful in contraction arguments. The maximal p−Wasserstein metric, W p = sup(s,a) Wp (Z(s, a), Z 0 (s, a)) is defined over state-action tuples (s, a) ∈ S × A for any two value distributions Z, Z 0 . It has been shown that while T π is a contraction under the maximal p−Wasserstein metric , the Bellman optimality operator T is not necessarily a contraction. This result implies that the control

setting requires a treatment different from prediction, which is done in through the C algorithm in order to guarantee proper convergence.

2.3 Generative Models

Generative models such as hidden Markov models (HMMs), restricted Boltzmann machines (RBMs)

, variational auto-encoders (VAEs) and generative adversarial networks (GANs) learn the distribution of data for all classes. Moreover, they provide a mechanism which allows to sample new observations from the learned distribution. In this work, we make use of generative adversarial networks which consist of two neural networks playing a zero-sum game against each other. The generator network G : Z → X is a mapping from a high-dimensional noise space Z = Rdz onto the input space X on which a target distribution fX

is defined. The generator’s task consists in fitting the underlying distribution of observed data fX as closely as possible. The discriminator network D : X → R ∩ [0, 1] scores each input as the probability of coming from the real data distribution fX or from the generator G. Both networks are gradually improved through alternating or simultaneous gradient descent updates.

The classical GAN algorithm minimizes the Jensen-Shannon divergence (JS) between the real and

generated distributions. Recently, suggested to replace the JS metric by the Wasserstein-1 or

Earth-Mover divergence. We make use of an improved version of this algorithm, Wasserstein GAN

with Gradient Penalty . It’s objective is given below: min max E [D(x)] + E [−D(x)] + p(λ), (5)

G D x∼fX (x) x∼G(z)

where p(λ) = λ(||∇x̃ D(x̃)|| − 1)2 , x̃ = εx + (1 − ε)G(Z), ε ∼ Uniform(0, 1), and Z ∼ fZ (z). Setting λ = 0 recovers the original WGAN objective.

3 GAN Q-learning

3.1 Motivation

We borrow the two-player game analogy from the original GAN paper : the generator network’s purpose is to produce realistic samples of the optimal state-action value distribution (estimate of Zπ∗ (s, a)). On the other hand, the discriminator network aims to distinguish real samples of T Z(s, a) from the samples Z(s, a) outputted by G. The generator network improves its performance through the signal from the discriminator, which is reminiscent of the actor-critic architecture .

3.2 Algorithm

At each timestep, G receives stochastic noise z ∼ N (0, 1) and a state s as input and returns a sample G(z|(s, a)) for every action a from the current estimate of Z(s, a). We then select the action a∗ = max G(z|(s, a)). The agent then applies the chosen action a∗ , receives a reward r and a transitions to state s . The tuple (s, a, r, s ) is saved into a replay buffer B as done in . Each

update step consists in sampling a tuple (s, a, r, s ) uniformly from the buffer B and proceed to update the generator and discriminator according to Eq 5. Values obtained from the Bellman backup operator x ≡ x(s, a, r, s ) = r + γmaxG(z|(s , a)) are a considered as coming from the real distribution. The discriminator’s goal is to differentiate between the Bellman target x = r + γmaxG(z|(s , a)) and the output produced by Z(s, a). We obtain the a following updates for G and D, respectively:

  E [DwD (x|(s, a))] − E [DwD (X|(s, a))] + p(λ) , (s,a)∼B (s,a,r,s

 )∼B

X∼Gw (z|(s,a))

L(wD , wG ) = G

(6)  E [−D wD (X|(s, a))] , (s,a)∼B    z∼N (0,1) 

X∼Gw (z|(s,a))

where wG , wD are weights of the generator and discriminator networks with respect to which the gradient updates w(t+1) ← w(t) − αt ∇w(t) L(w(t) ) are taken. Note that to further stabilize the training process, one can use a target network G updated every k epochs as in . Due to the nature of GANs, we advise training the model in a batch setting using experience replay and a second (target) network for a more stable training process.

Algorithm 1 GAN Q-learning

Input: MDP M , discriminator and generator networks D, G, learning rate α,ndisc the number of updates of the discriminator, ngen the number of updates of the generator, gradient penalty coefficient λ, batch size m. Initialize replay buffer B to capacity N, D and G with random weights, Z, s , a . t←0 for episode = 1, ..., M do for t = 1, ..., Tmax do sample a batch z ∼ N (0, 1) at ← maxG(z|(st , a)) a sample st+1 ∼ P (·|st , at )

Store transition (st ,at ,rt ,st+1 ) in B

{Updating Discriminator} for n = 1, ..., ndisc do

Sample minibatch {s, a, r, s }m i=1 from B

m sample batch ( {z}i=1 ∼ N (0, 1) ri , s terminal

Set yi = r + γmax G(z |(s , a )), otherwise

i i i i ai sample a batch {}m i=1 ∼ N (0, 1) x̃i ← i yi + (1 − i )max G(zi |(s0i , ai )) ai

L(i) ← D(G(zi |(si , a∗i )|(sP ∗ ∗ ∗

i , ai )) − D(yi |(si , ai )) + λ(|∇x̃ D(x̃i |(si , ai ))| − 1)

1 m

wD ← RMSProp(∇wD m i=1 L(i) , α ) end for {Updating Generator} for n = 1, ..., ngen do sample a batch of {z (i) }m i=1 P ∼ N (0, 1)

1 m (i)

wG ← RMSProp(−∇wG m i=1 L , α ) end for end for end for

Here, the objective function L is identical to Eq.5, where fX (x) is taken to be r(s, a) + γmax G(z|(s , a)). a

Using a generative model to represent the state-action value distribution allows for an alternative

to explicit exploration strategies. Indeed, at the beginning of the training process, taking an action

ment. Here, π denotes the greedy policy.

is analogous to using a decaying exploration strategy (since G has not clearly sepa

Continued Discussion and Extended Analysis (Part 3)

GAN Q-learning

Thang Doan∗ Bogdan Mazoure∗

Clare Lyle

School of Computer Science

Abstract

Distributional reinforcement learning (distributional RL) has seen empirical success

in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose

GAN Q-learning, a novel distributional RL method based on generative adversarial

networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.

1 Introduction

Reinforcement learning (RL) has recently had great success in solving complex tasks with continuous

control [21, 22]. However, as these methods often have high variance results while dealing with unstable environments, distributional perspectives on the state-value function in RL have begun to gain popularity . Note that the distributional perspective is distinct from Bayesian approaches to the RL problem as the former models the inherent variability of the returns from a state and not the agent’s confidence in its prediction of the average return. Up to now, deep learning methods in RL used multiple function approximators (typically a network

with shared hidden layers) to fit a state value or state-action value distribution. For instance, used k-heads on the state-action value function Q for every available action and used it to model a distribution. In , a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) instead of the critic, hence allowing for a closed-form derivation of update rules. More recently, introduced a distributional algorithm C which aimed to solve the RL problem by

learning a categorical probability vector over returns Q. Unlike which uses a generative network to learn the underlying transition model of the environment, we utilize a generative network to model the distribution approximation of the Bellman updates. In this work, we build on top of the aforementioned distributional RL methods and introduce a novel way to learn the state-value distribution. Inspired by the analogy between the actor-critic architecture and generative adversarial networks (GANs) , we leverage the later in order to implicitly represent

the distribution of the Bellman target update through a generator/discriminator architecture. We show that, although sometimes volatile, our proposed algorithm is a viable alternative to now considered classical deep Q networks (DQN). We aim to provide a unifying view on distributional RL through ∗ These authors contributed equally.

the minimization of the Earth-Mover distance and without explicitly using the distributional Bellman projection operator on the support of the Q values.

2 Related Work

2.1 Background

Multiple tasks in machine learning require finding an optimal behaviour in a given setting, i.e. solving the reinforcement learning problem. We proceed to formulate the task as follows. Let (S,A,R,P ,s ,γ) be a 5-tuple representing a Markov decision process (MDP) where S is the set of states, A the set of allowed actions, R : S × A → R is the (deterministic or stochastic) reward function, P are the environment transition probabilities and s ⊆ S is the set of initial states. At a

given time step t = 1, 2, 3, ... an agent acts according to a policy π(a|s) = P [at |st ]. The environment is characterized by its set of initial states in which the agent starts, as well as the transition model which encodes the mechanics of moving from one state to another. In order to compare states, we introduce the state value function P V : S → R which gives the expected sum of discounted rewards in ∞ that state. That is, V (s) = Eπ,P [ t=1 γ t Rt |S = s]. The reinforcement learning problem is two-fold: (1) given a fixed policy π we would like to obtain

the correct state value function Vπ (s), ∀s ∈ S and (2) we wish to find the optimal policy π ∗ (a) which yields the highest V (s) for all states of the MDP or, equivalently, π ∗ = argmaxπ Vπ (s). The first task is known in the reinforcement learning literature as prediction and the second as control. In order to find the value function for each state, we need to solve the Bellman equations : Vπ (s) = Eπ,P [R(st , at ) + γVπ (st+1 )|st = s], (1)

for Vπ = {Vπ (s ), ..., Vπ (sn )} and R = {R(s , a ), ..., R(sP n , an )}. If we define a state-action ∞ value function Qπ (s, a) : S × A → R as Qπ (s, a) = Eπ,P [ t=0 γ t R(St , At )|S = s, A = a], then we can rewrite Eq.1 as:

Qπ (s, a) = Eπ,P [R(s, a) + γQπ (s , a )] or

(2) Qπ = R + γPπ Qπ , for Qπ = {Qπ (s , a ), ..., Qπ (sn , an )}. While both Eq.1 and Eq.2 have the following direct solution obtained by matrix inversion: Vπ = (I − γP )−1 R. However it should be noted that this is only well-defined for finite-state MDPs, and further as this computation has O(n2.4 ) complexity , it is only computationally feasible for small

MDPs. Therefore, sample-based algorithms such as Temporal Difference (TD) for prediction

and SARSA or Q-learning [18, 25] for control are preferred for more general classes of environments.

2.2 Distributional Reinforcement Learning

In the setting of distributional reinforcement learning , we seek to learn the distribution of returns from a state, rather than the mean of the returns. We translate the Bellman operator on points to an operator on distributions. The vector of mean rewards therefore becomes a function of reward distributions R(s, a). We can thus represent Qπ (s, a) as Zπ (s, a), a random variable whose law is the returns following a state. Both expected and distributional distributional quantities are linked through

the following relation: Qπ (s, a) = E[Zπ (s, a)]. We finally arrive to the distributional Bellman equations: d T π Zπ (s, a) ≡ R(s, a) + γZπ (S 0 , A ) with fixed point: (3) d Zπ (s, a) = T π Zπ (s, a), ∀s ∈ S ∀a ∈ A where T π denotes an analogue to the well-known Bellman operator now defined over distributions. Eq.3 is the distributional counterpart of Eq.2, where equality holds for sequences of random variables.

In traditional reinforcement learning algorithms, we use experience in the MDP to improve an

estimated state value function in order to minimize the expected distance between the value function’s output for a state and the actual returns from that state. In distributional reinforcement learning, we still aim to minimize the distance between our output and the true distribution, but now we have more

freedom in how we choose our definition of "distance", as there are many metrics on probability distributions which have subtly different properties.

The p−Wasserstein metric between two real-valued random variables X and Y with cumulative

distribution functions FX and FY is given by  1  p inf E |X − Y |p 

Wp (FX , FY ) = (4)

(X,Y ) (X,Y )

For p = 1 we recover the widely known Earth-Mover distance. More generally, for any 1 ≤ p ≤ q ≤ ∞, Wp (FX , FY ) ≤ Wq (FX , FY ) holds and is useful in contraction arguments. The maximal p−Wasserstein metric, W p = sup(s,a) Wp (Z(s, a), Z 0 (s, a)) is defined over state-action tuples (s, a) ∈ S × A for any two value distributions Z, Z 0 . It has been shown that while T π is a contraction under the maximal p−Wasserstein metric , the Bellman optimality operator T is not necessarily a contraction. This result implies that the control

setting requires a treatment different from prediction, which is done in through the C algorithm in order to guarantee proper convergence.

2.3 Generative Models

Generative models such as hidden Markov models (HMMs), restricted Boltzmann machines (RBMs)

, variational auto-encoders (VAEs) and generative adversarial networks (GANs) learn the distribution of data for all classes. Moreover, they provide a mechanism which allows to sample new observations from the learned distribution. In this work, we make use of generative adversarial networks which consist of two neural networks playing a zero-sum game against each other. The generator network G : Z → X is a mapping from a high-dimensional noise space Z = Rdz onto the input space X on which a target distribution fX

is defined. The generator’s task consists in fitting the underlying distribution of observed data fX as closely as possible. The discriminator network D : X → R ∩ [0, 1] scores each input as the probability of coming from the real data distribution fX or from the generator G. Both networks are gradually improved through alternating or simultaneous gradient descent updates.

The classical GAN algorithm minimizes the Jensen-Shannon divergence (JS) between the real and

generated distributions. Recently, suggested to replace the JS metric by the Wasserstein-1 or

Earth-Mover divergence. We make use of an improved version of this algorithm, Wasserstein GAN

with Gradient Penalty . It’s objective is given below: min max E [D(x)] + E [−D(x)] + p(λ), (5)

G D x∼fX (x) x∼G(z)

where p(λ) = λ(||∇x̃ D(x̃)|| − 1)2 , x̃ = εx + (1 − ε)G(Z), ε ∼ Uniform(0, 1), and Z ∼ fZ (z). Setting λ = 0 recovers the original WGAN objective.

3 GAN Q-learning

3.1 Motivation

We borrow the two-player game analogy from the original GAN paper : the generator network’s purpose is to produce realistic samples of the optimal state-action value distribution (estimate of Zπ∗ (s, a)). On the other hand, the discriminator network aims to distinguish real samples of T Z(s, a) from the samples Z(s, a) outputted by G. The generator network improves its performance through the signal from the discriminator, which is reminiscent of the actor-critic architecture .

3.2 Algorithm

At each timestep, G receives stochastic noise z ∼ N (0, 1) and a state s as input and returns a sample G(z|(s, a)) for every action a from the current estimate of Z(s, a). We then select the action a∗ = max G(z|(s, a)). The agent then applies the chosen action a∗ , receives a reward r and a transitions to state s . The tuple (s, a, r, s ) is saved into a replay buffer B as done in . Each

update step consists in sampling a tuple (s, a, r, s ) uniformly from the buffer B and proceed to update the generator and discriminator according to Eq 5. Values obtained from the Bellman backup operator x ≡ x(s, a, r, s ) = r + γmaxG(z|(s , a)) are a considered as coming from the real distribution. The discriminator’s goal is to differentiate between the Bellman target x = r + γmaxG(z|(s , a)) and the output produced by Z(s, a). We obtain the a following updates for G and D, respectively:

  E [DwD (x|(s, a))] − E [DwD (X|(s, a))] + p(λ) , (s,a)∼B (s,a,r,s

 )∼B

X∼Gw (z|(s,a))

L(wD , wG ) = G

(6)  E [−D wD (X|(s, a))] , (s,a)∼B    z∼N (0,1) 

X∼Gw (z|(s,a))

where wG , wD are weights of the generator and discriminator networks with respect to which the gradient updates w(t+1) ← w(t) − αt ∇w(t) L(w(t) ) are taken. Note that to further stabilize the training process, one can use a target network G updated every k epochs as in . Due to the nature of GANs, we advise training the model in a batch setting using experience replay and a second (target) network for a more stable training process.

Algorithm 1 GAN Q-learning

Input: MDP M , discriminator and generator networks D, G, learning rate α,ndisc the number of updates of the discriminator, ngen the number of updates of the generator, gradient penalty coefficient λ, batch size m. Initialize replay buffer B to capacity N, D and G with random weights, Z, s , a . t←0 for episode = 1, ..., M do for t = 1, ..., Tmax do sample a batch z ∼ N (0, 1) at ← maxG(z|(st , a)) a sample st+1 ∼ P (·|st , at )

Store transition (st ,at ,rt ,st+1 ) in B

{Updating Discriminator} for n = 1, ..., ndisc do

Sample minibatch {s, a, r, s }m i=1 from B

m sample batch ( {z}i=1 ∼ N (0, 1) ri , s terminal

Set yi = r + γmax G(z |(s , a )), otherwise

i i i i ai sample a batch {}m i=1 ∼ N (0, 1) x̃i ← i yi + (1 − i )max G(zi |(s0i , ai )) ai

L(i) ← D(G(zi |(si , a∗i )|(sP ∗ ∗ ∗

i , ai )) − D(yi |(si , ai )) + λ(|∇x̃ D(x̃i |(si , ai ))| − 1)

1 m

wD ← RMSProp(∇wD m i=1 L(i) , α ) end for {Updating Generator} for n = 1, ..., ngen do sample a batch of {z (i) }m i=1 P ∼ N (0, 1)

1 m (i)

wG ← RMSProp(−∇wG m i=1 L , α ) end for end for end for

Here, the objective function L is identical to Eq.5, where fX (x) is taken to be r(s, a) + γmax G(z|(s , a)). a

Using a generative model to represent the state-action value distribution allows for an alternative

to explicit exploration strategies. Indeed, at the beginning of the training process, taking an action

ment. Here, π denotes the greedy policy.

is analogous to using a decaying exploration strategy (since G has not clearly sepa

Continued Discussion and Extended Analysis (Part 4)

GAN Q-learning

Thang Doan∗ Bogdan Mazoure∗

Clare Lyle

School of Computer Science

Abstract

Distributional reinforcement learning (distributional RL) has seen empirical success

in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose

GAN Q-learning, a novel distributional RL method based on generative adversarial

networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.

1 Introduction

Reinforcement learning (RL) has recently had great success in solving complex tasks with continuous

control [21, 22]. However, as these methods often have high variance results while dealing with unstable environments, distributional perspectives on the state-value function in RL have begun to gain popularity . Note that the distributional perspective is distinct from Bayesian approaches to the RL problem as the former models the inherent variability of the returns from a state and not the agent’s confidence in its prediction of the average return. Up to now, deep learning methods in RL used multiple function approximators (typically a network

with shared hidden layers) to fit a state value or state-action value distribution. For instance, used k-heads on the state-action value function Q for every available action and used it to model a distribution. In , a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) instead of the critic, hence allowing for a closed-form derivation of update rules. More recently, introduced a distributional algorithm C which aimed to solve the RL problem by

learning a categorical probability vector over returns Q. Unlike which uses a generative network to learn the underlying transition model of the environment, we utilize a generative network to model the distribution approximation of the Bellman updates. In this work, we build on top of the aforementioned distributional RL methods and introduce a novel way to learn the state-value distribution. Inspired by the analogy between the actor-critic architecture and generative adversarial networks (GANs) , we leverage the later in order to implicitly represent

the distribution of the Bellman target update through a generator/discriminator architecture. We show that, although sometimes volatile, our proposed algorithm is a viable alternative to now considered classical deep Q networks (DQN). We aim to provide a unifying view on distributional RL through ∗ These authors contributed equally.

the minimization of the Earth-Mover distance and without explicitly using the distributional Bellman projection operator on the support of the Q values.

2 Related Work

2.1 Background

Multiple tasks in machine learning require finding an optimal behaviour in a given setting, i.e. solving the reinforcement learning problem. We proceed to formulate the task as follows. Let (S,A,R,P ,s ,γ) be a 5-tuple representing a Markov decision process (MDP) where S is the set of states, A the set of allowed actions, R : S × A → R is the (deterministic or stochastic) reward function, P are the environment transition probabilities and s ⊆ S is the set of initial states. At a

given time step t = 1, 2, 3, ... an agent acts according to a policy π(a|s) = P [at |st ]. The environment is characterized by its set of initial states in which the agent starts, as well as the transition model which encodes the mechanics of moving from one state to another. In order to compare states, we introduce the state value function P V : S → R which gives the expected sum of discounted rewards in ∞ that state. That is, V (s) = Eπ,P [ t=1 γ t Rt |S = s]. The reinforcement learning problem is two-fold: (1) given a fixed policy π we would like to obtain

the correct state value function Vπ (s), ∀s ∈ S and (2) we wish to find the optimal policy π ∗ (a) which yields the highest V (s) for all states of the MDP or, equivalently, π ∗ = argmaxπ Vπ (s). The first task is known in the reinforcement learning literature as prediction and the second as control. In order to find the value function for each state, we need to solve the Bellman equations : Vπ (s) = Eπ,P [R(st , at ) + γVπ (st+1 )|st = s], (1)

for Vπ = {Vπ (s ), ..., Vπ (sn )} and R = {R(s , a ), ..., R(sP n , an )}. If we define a state-action ∞ value function Qπ (s, a) : S × A → R as Qπ (s, a) = Eπ,P [ t=0 γ t R(St , At )|S = s, A = a], then we can rewrite Eq.1 as:

Qπ (s, a) = Eπ,P [R(s, a) + γQπ (s , a )] or

(2) Qπ = R + γPπ Qπ , for Qπ = {Qπ (s , a ), ..., Qπ (sn , an )}. While both Eq.1 and Eq.2 have the following direct solution obtained by matrix inversion: Vπ = (I − γP )−1 R. However it should be noted that this is only well-defined for finite-state MDPs, and further as this computation has O(n2.4 ) complexity , it is only computationally feasible for small

MDPs. Therefore, sample-based algorithms such as Temporal Difference (TD) for prediction

and SARSA or Q-learning [18, 25] for control are preferred for more general classes of environments.

2.2 Distributional Reinforcement Learning

In the setting of distributional reinforcement learning , we seek to learn the distribution of returns from a state, rather than the mean of the returns. We translate the Bellman operator on points to an operator on distributions. The vector of mean rewards therefore becomes a function of reward distributions R(s, a). We can thus represent Qπ (s, a) as Zπ (s, a), a random variable whose law is the returns following a state. Both expected and distributional distributional quantities are linked through

the following relation: Qπ (s, a) = E[Zπ (s, a)]. We finally arrive to the distributional Bellman equations: d T π Zπ (s, a) ≡ R(s, a) + γZπ (S 0 , A ) with fixed point: (3) d Zπ (s, a) = T π Zπ (s, a), ∀s ∈ S ∀a ∈ A where T π denotes an analogue to the well-known Bellman operator now defined over distributions. Eq.3 is the distributional counterpart of Eq.2, where equality holds for sequences of random variables.

In traditional reinforcement learning algorithms, we use experience in the MDP to improve an

estimated state value function in order to minimize the expected distance between the value function’s output for a state and the actual returns from that state. In distributional reinforcement learning, we still aim to minimize the distance between our output and the true distribution, but now we have more

freedom in how we choose our definition of "distance", as there are many metrics on probability distributions which have subtly different properties.

The p−Wasserstein metric between two real-valued random variables X and Y with cumulative

distribution functions FX and FY is given by  1  p inf E |X − Y |p 

Wp (FX , FY ) = (4)

(X,Y ) (X,Y )

For p = 1 we recover the widely known Earth-Mover distance. More generally, for any 1 ≤ p ≤ q ≤ ∞, Wp (FX , FY ) ≤ Wq (FX , FY ) holds and is useful in contraction arguments. The maximal p−Wasserstein metric, W p = sup(s,a) Wp (Z(s, a), Z 0 (s, a)) is defined over state-action tuples (s, a) ∈ S × A for any two value distributions Z, Z 0 . It has been shown that while T π is a contraction under the maximal p−Wasserstein metric , the Bellman optimality operator T is not necessarily a contraction. This result implies that the control

setting requires a treatment different from prediction, which is done in through the C algorithm in order to guarantee proper convergence.

2.3 Generative Models

Generative models such as hidden Markov models (HMMs), restricted Boltzmann machines (RBMs)

, variational auto-encoders (VAEs) and generative adversarial networks (GANs) learn the distribution of data for all classes. Moreover, they provide a mechanism which allows to sample new observations from the learned distribution. In this work, we make use of generative adversarial networks which consist of two neural networks playing a zero-sum game against each other. The generator network G : Z → X is a mapping from a high-dimensional noise space Z = Rdz onto the input space X on which a target distribution fX

is defined. The generator’s task consists in fitting the underlying distribution of observed data fX as closely as possible. The discriminator network D : X → R ∩ [0, 1] scores each input as the probability of coming from the real data distribution fX or from the generator G. Both networks are gradually improved through alternating or simultaneous gradient descent updates.

The classical GAN algorithm minimizes the Jensen-Shannon divergence (JS) between the real and

generated distributions. Recently, suggested to replace the JS metric by the Wasserstein-1 or

Earth-Mover divergence. We make use of an improved version of this algorithm, Wasserstein GAN

with Gradient Penalty . It’s objective is given below: min max E [D(x)] + E [−D(x)] + p(λ), (5)

G D x∼fX (x) x∼G(z)

where p(λ) = λ(||∇x̃ D(x̃)|| − 1)2 , x̃ = εx + (1 − ε)G(Z), ε ∼ Uniform(0, 1), and Z ∼ fZ (z). Setting λ = 0 recovers the original WGAN objective.

3 GAN Q-learning

3.1 Motivation

We borrow the two-player game analogy from the original GAN paper : the generator network’s purpose is to produce realistic samples of the optimal state-action value distribution (estimate of Zπ∗ (s, a)). On the other hand, the discriminator network aims to distinguish real samples of T Z(s, a) from the samples Z(s, a) outputted by G. The generator network improves its performance through the signal from the discriminator, which is reminiscent of the actor-critic architecture .

3.2 Algorithm

At each timestep, G receives stochastic noise z ∼ N (0, 1) and a state s as input and returns a sample G(z|(s, a)) for every action a from the current estimate of Z(s, a). We then select the action a∗ = max G(z|(s, a)). The agent then applies the chosen action a∗ , receives a reward r and a transitions to state s . The tuple (s, a, r, s ) is saved into a replay buffer B as done in . Each

update step consists in sampling a tuple (s, a, r, s ) uniformly from the buffer B and proceed to update the generator and discriminator according to Eq 5. Values obtained from the Bellman backup operator x ≡ x(s, a, r, s ) = r + γmaxG(z|(s , a)) are a considered as coming from the real distribution. The discriminator’s goal is to differentiate between the Bellman target x = r + γmaxG(z|(s , a)) and the output produced by Z(s, a). We obtain the a following updates for G and D, respectively:

  E [DwD (x|(s, a))] − E [DwD (X|(s, a))] + p(λ) , (s,a)∼B (s,a,r,s

 )∼B

X∼Gw (z|(s,a))

L(wD , wG ) = G

(6)  E [−D wD (X|(s, a))] , (s,a)∼B    z∼N (0,1) 

X∼Gw (z|(s,a))

where wG , wD are weights of the generator and discriminator networks with respect to which the gradient updates w(t+1) ← w(t) − αt ∇w(t) L(w(t) ) are taken. Note that to further stabilize the training process, one can use a target network G updated every k epochs as in . Due to the nature of GANs, we advise training the model in a batch setting using experience replay and a second (target) network for a more stable training process.

Algorithm 1 GAN Q-learning

Input: MDP M , discriminator and generator networks D, G, learning rate α,ndisc the number of updates of the discriminator, ngen the number of updates of the generator, gradient penalty coefficient λ, batch size m. Initialize replay buffer B to capacity N, D and G with random weights, Z, s , a . t←0 for episode = 1, ..., M do for t = 1, ..., Tmax do sample a batch z ∼ N (0, 1) at ← maxG(z|(st , a)) a sample st+1 ∼ P (·|st , at )

Store transition (st ,at ,rt ,st+1 ) in B

{Updating Discriminator} for n = 1, ..., ndisc do

Sample minibatch {s, a, r, s }m i=1 from B

m sample batch ( {z}i=1 ∼ N (0, 1) ri , s terminal

Set yi = r + γmax G(z |(s , a )), otherwise

i i i i ai sample a batch {}m i=1 ∼ N (0, 1) x̃i ← i yi + (1 − i )max G(zi |(s0i , ai )) ai

L(i) ← D(G(zi |(si , a∗i )|(sP ∗ ∗ ∗

i , ai )) − D(yi |(si , ai )) + λ(|∇x̃ D(x̃i |(si , ai ))| − 1)

1 m

wD ← RMSProp(∇wD m i=1 L(i) , α ) end for {Updating Generator} for n = 1, ..., ngen do sample a batch of {z (i) }m i=1 P ∼ N (0, 1)

1 m (i)

wG ← RMSProp(−∇wG m i=1 L , α ) end for end for end for

Here, the objective function L is identical to Eq.5, where fX (x) is taken to be r(s, a) + γmax G(z|(s , a)). a

Using a generative model to represent the state-action value distribution allows for an alternative

to explicit exploration strategies. Indeed, at the beginning of the training process, taking an action

ment. Here, π denotes the greedy policy.

is analogous to using a decaying exploration strategy (since G has not clearly sepa

Continued Discussion and Extended Analysis (Part 5)

GAN Q-learning

Thang Doan∗ Bogdan Mazoure∗

Clare Lyle

School of Computer Science

Abstract

Distributional reinforcement learning (distributional RL) has seen empirical success

in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose

GAN Q-learning, a novel distributional RL method based on generative adversarial

networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.

1 Introduction

Reinforcement learning (RL) has recently had great success in solving complex tasks with continuous

control [21, 22]. However, as these methods often have high variance results while dealing with unstable environments, distributional perspectives on the state-value function in RL have begun to gain popularity . Note that the distributional perspective is distinct from Bayesian approaches to the RL problem as the former models the inherent variability of the returns from a state and not the agent’s confidence in its prediction of the average return. Up to now, deep learning methods in RL used multiple function approximators (typically a network

with shared hidden layers) to fit a state value or state-action value distribution. For instance, used k-heads on the state-action value function Q for every available action and used it to model a distribution. In , a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) instead of the critic, hence allowing for a closed-form derivation of update rules. More recently, introduced a distributional algorithm C which aimed to solve the RL problem by

learning a categorical probability vector over returns Q. Unlike which uses a generative network to learn the underlying transition model of the environment, we utilize a generative network to model the distribution approximation of the Bellman updates. In this work, we build on top of the aforementioned distributional RL methods and introduce a novel way to learn the state-value distribution. Inspired by the analogy between the actor-critic architecture and generative adversarial networks (GANs) , we leverage the later in order to implicitly represent

the distribution of the Bellman target update through a generator/discriminator architecture. We show that, although sometimes volatile, our proposed algorithm is a viable alternative to now considered classical deep Q networks (DQN). We aim to provide a unifying view on distributional RL through ∗ These authors contributed equally.

the minimization of the Earth-Mover distance and without explicitly using the distributional Bellman projection operator on the support of the Q values.

2 Related Work

2.1 Background

Multiple tasks in machine learning require finding an optimal behaviour in a given setting, i.e. solving the reinforcement learning problem. We proceed to formulate the task as follows. Let (S,A,R,P ,s ,γ) be a 5-tuple representing a Markov decision process (MDP) where S is the set of states, A the set of allowed actions, R : S × A → R is the (deterministic or stochastic) reward function, P are the environment transition probabilities and s ⊆ S is the set of initial states. At a

given time step t = 1, 2, 3, ... an agent acts according to a policy π(a|s) = P [at |st ]. The environment is characterized by its set of initial states in which the agent starts, as well as the transition model which encodes the mechanics of moving from one state to another. In order to compare states, we introduce the state value function P V : S → R which gives the expected sum of discounted rewards in ∞ that state. That is, V (s) = Eπ,P [ t=1 γ t Rt |S = s]. The reinforcement learning problem is two-fold: (1) given a fixed policy π we would like to obtain

the correct state value function Vπ (s), ∀s ∈ S and (2) we wish to find the optimal policy π ∗ (a) which yields the highest V (s) for all states of the MDP or, equivalently, π ∗ = argmaxπ Vπ (s). The first task is known in the reinforcement learning literature as prediction and the second as control. In order to find the value function for each state, we need to solve the Bellman equations : Vπ (s) = Eπ,P [R(st , at ) + γVπ (st+1 )|st = s], (1)

for Vπ = {Vπ (s ), ..., Vπ (sn )} and R = {R(s , a ), ..., R(sP n , an )}. If we define a state-action ∞ value function Qπ (s, a) : S × A → R as Qπ (s, a) = Eπ,P [ t=0 γ t R(St , At )|S = s, A = a], then we can rewrite Eq.1 as:

Qπ (s, a) = Eπ,P [R(s, a) + γQπ (s , a )] or

(2) Qπ = R + γPπ Qπ , for Qπ = {Qπ (s , a ), ..., Qπ (sn , an )}. While both Eq.1 and Eq.2 have the following direct solution obtained by matrix inversion: Vπ = (I − γP )−1 R. However it should be noted that this is only well-defined for finite-state MDPs, and further as this computation has O(n2.4 ) complexity , it is only computationally feasible for small

MDPs. Therefore, sample-based algorithms such as Temporal Difference (TD) for prediction

and SARSA or Q-learning [18, 25] for control are preferred for more general classes of environments.

2.2 Distributional Reinforcement Learning

In the setting of distributional reinforcement learning , we seek to learn the distribution of returns from a state, rather than the mean of the returns. We translate the Bellman operator on points to an operator on distributions. The vector of mean rewards therefore becomes a function of reward distributions R(s, a). We can thus represent Qπ (s, a) as Zπ (s, a), a random variable whose law is the returns following a state. Both expected and distributional distributional quantities are linked through

the following relation: Qπ (s, a) = E[Zπ (s, a)]. We finally arrive to the distributional Bellman equations: d T π Zπ (s, a) ≡ R(s, a) + γZπ (S 0 , A ) with fixed point: (3) d Zπ (s, a) = T π Zπ (s, a), ∀s ∈ S ∀a ∈ A where T π denotes an analogue to the well-known Bellman operator now defined over distributions. Eq.3 is the distributional counterpart of Eq.2, where equality holds for sequences of random variables.

In traditional reinforcement learning algorithms, we use experience in the MDP to improve an

estimated state value function in order to minimize the expected distance between the value function’s output for a state and the actual returns from that state. In distributional reinforcement learning, we still aim to minimize the distance between our output and the true distribution, but now we have more

freedom in how we choose our definition of "distance", as there are many metrics on probability distributions which have subtly different properties.

The p−Wasserstein metric between two real-valued random variables X and Y with cumulative

distribution functions FX and FY is given by  1  p inf E |X − Y |p 

Wp (FX , FY ) = (4)

(X,Y ) (X,Y )

For p = 1 we recover the widely known Earth-Mover distance. More generally, for any 1 ≤ p ≤ q ≤ ∞, Wp (FX , FY ) ≤ Wq (FX , FY ) holds and is useful in contraction arguments. The maximal p−Wasserstein metric, W p = sup(s,a) Wp (Z(s, a), Z 0 (s, a)) is defined over state-action tuples (s, a) ∈ S × A for any two value distributions Z, Z 0 . It has been shown that while T π is a contraction under the maximal p−Wasserstein metric , the Bellman optimality operator T is not necessarily a contraction. This result implies that the control

setting requires a treatment different from prediction, which is done in through the C algorithm in order to guarantee proper convergence.

2.3 Generative Models

Generative models such as hidden Markov models (HMMs), restricted Boltzmann machines (RBMs)

, variational auto-encoders (VAEs) and generative adversarial networks (GANs) learn the distribution of data for all classes. Moreover, they provide a mechanism which allows to sample new observations from the learned distribution. In this work, we make use of generative adversarial networks which consist of two neural networks playing a zero-sum game against each other. The generator network G : Z → X is a mapping from a high-dimensional noise space Z = Rdz onto the input space X on which a target distribution fX

is defined. The generator’s task consists in fitting the underlying distribution of observed data fX as closely as possible. The discriminator network D : X → R ∩ [0, 1] scores each input as the probability of coming from the real data distribution fX or from the generator G. Both networks are gradually improved through alternating or simultaneous gradient descent updates.

The classical GAN algorithm minimizes the Jensen-Shannon divergence (JS) between the real and

generated distributions. Recently, suggested to replace the JS metric by the Wasserstein-1 or

Earth-Mover divergence. We make use of an improved version of this algorithm, Wasserstein GAN

with Gradient Penalty . It’s objective is given below: min max E [D(x)] + E [−D(x)] + p(λ), (5)

G D x∼fX (x) x∼G(z)

where p(λ) = λ(||∇x̃ D(x̃)|| − 1)2 , x̃ = εx + (1 − ε)G(Z), ε ∼ Uniform(0, 1), and Z ∼ fZ (z). Setting λ = 0 recovers the original WGAN objective.

3 GAN Q-learning

3.1 Motivation

We borrow the two-player game analogy from the original GAN paper : the generator network’s purpose is to produce realistic samples of the optimal state-action value distribution (estimate of Zπ∗ (s, a)). On the other hand, the discriminator network aims to distinguish real samples of T Z(s, a) from the samples Z(s, a) outputted by G. The generator network improves its performance through the signal from the discriminator, which is reminiscent of the actor-critic architecture .

3.2 Algorithm

At each timestep, G receives stochastic noise z ∼ N (0, 1) and a state s as input and returns a sample G(z|(s, a)) for every action a from the current estimate of Z(s, a). We then select the action a∗ = max G(z|(s, a)). The agent then applies the chosen action a∗ , receives a reward r and a transitions to state s . The tuple (s, a, r, s ) is saved into a replay buffer B as done in . Each

update step consists in sampling a tuple (s, a, r, s ) uniformly from the buffer B and proceed to update the generator and discriminator according to Eq 5. Values obtained from the Bellman backup operator x ≡ x(s, a, r, s ) = r + γmaxG(z|(s , a)) are a considered as coming from the real distribution. The discriminator’s goal is to differentiate between the Bellman target x = r + γmaxG(z|(s , a)) and the output produced by Z(s, a). We obtain the a following updates for G and D, respectively:

  E [DwD (x|(s, a))] − E [DwD (X|(s, a))] + p(λ) , (s,a)∼B (s,a,r,s

 )∼B

X∼Gw (z|(s,a))

L(wD , wG ) = G

(6)  E [−D wD (X|(s, a))] , (s,a)∼B    z∼N (0,1) 

X∼Gw (z|(s,a))

where wG , wD are weights of the generator and discriminator networks with respect to which the gradient updates w(t+1) ← w(t) − αt ∇w(t) L(w(t) ) are taken. Note that to further stabilize the training process, one can use a target network G updated every k epochs as in . Due to the nature of GANs, we advise training the model in a batch setting using experience replay and a second (target) network for a more stable training process.

Algorithm 1 GAN Q-learning

Input: MDP M , discriminator and generator networks D, G, learning rate α,ndisc the number of updates of the discriminator, ngen the number of updates of the generator, gradient penalty coefficient λ, batch size m. Initialize replay buffer B to capacity N, D and G with random weights, Z, s , a . t←0 for episode = 1, ..., M do for t = 1, ..., Tmax do sample a batch z ∼ N (0, 1) at ← maxG(z|(st , a)) a sample st+1 ∼ P (·|st , at )

Store transition (st ,at ,rt ,st+1 ) in B

{Updating Discriminator} for n = 1, ..., ndisc do

Sample minibatch {s, a, r, s }m i=1 from B

m sample batch ( {z}i=1 ∼ N (0, 1) ri , s terminal

Set yi = r + γmax G(z |(s , a )), otherwise

i i i i ai sample a batch {}m i=1 ∼ N (0, 1) x̃i ← i yi + (1 − i )max G(zi |(s0i , ai )) ai

L(i) ← D(G(zi |(si , a∗i )|(sP ∗ ∗ ∗

i , ai )) − D(yi |(si , ai )) + λ(|∇x̃ D(x̃i |(si , ai ))| − 1)

1 m

wD ← RMSProp(∇wD m i=1 L(i) , α ) end for {Updating Generator} for n = 1, ..., ngen do sample a batch of {z (i) }m i=1 P ∼ N (0, 1)

1 m (i)

wG ← RMSProp(−∇wG m i=1 L , α ) end for end for end for

Here, the objective function L is identical to Eq.5, where fX (x) is taken to be r(s, a) + γmax G(z|(s , a)). a

Using a generative model to represent the state-action value distribution allows for an alternative

to explicit exploration strategies. Indeed, at the beginning of the training process, taking an action

ment. Here, π denotes the greedy policy.

is analogous to using a decaying exploration strategy (since G has not clearly sepa