Ai Code Generation

This project focuses on ai code generation using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

Published as a conference paper at ICLR 20

DARTS: D IFFERENTIABLE A RCHITECTURE S EARCH

Hanxiao Liu∗ Karen Simonyan Yiming Yang

CMU DeepMind CMU

A BSTRACT

This paper addresses the scalability challenge of architecture search by formulating

the task in a differentiable manner. Unlike conventional approaches of applying evo- lution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures

for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

Our implementation has been made publicly available to facilitate further research

on efficient architecture search algorithms.

1 I NTRODUCTION

Discovering state-of-the-art neural network architectures requires substantial effort of human experts.

Recently, there has been a growing interest in developing algorithmic solutions to automate the

manual process of architecture design. The automatically searched architectures have achieved highly

The best existing architecture search algorithms are computationally demanding despite their remark-

able performance. For example, obtaining a state-of-the-art architecture for CIFAR-1 and ImageNet 2018), but the fundamental challenge of scalability remains. An inherent cause of inefficiency for the dominant approaches, e.g. based on RL, evolution, MCTS (Negrinho & Gordon, 2017), SMBO (Liu is treated as a black-box optimization problem over a discrete domain, which leads to a large number of architecture evaluations required. In this work, we approach the problem from a different angle, and propose a method for efficient

architecture search called DARTS (Differentiable ARchiTecture Search). Instead of searching over a discrete set of candidate architectures, we relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. The data efficiency of gradient-based optimization, as opposed to inefficient black-box search, allows DARTS to achieve competitive performance with the state of the art using orders of magnitude

less computation resources. It also outperforms another recent efficient architecture search method, yet it is generic enough handle both convolutional and recurrent architectures. The idea of searching architectures within a continuous domain is not new (Saxena & Verbeek, 2016; ∗ Current affiliation: Google Brain.

Published as a conference paper at ICLR 20

distinctions. While prior works seek to fine-tune a specific aspect of an architecture, such as filter shapes or branching patterns in a convolutional network, DARTS is able to learn high-performance architecture building blocks with complex graph topologies within a rich search space. Moreover, DARTS is not restricted to any specific architecture family, and is applicable to both convolutional and recurrent networks. In our experiments (Sect. 3) we show that DARTS is able to design a convolutional cell that achieves

2.7 ± 0.09% test error on CIFAR-1 for image classification using 3.3M parameters, which is three orders of magnitude more computation resources. The same convolutional cell also achieves 26.7% top-1 error when transferred to ImageNet (mobile setting), which is comparable to the best RL cell that achieves 55.7 test perplexity on Penn Treebank (PTB), outperforming both extensively tuned Our contributions can be summarized as follows:

• We introduce a novel algorithm for differentiable network architecture search based on bilevel optimization, which is applicable to both convolutional and recurrent architectures. • Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieves highly competitive results on CIFAR-1 and outperforms the state of the art on PTB. This is a very interesting result, considering that so far the best architecture search methods used non-differentiable search techniques,

• We achieve remarkable efficiency improvement (reducing the cost of architecture discovery to a few GPU days), which we attribute to the use of gradient-based optimization as opposed to non-differentiable search techniques. • We show that the architectures learned by DARTS on CIFAR-1 and PTB are transferable to ImageNet and WikiText-2, respectively.

The implementation of DARTS is available at https://github.com/quark0/darts

2 D IFFERENTIABLE A RCHITECTURE S EARCH

We describe our search space in general form in Sect. 2.1, where the computation procedure for an architecture (or a cell in it) is represented as a directed acyclic graph. We then introduce a simple continuous relaxation scheme for our search space which leads to a differentiable learning objective for the joint optimization of the architecture and its weights (Sect. 2.2). Finally, we propose an approximation technique to make the algorithm computationally feasible and efficient (Sect. 2.3).

2.1 S EARCH S PACE

cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or recursively connected to form a recurrent network. A cell is a directed acyclic graph consisting of an ordered sequence of N nodes. Each node x(i) is a latent representation (e.g. a feature map in convolutional networks) and each directed edge (i, j) is associated with some operation o(i,j) that transforms x(i) . We assume the cell to have two input

nodes and a single output node. For convolutional cells, the input nodes are defined as the cell outputs the current step and the state carried from the previous step. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. Each intermediate node is computed based on all of its predecessors: X x(j) = o(i,j) (x(i) ) (1) i<j

Published as a conference paper at ICLR 20

relaxation of the search space by placing a mixture of candidate operations on each edge. (c) Joint optimization of the mixing probabilities and the network weights by solving a bilevel optimization problem. (d) Inducing the final architecture from the learned mixing probabilities.

A special zero operation is also included to indicate a lack of connection between two nodes. The task of learning the cell therefore reduces to learning the operations on its edges.

2.2 C ONTINUOUS R ELAXATION AND O PTIMIZATION

Let O be a set of candidate operations (e.g., convolution, max pooling, zero) where each operation represents some function o(·) to be applied to x(i) . To make the search space continuous, we relax the categorical choice of a particular operation to a softmax over all possible operations: (i,j)

X exp(αo )

ō(i,j) (x) = P (i,j) o(x) (2) o∈O o ∈O exp(αo )

where the operation mixing weights for a pair of nodes (i, j) are parameterized by a vector α(i,j) of dimension |O|. The task of architecture search then reduces to learning a set of continuous variables α = α(i,j) , as illustrated in Fig. 1. At the end of search, a discrete architecture can be obtained by (i,j) replacing each mixed operation ō(i,j) with the most likely operation, i.e., o(i,j) = argmaxo∈O αo . In the following, we refer to α as the (encoding of the) architecture.

After relaxation, our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters). Analogous to architecture search using RL (Zoph where the validation set performance is treated as the reward or fitness, DARTS aims to optimize the validation loss, but using gradient descent. Denote by Ltrain and Lval the training and the validation loss, respectively. Both losses are deter- mined not only by the architecture α, but also the weights w in the network. The goal for architecture

search is to find α∗ that minimizes the validation loss Lval (w∗ , α∗ ), where the weights w∗ associated with the architecture are obtained by minimizing the training loss w∗ = argminw Ltrain (w, α∗ ). α as the upper-level variable and w as the lower-level variable:

min Lval (w∗ (α), α) (3) α s.t. w∗ (α) = argminw Ltrain (w, α) (4)

could be viewed as a special type of hyperparameter, although its dimension is substantially higher than scalar-valued hyperparameters such as the learning rate, and it is harder to optimize.

Published as a conference paper at ICLR 20

Algorithm 1: DARTS – Differentiable Architecture Search

Create a mixed operation ō(i,j) parametrized by α(i,j) for each edge (i, j)

1. Update architecture α by descending ∇α Lval (w − ξ∇w Ltrain (w, α), α)

(ξ = 0 if using first-order approximation)

2. Update weights w by descending ∇w Ltrain (w, α)

Derive the final architecture based on the learned α.

2.3 A PPROXIMATE A RCHITECTURE G RADIENT

Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimization. We therefore propose a simple approximation scheme as follows: ∇α Lval (w∗ (α), α) (5) ≈∇α Lval (w − ξ∇w Ltrain (w, α), α) (6) where w denotes the current weights maintained by the algorithm, and ξ is the learning rate for a step of inner optimization. The idea is to approximate w∗ (α) by adapting w using only a single training

step, without solving the inner optimization (equation 4) completely by training until convergence. the inner optimization and thus ∇w Ltrain (w, α) = 0. The iterative procedure is outlined in Alg. 1. While we are not currently aware of the convergence guarantees for our optimization algorithm, in practice it is able to reach a fixed point with a suitable choice of ξ 1 . We also note that when momentum is enabled for weight optimisation, the one-step unrolled learning objective in equation 6 is modified accordingly and all of our analysis still applies.

Applying chain rule to the approximate architecture gradient (equation 6) yields

∇α Lval (w , α) − ξ∇2α,w Ltrain (w, α)∇w Lval (w , α) (7) where w = w − ξ∇w Ltrain (w, α) denotes the weights for a one-step forward model. The expression above contains an expensive matrix-vector product in its second term. Fortunately, the complexity can be substantially reduced using the finite difference approximation. Let be a small scalar and w± = w ± ∇w Lval (w , α). Then: ∇α Ltrain (w+ , α) − ∇α Ltrain (w− , α) ∇2α,w Ltrain (w, α)∇w Lval (w , α) ≈ (8)

2 Evaluating the finite difference requires only two forward passes for the weights and two backward passes for α, and the complexity is reduced from O(|α||w|) to O(|α| + |w|).

First-order Approximation When ξ = 0, the second-order derivative in equation 7 will disappear. In this case, the architecture gradient is given by ∇α Lval (w, α), corresponding to the simple heuristic of optimizing the validation loss by assuming the current w is the same as w∗ (α). This leads to some speed-up but empirically worse performance, according to our experimental results in Table 1 and the gradient formulation with ξ > 0 as the second-order approximation.

2.4 D ERIVING D ISCRETE A RCHITECTURES

To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes. The strength exp(α(i,j) ) of an operation is defined as P o (i,j) . To make our derived architecture comparable with o ∈O exp(αo 0 )

A simple working strategy is to set ξ equal to the learning rate for w’s optimizer. We found = 0.01/k∇w Lval (w , α)k to be sufficiently accurate in all of our experiments.

Published as a conference paper at ICLR 20

= 0.7 3 = 0.5 when Lval (w, α) = αw − 2α + 1 and Ltrain (w, α) = = 0.3 2 =0 w −2αw+α , starting from (α(0) , w(0) ) = (2, −2). The

Weights (w)

analytical solution for the corresponding bilevel optimiza-

1 tion problem is (α∗ , w∗ ) = (1, 1), which is highlighted

0 in the red circle. The dashed red line indicates the fea-

sible set where constraint equation 4 is satisfied exactly (namely, weights in w are optimal for the given architec-

2 ture α). The example shows that a suitable choice of ξ

0.5 1.0 1.5 2.0 2.5 3.0 3.5 helps to converge to a better local optimum.

Architecture ( )

The zero operations are excluded in the above for two reasons. First, we need exactly k non-zero incoming edges per node for fair comparison with the existing models. Second, the strength of the zero operations is underdetermined, as increasing the logits of zero operations only affects the scale of the resulting node representations, and does not affect the final classification outcome due to the presence of batch normalization (Ioffe & Szegedy, 2015).

3 E XPERIMENTS AND R ESULTS

Our experiments on CIFAR-1 and PTB consist of two stages, architecture search (Sect. 3.1) and architecture evaluation (Sect. 3.2). In the first stage, we search for the cell architectures using DARTS, and determine the best cells based on their validation performance. In the second stage, we use these cells to construct larger architectures, which we train from scratch and report their performance on the test set. We also investigate the transferability of the best cells learned on CIFAR-1 and PTB by

evaluating them on ImageNet and WikiText-2 (WT2) respectively.

3.1 A RCHITECTURE S EARCH

3.1.1 S EARCHING FOR C ONVOLUTIONAL C ELLS ON CIFAR-1

We include the following operations in O: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero. All operations are of stride one (if applicable) and the convolved feature maps are padded to preserve their spatial resolution. We use the ReLU-Conv-BN order for convolutional operations, and each Our convolutional cell consists of N = 7 nodes, among which the output node is defined as the

depthwise concatenation of all the intermediate nodes (input nodes excluded). The rest of the setup stacking multiple cells together. The first and second nodes of cell k are set equal to the outputs of cell k − 2 and cell k − 1, respectively, and 1 × 1 convolutions are inserted as necessary. Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two. The architecture encoding therefore is (αnormal , αreduce ), where

αnormal is shared by all the normal cells and αreduce is shared by all the reduction cells. Detailed experimental setup for this section can be found in Sect. A.1.1.

3.1.2 S EARCHING FOR R ECURRENT C ELLS ON P ENN T REEBANK

Our set of available operations includes linear transformations followed by one of tanh, relu, sigmoid activations, as well as the identity mapping and the zero operation. The choice of these candidate Our recurrent cell consists of N = 1 nodes. The very first intermediate node is obtained by linearly transforming the two input nodes, adding up the results and then passing through a tanh activation

Published as a conference paper at ICLR 20

CIFAR-1 CIFAR-1 7

Penn Treebank 7

Penn Treebank

1 DARTS 1 DARTS (run 1) DARTS DARTS (run 1) AmoebaNet-A DARTS (run 2) 7 ENAS 7 DARTS (run 2) NASNet-A DARTS (run 3) DARTS (run 3)

Best Valid Perplexity So Far

1 1

Best Valid Error So Far (%)

DARTS (run 4) 6 6 DARTS (run 4) 1 1 AmoebaNet-A ENAS

Valid Perplexity

Valid Error (%)

NASNet-A 6 6

1 1 6 6 1 1 6 6 1 1 6 6

9 9 6 6 8 8 6 6 0 5 1 1 2 2 0 1 2 0 1 2 3 4 0 1 2 3 4

GPU Hours GPU Hours GPU Hours GPU Hours Penn Treebank. We keep track of the most recent architectures over time. Each architecture snapshot is re-trained from scratch using the training set (for 1 epochs on CIFAR-1 and for 3 epochs on PTB) and then evaluated on the validation set. For each task, we repeat the experiments for 4 times

with different random seeds, and report the median and the best (per run) validation performance of the architectures over time. As references, we also report the results (under the same evaluation setup; with comparable number of parameters) of the best existing cells discovered using RL or evolution,

the cell output is defined as the average of all the intermediate nodes. As in ENAS, we enable batch normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. Our recurrent network consists of only a single cell, i.e. we do not assume any repetitive patterns within the recurrent architecture. Detailed experimental setup for this section can be found in Sect. A.1.2. sep_conv_3x x_{t} h_{t-1}

skip_connect 2 0 sep_conv_3x 1 sigmoid sep_conv_3x c_{k} c_{k-2} 0 1

sep_conv_3x dil_conv_3x skip_connect relu identity

2 relu

tanh 4 max_pool_3x skip_connect 2 5 3 c_{k-2} max_pool_3x

c_{k} sigmoid relu tanh

0 skip_connect

max_pool_3x 3 skip_connect 6 8 7 c_{k-1} max_pool_3x max_pool_3x h_{t}

3.2 A RCHITECTURE E VALUATION

To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for

Published as a conference paper at ICLR 20

a short period (1 epochs on CIFAR-1 and 3 epochs on PTB). This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive (Fig. 3).

To evaluate the selected architecture, we randomly initialize its weights (weights learned during the

search process are discarded), train it from scratch, and report its performance on the test set. We note the test set is never used for architecture search or architecture selection.

Detailed experimental setup for architecture evaluation on CIFAR-1 and PTB can be found in

Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-1 and PTB, we further investigated the

transferability of our best convolutional cell (searched on CIFAR-10) and recurrent cell (searched on

PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of

the transfer learning experiments can be found in Sect. A.2.3 and Sect. A.2.4.

Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (1.5 GPU days).

Test Error Params Search Cost Search

Architecture #ops

(%) (M) (GPU days) Method

Random search baseline‡ + cutout 3.2 ± 0.1 3.2 4 7 random

DARTS (first order) + cutout 3.0 ± 0.1 3.3 1.5 7 gradient-based

DARTS (second order) + cutout 2.7 ± 0.0 3.3 4 7 gradient-based

* Obtained by repeating ENAS for 8 times using the code publicly released by the authors. The cell for final evaluation is chosen according to the same selection protocol as for DARTS. † Obtained by training the corresponding architectures using our setup. ‡ Best architecture among 2 samples according to the validation error after 1 training epochs.

the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (3 GPU days).

Perplexity Params Search Cost Search

Architecture #ops

valid test (M) (GPU days) Method

NAS (Zoph & Le, 2017) – 64.0 2 1e CPU days 4 RL

Random search baseline‡ 61.8 59.4 2 2 4 random

DARTS (first order) 60.2 57.6 2 0.5 4 gradient-based

DARTS (second order) 58.1 55.7 2 1 4 gradient-based

* † Obtained by training the corresponding architecture using our setup. ‡ Best architecture among 8 samples according to the validation perplexity after 3 training epochs.

Published as a conference paper at ICLR 20

Test Error (%) Params +× Search Cost Search

Architecture top-1 top-5 (M) (M) (GPU days) Method

DARTS (searched on CIFAR-10) 26.7 8.7 4.7 5 4 gradient-based

3.3 R ESULTS A NALYSIS

The CIFAR-1 results for convolutional architectures are presented in Table 1. Notably, DARTS

three orders of magnitude less computation resources (i.e. 1.5 or 4 GPU days vs 20 GPU days for

NASNet and 31 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS

parameters. The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings presented in the source material.

Speaker Verification Xvector

Ai Code Generation

Published as a conference paper at ICLR 20

DARTS: D IFFERENTIABLE A RCHITECTURE S EARCH

Hanxiao Liu∗ Karen Simonyan Yiming Yang

CMU DeepMind CMU

A BSTRACT

This paper addresses the scalability challenge of architecture search by formulating

Our implementation has been made publicly available to facilitate further research

1 I NTRODUCTION

Recently, there has been a growing interest in developing algorithmic solutions to automate the

The best existing architecture search algorithms are computationally demanding despite their remark-

Published as a conference paper at ICLR 20

The implementation of DARTS is available at https://github.com/quark0/darts

2 D IFFERENTIABLE A RCHITECTURE S EARCH

2.1 S EARCH S PACE

Published as a conference paper at ICLR 20

2.2 C ONTINUOUS R ELAXATION AND O PTIMIZATION

X exp(αo )

Published as a conference paper at ICLR 20

Algorithm 1: DARTS – Differentiable Architecture Search

Create a mixed operation ō(i,j) parametrized by α(i,j) for each edge (i, j)

1. Update architecture α by descending ∇α Lval (w − ξ∇w Ltrain (w, α), α)

2. Update weights w by descending ∇w Ltrain (w, α)

2.3 A PPROXIMATE A RCHITECTURE G RADIENT

Applying chain rule to the approximate architecture gradient (equation 6) yields

2.4 D ERIVING D ISCRETE A RCHITECTURES

Published as a conference paper at ICLR 20

Weights (w)

1 tion problem is (α∗ , w∗ ) = (1, 1), which is highlighted

0 in the red circle. The dashed red line indicates the fea-

2 ture α). The example shows that a suitable choice of ξ

Architecture ( )

3 E XPERIMENTS AND R ESULTS

3.1 A RCHITECTURE S EARCH

3.1.1 S EARCHING FOR C ONVOLUTIONAL C ELLS ON CIFAR-1

3.1.2 S EARCHING FOR R ECURRENT C ELLS ON P ENN T REEBANK

Published as a conference paper at ICLR 20

CIFAR-1 CIFAR-1 7

Penn Treebank 7

Penn Treebank

Best Valid Perplexity So Far

Best Valid Error So Far (%)

Valid Perplexity

Valid Error (%)

NASNet-A 6 6

2 relu

0 skip_connect

3.2 A RCHITECTURE E VALUATION

Published as a conference paper at ICLR 20

To evaluate the selected architecture, we randomly initialize its weights (weights learned during the

Detailed experimental setup for architecture evaluation on CIFAR-1 and PTB can be found in

Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-1 and PTB, we further investigated the

PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of

Test Error Params Search Cost Search

Architecture #ops

Random search baseline‡ + cutout 3.2 ± 0.1 3.2 4 7 random

DARTS (first order) + cutout 3.0 ± 0.1 3.3 1.5 7 gradient-based

DARTS (second order) + cutout 2.7 ± 0.0 3.3 4 7 gradient-based

Perplexity Params Search Cost Search

Architecture #ops

NAS (Zoph & Le, 2017) – 64.0 2 1e CPU days 4 RL

Random search baseline‡ 61.8 59.4 2 2 4 random

DARTS (first order) 60.2 57.6 2 0.5 4 gradient-based

DARTS (second order) 58.1 55.7 2 1 4 gradient-based

Published as a conference paper at ICLR 20

Test Error (%) Params +× Search Cost Search

DARTS (searched on CIFAR-10) 26.7 8.7 4.7 5 4 gradient-based

3.3 R ESULTS A NALYSIS

The CIFAR-1 results for convolutional architectures are presented in Table 1. Notably, DARTS

NASNet and 31 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS

Frequently Asked Questions

What is this project about?