Ai Parental Controls

This project focuses on ai parental controls using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Mingxing Tan 1 Quoc V. Le 1

Abstract EfficientNet-B7

84 B6 AmoebaNet-C

Convolutional Neural Networks (ConvNets) are B5 AmoebaNet-A B4 commonly developed at a fixed resource budget, NASNet-A SENet

and then scaled up for better accuracy if more B3 resources are available. In this paper, we sys- ResNeXt-101

80 Inception-ResNet-v2

tematically study model scaling and identify that carefully balancing network depth, width, and res- Xception olution can lead to better performance. Based 78 ResNet-152

Top1 Acc. #Params

EfficientNet-B1 79.1% 7.8M

method that uniformly scales all dimensions of 76 EfficientNet-B3 81.6% 12M

EfficientNet-B4 82.9% 19M

effective compound coefficient. We demonstrate Inception-v2

NASNet-A EfficientNet-B7 84.3% 66M

the effectiveness of this method on scaling up †

Not plotted

ResNet-34 MobileNets and ResNet. 0 20 40 60 80 100 120 140 160 180

Number of Parameters (Millions)

To go even further, we use neural architec-

ture search to design a new baseline network for single-crop, single-model. Our EfficientNets significantly out- and scale it up to obtain a family of models, perform other ConvNets. In particular, EfficientNet-B7 achieves called EfficientNets, which achieve much new state-of-the-art 84.3% top-1 accuracy but being 8.4x smaller better accuracy and efficiency than previous and 6.1x faster than GPipe. EfficientNet-B1 is 7.6x smaller and ConvNets. In particular, our EfficientNet-B7 5.7x faster than ResNet-152. Details are in Table 2 and 4.

achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing time larger. However, the process of scaling up ConvNets ConvNet. Our EfficientNets also transfer well and has never been well understood and there are currently many achieve state-of-the-art accuracy on CIFAR-100 ways to do it. The most common way is to scale up Con- learning datasets, with an order of magnitude Komodakis, 2016). Another less common, but increasingly

fewer parameters. Source code is at https: popular, method is to scale up models by image resolution master/models/official/efficientnet. only one of the three dimensions – depth, width, and image size. Though it is possible to scale two or three dimensions arbitrarily, arbitrary scaling requires tedious manual tuning and still often yields sub-optimal accuracy and efficiency.

1. Introduction

In this paper, we want to study and rethink the process

Scaling up ConvNets is widely used to achieve better accu- of scaling up ConvNets. In particular, we investigate the up from ResNet-18 to ResNet-200 by using more layers; ConvNets that can achieve better accuracy and efficiency? geNet top-1 accuracy by scaling up a baseline model four dimensions of network width/depth/resolution, and surpris- ingly such balance can be achieved by simply scaling each Google Research, Brain Team, Mountain View, CA. Corre- of them with constant ratio. Based on this observation, we

propose a simple yet effective compound scaling method. Proceedings of the 36 th International Conference on Machine Unlike conventional practice that arbitrary scales these fac- Learning, Long Beach, California, PMLR 97, 2019. tors, our method uniformly scales network width, depth,

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

higher higher resolution HxW resolution resolution

(a) baseline (b) width scaling (c) depth scaling (d) resolution scaling (e) compound scaling

width, depth, or resolution. (e) is our proposed compound scaling method that uniformly scales all three dimensions with a fixed ratio.

and resolution with a set of fixed scaling coefficients. For of-the-art accuracy on 5 out of 8 widely used datasets, while example, if we want to use 2N times more computational reducing parameters by up to 21x than existing ConvNets. resources, then we can simply increase the network depth by αN , width by β N , and image size by γ N , where α, β, γ are 2. Related Work constant coefficients determined by a small grid search on between our scaling method and conventional methods. 2012) won the 2012 ImageNet competition, ConvNets have

become increasingly more accurate by going bigger: while

Intuitively, the compound scaling method makes sense be-

cause if the input image is bigger, then the network needs achieves 74.8% top-1 accuracy with about 6.8M parameters, more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image. In 82.7% top-1 accuracy with 145M parameters. Recently, and empirical results (Zagoruyko & Komodakis, 2016) both show that there exists certain relationship between network parameters: it is so big that it can only be trained with a width and depth, but to our best knowledge, we are the

specialized pipeline parallelism library by partitioning the first to empirically quantify the relationship among all three network and spreading each part to a different accelera- dimensions of network width, depth, and resolution. tor. While these models are mainly designed for ImageNet, We demonstrate that our scaling method work well on exist- recent studies have shown better ImageNet models also per- go even further, we use neural architecture search (Zoph Although higher accuracy is critical for many applications,

network, and scale it up to obtain a family of models, called further accuracy gain needs better efficiency.

EfficientNets. Figure 1 summarizes the ImageNet perfor-

mance, where our EfficientNets significantly outperform ConvNet Efficiency: Deep ConvNets are often over- using 8.4x fewer parameters and running 6.1x faster on in- duce model size by trading accuracy for efficiency. As mo- 2016), our EfficientNet-B4 improves the top-1 accuracy craft efficient mobile-size ConvNets, such as SqueezeNets

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

tecture search becomes increasingly popular in designing the spatial dimension is gradually shrunk but the channel 2019), and achieves even better efficiency than hand-crafted input shape h224, 224, 3i to final output shape h7, 7, 512i. mobile ConvNets by extensively tuning the network width,

Unlike regular ConvNet designs that mostly focus on find-

depth, convolution kernel types and sizes. However, it is ing the best layer architecture Fi , model scaling tries to ex- unclear how to apply these techniques for larger models that pand the network length (Li ), width (Ci ), and/or resolution have much larger design space and much more expensive (Hi , Wi ) without changing Fi predefined in the baseline tuning cost. In this paper, we aim to study model efficiency network. By fixing Fi , model scaling simplifies the design for super large ConvNets that surpass state-of-the-art accu-

problem for new resource constraints, but it still remains racy. To achieve this goal, we resort to model scaling. a large design space to explore different Li , Ci , Hi , Wi for Model Scaling: There are many ways to scale a Con- each layer. In order to further reduce the design space, we 2016) can be scaled down (e.g., ResNet-18) or up (e.g., stant ratio. Our target is to maximize the model accuracy ResNet-200) by adjusting network depth (#layers), while for any given resource constraints, which can be formulated

WideResNet (Zagoruyko & Komodakis, 2016) and Mo- as an optimization problem: width (#channels). It is also well-recognized that bigger max Accuracy N (d, w, r) d,w,r input image size will help accuracy with the overhead of K

F̂id·L̂i Xhr·Ĥi ,r·Ŵi ,w·Ĉi i

i=1...s 2018) have shown that network depth and width are both Memory(N ) ≤ target memory important for ConvNets’ expressive power, it still remains FLOPS(N ) ≤ target flops an open question of how to effectively scale a ConvNet to (2) achieve better efficiency and accuracy. Our work systemati- where w, d, r are coefficients for scaling network width,

cally and empirically studies ConvNet scaling for all three depth, and resolution; F̂i , L̂i , Ĥi , Ŵi , Ĉi are predefined pa- dimensions of network width, depth, and resolutions. rameters in baseline network (see Table 1 as an example).

3. Compound Model Scaling 3.2. Scaling Dimensions

In this section, we will formulate the scaling problem, study The main difficulty of problem 2 is that the optimal d, w, r different approaches, and propose our new scaling method. depend on each other and the values change under different resource constraints. Due to this difficulty, conventional

3.1. Problem Formulation methods mostly scale ConvNets in one of these dimensions:

A ConvNet Layer i can be defined as a function: Yi =

Depth (dd): Scaling network depth is the most common way

Fi (Xi ), where Fi is the operator, Yi is output tensor, Xi is

input tensor, with tensor shape hHi , Wi , Ci i1 , where Hi and Wi are spatial dimension and Ci is the channel dimension.

ConvNet can capture richer and more complex features, and

A ConvNet N can be represented by aJ list of composed lay-

generalize well on new tasks. However, deeper networks ers: N = Fk ... F2 F1 (X1 ) = j=1...k Fj (X1 ). In are also more difficult to train due to the vanishing gradient practice, ConvNet layers are often partitioned into multiple problem (Zagoruyko & Komodakis, 2016). Although sev- stages and all layers in each stage share the same architec- and batch normalization (Ioffe & Szegedy, 2015), alleviate and all layers in each stage has the same convolutional type the training problem, the accuracy gain of very deep network

except the first layer performs down-sampling. Therefore, diminishes: for example, ResNet-1000 has similar accuracy we can define a ConvNet as: as ResNet-101 even though it has much more layers. Figure 3 (middle) shows our empirical study on scaling a baseline K

FiLi XhHi ,Wi ,Ci i

N = (1)

i=1...s model with different depth coefficient d, further suggesting the diminishing accuracy return for very deep ConvNets. where FiLi denotes layer Fi is repeated Li times in stage i, hHi , Wi , Ci i denotes the shape of input tensor X of layer

Width (w w ): Scaling network width is commonly used for

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

81 81 81

80 w=5.0 80 80

w=3.8 d=6.0 d=8.0 w=2.6 r=2.2 r=2.5 79 79 79 r=1.9 d=3.0d=4.0 r=1.7

w=1.8 d=2.0 78 78 78 r=1.5 r=1.3 w=1.4 77 77 77

76 w=1.0 76 d=1.0 76 r=1.0

75 75 75 0 2 4 6 8 0 1 2 3 4 0 1 2 3 FLOPS (Billions) FLOPS (Billions) FLOPS (Billions)

networks with larger width, depth, or resolution tend to achieve higher accuracy, but the accuracy gain quickly saturate after reaching 80%, demonstrating the limitation of single dimension scaling. Baseline network is described in Table 1.

modakis, 2016), wider networks tend to be able to capture more fine-grained features and are easier to train. However, 81

extremely wide but shallow networks tend to have difficul- ties in capturing higher level features. Our empirical results 80 in Figure 3 (left) show that the accuracy quickly saturates when networks become much wider with larger w. 79

Resolution (rr ): With higher resolution input images, Con- 78 d=1.0, r=1.0 vNets can potentially capture more fine-grained patterns. d=1.0, r=1.3 Starting from 224x224 in early ConvNets, modern Con- 77 d=2.0, r=1.0 vNets tend to use 299x299 (Szegedy et al., 2016) or 331x331 d=2.0, r=1.3

0 5 10 15 20 25 curacy with 480x480 resolution. Higher resolutions, such as 600x600, are also widely used in object detection ConvNets Figure 4. Scaling Network Width for Different Baseline Net- results of scaling network resolutions, where indeed higher coefficient (w). All baseline networks are from Table 1. The first baseline network (d=1.0, r=1.0) has 18 convolutional layers with

resolutions improve accuracy, but the accuracy gain dimin- resolution 224x224, while the last baseline (d=2.0, r=1.3) has 36 ishes for very high resolutions (r = 1.0 denotes resolution layers with resolution 299x299. 224x224 and r = 2.5 denotes resolution 560x560). The above analyses lead us to the first observation: order to capture more fine-grained patterns with more pixels Observation 1 – Scaling up any dimension of network in high resolution images. These intuitions suggest that we

width, depth, or resolution improves accuracy, but the accu- need to coordinate and balance different scaling dimensions racy gain diminishes for bigger models. rather than conventional single-dimension scaling.

To validate our intuitions, we compare width scaling under

3.3. Compound Scaling different network depths and resolutions, as shown in Figure

4. If we only scale network width w without changing

We empirically observe that different scaling dimensions are

depth (d=1.0) and resolution (r=1.0), the accuracy saturates not independent. Intuitively, for higher resolution images, quickly. With deeper (d=2.0) and higher resolution (r=2.0), we should increase network depth, such that the larger re- width scaling achieves much better accuracy under the same ceptive fields can help capture similar features that include FLOPS cost. These results lead us to the second observation: more pixels in bigger images. Correspondingly, we should also increase network width when resolution is higher, in

Observation 2 – In order to pursue better accuracy and

In some literature, scaling number of channels is called “depth efficiency, it is critical to balance all dimensions of network multiplier”, which means the same as our width coefficient w. width, depth, and resolution during ConvNet scaling.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

have already tried to arbitrarily balance network width and a stage i with L̂i layers, with input resolution hĤi , Ŵi i and output depth, but they all require tedious manual tuning. channels Ĉi . Notations are adopted from equation 2.

Stage Operator Resolution #Channels #Layers

In this paper, we propose a new compound scaling method, i F̂i Ĥi × Ŵi Ĉi L̂i which use a compound coefficient φ to uniformly scales

1 Conv3x3 224 × 224 32 1

network width, depth, and resolution in a principled way: 2 MBConv1, k3x3 112 × 112 16 1

3 MBConv6, k3x3 112 × 112 24 2

depth: d = αφ 4 MBConv6, k5x5 56 × 56 40 2

5 MBConv6, k3x3 28 × 28 80 3

width: w = β φ 6 MBConv6, k5x5 14 × 14 112 3

7 MBConv6, k5x5 14 × 14 192 4

resolution: r = γ φ (3) 8 MBConv6, k3x3 7×7 320 1 2 2 9 Conv1x1 & Pooling & FC 7×7 1280 1 s.t. α · β · γ ≈ 2 α ≥ 1, β ≥ 1, γ ≥ 1

Net, except our EfficientNet-B0 is slightly bigger due to

where α, β, γ are constants that can be determined by a the larger FLOPS target (our FLOPS target is 400M). Ta- small grid search. Intuitively, φ is a user-specified coeffi- ble 1 shows the architecture of EfficientNet-B0. Its main cient that controls how many more resources are available building block is mobile inverted bottleneck MBConv (San- spectively. Notably, the FLOPS of a regular convolution op Starting from the baseline EfficientNet-B0, we apply our

is proportional to d, w2 , r2 , i.e., doubling network depth compound scaling method to scale it up with two steps: will double FLOPS, but doubling network width or resolu- tion will increase FLOPS by four times. Since convolution • STEP 1: we first fix φ = 1, assuming twice more re- ops usually dominate the computation cost in ConvNets, sources available, and do a small grid search of α, β, γ scaling a ConvNet with equation 3 will approximately in- based on Equation 2 and 3. In particular, we find

φ crease total FLOPS by α · β 2 · γ 2 . In this paper, we the best values for EfficientNet-B0 are α = 1.2, β = constraint α · β 2 · γ 2 ≈ 2 such that for any new φ, the total 1.1, γ = 1.15, under constraint of α · β 2 · γ 2 ≈ 2. FLOPS will approximately3 increase by 2φ . • STEP 2: we then fix α, β, γ as constants and scale up baseline network with different φ using Equation 3, to

4. EfficientNet Architecture obtain EfficientNet-B1 to B7 (Details in Table 2).

Since model scaling does not change layer operators F̂i Notably, it is possible to achieve even better performance by in baseline network, having a good baseline network is searching for α, β, γ directly around a large model, but the also critical. We will evaluate our scaling method using search cost becomes prohibitively more expensive on larger existing ConvNets, but in order to better demonstrate the models. Our method solves this issue by only doing search

effectiveness of our scaling method, we have also developed once on the small baseline network (step 1), and then use a new mobile-size baseline, called EfficientNet. the same scaling coefficients for all other models (step 2). work by leveraging a multi-objective neural architecture 5. Experiments search that optimizes both accuracy and FLOPS. Specifi- and use ACC(m)×[F LOP S(m)/T ]w as the optimization existing ConvNets and the new proposed EfficientNets.

goal, where ACC(m) and F LOP S(m) denote the accu- racy and FLOPS of model m, T is the target FLOPS and 5.1. Scaling Up MobileNets and ResNets w=-0.07 is a hyperparameter for controlling the trade-off

As a proof of concept, we first apply our scaling method

tency since we are not targeting any specific hardware de- shows the ImageNet results of scaling them in different vice. Our search produces an efficient network, which we ways. Compared to other single-dimension scaling methods, name EfficientNet-B0. Since we use the same search space our compound scaling method improves the accuracy on all these models, suggesting the effectiveness of our proposed FLOPS may differ from theoretical value due to rounding. scaling method for general existing ConvNets.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

baseline EfficientNet-B0 using different compound coefficient φ in Equation 3. ConvNets with similar top-1/top-5 accuracy are grouped together for efficiency comparison. Our scaled EfficientNet models consistently reduce parameters and FLOPS by an order of magnitude (up to 8.4x parameter reduction and up to 16x FLOPS reduction) than existing ConvNets.

Model Top-1 Acc. Top-5 Acc. #Params Ratio-to-EfficientNet #FLOPs Ratio-to-EfficientNet EfficientNet-B0 77.1% 93.3% 5.3M 1x 0.39B 1x EfficientNet-B1 79.1% 94.4% 7.8M 1x 0.70B 1x

Xception (Chollet, 2017) 79.0% 94.5% 23M 3.0x 8.4B 12x EfficientNet-B2 80.1% 94.9% 9.2M 1x 1.0B 1x EfficientNet-B3 81.6% 95.7% 12M 1x 1.8B 1x

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings presented in the source material. The practical implications and implementation considerations are further elaborated to support complete understanding of the subject matter across different application domains and use cases.

Speaker Verification Xvector

Ai Parental Controls

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Mingxing Tan 1 Quoc V. Le 1

Abstract EfficientNet-B7

84 B6 AmoebaNet-C

80 Inception-ResNet-v2

Top1 Acc. #Params

EfficientNet-B1 79.1% 7.8M

EfficientNet-B4 82.9% 19M

NASNet-A EfficientNet-B7 84.3% 66M

Not plotted

Number of Parameters (Millions)

To go even further, we use neural architec-

1. Introduction

In this paper, we want to study and rethink the process

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Intuitively, the compound scaling method makes sense be-

EfficientNets. Figure 1 summarizes the ImageNet perfor-

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Unlike regular ConvNet designs that mostly focus on find-

F̂id·L̂i Xhr·Ĥi ,r·Ŵi ,w·Ĉi i

3. Compound Model Scaling 3.2. Scaling Dimensions

3.1. Problem Formulation methods mostly scale ConvNets in one of these dimensions:

A ConvNet Layer i can be defined as a function: Yi =

Depth (dd): Scaling network depth is the most common way

Fi (Xi ), where Fi is the operator, Yi is output tensor, Xi is

ConvNet can capture richer and more complex features, and

A ConvNet N can be represented by aJ list of composed lay-

FiLi XhHi ,Wi ,Ci i

N = (1)

Width (w w ): Scaling network width is commonly used for

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

80 w=5.0 80 80

To validate our intuitions, we compare width scaling under

4. If we only scale network width w without changing

We empirically observe that different scaling dimensions are

Observation 2 – In order to pursue better accuracy and

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Stage Operator Resolution #Channels #Layers

1 Conv3x3 224 × 224 32 1

3 MBConv6, k3x3 112 × 112 24 2

5 MBConv6, k3x3 28 × 28 80 3

7 MBConv6, k5x5 14 × 14 192 4

Net, except our EfficientNet-B0 is slightly bigger due to

4. EfficientNet Architecture obtain EfficientNet-B1 to B7 (Details in Table 2).

As a proof of concept, we first apply our scaling method

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Frequently Asked Questions

What is this project about?