Devops Image Caching Optimization

Devops Automation Of Software Deployment Pipelin · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: anomaly detection / classification using IoT sensor streams and timestamped device measurements.

AIDA: Associative DNN Inference Accelerator

L. Yavits, R. Kaplan and R. Ginosar

Abstract— We propose AIDA, an inference engine for accelerating fully-connected (FC) layers of Deep Neural Network (DNN).

AIDA is an associative in-memory processor, where the bulk of data never leaves the confines of the memory arrays, and processing is performed in-situ. AIDA’s area and energy efficiency strongly benefit from sparsity and lower arithmetic precision. We show that AIDA outperforms the state of art inference accelerator, EIE, by 14.5× (peak performance) and 2.5× (throughput).

Index Terms—Deep Neural Network, RNN, associative processing in memory, matrix-vector multiplication, accelerator.

1 INTRODUCTION

F ully-connected (FC) layers are extensively used in Con-

volutional, Recurrent (Long Short-Term Memory) and

Multi-Layer Perceptron Neural Networks . FC

power efficiency of AIDA to those of ASIC and FPGA based accelerators.

The rest of this paper is organized as follows. Section 2

layers perform matrix-vector multiplication (M×V) and introduces AIDA architecture. Section 3 discusses the FC non-linear activation function calculation. A number of layer implementation. Section 4 presents the evaluation ASIC and FPGA based platforms that accelerate FC layers and Section 5 offers conclusions. exist .

We propose AIDA, an associative processing in-

2 ACCELERATOR DESIGN

memory based inference engine. AIDA is a general-pur- pose massively parallel accelerator that calculates M×V 2.1 Associative Processing (both sparse and dense matrices and vectors are sup- Associative processor (AP) is a non-von-Neumann in- ported) and non-linear activation function (typically, memory computer . AP is based on Content Addressa- RELU, sigmoid or tanh), and implements FC of networks ble Memory (CAM), which allows comparing the entire

such as AlexNet and CTC-3L-421H-UNI , outper- dataset to a search pattern (key), tagging the matching row, forming the EIE in terms of peak performance and and writing another pattern to all tagged rows. AP per- throughput by 14.5× and 2.5× respectively. forms no computations in conventional sense. Most arith- The main sources of AIDA performance and efficiency metic and logic operations can be structured as series of

are (1) ultra-high internal bandwidth achieved by each Boolean functions, which are implemented by the AP by memory bit being directly connected to processing transis- perfect induction approach as follows. tors, (2) associative access that provides intrinsic support The dataset is stored in CAM, typically one data ele- for dense and sparse datasets, and (3) associative arithme- ment per CAM row (constituting a Processing Unit, PU).

tic that natively supports variable wordlength and preci- AP controller sequentially matches all possible input com- sions, as well as perfect induction approach to implementing binations (hence the name perfect induction) of the argu- arithmetic functions, from multiplication to non-linear ac- ments against the entire CAM content. The matching CAM tivation functions. rows are tagged, and the corresponding function values

This paper makes the following contributions: (precalculated and embedded in AP microcode), are writ-

• We present the first associative processing in- ten into the designated output fields of the tagged rows. memory accelerator for FC layers of DNN, that can For an 𝑚-bit argument 𝑥 (𝑥 ∈ dataset), any Boolean efficiently support dense and sparse fixed point function 𝑏(𝑥) has at most 2𝑚 possible values. Therefore, a arithmetic networks; perfect induction approach would incur up to 𝑂(2𝑚 ) cy-

• We introduce the associative CSR (ACSR) format, a cles, regardless of the dataset size. modified CRS format that enables sparsity utiliza- Arithmetic operations can sometimes (when there are tion in associative processor-in-memory. few “don’t cares” in the truth table) be more efficiently im- • We present an algorithm of massively parallel asso- plemented in a bit-serial, word-parallel manner, reducing ciative in-memory implementation of FC layers. time complexity from 𝑂(2𝑚 ) to 𝑂(𝑚) .

• We evaluate AIDA on FC layers of CNN and RNN, and compare the performance, throughput and 2.2 Hardware Design and Instruction Set ———————————————— The architecture of AIDA is presented in Fig. 1. The • Leonid Yavits, E-mail: [email protected]. CAM Array comprises bit cells (further described below) • Roman Kaplan, E-mail: [email protected]. organized in bit-columns and word-rows. Several special

• Ran Ginosar, E-mail: [email protected]. registers are appended to the CAM array. The COMPARE Authors are with the Department of Electrical Engineering, Technion-Israel and WRITE KEY registers contains the patterns (keys) to Institute of Technology, Haifa 3200000, Israel. Manuscript submitted: 20-July-2018. Manuscript accepted: XX-Xxx- be compared against and written. The MASK register de-

XXXX. Final manuscript received: YY-Yyy-YYYY

fines the active fields for write and read operations, ena- 𝑊 are multiplied by nonzero elements of an input activa- bling bit selectivity. The TAG register marks the rows that tion vector 𝐵, and dot products are spatially accumulated are matched by the compare operation and are to be af- (reduced) to produce an output activation vector element fected by consecutive parallel write. A static associative 𝐶. The inner product and activation function are calculated

NOR bitcell is shown in Fig. 1(b). Its two main components in parallel for all rows of weight matrix and all elements of are the 6-Transistors (6T) SRAM bit cell (two inverters and output activation vector, respectively.

2 write enable transistors) and the 4T wired NMOS XOR. The weight matrix 𝑊 is stored in AIDA’s CAM, a single

nonzero matrix element per CAM row, in a modified CSR format, which we call associative CSR (ACSR) (Fig. 2). The value and the column index vectors are identical to those of conventional CSR, while the row pointer is replaced by a two-bit row flag vector, the elements of which are stored alongside 𝑊’s values and column indices. Row flag signals the 1st and the last element of a matrix row, as shown in

Fig. 2. Every CAM row, a PU, stores nonzero weight ma-

trix, input activation and output activation elements. There is an optional temporary field for storing temporary varia- bles.

Fig. 1: (a) AIDA Architecture, (b) 10T NOR CAM cell, (c) TAG Logic.

To compare the key data word against the data stored

in the CAM array (the entire row, a number of bits or a sin- gle bit), the Match line is precharged and the inverted key is set on CBL and ¬CBL lines. In the columns that should be ignored during comparison, CBL and ¬CBL lines are set to ‘0’. If all unmasked bits in a row match the key, the Match line remains high and a ‘1’ is sampled into the correspond- ing TAG FF. If the key differs from the row data (even in one bit), the Match line discharges and a ‘0’ is written into the TAG FF.

In AIDA, compare is typically followed by a parallel Fig. 2: AIDA CAM map example. The CAM depth is scaled to the weight write into the unmasked bits of all tagged words. To write matrix nnz. Base addresses are detailed in Fig. 3. data (from the WRITE KEY register) into CAM, each TAG

The AIDA FC layer algorithm is presented in Fig. 3. It

FF (set earlier by the compare) is connected to the corre-

consists of four stages: activation broadcast, multiplica- sponding Write line. If a row matched during the compare, tion, soft reduction and activation function. the key data set on WBL and ¬WBL lines is written into it

During activation broadcast, each activation either

in accordance with the MASK pattern. Otherwise (in the comes from outside (in the case of the first FC layer), or is case of mismatch), the write does not affect the row. AIDA read from AIDA CAM memory, from the output activation allows simultaneous execution of compare and write oper-

C field (for the subsequent FC layers). It is matched (line 3

ations. of Fig. 3) against and stored in B field (line 4) along all pos-

To read data from memory, the WBL and ¬WBL lines are

sible multiplication candidates in the weight matrix. Lines precharged and the Write line is asserted.

3 and 4 are executed in parallel. Broadcast is done sequen-

Two additional AIDA instructions are if_match, that

tially, activation by activation, but in parallel to all loca- signals ‘1’ if there is at least one match in the entire CAM tions of each activation. array, and Move (dir, step), that moves tag vector up or

In the multiplication step, all nonzero weight-activation

down (set by dir), by a step, which has two values, pairs are multiplied in parallel. Associative multiplication short_step (1) and long_step (for example, 16, as in Fig. is a bit-serial operation the dataflow of which is illustrated 1(c)). in Fig. 4(c). The loops in lines 7 and 8 of Fig. 3 go over all bits of the operands. A single-bit multiplication is per-

3 ALGORITHM formed in two steps, a bitwise AND operation and a full

AIDA implements the following computation: addition (lines 9 and 10). A separate bit-column in the tem- porary field is allocated to store the carry bit during addi- 𝐶 = 𝑓(𝑊 × 𝐵) (1) tion. Both bitwise AND and full addition are implemented where 𝑊 × 𝐵 is M×V, and 𝑓 is a non-linear activation func- using the perfect induction approach, by matching each tion, typically RELU for CNN and sigmoid or tanh for entry of a truth table against the relevant B and W bit-col-

RNN. AIDA implements M×V by inner product method, umns, and substituting the result (the value of the func- where all nonzero elements of each row of a weight matrix tion) into the corresponding bits in the temporary field in all matching rows.

Algorithm: AIDA FC Layer Implementation

Let W be a weight matrix of nnz nonzero elements, stored in ACSR for-

mat, as follows: • WROW FLAG(nnz) is a 2-bit row flag vector, beginning at the row_flag_base; • WCOL INDEX(nnz) is a vector of column pointers, beginning at col_in- dex_base; • WVALUE(nnz) is a vector of nonzero elements of W of wordlength 𝑚, beginning at W_base; • Row j of W is held in consecutive CAM rows (PUs), where the 1st PU is marked by Wrow flag=’01’ and the last PU is marked by Wrow Fig. 4: AIDA bit serial associative multiplication: Perfect induction ap-

flag=’10’; if there is only one element in row 𝑗, it is marked by Wrow proach. flag=’11’ Let B be an input activation vector of nnzB nonzero elements, presented Soft reduction is performed in parallel for all output ac- in the following format: tivation elements, where for each such element, we accu- • BINDEX is a vector of indices; mulate the dot products in a binary tree fashion: first, all

• BVALUE is a vector of nonzero elements of B of wordlength 𝑛, begin- odd and even dot products are summed up; then the odd ning at B_base;

Let C be an output activation vector of wordlength 𝑘, beginning at

and even partial results are summed up, and so on, until C_base; Its sign bit resides at C_sign_bit base; the last pair of partial results are summed up. Soft reduc- Let Temporary field begin at T_base. tion requires (1) placing of odd and even dot products and Main: partial results in the same PU, and (2) adding them up.

1. Activation broadcast: Lines 1 through 2 are responsible for moving the odd dot

// performed in all AIDA PUs in parallel product or partial result to the PU where the even one is

2. for (p = 0 ; p < nnzB; p++) { located. Such move is carried out bit-serially, but in paral-

3. Compare (BINDEX ≡ col_index_base); lel in all PUs, in three steps. First, a relevant bit of the

4. Write (B_base←BVALUE);

source data element is copied to the tag (line 15), then it is 5. } shifted to the tag of the destination PU (lines 1 through

6. Multiplication:

19), and finally it is written to the destination bit-column // performed in all AIDA PUs in parallel

7. for (j = 0 ; j < n; j++) { of the destination PU (line 20). The MSB of the correspond-

8. for (i=0; i < m; i++) { ing row flag element is shifted along with data. Lines 2

9. Bit_AND(T_base+i+j, W_base+i, B_base+j); and 2 comprise the addition operation (performed bit-se-

10. Bit_ADD(C_base+i+j, C_base+i, T_base+j); rially in parallel in all PUs). The reduction is performed

11. } until all “01” patterns in the row flag vector (signaling the 12. } 1st element of each weight matrix row) are replaced by “11”

13. Soft Reduction: pattern which signals the end of reduction (lines 2 and

// performed in all AIDA PUs in parallel 26).

14. for (j = 0 ; j < k || j== W_row_flag_base+1; j++) {

15. Compare (C_base+j ≡ 1);

In the Activation Function (RELU) stage, we check the

16. if (2𝑗 <long_step) sign of all output activations and reset the negative ones

17. Move(up, short_step); (lines 2 and 29). Other activation functions, such as tanh

18. else or sigmoid, can be implemented using perfect induction

19. Move(up, long_step); approach.

20. Write (B_base+j ← ‘1’);

22. for (j = 0 ; j < k; j++) { 4 EVALUATION

23. Bit_ADD (C_base+j, C_base+j, B_base+j);

24. } 4.1 Methodology

25. Compare (W_row_flag≡’10’); We implemented a custom simulator of AIDA, used for

26. if_match goto ‘Soft Reduction’

performance and power simulation and design space ex-

27. Activation Function (RELU): ploration. We compare AIDA with several state of art ded-

// performed in all AIDA PUs in parallel icated hardware (FPGA and ASIC) platforms. A large

28. Compare (C_sign_bit≡’1’);

29. Write (C_base← ‘000…0’)

number of CNN accelerators (DaDianNao , Angel-Eye Fig. 3: AIDA FC layer implementation algorithm , True North ), some of which share with AIDA archi- tectural techniques such as bit-serial arithmetic (Cnvlutin If the number of unique multipliers and multiplicands ), or processing in memory (PRIME ), implement is limited, which frequently occurs in compressed DNN both convolutional and FC layers, making comparison dif-

(for example to 1 ), the perfect induction method can ficult. We therefore focus on accelerators where FC layer be applied in a bit-parallel manner, by traversing all possi- implementation details are possible to extract. The bench- ble multiplier-multiplicand combinations and substituting mark networks for AIDA are AlexNet and CTC-3L- the product values in the matching CAM rows. 421H-UNI .

The third stage is soft reduction, where the dot products

received in the previous stage are accumulated to a final 4.2 Experimental results output activation. We synthesized the TAG logic (Fig. 1(c)) using Synopsis

Design Compiler with the 45nm FreePDK open cell library AIDA. The CAM depth scales to the number of nonzero and scaled the area and power figures to 28nm technology. weights. The CAM width scales according to the weight A single TAG cell area in 28nm is 7.1µm2, and its average and activation wordlength. AIDA performance increases energy consumption is 5.6fJ. The 10T NOR CAM bitcell sub-linearly to the network size, since the execution time

area is 0.1 µm in 28nm. The estimated area of AIDA ca- of at least one step of AIDA algorithm, the soft reduction, pable of storing and processing the compressed FC layers increases with the number of nonzero output activations, of AlexNet and CTC-3L-421H-UNI is 44.6mm2. which is likely to grow in larger networks.

TABLE 1. COMPARISON WITH EXISTING PLATFORMS

C-Munk DNPU A-eye AIDA

Platform EIE (ours)

Network C,R R C,R R C C,R

Type ASIC ASIC ASIC FPGA FPGA GP/A

Memory, B <8.3M 82K 10K 332K 6.4M

Freq, MHz 8 1 2 1 1 10

Quant, bit 4/1 8 4-7 5-1 1 16/1

Area, mm 20.9(1) 0.27(1) 44.5 Fig. 5: Effects of sparsity (a) and arithmetic precision (b)

Power, W 0.3 0.0 0.0 1 9.6(2) 7.1

PP, GOPs 1 32.3 2 1 1.1 14 5 CONCLUSIONS

EE, GOP/J 27 11 11 15.2 1 (2) 2

This paper presents AIDA, the associative in-memory

Thrpt, Inf/s 819 12 3 2045 accelerator of fully connected (FC) layer of neural network. Network, C=CNN, R=RNN; Memory=On chip memory; PP=Peak Perfor- AIDA provides 14.5× higher peak performance and 2.5× mance, EE=Energy Efficiency, Quant=Quantization, Thrpt=Throughput; higher throughput than EIE . In terms of area and en- (1) Area and power are scaled to 28nm ergy efficiency, AIDA strongly benefits from sparsity and (2) Total power (including convolutional and pooling layers)

lower arithmetic precision. Although in this work we use

Table 1 presents performance, throughput, power and AIDA as FC layer accelerator, AIDA is in fact a general

area comparison. All area figures in Table 1 are scaled to purpose associative processor in memory, which can effi- 28nm. Large discrepancies in area and power among exist- ciently implement a wide variety of data intensive applica- ing platforms are mainly due to the amount of on-chip tions, including machine learning, graph processing and memory. sequence alignment.

Similar to EIE , AIDA utilizes weight and activation

sparsity to fit all network parameters in an on-chip REFERENCES memory. AIDA yields the highest throughput, outper- S. Han et al., “EIE: efficient inference engine on compressed deep neural forming EIE by 2.5×, however it is 7.7× less energy effi- network,” ISCA 2016. cient, because CAM is more power hungry than SRAM. A. Krizhevsky et al., “Imagenet classification with deep convolutional neural networks,” NIPS, 2012.

4.3 Design Space Exploration A. Graves et al., “Speech recognition with deep recurrent neural net-

Effects of sparsity: Fig. 5(a) shows AIDA area and en- works.” ICASSP 2013. ergy efficiency as function of weight and activation spar- L. Yavits et al., “Computer architecture with associative processor replac- ing last-level cache and SIMD accelerator,” IEEE Transactions on Com- sity, relative to the figures presented in Table 1. Both area puters, 6 (2), pp 368-381, 20 and energy efficiency increase almost linearly with weight L Yavits et al., “Sparse matrix multiplication on an associative processor”,

sparsity. IEEE Transactions on Parallel and Distributed Systems 2 (11), 3175-31 Effects of arithmetic precision: Fig. 5(b) shows AIDA J. Qiu, et al. "Going deeper with embedded FPGA platform for convolu- area and energy efficiency as function of weight and acti- tional neural network," FPGA 2016. vation wordlength, relative to the figures presented in Ta- D. Shin et al., “DNPU: An 8.1TOPS/W Reconfigurable CNN-RNN Pro-

cessor for General-Purpose Deep Neural Networks,” ISSCC 2017. ble 1. Best area and energy efficiency are achieved for bi-

F. Conti, et al. "Chipmunk: A systolically scalable 0.9 mm 2, 3.0

nary and ternary networks, and drop with growing weight Gop/s/mW @ 1.2 mW accelerator for near-sensor recurrent neural net- and activation wordlength. The super-linear growth of en- work inference." IEEE CICC, 2018. ergy efficiency is due to quadratic dependency between S. K. Esser and et al., “Convolutional networks for fast, energy efficient the associative bit-serial multiplication time and word- neuromorphic computing,” arXiv:1603.08270, 20

length. V. Rybalkin, et al “Hardware Architecture of Bidirectional Long Short- Scalability: AIDA operation can be parallelized in a Term Memory Neural Network for Optical Character Recognition,” IEEE DATE, 2017, pp. 1390–1395. number of ways. For example, the activation broadcast can Y. Chen et al., “DaDianNao: A machine-learning supercomputer,” MI- be performed in parallel with M×V calculations. To accom- CRO 20

plish that, the CAM array needs to be partitioned along its J. Albericio, et al. "Cnvlutin: Ineffectual-neuron-free deep neu

Continued Discussion and Extended Analysis (Part 2)

AIDA: Associative DNN Inference Accelerator

L. Yavits, R. Kaplan and R. Ginosar

Abstract— We propose AIDA, an inference engine for accelerating fully-connected (FC) layers of Deep Neural Network (DNN).

Index Terms—Deep Neural Network, RNN, associative processing in memory, matrix-vector multiplication, accelerator.

1 INTRODUCTION

F ully-connected (FC) layers are extensively used in Con-

volutional, Recurrent (Long Short-Term Memory) and

Multi-Layer Perceptron Neural Networks . FC

power efficiency of AIDA to those of ASIC and FPGA based accelerators.

The rest of this paper is organized as follows. Section 2

We propose AIDA, an associative processing in-

2 ACCELERATOR DESIGN

This paper makes the following contributions: (precalculated and embedded in AP microcode), are writ-

XXXX. Final manuscript received: YY-Yyy-YYYY

2 write enable transistors) and the 4T wired NMOS XOR. The weight matrix 𝑊 is stored in AIDA’s CAM, a single

Fig. 2. Every CAM row, a PU, stores nonzero weight ma-

trix, input activation and output activation elements. There is an optional temporary field for storing temporary varia- bles.

Fig. 1: (a) AIDA Architecture, (b) 10T NOR CAM cell, (c) TAG Logic.

To compare the key data word against the data stored

The AIDA FC layer algorithm is presented in Fig. 3. It

FF (set earlier by the compare) is connected to the corre-

During activation broadcast, each activation either

C field (for the subsequent FC layers). It is matched (line 3

ations. of Fig. 3) against and stored in B field (line 4) along all pos-

To read data from memory, the WBL and ¬WBL lines are

sible multiplication candidates in the weight matrix. Lines precharged and the Write line is asserted.

3 and 4 are executed in parallel. Broadcast is done sequen-

Two additional AIDA instructions are if_match, that

In the multiplication step, all nonzero weight-activation

3 ALGORITHM formed in two steps, a bitwise AND operation and a full

Algorithm: AIDA FC Layer Implementation

Let W be a weight matrix of nnz nonzero elements, stored in ACSR for-

• BVALUE is a vector of nonzero elements of B of wordlength 𝑛, begin- odd and even dot products are summed up; then the odd ning at B_base;

Let C be an output activation vector of wordlength 𝑘, beginning at

1. Activation broadcast: Lines 1 through 2 are responsible for moving the odd dot

// performed in all AIDA PUs in parallel product or partial result to the PU where the even one is

2. for (p = 0 ; p < nnzB; p++) { located. Such move is carried out bit-serially, but in paral-

3. Compare (BINDEX ≡ col_index_base); lel in all PUs, in three steps. First, a relevant bit of the

4. Write (B_base←BVALUE);

source data element is copied to the tag (line 15), then it is 5. } shifted to the tag of the destination PU (lines 1 through

6. Multiplication:

19), and finally it is written to the destination bit-column // performed in all AIDA PUs in parallel

7. for (j = 0 ; j < n; j++) { of the destination PU (line 20). The MSB of the correspond-

8. for (i=0; i < m; i++) { ing row flag element is shifted along with data. Lines 2

9. Bit_AND(T_base+i+j, W_base+i, B_base+j); and 2 comprise the addition operation (performed bit-se-

10. Bit_ADD(C_base+i+j, C_base+i, T_base+j); rially in parallel in all PUs). The reduction is performed

11. } until all “01” patterns in the row flag vector (signaling the 12. } 1st element of each weight matrix row) are replaced by “11”

13. Soft Reduction: pattern which signals the end of reduction (lines 2 and

// performed in all AIDA PUs in parallel 26).

14. for (j = 0 ; j < k || j== W_row_flag_base+1; j++) {

15. Compare (C_base+j ≡ 1);

In the Activation Function (RELU) stage, we check the

16. if (2𝑗 <long_step) sign of all output activations and reset the negative ones

17. Move(up, short_step); (lines 2 and 29). Other activation functions, such as tanh

18. else or sigmoid, can be implemented using perfect induction

19. Move(up, long_step); approach.

20. Write (B_base+j ← ‘1’);

22. for (j = 0 ; j < k; j++) { 4 EVALUATION

23. Bit_ADD (C_base+j, C_base+j, B_base+j);

24. } 4.1 Methodology

25. Compare (W_row_flag≡’10’); We implemented a custom simulator of AIDA, used for

26. if_match goto ‘Soft Reduction’

performance and power simulation and design space ex-

27. Activation Function (RELU): ploration. We compare AIDA with several state of art ded-

// performed in all AIDA PUs in parallel icated hardware (FPGA and ASIC) platforms. A large

28. Compare (C_sign_bit≡’1’);

29. Write (C_base← ‘000…0’)

The third stage is soft reduction, where the dot products

received in the previous stage are accumulated to a final 4.2 Experimental results output activation. We synthesized the TAG logic (Fig. 1(c)) using Synopsis

TABLE 1. COMPARISON WITH EXISTING PLATFORMS

C-Munk DNPU A-eye AIDA

Platform EIE (ours)

Network C,R R C,R R C C,R

Type ASIC ASIC ASIC FPGA FPGA GP/A

Memory, B <8.3M 82K 10K 332K 6.4M

Freq, MHz 8 1 2 1 1 10

Quant, bit 4/1 8 4-7 5-1 1 16/1

Area, mm 20.9(1) 0.27(1) 44.5 Fig. 5: Effects of sparsity (a) and arithmetic precision (b)

Power, W 0.3 0.0 0.0 1 9.6(2) 7.1

PP, GOPs 1 32.3 2 1 1.1 14 5 CONCLUSIONS

EE, GOP/J 27 11 11 15.2 1 (2) 2

This paper presents AIDA, the associative in-memory

lower arithmetic precision. Although in this work we use

Table 1 presents performance, throughput, power and AIDA as FC layer accelerator, AIDA is in fact a general

Similar to EIE , AIDA utilizes weight and activation

4.3 Design Space Exploration A. Graves et al., “Speech recognition with deep recurrent neural net-

cessor for General-Purpose Deep Neural Networks,” ISSCC 2017. ble 1. Best area and energy efficiency are achieved for bi-

F. Conti, et al. "Chipmunk: A systolically scalable 0.9 mm 2, 3.0

plish that, the CAM array needs to be partitioned along its J. Albericio, et al. "Cnvlutin: Ineffectual-neuron-free deep neu

Continued Discussion and Extended Analysis (Part 3)

AIDA: Associative DNN Inference Accelerator

L. Yavits, R. Kaplan and R. Ginosar

Abstract— We propose AIDA, an inference engine for accelerating fully-connected (FC) layers of Deep Neural Network (DNN).

Index Terms—Deep Neural Network, RNN, associative processing in memory, matrix-vector multiplication, accelerator.

1 INTRODUCTION

F ully-connected (FC) layers are extensively used in Con-

volutional, Recurrent (Long Short-Term Memory) and

Multi-Layer Perceptron Neural Networks . FC

power efficiency of AIDA to those of ASIC and FPGA based accelerators.

The rest of this paper is organized as follows. Section 2

We propose AIDA, an associative processing in-

2 ACCELERATOR DESIGN

This paper makes the following contributions: (precalculated and embedded in AP microcode), are writ-

XXXX. Final manuscript received: YY-Yyy-YYYY

2 write enable transistors) and the 4T wired NMOS XOR. The weight matrix 𝑊 is stored in AIDA’s CAM, a single

Fig. 2. Every CAM row, a PU, stores nonzero weight ma-

trix, input activation and output activation elements. There is an optional temporary field for storing temporary varia- bles.

Fig. 1: (a) AIDA Architecture, (b) 10T NOR CAM cell, (c) TAG Logic.

To compare the key data word against the data stored

The AIDA FC layer algorithm is presented in Fig. 3. It

FF (set earlier by the compare) is connected to the corre-

During activation broadcast, each activation either

C field (for the subsequent FC layers). It is matched (line 3

ations. of Fig. 3) against and stored in B field (line 4) along all pos-

To read data from memory, the WBL and ¬WBL lines are

sible multiplication candidates in the weight matrix. Lines precharged and the Write line is asserted.

3 and 4 are executed in parallel. Broadcast is done sequen-

Two additional AIDA instructions are if_match, that

In the multiplication step, all nonzero weight-activation

3 ALGORITHM formed in two steps, a bitwise AND operation and a full

Algorithm: AIDA FC Layer Implementation

Let W be a weight matrix of nnz nonzero elements, stored in ACSR for-

• BVALUE is a vector of nonzero elements of B of wordlength 𝑛, begin- odd and even dot products are summed up; then the odd ning at B_base;

Let C be an output activation vector of wordlength 𝑘, beginning at

1. Activation broadcast: Lines 1 through 2 are responsible for moving the odd dot

// performed in all AIDA PUs in parallel product or partial result to the PU where the even one is

2. for (p = 0 ; p < nnzB; p++) { located. Such move is carried out bit-serially, but in paral-

3. Compare (BINDEX ≡ col_index_base); lel in all PUs, in three steps. First, a relevant bit of the

4. Write (B_base←BVALUE);

source data element is copied to the tag (line 15), then it is 5. } shifted to the tag of the destination PU (lines 1 through

6. Multiplication:

19), and finally it is written to the destination bit-column // performed in all AIDA PUs in parallel

7. for (j = 0 ; j < n; j++) { of the destination PU (line 20). The MSB of the correspond-

8. for (i=0; i < m; i++) { ing row flag element is shifted along with data. Lines 2

9. Bit_AND(T_base+i+j, W_base+i, B_base+j); and 2 comprise the addition operation (performed bit-se-

10. Bit_ADD(C_base+i+j, C_base+i, T_base+j); rially in parallel in all PUs). The reduction is performed

11. } until all “01” patterns in the row flag vector (signaling the 12. } 1st element of each weight matrix row) are replaced by “11”

13. Soft Reduction: pattern which signals the end of reduction (lines 2 and

// performed in all AIDA PUs in parallel 26).

14. for (j = 0 ; j < k || j== W_row_flag_base+1; j++) {

15. Compare (C_base+j ≡ 1);

In the Activation Function (RELU) stage, we check the

16. if (2𝑗 <long_step) sign of all output activations and reset the negative ones

17. Move(up, short_step); (lines 2 and 29). Other activation functions, such as tanh

18. else or sigmoid, can be implemented using perfect induction

19. Move(up, long_step); approach.

20. Write (B_base+j ← ‘1’);

22. for (j = 0 ; j < k; j++) { 4 EVALUATION

23. Bit_ADD (C_base+j, C_base+j, B_base+j);

24. } 4.1 Methodology

25. Compare (W_row_flag≡’10’); We implemented a custom simulator of AIDA, used for

26. if_match goto ‘Soft Reduction’

performance and power simulation and design space ex-

27. Activation Function (RELU): ploration. We compare AIDA with several state of art ded-

// performed in all AIDA PUs in parallel icated hardware (FPGA and ASIC) platforms. A large

28. Compare (C_sign_bit≡’1’);

29. Write (C_base← ‘000…0’)

The third stage is soft reduction, where the dot products

received in the previous stage are accumulated to a final 4.2 Experimental results output activation. We synthesized the TAG logic (Fig. 1(c)) using Synopsis

TABLE 1. COMPARISON WITH EXISTING PLATFORMS

C-Munk DNPU A-eye AIDA

Platform EIE (ours)

Network C,R R C,R R C C,R

Type ASIC ASIC ASIC FPGA FPGA GP/A

Memory, B <8.3M 82K 10K 332K 6.4M

Freq, MHz 8 1 2 1 1 10

Quant, bit 4/1 8 4-7 5-1 1 16/1

Area, mm 20.9(1) 0.27(1) 44.5 Fig. 5: Effects of sparsity (a) and arithmetic precision (b)

Power, W 0.3 0.0 0.0 1 9.6(2) 7.1

PP, GOPs 1 32.3 2 1 1.1 14 5 CONCLUSIONS

EE, GOP/J 27 11 11 15.2 1 (2) 2

This paper presents AIDA, the associative in-memory

lower arithmetic precision. Although in this work we use

Table 1 presents performance, throughput, power and AIDA as FC layer accelerator, AIDA is in fact a general

Similar to EIE , AIDA utilizes weight and activation

4.3 Design Space Exploration A. Graves et al., “Speech recognition with deep recurrent neural net-

cessor for General-Purpose Deep Neural Networks,” ISSCC 2017. ble 1. Best area and energy efficiency are achieved for bi-

F. Conti, et al. "Chipmunk: A systolically scalable 0.9 mm 2, 3.0

plish that, the CAM array needs to be partitioned along its J. Albericio, et al. "Cnvlutin: Ineffectual-neuron-free deep neu

FAQ

Typical stacks include Git, a CI engine (Jenkins/GitHub Actions/GitLab CI), Docker, Kubernetes or a cloud PaaS, infrastructure-as-code (Terraform/Ansible), and monitoring (Prometheus/Grafana). Exact tools for Devops Image Caching Optimization depend on the chosen cloud and delivery model.

It can be delivered as a fully documented simulation on local/kind clusters or as a guided deployment on a cloud free tier. Both approaches are acceptable for academic evaluation when metrics, logs and diagrams are captured.

Pipeline screenshots, successful/failed run history, deployment frequency charts, rollback demos, monitoring dashboards, IaC plans/applies, and a short viva demo script are commonly included.