Ai In Construction

This project focuses on ai in construction using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

Simple, Efficient, and Neural Algorithms for Sparse Coding

Sanjeev Arora ∗ and Rong Ge † and Tengyu Ma ‡ and Ankur Moitra §

Abstract

Sparse coding is a basic task in many fields including signal processing, neuroscience and

machine learning where the goal is to learn a basis that enables a sparse representation of a given set of data, if one exists. Its standard formulation is as a non-convex optimization problem which is solved in practice by heuristics based on alternating minimization. Re- cent work has resulted in several algorithms for sparse coding with provable guarantees, but somewhat surprisingly these are outperformed by the simple alternating minimization heuristics. Here we give a general framework for understanding alternating minimization

which we leverage to analyze existing heuristics and to design new ones also with provable guarantees. Some of these algorithms seem implementable on simple neural architectures, which was the original motivation of Olshausen and Field (1997a) in introducing sparse coding. We also give the first efficient algorithm for sparse coding that works almost up to the information theoretic limit for sparse recovery on incoherent dictionaries. All previous algorithms that approached or surpassed this limit run in time exponential in some natural

parameter. Finally, our algorithms improve upon the sample complexity of existing ap- proaches. We believe that our analysis framework will have applications in other settings where simple iterative algorithms are used.

1. Introduction

Sparse coding or dictionary learning consists of learning to express (i.e., code) a set of input vectors, say image patches, as linear combinations of a small number of vectors chosen from a large dictionary. It is a basic task in many fields. In signal processing, a wide variety of signals turn out to be sparse in an appropriately chosen basis (see references in Mallat (1998)). In neuroscience, sparse representations are believed to improve energy efficiency of the brain by allowing most neurons to be inactive at any given time. In machine

learning, imposing sparsity as a constraint on the representation is a useful way to avoid over-fitting. Additionally, methods for sparse coding can be thought of as a tool for feature extraction and are the basis for a number of important tasks in image processing such as segmentation, retrieval, de-noising and super-resolution (see references in Elad (2010)), as § tational Intractability. This work was supported in part by NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, Simons Investigator Award, and Simons Collaboration Grant.

§ § tational Intractability. This work was supported in part by NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, Simons Investigator Award, and Simons Collaboration Grant. § work was supported in part by a grant from the MIT NEC Corporation and a Google Research Award.

also a basic problem in linear algebra itself since it involves finding a better basis.

The notion was introduced by neuroscientists Olshausen and Field (1997a) who formal-

ized it as follows: Given a dataset y (1) , y (2) , . . . , y (p) ∈ <n , our goal is to find a set of basis vectors A1 , A2 , . . . , Am ∈ <n and sparse coefficient vectors x(1) , x(2) , . . . , x(p) ∈ <m that minimize the reconstruction error p

X p

X (i) ky − A · x(i) k22 + S(x(i) ) (1) i=1 i=1

where A is the n×m coding matrix whose jth column is Aj and S(·) is a nonlinear penalty function that is used to encourage sparsity. This function is nonconvex because both A and the x(i) ’s are unknown. Their paper as well as subsequent work chooses m to be larger than n (so-called overcomplete case) because this allows greater flexibility in adapting the representation to the data. We remark that sparse coding should not be confused with the related — and usually easier — problem of finding the sparse representations of the y (i) ’s

given the coding matrix A, variously called compressed sensing or sparse recovery Candes

Olshausen and Field also gave a local search/gradient descent heuristic for trying to min-

imize the nonconvex energy function (1). They gave experimental evidence that it produces coding matrices for image patches that resemble known features (such as Gabor filters) in V 1 portion of the visual cortex. A related paper of the same authors Olshausen and Field (1997b) (and also Lewicki and Sejnowski (2000)) places sparse coding in a more familiar generative model setting whereby the data points y (i) ’s are assumed to be probabilistically generated according to a model y (i) = A∗ · x∗(i) + noise where x∗(1) , x∗(2) , . . . , x∗(p) are sam-

ples from some appropriate distribution and A∗ is an unknown code. Then one can define the maximum likelihood estimate, and this leads to a different and usually more complicated energy function — and associated heuristics — compared to (1).

Surprisingly, maximum likelihood-based approaches seem unnecessary in practice and

local search/gradient descent on the energy function (1) with hard constraints works well, In fact these methods are so effective that sparse coding is considered in practice to be a solved problem, even though it has no polynomial time algorithm per se.

Efficient Algorithms vs Neural Algorithms. Recently, there has been rapid progress

on designing polynomial time algorithms for sparse coding with provable guarantees (the relevant papers are discussed below). All of these adopt the generative model viewpoint sketched above. But the surprising success of the simple descent heuristics has remained largely unexplained. Empirically, these heuristics far out perform — in running time, sample complexity, and solution quality — the new algorithms, and this (startling) observation was in fact the starting point for the current work.

Of course, the famous example of simplex vs ellipsoid for linear programming reminds us

that it can be much more challenging to analyze the behavior of an empirically successful algorithm than it is to to design a new polynomial time algorithm from scratch! But for sparse coding the simple intuitive heuristics are important for another reason beyond just their algorithmic efficiency: they appear to be implementable in neural architectures.

Simple, Efficient, and Neural Algorithms for Sparse Coding

(Roughly speaking, this means that the algorithm stores the code matrix A as synapse weights in a neural network and updates the entries using differences in potentials of the synapse’s endpoints.) Since neural computation — and also deep learning — have proven to be difficult to analyze in general, analyzing sparse coding thoroughly seems to be a natural first step for theory. Our algorithm is a close relative of the Olshausen-Field algorithm and thus inherits its neural implementability; see Appendix E for further discussion.

Here we present a rigorous analysis of the simple energy minimization heuristic, and as

a side benefit this yields bounds on running time and sample complexity for sparse coding that are better (in some cases, dramatically so) than the algorithms in recent papers. This program that is known to work too, and in our setting, the only known convex program

1.1. Recent Work

A common thread in recent work on sparse coding is to assume a generative model; the precise details vary, but each has the property that given enough samples the solution is √ column rank (in particular m ≤ n) which works up to sparsity roughly n. However this assuming that A∗ is µ-incoherent (which we define in the next section). The former gave an algorithm that works up to sparsity n1/2−γ /µ for any γ > 0 but the running time is nΘ(1/γ) ; n1/6 /µ depending on the particular assumptions on the model. These works also analyze

alternating minimization but assume that it starts from an estimate A that is column-wise 1/poly(n)-close to A∗ , in which case the objective function is essentially convex. works for sparsity up to n1−γ for any γ > 0. But in order to output an estimate that is

O(1)

column-wise -close to A∗ the running time of the algorithm is n1/ . In most applications, one needs to set (say) = 1/k in order to get a useful estimate. However in this case their algorithm runs in exponential time. The sample complexity of the above algorithms is also rather large, and is at least Ω(m2 ) if not much larger. Here we will give simple and more efficient algorithms based on alternating minimization whose column-wise error decreases geometrically, and that work for sparsity up to n1/2 /µ log n. We remark that even

empirically, alternating minimization does not appear to work much beyond this bound.

1.2. Model, Notation and Results

We will work with the following family of generative models (similar to those in earlier papers)1 :

1. The casual reader should just think of x∗ as being drawn from some distribution that has independent

coordinates. Even in this simpler setting —which has polynomial time algorithms using Independent

Component Analysis—we do not know of any rigorous analysis of heuristics like Olshausen-Field. The

Our Model Each sample is generated as y = A∗ x∗ + noise where A∗ is a ground truth dictionary and x∗ is drawn from an unknown distribution D where

(1) the support S = supp(x∗ ) is of size at most k, Pr[i ∈ S] = Θ(k/m) and Pr[i, j ∈

S] = Θ(k 2 /m2 )

(2) the distribution is normalized so that E[x∗i |x∗j 6= 0] = 0; E[x∗i 2 |x∗i 6= 0] = 1 and when x∗i 6= 0, |x∗i | ≥ C for some constant C ≤ 1 and

(3) the non-zero entries are pairwise independent and subgaussian, conditioned on the support.

(4) The noise is Gaussian and independent across coordinates.

Such models are natural since the original motivation behind sparse coding was to discover a code whose representations have the property that the coordinates are almost independent. We can relax most of the requirements above, at the expense of further restricting the sparsity, but will not detail such tradeoffs. The rest of the paper ignores the iid noise: it has little effect on our basic steps like computing inner products of samples or taking singular vectors, and easily tolerated so long

as it stays smaller than the “signal.”

We assume A∗ is an incoherent dictionary, since these are widespread in signal processing

Elad (2010) and statistics Donoho and Huo (1999), and include various families of wavelets, Gabor filters as well as randomly generated dictionaries.

Definition 1 An n × m matrix A whose columns are unit vectors is µ-incoherent if for all √ i 6= j we have hAi , Aj i ≤ µ/ n.

We also require that kA∗ k = O( m/n). However this can be relaxed within polylogarithmic p

factors by tightening the bound on the sparsity by the same factor. Throughout this paper we will say that As is (δ, κ)-near to A∗ if after a permutation and sign flips its columns are within distance δ and we have kAs − A∗ k ≤ κkA∗ k. See also Definition 8. We will use this notion to measure the progress of our algorithms. Moreover we will use g(n) = O∗ (f (n)) to signify that g(n) is upper bounded by Cf (n) for some small enough constant C. Finally, √ throughout this paper we will assume that k ≤ O∗ ( n/µ log n) and m = O(n). Again, m

can be allowed to be higher by lowering the sparsity. We assume all these conditions in our main theorems.

Main Theorems In Section 2 we give a general framework for analyzing alternating

minimization. Instead of thinking of the algorithm as trying to minimize a known non- convex function, we view it as trying to minimize an unknown convex function. Various update rules are shown to provide good approximations to the gradient of the unknown function. See Lemma 11, Lemma 28 and Lemma 33 for examples. We then leverage our framework to analyze existing heuristics and to design new ones also with provable guarantees. In Section 3, we prove: earlier papers were only interested in polynomial-time algorithms, so did not wish to assume indepen-

Simple, Efficient, and Neural Algorithms for Sparse Coding

Theorem 2 There is a neurally plausible algorithm which when initialized with an esti-

mate A0 that is (δ, 2)-near to A∗pfor δ = O∗ (1/ log n), converges at a geometric rate to A∗ until the column-wise error is O( k/n). Furthermore the running time is O(mnp) and the sample complexity is p = O(mk) e for each step.

Additionally we give a neural architecture implementing our algorithm in Appendix E. To

the best of our knowledge, this is the first neurally plausible algorithm for sparse coding with provable convergence. Having set up our general framework and analysis technique we can use it on other variants of alternating minimization. Section 3.1.2 gives a new update rule whose bias (i.e., error) is negligible:

Theorem 3 There is an algorithm which when initialized with an estimate A0 that is (δ, 2)- near to A∗ for δ = O∗ (1/ log n), converges at a geometric rate to A∗ until the column-wise error is O(n−ω(1) ). Furthermore each step runs in time O(mnp) and the sample complexity p is polynomial.

This algorithm is based on a modification where we carefully project out components along

the column currently being updated. We complement the above theorems by revisiting the

Olshausen-Field rule and analyzing a variant of it in Section 3.1.1 (Theorem 12). However

its analysis is more complex because we need to bound some quadratic error terms. It uses convex programming. What remains is to give a method to initialize these iterative algorithms. We give a new approach based on pair-wise reweighting and we prove that it returns an estimate A0 that is (δ, 2)-near to A∗ for δ = O∗ (1/ log n) with high probability. As an additional benefit, this algorithm can be used even in settings where m is not known and this could help solve another problem in practice — that of model selection. In Section 5 we prove:

Theorem 4 There is an algorithm which returns an estimate A0 that is (δ, 2)-near to A∗ for δ = O∗ (1/ log n). Furthermore the running time is O(mn e 2 p) and the sample complexity

This algorithm also admits a neural implementation, which is sketched in Appendix E. The

proof currently requires a projection step that increases the run time though we suspect it is not needed. √ We remark that these algorithms work up to sparsity O∗ ( n/µ log n) which is within a logarithmic factor of the information theoretic threshold for sparse recovery on incoherent dictionaries Donoho and Huo (1999); Gribonval and Nielsen (2003). All previous known run in time exponential in some natural parameter. Moreover, our algorithms are simple to describe and implement, and involve only basic operations. We believe that our framework

will have applications beyond sparse coding, and could be used to show that simple, iterative algorithms can be powerful in other contexts as well by suggesting new ways to analyze them.

Algorithm 1 Generic Alternating Minimization Approach

Initialize A0

Repeat for s = 0, 1, ..., T

Decode: Find a sparse solution to As x(i) = y (i) for i = 1, 2, ..., p Set X s such that its columns are x(i) for i = 1, 2, ..., p Update: As+1 = As − ηg s where g s is the gradient of E(As , X s ) with respect to As

2. Our Framework, and an Overview

Here we describe our framework for analyzing alternating minimization. The generic scheme

we will be interested in is given in Algorithm 1 and it alternates between updating the estimates A and X. It is a heuristic for minimizing the non-convex function in (1) where the penalty function is a hard constraint. The crucial step is if we fix X and compute the gradient of (1) with respect to A, we get: p X ∇A E(A, X) = −2(y (i) − Ax(i) )(x(i) )T . i=1

We then take a step in the opposite direction to update A. Here and throughout the paper η is the learning rate, and needs to be set appropriately. The challenge in analyzing this general algorithm is to identify a suitable “measure of progress”— called a Lyapunov function in dynamical systems and control theory — and show that it improves at each step (with high probability over the samples). We will measure the progress of our algorithms by the maximum column-wise difference between A and A∗ .

In the next subsection, we identify sufficient conditions that guarantee progress. They

are inspired by proofs in convex optimization. We view Algorithm 1 as trying to minimize an unknown convex function, specifically f (A) = E(A, X ∗ ), which is strictly convex and hence has a unique optimum that can be reached via gradient descent. This function is unknown since the algorithm does not know X ∗ . The analysis will show that the direction of movement is correlated with A∗ − As , which in turn is the gradient of the above func- analysing EM algorithms for hidden variable models. The difference is that their condition

is really about the geometry of the objective function, though ours is about the property of the direction of movement. Therefore we have the flexibility to choose different decoding procedures. This flexibility allows us to have a closed form of X s and obtain a useful func- tional form of g s . The setup is reminiscent of stochastic gradient descent, which moves in a direction whose expectation is the gradient of a known convex function. By contrast, here the function f () is unknown, and furthermore the expectation of g s is not the true gradient

and has bias. Due to the bias, we will only be able to prove that our algorithms reach an approximate optimum up to some error whose magnitude is determined by the bias. We can make the bias negligible using more complicated algorithms.

Simple, Efficient, and Neural Algorithms for Sparse Coding

Approximate Gradient Descent

Consider a general iterative algorithm that is trying to get to a desired solution z ∗ (in our case z ∗ = A∗i for some i). At step s it starts with a guess z s , computes some direction g s , and updates its estimate as: z s+1 = z s − ηg s . The natural progress measure is kz ∗ − z s k2 , and below we will identify a sufficient condition for it to decrease in each step:

Definition 5 A vector g s is (α, β, s )-correlated with z ∗ if

hg s , z s − z ∗ i ≥ αkz s − z ∗ k2 + βkg s k2 − s .

Remark: The traditional analysis of convex optimization corresponds to the setting where

z ∗ is the global optimum of some convex function f , and s = 0. Specifically, if f (·) is 2α-strongly convex and 1/(2β)-smooth, then g s = ∇f (z s ) (α, β, 0)-correlated with z ∗ . Also we will refer to s as the bias.

Theorem 6 Suppose g s satisfies Definition 5 for s = 1, 2, . . . , T , and η satisfies 0 < η ≤ 2β and = maxTs=1 s . Then for any s = 1, . . . , T ,

kz s+1 − z ∗ k2 ≤ (1 − 2αη)kz s − z ∗ k2 + 2ηs

In particular, the update rule above converges to z ∗ geometrically with systematic error /α in the sense that kz s − z ∗ k2 ≤ (1 − 2αη)s kz 0 − z ∗ k2 + /α. Furthermore, if s < α2 kz s − z ∗ k2 for s = 1, . . . , T , then

kz s − z ∗ k2 ≤ (1 − αη)s kz 0 − z ∗ k2 .

The proof closely follows existing proofs in convex optimization:

Proof:[Proof of Theorem 6] We expand the error as

kz s+1 − z ∗ k2 = kz s − z ∗ k2 − 2ηg sT (z s − z ∗ ) + η 2 kg s k2 = kz s − z ∗ k2 − η 2g sT (z s − z ∗ ) − ηkg s k2

≤ kz s − z ∗ k2 − η 2αkz s − z ∗ k2 + (2β − η)kg s k2 − 2s (Definition 5 and η ≤ 2β) ≤ kz s − z ∗ k2 − η 2αkz s − z ∗ k2 − 2s

≤ (1 − 2αη)kz s − z ∗ k2 + 2ηs

Then solving this recurrence we have kz s+1 −z ∗ k2 ≤ (1−2αη)s+1 R2 + α where R = kz 0 −z ∗ k. And furthermore if s < α2 kz s − z ∗ k2 we have instead

kz s+1 − z ∗ k2 ≤ (1 − 2αη)kz s − zk2 + αηkz s − zk2 = (1 − αη)kz s − zk2

and this yields the second part of the theorem too.

In fact, we can extend the analysis above to obtain identical results for the case of constrained optimization. Suppose we are interested in optimizing a convex function f (z)

over a convex set B. The standard approach is to take a step in the direction of the gradient (or g s in our case) and then project into B after each iteration, namely, replace z s+1 by ProjB z s+1 which is the closest point in B to z s+1 in Euclidean distance. It is well-known that if z ∗ ∈ B, then kProjB z − z ∗ k ≤ kz − z ∗ k. Therefore we obtain the following as an immediate corollary to the above analysis:

Corollary 7 Suppose g s satisfies Definition 5 for s = 1, 2, . . . , T and set 0 < η ≤ 2β and = maxTs=1 s . Further suppose that z ∗ lies in a convex set B. Then the update rule z s+1 = ProjB (z s − ηg s ) satisfies that for any s = 1, . . . , T ,

kz s − z ∗ k2 ≤ (1 − 2αη)s kz 0 − z ∗ k2 + /α

In particular, z s converges to z ∗ geometrically with systematic error /α. Additionally if s < α2 kz s − z ∗ k2 for s = 1, . . . , T , then

kz s − z ∗ k2 ≤ (1 − αη)s kz 0 − z ∗ k2

What remains is to derive a functional form for various update rules and show that these rules move in a direction g s that approximately points in the direction of the desired solution z ∗ (under the assumption that our data is generated from a stochastic model that meets certain conditions).

An Overview of Applying Our Framework

Our framework clarifies that any improvement step meeting Definition 5 will also converge

to an approximate optimum, which enables us to engineer other update rules that turn out to be easier to analyze. Indeed we first analyze a simpler update rule with g s = E[(y − As x)sgn(xT )] in Section 3. Here sgn(·) is the coordinate-wise sign function. We then return to the Olshausen-Field update rule and analyze a variant of it in Section 3.1.1 using approximate projected gradient descent. Finally, we design a new update rule in

Section 3.1.2 where we carefully project out components along the column currently being

updated. This has the effect of replacing one error term with another and results in an update rule with negligible bias. The main steps in showing that these update rules fit into our framework are given in Lemma 11, Lemma 28 and Lemma 33. How should the algorithm update X? The usual approach is to solve a sparse recovery problem with respect to the current code matrix A. However many of the standard basis pursuit algorithms (such as solving a linear program with an `1 penalty) are difficult to

analyze when there is error in the code itself. This is in part because the solution does not have a closed form in terms of the code matrix. Instead we take a much simpler approach to solving the sparse recovery problem which uses matrix-vector multiplication followed by thresholding: In particular, we set x = thresholdC/2 ((As )T y), where thresholdC/2 (·) keeps only the coordinates whose magnitude is at least C/2 and zeros out the rest. Recall that the non-zero coordinates in x∗ have magnitude at least C. This decoding rule recovers the signs

and support of x correctly provided that A is column-wise δ-close to A∗ for δ = O∗ (1/ log n). See Lemma 10. The rest of the analysis can be described as follows: If the signs and support of x are recovered correctly, then alternating minimization makes progress in each step. In fact this

Simple, Efficient, and Neural Algorithms for Sparse Coding

holds each for much larger values of k than we consider; as high as n/(log n)O(1) . (However, √ the explicit decoding rule fails for k > n/µ log n.) Thus it only remains to properly initialize A0 so that it is close enough to A∗ to let the above decoding rule succeed. In

Section 5 we give a new initialization procedure based on pair-wise reweighting that we

prove works with high probability. This section may be of independent interest, since this algorithm can be used even in settings where m is not known and could help solve another problem in practice — that of model selection. See Lemma 20.

3. A Neurally Plausible Algorithm with Provable Guarantees

Here we will design and analyze a neurally plausible algorithm for sparse coding which is given in Algorithm 2, and we give a neural architecture implementing our algorithm in

Appendix E. The fact that such a simple algorithm provably works sheds new light on how

sparse coding might be accomplished in nature. Here and throughout this paper we will work with the following measure of closeness:

Definition 8 A is δ-close to A∗ if there is a permutation π : [m] → [m] and a choice of signs σ : [m] → {±1} such that kσ(i)Aπ(i) − A∗i k ≤ δ for all i We say A is (δ, κ)-near to A∗ if in addition kA − A∗ k ≤ κkA∗ k too.

This is a natural measure to use, since we can only hope to learn the columns of A∗ up to relabeling and sign-flips. In our analysis, we will assume throughout that π(·) is the identity permutation and σ(·) ≡ +1 because our family of generative models is invariant under this relabeling and it will simplify our notation. Let sgn(·) denote the coordinate-wise sign function and recall that η is the learning rate, which we will soon set. Also we fix both δ, δ0 = O∗ (1/ log n). We will also assume that in

each iteration, our algorithm is given a fresh set of p samples. Our main theorem is:

Theorem 9 Suppose that A0 is (2δ, 2)-near to A∗ and that η = Θ(m/k). Then if each update step in Algorithm 2 uses p = Ω(mk) e fresh samples, we have s ∗ 2 s 0 ∗ 2 E[kAi − Ai k ] ≤ (1 − τ ) kAi − Ai k + O(k/n)

for some 0 < τ < 1/2 and for any s = 1, ∗ p2, ..., T . In particular it converges to A geometri- cally, until the column-wise error is O( k/n).

Our strategy is to prove that gbs is (α, β, )-correlated (see Definition 5) with the desired solution A∗ , and then to prove that kAk never gets too large. We will first prove that if A is somewhat close to A∗ then the estimate x for the representation almost always has the correct support. Here and elsewhere in the paper, we use “very high probability” to mean that an event happens with probability at least 1 − 1/nω(1) .

Lemma 10 Suppose that As is δ-close to A∗ . Then with very high probability over the choice of the random sample y = A∗ x∗ :

sgn(thresholdC/2 ((As )T y)) = sgn(x∗ )

Algorithm 2 Neurally Plausible Update Rule

Initialize A0 that is (δ0 , 2)-near to A∗

Repeat for s = 0, 1, ..., T

Decode: x(i) = thresholdC/2 ((As )T y (i) ) for i = 1, 2, ..., p p s+1 s s 1 X (i) s

Update: A = A − ηb

g where gb = · (y − As x(i) )sgn(x(i) )T p i=1

We prove a more general version of this lemma (Lemma 23) in Appendix A; it is an ingredient in analyzing all of the update rules we consider in this paper. However this is just one step on the way towards proving that gbs is correlated with the true solution. The next step in our proof is to use the properties of the generative model to derive a new formula for gbs that is more amenable to analysis. We define g s to be the expectation of gbs

g s := E[b g s ] = E[(y − As x)sgn(x)T ] (2)

where x := thresholdC/2 ((As )T y) is the decoding of y. Let qi = Pr[x∗i 6= 0] and qi,j = Pr[x∗i x∗j 6= 0], and define pi = E[x∗i sgn(x∗i )|x∗i 6= 0]. Here and in the rest of the paper, we will let γ denote any vector whose norm is negligible (i.e. smaller than 1/nC for any large constant C > 1). This will simplify our calculations. Also let A∗−i denote the matrix obtained from deleting the ith column of A∗ . The following lemma is the main step in our analysis. Lemma 11 Suppose that As is (2δ, 2)-near to A∗ . Then the update step in Algorithm 2

takes the form E[As+1 i ] = Asi − ηgis where gis = pi qi (λsi Asi − A∗i + si ± γ), and λsi = hAsi , A∗i i and T ∗ si = As−i diag(qi,j ) As−i Ai /qi Moreover the norm of si can be bounded as ksi k ≤ O(k/n). Note that pi qi is a scaling constant and λi ≈ 1; hence from the above formula we should expect that gis is well-correlated with Asi − A∗i . Proof: Since As is (2δ, 2)-near to A∗ , As is 2δ-close to A∗ . We can now invoke Lemma 10

and conclude that with high probability, sgn(x∗ ) = sgn(x). Let Fx∗ be the event that sgn(x∗ ) = sgn(x), and let 1Fx∗ be the indicator function of this event.

To avoid the overwhelming number of appearances of the superscripts, let B = As

throughout this proof. Then we can write gis = E[(y − Bx)sgn(xi )]. Using the fact that 1Fx∗ + 1F̄x∗ = 1 and that Fx∗ happens with very high probability:

gis = E[(y − Bx)sgn(xi )1Fx∗ ] + E[(y − Bx)sgn(xi )1F x∗ ] = E[(y − Bx)sgn(xi )1Fx∗ ] ± γ (3)

The key is that this allows us to essentially replace sgn(x) with sgn(x∗ ). Moreover, let S = supp(x∗ ). Note that when Fx∗ happens S is also the support of x. Recall that

Simple, Efficient, and Neural Algorithms for Sparse Coding

according to the decoding rule (where we have replaced As by B for notational simplicity) x = thresholdC/2 (B T y). Therefore, xS = (B T y)S = BST y = BST A∗ x∗ . Using the fact that the support of x is S again, we have Bx = BST BS A∗ x∗ . Plugging it into equation (3): gis = E[(y − Bx)sgn(xi )1Fx∗ ] ± γ = E[(I − BS BST )A∗ x∗ · sgn(x∗i )1Fx∗ ] ± γ = E[(I − BS BST )A∗ x∗ · sgn(x∗i )] − E[(I − BS BST )A∗ x∗ · sgn(xi )1F̄x∗ ] ± γ = E[(I − BS BST )A∗ x · sgn(x∗i )] ± γ where again we have used the fact that Fx∗ happens with very high probability. Now we

rewrite the expectation above using subconditioning where we first choose the support

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings.

Speaker Verification Xvector

Ai In Construction

Simple, Efficient, and Neural Algorithms for Sparse Coding

Abstract

Sparse coding is a basic task in many fields including signal processing, neuroscience and

1. Introduction

The notion was introduced by neuroscientists Olshausen and Field (1997a) who formal-

X p

Olshausen and Field also gave a local search/gradient descent heuristic for trying to min-

Surprisingly, maximum likelihood-based approaches seem unnecessary in practice and

Efficient Algorithms vs Neural Algorithms. Recently, there has been rapid progress

Of course, the famous example of simplex vs ellipsoid for linear programming reminds us

Simple, Efficient, and Neural Algorithms for Sparse Coding

Here we present a rigorous analysis of the simple energy minimization heuristic, and as

1.1. Recent Work

O(1)

1.2. Model, Notation and Results

1. The casual reader should just think of x∗ as being drawn from some distribution that has independent

Component Analysis—we do not know of any rigorous analysis of heuristics like Olshausen-Field. The

S] = Θ(k 2 /m2 )

We assume A∗ is an incoherent dictionary, since these are widespread in signal processing

Main Theorems In Section 2 we give a general framework for analyzing alternating

Simple, Efficient, and Neural Algorithms for Sparse Coding

Theorem 2 There is a neurally plausible algorithm which when initialized with an esti-

Additionally we give a neural architecture implementing our algorithm in Appendix E. To

This algorithm is based on a modification where we carefully project out components along

Olshausen-Field rule and analyzing a variant of it in Section 3.1.1 (Theorem 12). However

This algorithm also admits a neural implementation, which is sketched in Appendix E. The

Algorithm 1 Generic Alternating Minimization Approach

Initialize A0

Repeat for s = 0, 1, ..., T

2. Our Framework, and an Overview

Here we describe our framework for analyzing alternating minimization. The generic scheme

In the next subsection, we identify sufficient conditions that guarantee progress. They

Simple, Efficient, and Neural Algorithms for Sparse Coding

Approximate Gradient Descent

Remark: The traditional analysis of convex optimization corresponds to the setting where

Proof:[Proof of Theorem 6] We expand the error as

An Overview of Applying Our Framework

Our framework clarifies that any improvement step meeting Definition 5 will also converge

Section 3.1.2 where we carefully project out components along the column currently being

Simple, Efficient, and Neural Algorithms for Sparse Coding

Section 5 we give a new initialization procedure based on pair-wise reweighting that we

3. A Neurally Plausible Algorithm with Provable Guarantees

Appendix E. The fact that such a simple algorithm provably works sheds new light on how

Algorithm 2 Neurally Plausible Update Rule

Initialize A0 that is (δ0 , 2)-near to A∗

Repeat for s = 0, 1, ..., T

Update: A = A − ηb

To avoid the overwhelming number of appearances of the superscripts, let B = As

Simple, Efficient, and Neural Algorithms for Sparse Coding

Frequently Asked Questions

What is this project about?