Cochlear Implant Signal Processing

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

Learning For Speech Enhancement

Reinhold Haeb-Umbach∗, Tomohiro Nakatani†, Marc Delcroix†, Christoph Boeddeker∗,

Abstract

Multi-channel acoustic signal processing is a well-established and powerful tool to exploit the spatial diversity between a target signal and non-target or noise sources for signal enhancement. However, the textbook solutions for optimal data-dependent spatial filtering rest on the knowledge of second-order statistical moments of the signals, which have traditionally been difficult to acquire. In this contribution, we compare model-based, purely data-driven, and hybrid approaches to parameter estimation and filtering, where the latter tries to combine the benefits of model-based signal processing and data-driven deep learning to overcome their individual deficiencies. We illustrate the underlying design principles with examples from noise reduction, source separation, and dereverberation.

Index Terms

Microphone array processing, speech enhancement, deep learning, source separation

I. Introduction

Speech enhancement has been a research topic for many years. It is concerned with removing dis- tortions, such as ambient noise, from a degraded speech signal. Its various applications fall into two classes: improving human-to-human communication and improving human-to-machine communication, i.e., automatic speech recognition (ASR). In the first case, where the human is the consumer of the enhanced signal, the focus is on enhancing the quality, and ideally also the intelligibility, of the signal captured by the microphone(s). Examples of applications are telecommunications, hearing aids, media archive enhancement, etc. –. If the enhancement stage is followed by an ASR system, the goal is to improve the recognition accuracy of the recognizer.

Arxiv:2501.07215V1 [Eess.As] 13 Jan 2025

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

2

There are many ways in which a speech signal can be distorted. It can be degraded by the superpo- sition of other acoustic signals, such as by environmental noise or competing speakers. The multi-path propagation from the speech source to the microphones, known as reverberation, leads to degraded intelligibility and ASR performance. And there are many other distortions, e.g., line or acoustic echoes, signal clipping, etc. Consequently, the field of speech enhancement is large, and a plethora of techniques have been proposed to clean up a speech signal.

Those techniques can be classified according to the enhancement task to be solved. In this contribution, we concentrate on noise reduction, source separation, and dereverberation. Another classification is into methods using a single microphone (single-channel) or multiple microphones (multi-channel). Here, we are concerned with multi-channel approaches, assuming that the signals are captured by a compact microphone array. Finally, and this is the focus of this study, methods can be categorized into model-based and data-driven methods, as well as hybrid methods that combine the two.

Research in speech enhancement has a long tradition, with the spectral subtraction method by Boll being an early hallmark. This heuristic approach was soon complemented by statistically motivated methods . For years, research was dominated by such model-based approaches, which use statistical signal processing methods derived from a physical model of the degradation and a probabilistic model of the involved signals. With the advent of deep learning, those model-based techniques have been challenged by data-driven approaches that learn the enhancement operation from data. This is typically done by training a deep neural network (DNN) in a supervised manner with the observed degraded signal at its input and the desired signal or a quantity derived from it as the training target. A third class of techniques, hybrid approaches, blend model-based with data-driven methods. They aim to combine the benefits of both approaches while overcoming their individual shortcomings.

This paper is not about giving a comprehensive overview of the various approaches to speech en- hancement. Its purpose is rather to highlight the advantages and shortcomings of the above classes of enhancement techniques mentioned in the last section and discuss those with a few concrete examples from the field of multi-channel enhancement. We look into noise reduction, dereverberation, and source separation, for which we can juxtapose the three classes of enhancement methods, as outlined in Table I.

Although many presented methods have more applications than being used for speech, e.g., for music signals, we focus here on speech for brevity. We will start with the presentation of commonly used signal models. The models have a decreasing degree of complexity. We continue by describing the properties of model-based and data-driven enhance- ment approaches. We then put an emphasis on hybrid methods, for which we offer a taxonomy. In recent research, we are observing a trend towards data-driven methods, replacing more and more components

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

Table I

EXAMPLES OF MODEL-BASED, DATA-DRIVEN, AND HYBRID SPEECH ENHANCEMENT APPROACHES FOR NOISE REDUCTION (NR), DEREVERBERATION (DR), AND SOURCE SEPARATION (SS). THE RELATED SECTIONS IN THIS PAPER AND PROS AND

Enhancement

Fig. 1. Illustration of the considered scenario: A reverberant enclosure with desired and undesired acoustic sources. The signal is captured by a microphone array and processed by a speech enhancement system to output one or more cleaned-up signals. of the enhancement operation with deep learning-based alternatives. We exemplify this trend with the example of acoustic beamforming and finish with a critical discussion of this trend.

Ii. Signal Models

At the outset of any speech enhancement method is a physical and/or statistical model that describes the properties of the desired signal, the distortion, and how the distortion impacts the desired signal. As shown in Fig. 1, we consider a scenario where the microphone signal is distant from the speakers, i.e., a far-field scenario, as opposed to a close-talking scenario.

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

4

The signals at the microphones are modeled as a linear superposition of the convolution of in general K source signals ˜sk,ℓ, k = 1, . . . , K (including, of course, the case of a single source, i.e., K = 1) with their respective acoustic impulse responses (AIRs) ˜hk,˜ℓfrom a source k to the D microphones and an

(1)

where ℓand ˜ℓare the discrete time and the AIR’s tap indices, respectively. Here, ˜yℓ, ˜hk,˜ℓand ˜nℓare D-dimensional vectors of the time-domain signals at the microphones, the AIR, and the noise.

The

AIRs represent the propagation of the speech signal from the source to the microphones. This is in general a multi-path propagation, seen in the equation by the convolution of the source signal with the AIR, resulting in a reverberated signal at the sensors. Note that we assumed a time-invariant AIR for simplicity.

The speech signals are either treated as deterministic variables or as random processes, where Gaussian and supergaussian probability density functions (PDFs) are typical assumptions. The noise ˜nℓis usually modeled as a Gaussian random process, which is independent of the speech.

With Eq. (1), the different enhancement tasks can be explained: • Noise reduction refers to removing ˜n from ˜y. If this is done by exploiting the multi-channel recording instead of only a single channel, it is called beamforming or spatial filtering.

• Source separation is concerned with isolating the individual source signals ˜sk, k = 1, . . . , K or the reverberated source signals at the microphones, from the mixture ˜y. If we are only interested in one of the sources, say k = 1, we call it source extraction. In that case, the remaining source signals ˜sk, k = 2, . . . , K are viewed as additive distortions.

• Dereverberation is about reducing the reverberation components in the AIR ˜hk,˜ℓ, such that the estimated source signal is not corrupted by delayed versions of it. While Eq. (1) can be considered a faithful model, it is often too complex to be used in practice for deriving enhancement algorithms. According to the well-known saying of the statistician George Box that “Essentially all models are wrong, but some are useful” , virtually all models are only a more or less exact approximation to the reality, and it is important to know the assumptions a model makes.

As a first step towards tractability, the signals are represented in the short time Fourier transform (STFT) domain. Since speech is a broadband signal, its processing is simplified if done in the STFT domain, where the time-domain convolution can be approximated by a convolution over the frame index

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

5

t employing a much shorter acoustic transfer function (ATF) h (so-called convolutive transfer function



+ nt,f ∈CD.

(4)

Here, t and f are the time frame and frequency bin indices, and sk,t,f, yt,f nt,f are the STFT of the k-th source signal and the STFT vectors of the microphone signal and the noise, respectively, while hk,t,f is the vector of ATFs. The time lag ∆separates the direct signal and early reflections from the late reverberation. Dereverberation is typically concerned with removing the late reverberation only.

Equation (2) is an approximation to Eq. (1), because the cross-band filters, which are used for canceling the aliasing effects caused by subsampling from sampling rate to frame rate, have been neglected, see for a discussion. We further introduced the notation xk,t,f for the spatial image of the source signal sk,t,f at the microphone array, which can be divided in x(early)

And X(Late)

k,t,f, the late reverberation. The model can be further simplified by approximating the convolution over the frame index by a multiplication (so-called multiplicative transfer function (MTF) or narrowband approximation , ):

K=1

xk,t,f + nt,f.

(5)

This simplification is justified if the STFT frame length is (significantly) larger than the length of the AIR, i.e., in low reverberant situations. The model can be even further simplified by assuming an anechoic environment, i.e., neglecting reverberation completely. Employing the far-field approximation, which says that the distance of the source to the microphones is much larger than the inter-microphone distance, the attenuation through signal propagation is the same for all elements and can be neglected for the model. Then, the vector of ATFs consists of complex exponentials representing delays caused by the different propagation path lengths from the sources to the sensors of the microphone array, which depend on the directions of arrival

(Doas) Of The Sources:

hk,f = [e −j2π ˜fτk,1, e −j2π ˜fτk,2, . . . , e −j2π ˜fτk,D]T.

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

Test / Inference

Fig. 2. A speech enhancement system consists of two operations, the estimation of the parameters θ of the enhancement operation (dashed boxes) and the actual enhancement (solid boxes). Left: the parameters are estimated from the same signal that has to be enhanced. This is the typical scenario in model-based approaches. Right: Parameters are estimated in a training stage and then used in the enhancement operation on the test data. This is typical for data-driven approaches.

Here, τk,d is the propagation delay from the k-th source to the d-th microphone, which depends on the DoA, and ˜f = f · (fs/F) is the frequency in Hertz (fs: sampling frequency, F: DFT size). Another simplification of (5) is to neglect the noise and concentrate on reverberation:

X(Late)

k,t,f.

(7)

Eqs. (1) - (7) are physical models of different degrees of complexity. A model of decreasing complexity goes hand in hand with the model having an increasing degree of approximations. Obviously, the desire for a simple, tractable model collides here with the desire of high faithfulness.

Iii. Model-Based And Data-Driven Approaches To Enhancement

Speech enhancement, as many other signal processing tasks, consists of two subtasks: the estimation of the parameters of the enhancement algorithm and the actual enhancement operation, see Fig. 2. For the first task, there are two options: either the parameters are estimated from the very same signal that is to be enhanced, or the parameters are estimated in a separate training stage. Of course, there is also the option that the parameters estimated in the training stage are adapted during the enhancement of the test data. We will come back to this option later on.

Model-based parameter estimation and enhancement is done as illustrated in the left figure, while data-driven estimation and enhancement take the route illustrated in the right figure. Hybrid methods blend model-based with data-driven approaches, where this blending can be done either in the parameter estimation, in the enhancement stage, or in both.

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

7

A. Model-based parameter estimation and enhancement For years, many sophisticated model-based enhancement techniques have been developed. Given a signal model, estimation theory is employed to derive statistically optimal estimators. Being able to prove the optimality of an estimator is an important property, which most data-driven techniques lack.

Note, however, that optimality rests upon certain assumptions, foremostly the validity of the model and the knowledge of its parameters. Below, we introduce representative examples of model-based approaches for denoising, source separation, and dereverberation.

1) Acoustic beamforming for denoising and source separation: Beamforming exploits the spatial diversity of the sources and points a beam towards the source of interest and enhances that source signal, while suppressing signals with a different spatial signature. It performs denoising and/or source separation/extraction depending on whether the signals to be suppressed include noise and/or interfering source signals. For this problem, optimal solutions have been derived . A common choice for the beamformer is the minimum variance distortionless response (MVDR) beamformer. Its coefficients can

(8)

where u1 is an all-zero vector with a one at the position of the reference microphone, and tr{·} is the trace operator. In these equations, Φx,f = E[xt,fxH

T,F] Are The Spatial Covariance

matrices (SCMs) of the speech and the noise part of the microphone signals, respectively. The key problem in beamforming is to estimate the beamformer coefficients wf that allow the extraction of the desired signal. This boils down to estimating the SCMs Φx,f and Φn,f of the desired signal and the noise, respectively.

One popular approach to estimating these quantities is an indirect one, which first estimates a signal activity parameter, from which the variance parameters can then be computed. This indirect method is based on the sparseness property and the w-disjoint orthogonality of speech in the STFT domain , which say that a time-frequency (tf)-bin is either dominated by speech or by noise, and in case of multiple speakers, the individual speakers are dominant in disjoint sets of tf-bins. This can be expressed by introducing a latent one-hot vector zt,f ∈{0, 1}K+1, where zk,t,f = 1 means that source k is dominant in the tf-bin (t, f). Giving noise the index 0 we have:

Nt,F

z0,t,f = 1.

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

1

Fig. 3. A 4.25 seconds long segment of speech from the LibriCSS corpus . From left (1) to right (4): (1) Observation, (2) mask of first active speaker, (3) mask of second active speaker, (4) the noise mask. Masks were estimated by a DNN. This latent variable can be estimated using the expectation maximization (EM) algorithm applied to a

(10)

where πk is the a priori probability that zk,t,f = 1, which can be chosen time frame or frequency dependent. Further, p(yt,f; θk) is a distribution that models the spatial properties of yt,f if it is from source k . The EM algorithm estimates the posterior probability mk,t,f = E[zk,t,f|yt,f] ∈[0, 1].

(11)

This, or a quantized version of it, is called a mask. Figure 3 shows example masks computed from a segment of audio, which consists of partially overlapping speech of two speakers and additive noise. With the estimated masks, the SCMs can be estimated as follows:

ˆM0,Τ,Fyτ,Fyh

τ,f.

(13)

With the SCM estimates, the beamformer coefficients of the various statistically optimum beamformers can be readily computed. Applying the beamformer as in Eq. (8) realizes denoising and/or source separation.

Another important class of model-based source separation technique is blind source separation (BSS) based on independent component analzysis (ICA). It estimates spatial filters that separate a given number of sources only from the observation, assuming its underlying physical model and statistical independence between the sources. Many techniques have been developed for speech source separation based on these principles , .

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

9

2) Dereverberation: The goal of dereverberation is the deconvolution of the speech signal to remove the effect of the reverberant environment. We here introduce two model-based methods, the first is based on a transversal and the second on an autoregressive model of the reverberated signal.

Equation (2) can be expressed in vector form (let K = 1 for ease of exposition):

(16)

If we model st,f as complex Gaussian random variable, p(st,f; λ(s)

T,F), We Arrive At A

Gaussian random process for ˘st,f with the following state equation ˘st,f = At,f˘st−1,f + wt,f.

(17)

The state transition matrix A and the system noise vector w can readily be identified by comparing this state space model with Eq. (16). Based on this view, a dereverberation algorithm has been developed, which consists of an EM algorithm with parameter estimation in the M-step, while the E-step is a Kalman filter for the actual enhancement .

The second dereverberation algorithm, the weighted prediction error (WPE) method , approximates the reverberated speech, Eq. (7), by an auto-regressive (AR) model (also referred to as a linear prediction

(18)

where the matrix Gt,f ∈CD×D is a matrix with the coefficients of the AR process yt,f. However, speech is an AR process by itself. To not destroy that property and capture only the unwanted reverberation in

X(Late)

t,f , the lag ∆is introduced. Thus, the dereverberated speech can be estimated via

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

10

where ˆG is an estimate of G. WPE maximizes the likelihood of the model under the assumption that

T,F

is a realization of a zero-mean complex Gaussian process with an unknown channel independent

T,F

, 0, λt,fI).

(20)

Here, I is the identity matrix. An iterative estimation algorithm has been developed for alternately estimating G and λt,f . 3) Remarks: The presented model-based approaches to beamforming or dereverberation estimate their parameters from the microphone signal to be enhanced. Since there is no separate training phase, there is no risk of a train-test mismatch, i.e., that the training data have different statistics than the data to be enhanced. In fact, they can adapt to a new acoustic environment with only a few seconds of observed data.

Since the model parameters to be estimated are not directly observable, the EM is a good starting point to derive an estimation algorithm. It is an iterative batch algorithm and thus not readily usable for online, low-latency processing. However, recursive EM formulations exist, which then allow low-latency processing .

Model-based techniques have been shown to be very robust. For example, the spatial mixture model- based source activity estimation and the WPE dereverberation have been chosen for the CHiME- 6 and CHiME-7 challenge baseline systems. This data set on dinner party transcription is known for its extremely challenging acoustic conditions. The high robustness of the model-based approach is probably due to the physical and statistical assumptions that are implemented in the model with relatively few parameters.

Given an underlying physical model, the derived algorithms often enjoy explainability: the effect of certain parameters on the outcome can often be well predicted. For example, for the above beamformers, the signal-to-noise ratio gain through beamforming can be computed, and the effect of the noise statistics on that gain can be assessed analytically. In case of Bayesian estimators one may even be able to predict the reliability of the estimate.

The above examples can only give a glimpse into the rich literature on model-based enhancement. They have been chosen to highlight the typical properties of model-based enhancement. We are aware of the fact that many important techniques had to be left out from our discussion due to space limitations.

B. Data-driven parameter estimation and enhancement With data-driven techniques, we refer to machine-learning-based methods. They take a different route than the model-based approach: instead of defining a physical model from which the enhancement

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

11

operation can be derived, the enhancement function is estimated from examples of input-output pairs (y, s), with y being the input to the enhancement stage and s being the desired output. The enhancement operator is nowadays realized by a DNN. Clearly, a DNN is also a model, although not physically inspired. The DNN has in general many more parameters than the physically and sta- tistically inspired models discussed above. To discern it from those, we won’t call it a “model” in the following.

1) Denoising and source separation: There are many examples of data-driven approaches to speech enhancement . The vast majority of neural networks are real-valued machines. However, the STFT representations are complex-valued variables. Thus, neural networks were initially not good at representing phase information. Consequently, the first neural regression-based enhancement techniques only estimated the magnitude spectrum of the desired signal from the STFT magnitude representation of the input signal, disregarding the phase . Later it was found that stacking the real and imaginary components of the input STFT was a simple but effective means to represent the complex-valued input signal. So-called complex spectral mapping computes the target real and imaginary (RI) components based on the RI components of the input signal with a regression neural network . Or the network is trained to estimate a complex-valued mask to do so-called complex ratio masking .

An alternative way to avoid the issue with representing complex-valued variables is to do “time- domain” processing. Here, the STFT and inverse STFT are replaced by learnt encoder-decoder modules . Using very short analysis windows of, e.g., 2 ms, these techniques achieved excellent single-channel source separation performance.

DNNs were initially also poor at modeling spatial information available in multi-channel data, because this manifests itself in phase differences between the frequency domain representations of the channels of a compact microphone array, see Eq. (6). This was in part overcome by computing spatial features, such as inter-channel phase differences between the STFTs of the microphone signals and appending these features with spectral features at the input of the DNN. Furthermore, the above technique of stacking real and imaginary parts to a real-valued vector of double the size proved to be effective to capture spatial cues as well .

For processing in the STFT domain, a typical representation of multi-channel data at the input to the network is a three-dimensional tensor with time, frequency and channel axes, where the latter consists of 2D components, capturing the real and imaginary values of the signals in the D microphone channels.

This is complemented by a fourth dimension, the batch dimension. An example network architecture for multi-channel speech enhancement consists of alternating layers for temporal processing and spectral processing . Temporal processing is achieved by a recurrent

Draft

DRAFT – MICROPHONE ARRAY SIGNAL PROCESSING AND DEEP LEARNING FOR SPEECH ENHANCEMENT

(Channel)

Fig. 4. An illustration of spectral (fullband) processing within a frame (left) and temporal (subband) processing across frames (right). The red arrows indicate the sequence axis. network layer (e.g., a bi-directional long short-term memory (BLSTM) layer) whose sequence axis is the time axis and which operates independently for each frequency bin. Spectral processing is done with a network layer (e.g., a recurrent BLSTM layer) that operates along the frequency axis, for each time frame separately. This is sometimes called “fullband processing” and is known to be particularly effective in capturing spatial information, because the DoA results in a characteristic phase change pattern along the frequency axis.

For denoising, the network predicts masks for the speech and noise. Since the two have distinct spectral patterns, a DNN can learn easily which of its outputs corresponds to the speech and which to the noise masks. However, for source separation, we should output masks for each speaker. Speech signals have similar spectral patterns, which causes a permutation ambiguity, i.e., the mapping between DNN outputs and sources are arbitrary. Permutation invariant training (PIT) was introduced as a way to circumvent this ambiguity to allow training DNNs for speech separation . The idea is simply to compute the training loss with the optimal permutation of the sources with respect to the loss. PIT is used in most DNN-based speech separation approaches.

2) Dereverberation: As late reverberation can be viewed as an additive distortion of the desired signal, see Eq. (3), DNN-based networks for dereverberation are similar to those for denoising. One of the earliest works was , where a recurrent neural network was employed in an autoencoder structure with the magnitude spectrum of the reverberated speech at the input and the magnitude spectrum of the non- reverberant speech as the training target at the output. Many more deep learning-based systems have been proposed, some of which are capable of jointly carrying out dereverberation, denoising, and source separation .

3) Remarks: Parameter estimation of DNNs is also guided by a statistically motivated objective function, typically the cross entropy in case of a network for classification (e.g., for mask estimation),

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.