Devops Failover Automation Project

Reliability Engineering And Resilience Automatio · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: anomaly detection / classification using IoT sensor streams and timestamped device measurements.

Error estimates for POD-DL-ROMs: a deep learning

framework for reduced order modeling of nonlinear parametrized PDEs enhanced by proper orthogonal

Simone Brivio , Stefania Fresca , Nicola R. Franco , Andrea Manzoni

MOX - Dipartimento di Matematica, Politecnico di Milano,

P.zza Leonardo da Vinci 32, I-201 Milano, Italy

Abstract

POD-DL-ROMs have been recently proposed as an extremely versatile strat-

egy to build accurate and reliable reduced order models (ROMs) for nonlinear parametrized partial differential equations, combining (i) a preliminary dimen- sionality reduction obtained through proper orthogonal decomposition (POD) for the sake of efficiency, (ii) an autoencoder architecture that further reduces the dimensionality of the POD space to a handful of latent coordinates, and (iii) a dense neural network to learn the map that describes the dynamics of the latent coordinates as a function of the input parameters and the time vari-

able. Within this work, we aim at justifying the outstanding approximation capabilities of POD-DL-ROMs by means of a thorough error analysis, showing how the sampling required to generate training data, the dimension of the POD space, and the complexity of the underlying neural networks, impact on the so- lution accuracy. This decomposition, combined with the constructive nature of the proofs, allows us to formulate practical criteria to control the relative error in the approximation of the solution field of interest, and derive general error

estimates. Furthermore, we show that, from a theoretical point of view, POD-

DL-ROMs outperform several deep learning-based techniques in terms of model

complexity. Finally, we validate our findings by means of suitable numerical experiments, ranging from parameter-dependent operators analytically defined to several parametrized PDEs.

Keywords Operator Learning, Neural Networks, Approximation bounds, Reduced order

modeling, parametrized PDEs, deep learning-based reduced order modeling

1 Introduction

Solutions to partial differential equations (PDEs) are not usually available in analytic

form and need to be approximated by suitable high-fidelity methods, such as the Fi- nite Element Method (FEM) [33, 35]. The latter usually entails a suitable spatial discretization of the (bounded, compact) computational domain Ω ⊂ Rd , d = 1, 2, 3, regulated by the step size h > 0 and yielding a set of Nh degrees of freedom, that

in some cases might correspond to the vertices of the elements providing the do- main discretization. High-fidelity methods are usually referred to as full order models (FOMs) as they provide very accurate solutions, however resulting in computation- ally demanding strategies in terms of either time or resources. Within this work, we focus on a parametric setting, where in general the PDE solution u depends not only on the spatial coordinate x ∈ Ω and the time variable t ∈ T = [0, T ], but also on a

parameter vector µ ∈ P – being the parameter space P ⊂ Rp a compact set – namely u = u(x, µ, t). Once the problem has been discretized in space, we aim at exploring the solution manifold SNh = {u(µ, t) = [u(xi , µ, t)]N i=1 ∈ R h Nh : (µ, t) ∈ P × T }, evaluating the problem solution in multiple scenarios, for different parameter values. To carry out this task efficiently, as well as to tackle other multi-query tasks such as those involving Uncertainty Quantification and to perform real-time numerical

simulations, FOMs must be replaced by efficient and reliable reduced order models (ROMs), a wide class of strategies providing very efficient results yet retaining an adequate representation of the solution manifold SNh .

Linear projection-based ROMs, such as the reduced basis (RB) method relying on

either greedy algorithms or the Proper Orthogonal Decomposition (POD) to build a low-dimensional linear trial subspace, are widely used in the context of parametrized

PDEs. Usually relying on a (Petrov-)Galerkin projection to generate the correspond-

ing ROM by enforcing at the reduced order level the physical constraints expressed by the FOM, these strategies feature however several drawbacks, especially when dealing with time-dependent, nonlinear, and nonaffine problems, ultimately requiring suit- able hyper-reduction strategies such as the Empirical Interpolation Method (EIM) [1, 8, 34] or the Discrete EIM (DEIM, ). Despite being very general, and widely applied, hyper-reduction techniques usually feature an intrusive nature, require to

handle algebraic arrays extracted from the FOM, ultimately resulting in overwhelm- ing computational costs when dealing with nonlinear time-dependent parametrized PDEs.

To overcome these limitations, data-driven Deep Learning-based ROMs (DL-

ROMs) were recently proposed in [10, 12] and similar works [26, 29, 32, 43] to exploit the power of DNNs to both perform dimensionality reduction of a set of high-dimensional snapshots data (obtained by sampling the solution manifold) and learn parameter-to-solution maps nonintrusively. Unfortunately, these techniques re- quire to train complex architectures and might become unfeasible to train as soon as the FOM dimension Nh increases, suffering from the curse of dimensionality in their

vanilla version. To counter this issue, POD-DL-ROMs were then introduced in , leveraging on the power of DL-ROMs and the physically-consistent dimensionality reduction achieved through POD, and then training a DL-ROM network using FOM data projected on a (possibly, large dimensional) POD space: overall, POD-DL-ROMs are capable of lower training efforts in terms of both memory storage and computa- tional time. The POD-DL-ROM paradigm has been tested against several problems, showing remarkable approximation capabilities in the numerical simulation of, e.g.,

fluid flows and fluid-structure interaction problems [15, 14], cardiac electrophysiology , and micro-electromechanical systems among others.

However, a thorough numerical analysis of the POD-DL-ROM technique – con-

necting, e.g., the complexity of the NN architectures involved in a POD-DL-ROM, the sampling error entailed by the selection of training data, the POD error gener- ated while projecting those data onto a POD space, with the overall accuracy of the computed solution – is still lacking. Within this work, we aim at addressing these

questions in light of a solid theoretical analysis, providing general error estimates for the POD-DL-ROM technique, assessing their validity in a series of numerical experi- ments involving different parametrized problems.

1.1 Literature review and existing results

Thanks to the flourishing and rapidly evolving literature of Approximation Theory, many Deep Learning-based approaches to reduced order modeling are now being justified with rigorous theoretical results and error estimates. The majority of these are grounded on a notorious result by Yarotski (2017) , which we report below. Theorem (Yarotski ). Let b ∈ N, b ≥ 1 and 0 < ε < 1/2. Any f ∈ W s,+∞ ([0, 1]b ) can be approximated uniformly with an error of at most ε by a ReLU Deep Neural

Network (DNN) having at most c log(1/ε) layers and cε−b/s log(1/ε) weights, where

Indeed, this result and its subsequent generalizations, see e.g. [41, 20], constitute

the foundation of many recent works, for instance: (i) in , the authors exploited these results to formulate an error analysis for general DL-ROMs. However, their analysis is limited to the time-independent case and does not resolve the curse of dimensionality, as it binds the complexity of DL-ROMs linearly with the FOM dimension Nh ; (ii) Yarotski’s Theorem was also considered in , where the authors investigated the approximation capabilities of Convolutional Neural Networks (CNNs), sug-

gesting a strong connection between these architectures and the Fourier trans- form; (iii) similarly, the results in are fundamental for the derivation of the approxima- tion bounds reported in , which, instead, concern the DeepONet paradigm, an approach first proposed by Lu et al. in ; (iv) finally, Yarotski’s Theorem and its generalizations were also employed to derive approximation bounds for deep learning-based ROM strategies that couple POD and feedforward neural networks, see, e.g., .

Here, we aim at proposing a similar analysis for POD-DL-ROMs, emphasizing the

main differences between this approach and the existing literature.

1.2 Overall idea and paper structure

We analyze the overall approximation error entailed by the use of POD-DL-ROMs

when dealing with the solution of both linear and nonlinear time-dependent parametrized

PDEs by highlighting two separate error contributions: one, coming from the prelim-

inary dimensionality reduction obtained through POD, and one entailed by the use of neural networks. In brief, the idea goes as follows. First, we show that in the finite data regime, the overall error of a POD-DL-ROM, ER , can be decomposed as

ER ≤ ES + EP OD + EN N ,

where ES is the sampling error, EP OD is the POD projection error, and EN N is the approximation error of the neural network model in the DL-ROM pipeline. Then, we address each of the three contributions separately.

For the first two, we rely on classical arguments that bind together the discrete

and the continuous formulation of POD, see e.g. [8, 34], ultimately showing that the sampling error vanishes as a function of the sample size, while EP OD is uniquely characterized by the eigenvalue decay of the data correlation matrix. In this sense, our analysis is strictly related to the one proposed in . To study the neural network error, instead, we consider a specific construction that reflects the general philosophy of DL-ROM techniques. More precisely, we emphasize the fact that POD-

DL-ROMs use a neural network architecture that is obtained through the combination

of two networks: a feature map, φ, which captures the roughness in the parameter- to-solution operator, and a smoother decoder φ. In particular, we base our proof on a generalization of Yarotski’s Theorem, due to Gühring et al. , which, during the composition step, allows us to keep the approximation error under control. For the sake of better readability, we report the latter result below. Theorem (Gühring et al. ). Let b, s ∈ N, with b ≥ 1, s ≥ 2 and n ∈ {0, 1}. For any tollerance 0 < ε < 1/2 and any f ∈ W s,+∞ ([0, 1]b ), there exists and a

ReLU DNN φ having at most c log(1/ε) layers and cε−b/(n−s) log(1/ε) weights, where

c = c(s, b, f, n) is a constant, such that

All of this ultimately allows us to characterize the accuracy of POD-DL-ROMs in

terms of their complexity, providing explicit error bounds that we later compare with the existing literature and verify numerically.

The paper is organized as follows: in Section 2 we formulate the problem, de-

scribing rigorously the POD-DL-ROM approach and the reducibility measures for the framework at hand; Section 3 contains the main results of this work, namely the error decomposition formula, a lower bound result and an upper bound result for the approximation error. Section 4 then demonstrates advantages of POD-DL-ROMs when compared to similar deep learning-based frameworks, such as, e.g., POD+DNN and DeepONets. Finally, a series of numerical experiments that validate the theoret-

ical analysis is shown in Section 5, while the last section draws some conclusions and summarizes possible further developments.

2 An overview of the POD-DL-ROM technique

POD-DL-ROMs provide a general-purpose ROM approach combining a data dimen-

sionality reduction obtained through POD with the DL-ROM approach. After in- troducing the general class of problems we deal with, we overview the main building blocks of the POD-DL-ROM technique. For further details regarding, e.g., detailed algorithms for the offline (or training) and the online query (or testing) stages, the interested reader can refer to, e.g., . An extension of the POD-DL-ROM technique in view of time forecasts of the problem solution out of the training time window has

2.1 Problem formulation

Within this work, we consider time-dependent parametric PDEs of the following type

∂u   ∂t + L(µ)u(µ, t) + N (u(µ, t), µ) = f (µ, t), in Ω × (0, T ]  

B(µ)u(µ, t) = g(µ, t), on ∂Ω × (0, T ] (1)

   u(µ, 0) = u (µ), in Ω,

where: • u = u(x, µ, t) is the PDE solution ∀x ∈ Ω. Here we highlight the explicit dependence of u on the time variable t ∈ T = [0, T ] (for some T > 0) and the input parameter vector µ ∈ P ⊂ Rp , P compact; • L is a linear operator, whereas N is a nonlinear operator and B is the boundary operator; virtually, all these operators might be µ-dependent • u = u (µ) is the initial condition; • Ω is the (bounded) spatial domain where the problem is set.

Depending on the nature of the problem, input parameter can refer to either physical

or geometrical properties of the problem at hand. We considered the formulation (2) as general framework since it describes a wide variety of problems ranging in the fields of engineering, physics, and life sciences, just to make a few examples. Introducing a computational mesh over Ω with mesh size h > 0 and a corresponding space discretization of the problem (1) having Nh degrees of freedom (dofs) obtained through, e.g., the finite element method, the finite-dimensional counterpart of problem

(1) provides our FOM and reads as follows:

M(µ) ∂u (µ, t) + A(µ)u(µ, t) + N(u(µ, t), µ) = f (µ, t), t ∈ (0, T ] 

∂t (2) u(µ, 0) = u (µ), 

where u(µ, t) ∈ RNh denotes the vector of the Nh dofs of the FOM solution, M(µ) ∈ RNh ×Nh the mass matrix, A(µ) ∈ RNh ×Nh the stiffness matrix, N(·, µ) : RNh → RNh a nonlinear map, f (µ, t) ∈ RNh the source term and u (µ) ∈ RNh the initial data. The

FOM (2) is then discretized in time, introducing a suitable time advancing scheme

over a partition of T made by Nt time steps {tk }Nk=1 . t

To explore efficiently the solution manifold SNh = {u(µ, t) : (µ, t) ∈ P × T } we employ the POD-DL-ROM technique, performing a two-step dimensionality re- duction: first, POD (realized through randomized SVD) is applied on a set of FOM snapshots; then, a DL-ROM is built to approximate the map between (µ, t) and the

POD generalized coordinates. This latter task can be achieved by relying on two neu-

ral network architectures, (i) a deep autoencoder – possibly involving convolutional layers – that extracts a set of few, latent coordinates, ultimately representing the reduced-order coordinates of the ROM, and (ii) a deep feedforward neural network, to learn the map between (µ, t) and these latent coordinates. Below, we report the main building blocks of a POD-DL-ROM, originally proposed in :

(i) the snapshot matrix for the parameter vectors µj , j = 1, . . . , Ns is collected, thus obtaining Uj = [u(µj , tk )]N k=1 ∈ R t Nh ×Nt ;

(ii) the whole snapshot matrix is obtained stacking Uj , j = 1, . . . , Ns , namely

U = [Uj ]N

j=1 ∈ R s Nh ×Ndata , where Ndata = Ns Nt ;

(iii) a singular value decomposition (SVD) is performed on the snapshot matrix U, and the first N left singular vectors are retained, thus yielding U ≈ VΣWT , where V ∈ RNh ×N , Σ ∈ RN ×N and W ∈ RN ×Ndata . Then, projecting U on the reduced linear subspace V ∈ RNh ×N , we obtain a snapshot matrix for the POD coefficients Q = VT U;

(iv) the POD coefficient vectors q(µj , tk ), j = 1, . . . , Ns , k = 1, . . . , Nt , obtained from the columns of Q, along with the parameters vector µj and the time instants tk , are used to train a DL-ROM. This latter consists of a deep autoen- coder Ψ ◦ Ψ′ and a deep feedforward neural network (to which we refer to as reduced network) φ, defined as follows:

 DY N

 z = φ(θDY N ; µj , tk ) 

EN C

z = Ψ′ (θEN C ; q(µj , tk ))  q̂ = Ψ(θDEC ; z DY N (θDY N , µj , tk )), 

where φ, Ψ′ , Ψ are the reduced network, the encoder and the decoder, respec- tively, while θDY N , θEN C , θDEC are their corresponding neural network weights and biases (they are omitted, hereon, for the sake of readability). The three networks are trained according to the per-example loss function below,

Ns X

X Nt

LN = kq̂(µj , tk ) − q(µj , tk )k , j=1 k=1

Ns X

X Nt

Ln = kz EN C (µj , tk ) − z DY N (µj , tk )k , j=1 k=1

and n denotes the latent dimension of the architecture. As a matter of notation, from hereon we equip any finite dimensional space Rb (for some b ∈ N) with the ℓ norm: thus, unless otherwise stated, we define k · k := k · k . It is worth to remark that LN penalizes high reconstruction errors and Ln ensures a good representation in the latent space.

Recalling that (µ, t) → Vq̂(µ, t) ≈ u(µ, t) provides the POD-DL-ROM approxi-

mation, the objective of the present work is to characterize the relative error ã1/2 ku(µ, t) − Vq̂(µ, t)k ÅZ

ER := d(µ, t)

P×T ku(µ, t)k

in terms of the POD-DL-ROMs complexity. Here, we choose to focus on analyzing ER since it is a common measure for the accuracy in the ROM literature. Moreover, we

highlight that the entire workflow yielding the error estimate we propose in this work is only based on the approximation error, without considering the contribution carried by the training error. The extension to more general vector energy norms including the contribution of symmetric positive definite mass matrices to define the counterpart of norms in functional spaces like, e.g., L (Ω) or H 1 (Ω), is also straightforward and is not considered here for the sake of simplicity.

2.2 POD: from the discrete to the continuous formulation

Before proceeding towards the thorough analysis of ER , we have to appropriately

define the working setting, which depends on the linear dimensionality reduction.

First, we notice that even though within the POD-DL-ROM pipeline we computed

the POD matrix V through the (randomized) SVD algorithm, thus using a fully data- driven procedure that employs a set of training data, the relative error ER aims at measuring the approximation capabilities over the entire time-parameter space P ×T , taking advantage of a continuous formulation. Within this section, we aim at filling the gap between the discrete and the continuous formulation of POD, highlighting links and bounds, focusing initially only on the source of error coming from the

projection phase, rather than directly considering ER : this allows us to set the ground upon which the more complex approximation results of POD-DL-ROM are based. We start by considering the (P × T )-discrete setting, and the fact that V results from the solution of a minimization problem; indeed, denoting by

|P × T |

K= UUT ∈ RNh ×Nh

the (discrete) correlation matrix and by σk its eigenvalues, it holds that Ndata

X |P × T | X

σk = kuj − VVT uj k

Ndata j=1

|P × T | X

= min kuj − WWT uj k ,

W∈RNh ×N :WT W=I Ndata

where N is the chosen POD dimension and uj is the solution vector that corresponds to the tuple (µ, t)j . We can proceed analogously for the (P × T )-continuous setting, by considering Z

K∞ = u(µ, t)u(µ, t)T d(µ, t) ∈ RNh ×Nh (3)

P×T as the (continuous) correlation matrix and denoting by σk,∞ its eigenvalues; similarly, we can prove that there exists an optimal rank-N matrix V∞ ∈ RNh ×N such that

X Z

2 T

σk,∞ = ku(µ, t) − V∞ V∞ u(µ, t)k d(µ, t) k>N P×T Z = min ku(µ, t) − WWT u(µ, t)k d(µ, t)

W∈RNh ×N :WT W=I P×T

From the considerations above, we can infer that

X Z

2 T

σk,∞ = ku(µ, t) − V∞ V∞ u(µ, t)k d(µ, t) k>N P×T Z ≤ ku(µ, t) − VVT u(µ, t)k d(µ, t); P×T

from the inequality above, we can remark that the data-driven POD matrix V is not optimal for the continuous formulation, which stems from the hypothesis of having infinite data samples, while being the best orthogonal matrix in terms of explained variability with respect the training data at hand. In other words, even though V is optimal for the training data, we have no guarantee that it is optimal for the test data, too; however, since in practice we are not able to obtain the matrix V∞ , we

must necessarily rely on V also in the online testing phase.

Finally, we show how the discrete and the continuous POD formulations are re-

lated: indeed, denoting by [·]i the i-th entry of a vector, and extending this notation to matrices, we have that ∀k, l = 1, . . . , Nh Ndata

|P × T | X

Z [K∞ − K]kl = [u]k [u]l d(µ, t) − [uj ]k [uj ]l ,

P×T Ndata j=1

recalling that uj is the solution vector that corresponds to the tuple (µ, t)j . Upon requiring integrability (easily verified for non-trivial bounded solutions), we can use a.s. the Strong Law of Large Numbers and obtain [K − K∞ ]kl −−→ 0 as Ns , Nt → a.s. ∞, ∀k, l = 1, . . . , Nh , which implies that kK−K∞ k −−→ 0, being kZk any 1-norm of the squared matrix Z. By employing Bauer-Fike’s theorem with the 1-norm, we can state that, upon ordering, for any σk,∞ , there exists σk belonging to the spectrum

|σk − σk,∞ | ≤ K (X)kK − K∞ k , ∀k = 1, . . . , Nh

where X is the matrix collecting the right eigenvectors of K, and K (X) denotes its condition number. Thus, we can conclude that, setting N as the POD dimension, it holds that a.s.

X X

σk −−→ 2 σk,∞ , Ns , Nt → ∞. k>N k>N

2.3 An overlook over the reducibility measures for POD-DL-

POD-DL-ROMs couple POD, for the sake of a preliminary dimensionality reduc-

tion, with an autoencoder-based architecture to reconstruct the parameter-to-POD- coefficients map. Thus, at first it is evident that the projection-based nature of the paradigm invokes the definition of a linear reducibility measure to account for the FOM-to-POD dimensionality reduction task. Definition 1. Let SNh = {u(µ, t) ∈ RNh : (µ, t) ∈ P × T } be the solution manifold.

The linear Kolmogorov N-width of SNh is defined as

dN (SNh ) = inf sup inf ku − vk.

VN ⊂RNh :dim(VN )=N u∈SN v∈VN

It is worth to notice that the linear Kolmogorov N -width is strictly related to the eigenvalues decay of the correlation matrix K∞ ∈ RNh ×Nh . In fact, following the same notation of Subsection 2.2, we have that:

X ÅZ ã1/2

T σk,∞ = ku(µ, t) − V∞ V∞ u(µ, t)k d(µ, t) k>N P×T

ÅZ ã1/2

≤ ku(µ, t) − WWT u(µ, t)k d(µ, t) P×T

≤ |P × T |1/2 sup ku(µ, t) − WWT u(µ, t)k (µ,t)∈P×T

for any W ∈ RNh ×N ; thus, X σk,∞ ≤ |P × T |1/2 dN (SNh ). k>N

The above relationship shows that the eigenvalue decay is an alternative (and more

practical) measure of reducibility, with respect to a P weaker norm. P However, notice that in practice we can only approximate the quantity k>N σk,∞ ≈ k>N σk , which is consistent with the theory thanks to the convergence result presented in Subsection 2.2.

The autoencoder-based architecture of a POD-DL-ROM introduces a second level

of dimensionality reduction, which operates a further compression of the information coming from the parameter-to-POD-coefficients map Q : (µ, t) → VT u(µ, t). The nonlinear nature of the dimensionality reduction performed through the autoencoder Ψ ◦ Ψ′ (being Ψ′ , Ψ the encoder and the decoder, respectively) induces a nonlinear analogue of the Kolmogorov n-width .

Definition 2. The nonlinear Kolmogorov n-width of the reduced manifold SN =

{q(µ, t) = VT u(µ, t) ∈ RN : (µ, t) ∈ P × T } is defined as

δn (SN ) = inf sup ku − Ψ(Ψ′ (u))k.

Ψ∈C(RN ,Rn ) u∈SNh

Ψ′ ∈C(Rn ,RN )

Now, to deal with nonlinear approximation methods, we state another fundamen-

tal definition upon which the main results of this work are based. Definition 3. The reduced manifold SN = {q(µ, t) = VT u(µ, t) ∈ RN : (µ, t) ∈ P × T } enjoys the perfect embedding Assumption with regularity s, s′ if the infimum ′ in Definition 2 is attained, namely there exist Ψ∗ ∈ C s (RN , Rn ), Ψ′∗ ∈ C s (Rn , RN ) such that Ψ∗ (Ψ′∗ (q(µ, t)) = q(µ, t) ∀(µ, t) ∈ P × T . In conclusion, as we did with the POD dimension N , we need to characterize the latent dimension n with a practical criterion. To do that, an extension of Theorem 3

provided in shows that if the parameter-to-solution map G : (µ, t) → u(µ, t) and thus the parameter-to-POD-coefficients map Q : (µ, t) → VT u(µ, t) are Lipschitz- continuous, there exists n ≤ 2p + 3 such that δn (SN ) = 0.

3 Main results

Before stating the main result of this work, namely an upper bound result, that

concerns only POD-DL-ROMs, we make some preliminary reasoning that, instead, applies to any POD+DNN approach, i.e. we do not constrain the neural network q̂, that approximates the parameter-to-POD-coefficients map, to be a DL-ROM. For this purpose, we briefly recall that the POD+DNN technique involves the reconstruction of the parameter-to-solution map through the approximation (µ, t) 7→ Vq̂(µ, t) ≈ u(µ, t), where q̂ is a generic (possibly dense) neural network.

In particular, we start by characterizing ER through an error decomposition for-

mula, that enables us to describe the various error contributions and formulate possi- ble strategies to control them. Secondly, we state a lower bound result, that highlights how, regardless of the architecture of neural network q̂, the relative error ER can be bounded from below by a quantity depending on the POD projection. Then, we move to our upper bound result, where we quantify how complex a POD-DL-ROM should be in order to achieve a specific bound on the relative error ER .

We initially remark that the computation of the error ER and other related quan-

tities hinges upon the evaluation of complex integrals, possibly in high dimensional spaces, which can be effectively handled through Monte Carlo methods. In this re- spect, we shall make the following assumptions, which we assume to hold true hereon. Assumption 1 (Sampling criterion). Let p > 0, assume that P ⊂ Rp is compact and denote T = [0, T ] for some T > 0. We assume that the training (and testing) snapshots are sampled uniformly and iid in the parameter space, µ ∼ U(P), while

a uniform grid is employed for the time variable, t ∈ {∆t, 2∆t, ..., Nt ∆t}, where Nt ∈ N≥2 and ∆t = T /Nt .

Assumption 2 (Parameter-to-solution map). Let G : P ×T → RNh be the parameter-

to-solution map, mapping (µ, t) 7→ u(µ, t). We assume that i) m = ess inf (µ,t)∈P×T ku(µ, t)k > 0, M = ess sup(µ,t)∈P×T ku(µ, t)k < ∞;

ii) G is Lipschitz-continuous with constant L > 0.

From these assumptions, one can easily derive a couple of auxiliary results, which

will be of practical interest in the remainder, and are reported below; for the sake of brevity, their proofs are postponed to Appendix A. Proposition 1. Let f ∈ L (P × T ). Under Assumption 1, one has Ns

Nt X

|P × T | X

Z E f (µ, t)d(µ, t) − f (µj , ti ) ≤ O(Ns−1/2 + Nt−1 ).

P×T Ndata i=1 j=1

where the expectation is taken across all the possible realizations of the data sampling procedure.

Proposition 2. Under the Assumption 2, define w(µ, t) = ku(µ, t)k−2 . Then

ÅZ ã1/2

k · kL2w = k · k w(µ, t)d(µ, t) P×T

3.1 The error decomposition formula

In the following, we state an error decomposition formula that is valid for any POD+DNN approach – and, in particular, for our POD-DL-ROM strategy. Given the more gen- eral nature of this result, its formulation is therefore not restricted to the technique at hand. Theorem 3.1. Let G : (µ, t) 7→ u(µ, t) for any (µ, t) ∈ P × T be the parameter-to- solution map. Consider a POD+DNN approximation of G as G(µ, t) ≈ Vq̂, where q̂ : Rp+1 → RN is a neural network trained over a given training set made by a

collection of input parameters (µi , ti )N data i=1 and the corresponding snapshot matrix U ∈ RNh ×Ndata , while V ∈ RNh ×N is the POD projection matrix. Then, under the

Assumptions 1 and 2, we have

ER ≤ ES + EP OD + EN N , (4)

where: a.s. • ES = ES (G, {(µi , ti )N i=1 }, N ) is the sampling error, that satisfies ES −−→ 0 as data

−1/4 1/2 Ns , Nt → ∞ and E[ES ] = O(Ns + Nt );

• EP OD = EP OD (G, {(µi , ti )N i=1 }, N ) is the POD projection error, that satis- data

a.s. fies EP OD −−→ EP OD,∞ as Ns , Nt → ∞, where EP OD,∞ = EP OD,∞ (G, N ) is independent of the sampling criterion;

• EN N = EN N (G, N, q̂) is the approximation error of the neural network, which is arbitrarily low depending of the approximation capabilities of the network q̂.

Proof. By means of the triangular inequality, we obtain

ER = d(µ, t) = ku(µ, t) − Vq̂(µ, t)kL2w

P×T ku(µ, t)k

≤ ku(µ, t) − VVT u(µ, t)kL2w + kVVT u(µ, t) − Vq̂(µ, t)kL2w . (5)

According to the notation of Section 2, let q(µ, t) := VT u(µ, t). We define ã1/2 kVq(µ, t) − Vq̂(µ, t)k ÅZ

EN N := d(µ, t)

P×T ku(µ, t)k

and notice that EN N is the only error component that depends on the neural network approximation. Moreover, we can bound the remaining term in (5) as

ÅZ ã1/2

T −1 T 2

ku(µ, t) − VV u(µ, t)kL2w ≤ m ku(µ, t) − VV u(µ, t)k d(µ, t) . P×T

−1 Let now K = |P × T |Ndata UUT ∈ RNh ×Nh be the discrete correlation matrix and let σk be its eigenvalues. By employing the triangular inequality and the trivial inequality

√ √ √ a + b ≤ a + b for a, b ≥ 0,

ÅZ ã1/2

−1 T 2

m ku(µ, t) − VV u(µ, t)k d(µ, t) ≤ P×T

ÅZ X X ã1/2

≤m−1 ku(µ, t) − VVT u(µ, t)k d(µ, t) − σk + σk ≤

P×T k>N k>N

ÅZ X X ã1/2

≤m−1 ku(µ, t) − VVT u(µ, t)k d(µ, t) − σk + σk ≤

P×T k>N k>N

Z X 1/2 X

−1 ≤m ku(µ, t) − VVT u(µ, t)k d(µ, t) − σk + m−1 σk .

P×T k>N k>N

In light of this, we define the sampling error as

Z X 1/2

ES := m−1 ku(µ, t) − VVT u(µ, t)k d(µ, t) − σk ,

P×T k>N

and the POD error as X EP OD := m−1 σk . k>N

Thus, we obtain the inequality in (4)

ER ≤ ES + EP OD + EN N .

In the last part of the proof we aim at showing the characteristic properties of ES and

EP OD ; recalling that

X |P × T | X

σk = kuj − VVT uj k ,

Ndata j=1

we can write the sampling error in a slightly different form Z −1

ES = m ku(µ, t) − VVT u(µ, t)k d(µ, t)−

P×T N data 1/2

|P × T | X

kuj − VVT uj k .

Ndata j=1

Moreover, thanks to the compactness hypothesis of Assumption 1 and the bounded-

ness hypothesis of Assumption 2 we have that f (µ, t) = ku(µ, t) − VVT u(µ, t)k ≤ M 2 kI − VVT k < +∞, so that f ∈ L (P × T ). Thus, by means of Proposition 1, we conclude that E[ES ] = −1/4 −1/2 O(Ns + Nt ). Finally, since ES and EP OD depend on the number of samples and snapshots in the training set, it is natural to verify their behavior in the infinite data limit. Thanks a.s. to Assumption 1, by the Strong Law of Large Numbers, it is evident that ES −−→ 0 as Ns , Nt → ∞ and, by means of the results in Section 2,

EP OD −−→ EP OD,∞ := m−1 2

Remark 1. The convergence rate for ES can be improved by modifying Assumption 1.

Indeed, Monte Carlo sampling could be replaced by other strategies: for instance, using

Quasi-Monte Carlo techniques [30, 3], and under suitable regularity assumptions, one

p+1 −1/2 −1/2 has E[ES ] = O((log(Ns )) 2 Ns + Nt ).

3.2 Lower bound for the relative error

POD-DL-ROMs couple classical projection-based methods such as the POD with

Deep Learning-based techniques that allow to correctly reproduce the nonlinearity

of the parameter-to-POD-coefficient map Q. This means that we still need to rely on the linear transformation represented by the POD matrix V ∈ RNh ×N (or V∞ in the infinite data limit) to expand the neural network approximation of the POD coefficients.

This last consideration is crucial: indeed, the fact that the POD-DL-ROM tech-

nique hinges upon a linear decomposition forces the relative error to still depend on the eigenvalues decay of the correlation matrix; the mentioned dependence is high- lighted in the lower bound result provided Theorem 3.2. First, we derive a lower bound for ES + EP OD : we immediately prove that

ES + EP OD ≥

ÅZ ã1/2

≥ m−1 ku(µ, t) − VVT u(µ, t)k d(µ, t) ≥ P×T

Å Z ã1/2

≥ m−1 min ku(µ, t) − WWT u(µ, t)k d(µ, t) =

W∈RNh ×N :WT W=I P×T

ÅZ ã1/2

= m−1 T ku(µ, t) − V∞ V∞ u(µ, t)k d(µ, t) = P×T X = m−1 2 σk,∞ = EP OD,∞ , k>N

by trivially employing the definition of ES , EP OD , and V∞ . It is worth to remark that EP OD,∞ only depends on the eigenstructure of the continuous correlation matrix K∞ , while it is independent of the data sampling. Thus, in the following, we aim at showing that, up to a constant, EP OD,∞ represent a lower bound also for the relative error ER .

Theorem 3.2. Under the same assumptions of Theorem 3.1, we have that

ER ≥ EP OD,∞ .

M Proof. We immediately notice that, by optimality of projection coefficients,

ku(µ, t) − Vq̂(µ, t)k ku(µ, t) − VVT u(µ, t)k

Z Z

ER = d(µ, t) ≥ d(µ, t),

P×T ku(µ, t)k P×T ku(µ, t)k

where we recall that V is the POD matrix computed via SVD using the discrete

formulation and V∞ is relative to the continuous formulation. Then, X (EP OD,∞ )2 = m−2 2 σk,∞ k>N Z

−2 T

=m ku(µ, t) − V∞ V∞ u(µ, t)kd(µ, t)

ZP×T

−2 ≤m ku(µ, t) − VVT u(µ, t)k d(µ, t) P×T ku(µ, t) − VVT u(µ, t)k Z = m−2 ku(µ, t)k d(µ, t)

P×T ku(µ, t)k

M ku(µ, t) − VVT u(µ, t)k

Z ≤ 2 d(µ, t) m P×T ku(µ, t)k M

≤ (ER )2 ,

m from which the thesis follows. Remark 2. Since V∞ is not available in practice, we cannot compute exactly EP OD,∞ .

In practice we can use a stricter bound: leveraging on quantities emerging from the

ÅZ ã1/2

−1 T 2

ẼP OD := m ku(µ, t) − VV u(µ, t)k d(µ, t)

when we either compute analytically (if possible) or estimate via Monte-Carlo.

This result states that no matter how accurate the neural networks approximation

is the relative error ER is still bounded from below by the variance that is not explained by the POD projection. Additionally, the lower bound does not depend on how much data we gather for the supervised training phase. Of note, this is in agreement with the results provided in the analysis of other linear decomposition-based techniques, such as DeepONets .

3.3 Upper bound for the relative error

On the basis of the error decomposition and the perfect embedding hypothesis, we

aim at providing the main result of this work, which is contained in the Theorem 3.3 and is endowed with a constructive proof founded on the approximation results of . We remark that the present result is only valid for POD-DL-ROMs. Theorem 3.3. Let G : (µ, t) 7→ u(µ, t) for any (µ, t) ∈ P × T be the parameter- to-solution map and suppose valid Assumptions 1 and 2. Let δ > 0 and 0 < ε < 1; suppose to have collected Ndata = Ndata (δ, ε) data samples into the snapshot ma- trix U ∈ RNh ×Ndata . Consider the (P × T )-discrete correlation matrix K = |P ×

T |Ndata UUT ∈ RNh ×Nh and let σk be its eigenvalues. Moreover, choose

m 2 ß X ™ N = arg min j ∈ N : σk ≤ ε . k>j

We define the parameter-to-POD-coefficients map Q : (µ, t) 7→ q(µ, t) as Q(µ, t) = VT G(µ, t) for any (µ, t) ∈ P × T , where V ∈ RNh ×N is the reduced rank-N POD matrix computed via SVD. We assume that there exists n > 0, Ψ∗ : Rn → RN , Ψ′∗ : RN → Rn that are respectively s-times and s′ -times differentiable (with s ≫ s′ ≥ 2), such that they enjoy the perfect embedding assumption stated in Definition 3, namely

Ψ∗ (Ψ′∗ (q(µ, t)) = q(µ, t) ∀(µ, t) ∈ P × T .

We let

Then, there exists a constant c = c(P, T , L, C , C , p, n, s, s′ ) and a POD-DL-ROM architecture Vq̂ = Vψ ◦ φ : Rp+1 → RN composed of a decoder ψ : Rn → RN having at most: • Ln→N = c log(ε−1 ) layers, • wn→N = cN ε−n/(s−1) log(ε−1 ) active weights, and a reduced map φ : Rp+1 → Rn having at most: • L(p+1)→n = c log(ε−1 ) layers,

• w(p+1)→n = cnε−(p+1) log(ε−1 ) active weights, such that P{ER < ε} > 1 − δ.

Proof. We immediately notice that, choosing N as in the theorem statement, we

derive X ε EP OD = m−1 σk ≤ . k>N

Then, we aim at bounding ES = ES (Ns , Nt ); under the Assumption 1, by the Weak Law of Large Numbers we can infer the following statement:

∀δ > 0, ∀0 < ε < 1, ∃Ns , Nt : P{ES (Ns , Nt ) < ε/3} > 1 − δ.

Then, we are left to bound EN N : by means of the Cauchy-Schwarz and the Hölder inequalities, considering that kVk = 1, it is trivial that ã1/2 kVq(µ, t) − Vq̂(µ, t)k ÅZ

EN N = d(µ, t)

P×T ku(µ, t)k

ÅZ ã1/2

−1 2 ≤m kVq(µ, t) − Vq̂(µ, t)k d(µ, t) P×T

Å ã1/2

≤ m−1 |P × T | sup kVq(µ, t) − Vq̂(µ, t)k (6) (µ,t)∈P×T

Å ã1/2

−1 ≤m |P × T |kVk sup kq(µ, t) − q̂(µ, t)k 2 (µ,t)∈P×T

= m−1 |P × T |1/2 sup kq(µ, t) − q̂(µ, t)k, (µ,t)∈P×T

Therefore, we are left to bound the error due to the neural network approximation of the map Q, namely sup kq(µ, t) − q̂(µ, t)k. (µ,t)∈P×T

Firsly, we notice that we can take n ≤ 2p + 3, since G (and consequently Q) is

Lipschitz-continuous (see Theorem 3 in ). Then, we proceed as in , by employing

two different steps: • Consider the reduced manifold SN := {q = Q(µ, t) : (µ, t) ∈ P × T }; then Vn = Ψ′∗ (SN ) is such that diam(VN ) ≤ LC diam(P × T ), thanks to the Lipschitz- continuity hypothesis provided by Assumption 2. Thus, by Theorem due to Gühring et al. recalled in Section 1.2, there exists a ReLU DNN ψ : Rn →

RN such that

m sup kψ(v) − Ψ∗ (v)k < |P × T |−1/2 ε v∈Vn 6 (7) |(ψ − Ψ∗ )(v) − (ψ − Ψ∗ )(v′ )| m ess sup ′ < |P × T |−1/2 ε, v,v′ ∈Vn |v − v | 6

with Ln→N = c log(ε−1 ) layers and wn→N = cN ε−n/(s−1) log(ε−1 ) active weights.

Notice that the Lipschitz constant of ψ is bounded by the quantity C =

C + m |P × T |

−1/2 ; • Setting φ∗ (µ, t) = Ψ′∗ (q(µ, t)) ∀(µ, t) ∈ P × T , we notice that it is Lipschitz- continuous, with constant bounded by LC , and thus, by the Theorem due to Yarotski recalled in Section 1.1, there exists a ReLU DNN φ : Rp+1 → Rn such that m sup kφ(µ, t) − φ∗ (µ, t)k < |P × T |−1/2 ε, (8) (µ,t)∈P×T 6C 3

with L(p+1)→n = c log(ε−1 ) layers and w(p+1)→n = cnε−(p+1) log(ε−1 ) active weights. Moreover, let q̂ = ψ ◦ φ : Rp+1 → RN be the underlying neural network of the

POD-DL-ROM. Then, by means of the triangular inequality, the perfect embedding

Assumption, the definition of φ∗ , and the Lipschitz-continuity of ψ, we derive:

sup kq(µ, t) − q̂(µ, t)k (µ,t)∈P×T

≤ sup (kΨ∗ (Ψ′∗ (q(µ, t)) − ψ(φ∗ (µ, t))k + kψ(φ∗ (µ, t)) − ψ(φ(µ, t))k) (µ,t)∈P×T m ≤ sup kψ(v) − Ψ∗ (v)k + C sup kφ(µ, t) − φ∗ (µ, t)k < |P × T |−1/2 ε, v∈VN (µ,t)∈P×T 3

employing the bounds (7) and (8). Then, plugging the last inequality in (6) we can state that EN N < 3ε . Finally, by means of the error decomposition formula, we derive the desired bound ε ε ε ER ≤ EP OD + ES + EN N < + + = ε, 3 3 3 with probability greater than 1 − δ.

Remark 3. The DL-ROM paradigm proposed in and applied to cardiac elec-

trophysiology in , has been theoretically analyzed in , providing approximation bounds and a complexity analysis, which shows that in general DL-ROMs suffer from curse of dimensionality with respect the number of high-fidelity dofs Nh . Relying on the present Theorem 3.3, we demonstrate how the preliminary dimensionality reduc- tion through POD affects both the complexity of the POD-DL-ROM and its approx- imation capabilities. Indeed, POD-DL-ROMs avoid the curse of dimensionality of

the DL-ROMs at the cost of discarding the small scales contribution, which might be however relevant when considering, e.g., highly nonlinear problems showing a slow eigenvalue decay. On the other hand, POD-DL-ROMs provide a neural network ar- chitecture with a lower number of trainable weights, thus yielding a lighter training procedure in practice. Finally, we can highlight that the a priori choice of employing

DL-ROMs or POD-DL-ROMs must be based exclusively on the linear reducibility of

the problem and the availability of computational resources.

4 Comparative analysis with deep learning-based

existing strategies On the basis of the results of the previous section, we comment the advantages of

POD-DL-ROMs when compared with other deep learning-based existing strategies

present in the literature, namely: • simple DNNs to approximate the POD (or Kernel-POD) coefficients, that results in the widely used POD+DNN approach [6, 21, 36, 39];

• the POD-DeepONets architecture, which was proposed in and based on the classical DeepONets approach ;

• the technique presented in , which aims at reconstructing the parameter-to- solution map by coupling linear projection methods and residual networks and which we will hereon refer to as lin+ResNets;

• the CNNs architecture for operator learning proposed in , whose analysis is based on the Fourier decomposition.

4.1 POD-DL-ROMs vs POD+DNNs: a matter of regularity

The purpose of this subsection is to highlight how the POD-DL-ROM approach pro-

vides a suitable setting to establish tighter bounds on the model complexity when compared to generic POD+DNNs, especially when the parameter-to-solution map is not regular. It is worth to remark that, under the hypothesis of Theorem 3.3, the number of layers of the POD-DL-ROM network architecture is expected to scale as

while the total number of active weights behaves as wP OD−DL−ROM = wn→N + w(p+1)→n = O(N ε−n/(s−1) log(ε−1 )) + O(nε−(p+1) log(ε−1 )).

We expect that, in general n ≪ N = N (ε); moreover, n ≤ 2p + 3 since the parameter-to-POD-coefficients map is Lipschitz-continuous, due to Assumption 2. Thus, it is evident that the majority of the neural network complexity amounts to the decoder, which has to perform the most difficult task, namely, decoding the informa- tion provided by the latent coordinates. Instead, the reduced network only aims at providing an alternative representation of the time-parameters vector (µ, t) such that

it makes as easy as possible for the decoder to reconstruct the POD coefficients. Not- ing that wP OD−DL−ROM depends exponentially on s, we can control the complexity of the POD-DL-ROM by choosing s as large as possible, namely, s ≫ s′ ≥ 2.

Essentially, we aim at finding a representation of POD coefficients of the form

Ψ∗ (Ψ′∗ (q(µ, t)) = q(µ, t) ∀(µ, t) ∈ P × T , (9)

through the composition of an encoder Ψ′∗ that absorbs all the irregularity of the identity map I = Ψ∗ ◦ Ψ′∗ , and a decoder Ψ∗ that is extremely regular. We highlight that the perfect embedding Assumption stated in Definition 3 is critical; indeed, under the hypothesis of Theorem 3.3, leaving out only the perfect embedding assumption, we may be tempted to trivially use Yarotski’s Theorem to construct a ReLU DNN which has LP OD+DN N layers and wP OD+DN N active weights, where

LP OD+DN N = O(log(ε−1 ))

wP OD+DN N = O(N ε−(p+1) log(ε−1 )),

in order to control the relative error with ER < ε. Notice that: • the number of layers LP OD+DN N is of the same order as LP OD−DL−ROM ; • the estimate of the number of active weights wP OD+DN N can only take ad- vantage of mild regularity assumptions on G (and Q), that is only Lipschitz- continuous.

However, it is evident that Theorem 3.3 only provides a theoretical result offering

a different perspective in order to enhance the complexity estimate of POD+DNN.

Indeed, within the framework stated by Theorem 3.3, given an accuracy level ε one

could take advantage of the POD-DL-ROM theoretical setting, and thus the perfect embedding Assumption, to construct a proper architecture that approximates the parameter-to-solution map keeping ER < ε – and, then, notice that the resulting ar- chitecture is indeed in general a POD+DNN. The difference in practice is represented by the training procedure. Indeed, notice that training a network like the one involved in a POD+DNN with the classical supervised loss formulation, by letting ωn = 0 in

(2.1) and thus without taking advantage of the encoder, does not ensure to recover an adequate representation in the latent space. Instead, if we train the network relying on thje POD-DL-ROM paradigm, namely taking ωn > 0 in (2.1), we actually employ the encoder to implicitly enforce the architecture to satisfy the perfect embedding Assumption, and then discard the encoder in the online testing phase. Suppose now that N ≫ n: trivially, we have that wDN N & w(p+1)→n ; moreover,

n wP OD+DN N & wn→N , upon requiring that (s−1) < p + 1, that provides an estimate n for the regularity of the decoder in the representation (9), that is s > p+1 + 1. In practice, given that n ≤ 2p + 3, we can safely assume that s & 3 + p+1 and finally s & 4. Thus, if the parameter-to-solution map G is only Lipschitz-continuous, if the

perfect embedding Assumption is satisfied for s ≥ 4, POD-DL-ROMs achieve a tighter bound on the model complexity when compared to general POD+DNN approaches: this is due to the fact that there exists a better representation (in terms of regularity) φ∗ (µ, t) for the time-parameters vector (µ, t) that can be recovered by the reduced network.

Until now, we considered the case where the parameter-to-solution map is only

Lipschitz-continuous; however, it is interesting to consider cases where we can verify

that the map G shows higher regularity, and see how this increased regularity affects the complexity of both POD-DL-ROMs and POD+DNNs in terms of number of active weights. Indeed, by means of similar arguments employed previously, and thanks to the Theorem due to Yarotski recalled in Section 1.1, assuming that

G ∈ W r,+∞ (P × T ; RNh ), we obtain that

wP OD+DN N = O(N ε−(p+1)/r log(ε−1 )).

Thanks to the fact that the exact reduced map φ∗ of Theorem 3.3 now would be min{r, s′ }-times differentiable,

wP OD−DL−ROM = wn→N + w(p+1)→n ′ = O(N ε−n/(s−1) log(ε−1 )) + O(nε−(p+1)/ min{r,s } log(ε−1 ))

Assuming that N ≫ n, it is trivial to verify that wP OD+DN N & w(p+1)→n ; further- more, it is valid that wP OD+DN N & wn→N if (s−1) n > p+1 nr r , that is s > p+1 + 1, which gives the estimate s & (2 + p+1 )r + 1 and finally s & 3r + 1. The meaning of the last estimate is that, if the parameter-to-solution map is extremely regular (namely, r → ∞), it becomes more and more difficult for the POD-DL-ROMs to guarantee lower complexity than simple POD+DNNs, since the perfect embedding

Assumption should be verified for s → ∞. This is rather intuitive: indeed, if the parameter-to-solution map is extremely regular, we do not need a to recover a better representation φ∗ (µ, t) for the time-parameter vector (µ, t) in order to make it easier for the underlying neural network to learn the solution manifold.

4.2 POD-DeepONets and POD-DL-ROMs: a comparison

In this subsection, we aim at analyzing the POD-DeepONet architecture from a theo-

retical standpoint, showing the close relationship with POD-DL-ROMs when dealing with problems whose general formulation can be reduced to (2). We let X be a Ba- nach space and consider a compact subset K ⊂ X and a compact subset K ⊂ Rd , where d denotes the number of spatial (or spatio-temporal) dimensions of the prob- lem at hand. Defining W ⊂ C(K ) as a compact subset, we suppose that we aim at learning the operator G∞→∞ : W → C(K ), where the subscript highlights that the considered operator is a map between infinite-dimensional spaces. We first consider

a DeepONet architecture employed to reconstruct G∞→∞ , which in its unstacked formulation consists in the combination of the output of two different neural networks through the scalar product. In particular, we define the branch net b : W → RN as the neural network that processes information about the input function φ ∈ W , and the trunk net τ : Rd → RN , which aims at encoding the coordinate input y ∈ Rd in a set of basis functions. Then, we can define the DeepONet approximation as

G∞→∞ (φ)(y) ≈ Ĝ(φ)(y) = b(φ) · τ (y). (10)

and note that N describes the number of basis functions employed in the decomposi- tion (10); thus, N plays the same role as the POD dimension in the POD-DL-ROM architecture. Based on the analysis proposed in , we can split the DeepONet op- erator Ĝ : W → C(K ) into Ĝ = Rτ ◦ Am→N ◦ Em , where Em , Am→N and Rτ are defined as follows: • the encoder operator is defined as the map Em : C(K) → Rm , such that, given xi ∈ K , ∀i = 1, . . . , m:

Em (ψ) = [ψ(x ), ψ(x ), ..., ψ(xm )]T ∀ψ ∈ C(K ).

It is worth to notice that Em is well defined since any continuous function can be evaluated pointwise;

• Am→N : Rm → RN is the approximation operator; thus, we can decompose the branch net of the DeepONet operator as b = Am→N ◦ Em ;

• recalling that τ : Rd → RN is the trunk net, we define the τ -induced recon- structor operator Rτ : RN → C(K ) as

In a more compact formulation, we retrieve the classical architecture of the Deep-

Ĝ(φ) = Rτ ◦ (Am→N ◦ Em (φ)) = Rτ ◦ b(φ) = b(φ) · τ .

POD-DeepONets were recently introduced in and the test cases considered within

the paper confirm better approximation accuracy when compared with classical Deep-

ONets: the methodology consists in substituting the trunk net with the corresponding

row of the POD matrix. The drawback is that POD-DeepONets can only approximate operators defined as G∞→Nh : W → RNh , losing the capability of mapping between infinite-dimensional spaces.

Supposing to initially deal with stationary, time-independent problems and denot-

ing by vj ∈ RN the j-th row of the POD matrix V ∈ RNh ×N , we define the expansion operator Lvj : RN → RNh as

Lvj (ξ) = ξ · vj ∀ξ ∈ RN ,

and the POD-DeepONet operator as

[G∞→Nh (φ)]j ≈ [ĜP OD−DeepON et (φ)]j = q̂(φ) · vj = Lvj ◦ Am→N ◦ Em (φ),

∀j = 1, . . . , Nh , where q̂ is the corresponding branch net, which now approximates the underlying POD coefficients. It is worth to notice that, by employing the vector formulation, we can write:

G∞→Nh (φ) ≈ ĜP OD−DeepON et (φ) = Vq̂(φ).

Then, we need to adapt the POD-DeepONet framework to the problem considered

within this work (2), where even the input parameter space is finite-dimensional, thus

eliminating the need of the encoder operator Em . Indeed, POD-DeepONets for finite- dimensional-input problems involving the reconstruction of the map Gp→Nh : Rp →

RNh take the form

[Gp→Nh (µ)]j ≈ [ĜP OD−DeepON et (µ)]j = q̂(µ) · vj = Lvj ◦ Ap→N ,

∀j = 1, . . . , Nh , or in a more compact way

Gp→Nh (µ) ≈ ĜP OD−DeepON et (µ) = Vq̂(µ),

where µ ∈ P ⊂ Rp , P compact. It is worth to notice that in this case the branch net coincides with the approximation operator Am→N .

Finally, in order to include also the time-dependence, we could adopt two different

strategies: • we could treat the time t as a spatial coordinate in a DeepONet-like way, leading to a POD matrix of dimension Nh Nt × N , that however increases the possible impact of the curse of dimensionality, however offering the opportunity to deal with time-dependent basis functions; • alternatively, we may consider the time t as an additional parameter, a choice which reduces the computational requirements and is consistent with the POD-

DL-ROM approach, leading to the construction of time-independent global spa-

Within this comparison, for the sake of consistency, we choose to employ this latter

approach. Thus, aiming at reconstructing the map (µ, t) 7→ u(µ, t), we could employ different neural network architectures; for instance, if we choose to employ a DL-ROM architecture as the branch net of the POD-DeepONets, we retrieve the POD-DL-ROM approach, while employing a vanilla DNN as the branch net results in the POD+DNN approach. The comparison between POD-DL-ROM and POD+DNNs is extensively treated in the previous subsection.

Finally, inspired by the DeepONet approach, we notice that extending the content

of the present paper to the case of infinite-dimensional input parameters is straight- forward and introduces an additional source of error, namely the encoding error, that ultimately depends on the variability of the input parameters and their spatial dis- cretization; for a thorough discussion on the topic, we refer the reader to, e.g., .

4.3 Learning POD coefficients with ResNets

The ResNets-based approach proposed in couples linear decompositions and resid-

ual networks (ResNets) to reconstruct field-to-solution maps, an approach which is inherently close to POD-DL-ROMs. In this case, we start our analysis of the tech- nique by examining the proposed architecture, and by adapting it to the problem formulation considered within the present work.

Indeed, we immediately notice that the lin+ResNet architecture needs that ev-

ery residual layer has input dimension equal to the output dimension layer output dimension: iterating, for a fully residual network, we must require that the input of the network has the same dimension of the network output. Such a constraint in the architecture is managed in by projecting both the input fields and the output targets onto two linear subspace of equal dimension N ≪ Nh , where Nh is

the FOM dimension. Then, the output targets are numerically approximated on the same mesh and projected onto a subspace of dimension N , too. The approach results in the sequence of maps: lin.proj. residual residual residual lin.lif t. RNh −−−−−→ RN −−−−−→ RN −−−−−→ ... −−−−−→ RN −−−−−→ RNh ,

where the linear projection is usually carried out by employing POD, Karhunen-

Loève expansions or active subspaces . However, when dealing with finite

dimensional parameter inputs instead of fields (for instance (µ, t) ∈ Rp+1 with p+1 < N ), it may occur that the ResNet input dimension (p + 1) is different from the output dimension N ; to fill the gap, it is necessary to employ for instance a dense layer Rp+1 → RN as the first layer of the architecture. Thus, we will consider the sequence of maps: dense residual residual residual lin.lif t. Rp −−−−→ RN −−−−−→ RN −−−−−→ ... −−−−−→ RN −−−−−→ RNh .

The lin+ResNets approach ultimately aims at providing a constructive way to

build a neural network in terms of breadth and depth.

The breadth, which may be intuitively defined as the maximum number of neu-

rons per layer in the network, coincides with N , the characteristic dimension of the preliminary dimensionality reduction. In order to favour compressed representations, the authors of suggest keeping as low as possible the latent dimension k of the

ResNet, which can be identified with the dimension of the nonlinearity added at each

layer. Indeed, the residual map between the layer zl ∈ RN and zl+1 ∈ RN can be identified with zl+1 = zl + W1l σ(W0l zl + bl ), where W0l ∈ RN ×k , W1l ∈ Rk×N , bl ∈ Rk and σ is the activation function; the total number of weights per layer is then O(N k). However, in contrast to our approach, they did not propose a way to identify k: we remark that the discussion on the latent dimension n of the POD-DL-ROM architecture is fundamental because it allows to set a tighter bound on the complexity of the decoder network in terms of active weights.

Furthermore, the authors developed approximation bounds on the underlying

ResNet complexity in terms of its depth, employing the connection between ResNets, Neural ODE and control flows . The bound on the ResNet depth enable the user to control the ℓ error on the solution (and by extension the relative error too) with a suitable bound ε by employing O(ε−1 ) layers. Thus, we can straightforwardly state that, on the basis of the complexity analysis, POD-DL-ROMs outperform the ResNets-based approach in terms of number of layers:

Llin+ResN ets = O(ε−1 ) & O(log(ε−1 )) = LP OD−DL−ROM

wlin+ResN ets = O(N kε−1 ) & O(N ε−n/(s−1) log(ε−1 ))+ + O(nε−(p+1) log(ε−1 ))) = wP OD−DL−ROM ,

supposing for instance N ≫ n and s ≥ n + 1, which are reasonable assumptions.

Indeed, N ≫ n is satisfied when the nonlinear Kolmogorov n-width decays much

faster that the eigenvalue decay of the correlation matrix, a phenomenon that is

usually encountered in applications; the condition s ≥ n + 1 is valid by ensuring s ≥ 2p + 4, that is the decoder map must be sufficiently regular.

Despite the disadvantage on the complexity front, we remark that ResNets consti-

tute one of the most suitable paradigms to implement adaptive-depth architectures, since adding a layer to an already trained architecture can produce an arbitrary small perturbation on the network output; for a more detailed analysis on the lin+ResNets training, we refer the reader to .

4.4 The effect of the POD basis optimality on the network

complexity Within this subsection, our purpose is finally to show how choosing the POD basis as global spatial basis function in the linear decomposition leads to a reduced complexity of the underlying neural network, comparing in details CNNs for operator learning and POD-DL-ROMs. In particular, we notice that, within the POD-DL-ROM approach, the reconstruction of the approximated solution

P at the high-fidelity level depends on

the decomposition assumption u(µ, t) ≈ j<N q̂j (µ, t)vj , where N denotes the POD dimension. Analogously, the recent work on the approximation bounds for CNNs proposed in strives to reconstruct a decomposition between global spatial basis functions thatPare strictly related to the Fourier modes, and a set of coefficients, that is, u(µ, t) ≈ j<C âj (µ, t)fj , where the sum is over C terms (the number of channels in the input and output is O(C)). In the following, we assume that u(·, µ, t) ∈ C α (Ω) for any (µ, t) ∈ P × T , being

α ≥ 1 the spatial regularity, and ε > 0 is the desired accuracy level; we then describe the three main differences between the CNN-based approach and the POD-DL-ROM technique: • The convolutional block is limited to uniformly spaced mesh points (h is the spacing parameter) in square domains, while POD-DL-ROMs are more versatile both in terms of the domain shape and the mesh properties. • The architecture proposed in consists of two different blocks: the dense block is devoted to the parameter-dependent coefficient approximation, while

the convolutional block strives to reconstruct the spatial basis function. Instead,

POD-DL-ROMs compute the spatial basis before the training of neural networks

by means of SVD or randomized SVD through an unsupervised learning criterion: in principle, this means that POD-DL-ROMs do not need any active weights to reconstruct the spatial basis functions, while the CNN approach needs O(ε− 2α−1 log(h−1 )) weights to learn them (we refer the reader to Theorem 2 in ). • In the decomposition employed in , C plays the role of the reduced dimen- sion: it is an analogue of the POD-dimension N employed within the POD-DL-

ROM technique. In the following, we exploit an optimality result fulfilled by the

POD basis to show that the complexity of the neural network in the parameter-

to-coefficient map approximation is lower in the case of POD-DL-ROM when compared to the approach proposed in .

The quasi-optimality of the POD decomposition in its discrete formulation con-

firms that with a N -terms truncation, provided a sufficient amount of data have been

suitably sampled, no linear decomposition captures as much variance as the discrete formulation of the POD decomposition, so that the reduced dimension C of sat- isfies the inequality C > N with probability 1 − δ (see Subsection 2.2 and Appendix A). Furthermore, we assume that: (i) N ≫ n as usual, since we expect that the nonlinear Kolmogorov n-width decays (much) faster than the linear reduced dimension N ; (ii) u(·, µ, t) ∈ C α (Ω) for any (µ, t) ∈ P × T for some α ≥ 1 to comply with the

hypotheses of Theorem 2 of ; (iii) the parameter-to-solution map has regularity r, i.e. G ∈ W r,∞ (P × T ; RNh ); n (iv) the decoder map is adequately regular, namely s−1 > p+1 r (s ≥ 3r + 1 is sufficient, as in 4.1). We recall that Theorem 2 in provides the estimate C = O(ε− 2α−1 ). Therefore, in the worst case scenario N = O(ε− 2α−1 ); however, depending on the singular values decay that in some cases might be even exponential (e.g. stationary elliptic PDEs, analytic parameter-to-solution maps, see ) we actually obtain improved estimates.

wP OD−DL−ROM = O(N ε−n/(s−1) log(ε−1 )) + O(nε−(p+1) log(ε−1 )) ≈ O(N ε−n/(s−1) log(ε−1 )) . O(Cε−n/(s−1) log(ε−1 ))

2 n

. O(ε− 2α−1 [ε− (s−1) (log(ε−1 ) + log(h−1 )])

2 p+1

. O(ε− 2α−1 [ε− r (log(ε−1 ) + log(h−1 )]) = wCN N .

Thus, we can conclude that, if the hypotheses setting is verified, the overall com-

plexity of the POD-DL-ROMs in terms of active weights is lower (or equal) than the complexity of the CNN architecture proposed in .

5 Numerical experiments

Within this section, we present different numerical tests, aiming at validating the

theoretical analysis proposed in the previous Sections. In particular, we focus on (i) the error bounds of Theorems 3.2–3.3 and the error decomposition formula, as well as on (ii) the role of the reduced dimension N and the total number of snapshots Ndata and on (iii) the comparison against recent approaches proposed in the literature, in light of the theoretical results of Sections 3 and 4. In particular, the numerical experiments involve: a) a benchmark test case with an analytically defined operator that allows us to

know a priori the properties of the parametric operator (like, e.g., the regularity of the parameter-to-solution map) in order to validate the theoretical estimates on the network complexity;

b) a linear 1D Initial Boundary Value Problem (IBVP), to show how to select Ndata and N in order to minimize the a priori error (given by the sum of ES and EP OD ), then validating a posteriori the network complexity as a function of the relative error; c) a nonlinear 2D time-dependent IBVP in a non-conventional domain, to show the effectiveness of the POD-DL-ROM approach when dealing with more complex problems, validating also the lower bound and the upper bound on the relative

error ER , which stem from the theoretical analysis.

We remark that the complexity analysis of POD-DL-ROM and related approaches

is discussed from a theoretical point of view only in terms of the approximation error; however, when numerical experiments are addressed, we also have to take into account the training error, which plays a major role especially when the network is sufficiently deep or wide, or data are limited. For the same reasons, in our numerical experiments we mainly address the complexity study in terms of number of active weights w, since the latter is a quantity which is less sensitive (when compared to the depth L) to

the training error. Thus, the experimental complexity analysis presented here may not reflect exactly the estimates provided in the previous sections, but they validate qualitatively the theory. However, within the present section, aiming at mitigating the effect of the training error on the error estimates, we employ several ad hoc strategies, like, e.g., • we employ early stopping to prevent overfitting; • the approximation results in terms of network complexity are achieved in an

error range [ε , ε ] that is deemed appropriate for the chosen number of samples Ndata : in practice the training error depends on data availability; • for fixed number of active weights, we regulate the network architecture trying to randomly achieve the configuration that minimizes the training error; we keep the depth of the network as low as possible in order to ensure convergence to a suitable minimum and avoid expensive training loops; • starting from educated guesses, we look for the best training hyperparamenters

(which are the learning rate and the learning rate decay).

Finally, we remark that, in order to comply with the hypotheses of the Theorems

of Section 3, we limit the numerical experiments to generic dense layers and employ LeakyReLU as activation function: ® x, x≥0 LeakyReLUα (x) = . αx, x<0

Unless otherwise stated, we set α = 0.1. The optimization procedure is carried out

by employing the Adam algorithm .

5.1 Benchmark test case

We begin our experimental analysis by considering a benchmark test case similar to

the one described in , and involving the reconstruction of an analytically defined operator, namely uβ (x, µ) = µ |x − µ |β e−µ x , x ∈ [0, 1],

where µ = [µ , µ , µ ] ∈ P = [0, 1] × [0, 1] × [1, 2]. Within this numerical test we vary β ∈ {3/2, 7/3, 3} and we analyze the three resulting cases independently. Notice that the hyperparameter β > 0 controls the regularity of the parameter-to-solution map. Indeed, u3/2 (x, ·) ∈ W 1,+∞ (P) \ W 2,+∞ (P) u7/3 (x, ·) ∈ W 2,+∞ (P) \ W 3,+∞ (P) u (x, ·) ∈ W 3,+∞ (P) \ W 4,+∞ (P); thus β = 3/2, 7/3, 3 correspond to r = 1, 2, 3 respectively, where r is defined as the regularity of the parameter-to-solution map in agreement with this paper notation.

Furthermore, the problem does not depend on the time variable, thus we set Nt = 1, Ndata = Ns and p = 2 (instead of p = 3) to comply with the theoretical framework of the present work. Moreover, we discretize the problem in space by means of a uniform discretization with Nh = 1000. Selecting n = 5 ≤ 2p + 3 = 7 to ensure both a suitable compression and an adequate representation in the latent space, Ns = 500, and  20,  r=1

N = N (r) = 17, r=2

to control the variability retained by the preliminary linear dimensionality reduction.

We then proceed towards a complexity analysis, showing a comparison of the results

against the CNN approach considered in , the POD+DNN framework and the lin+ResNets technique. We remark that for the sake of fairness and consistency, we keep the batch size during training equal to B = 2 for every comparison considered in the benchmark test case. Then, for any r ∈ {1, 2, 3}, we estimate the approximation error ER on the respective test set consisting of Nstest = 1 samples. From a theoretical standpoint, we immediately notice G : µ 7→ u(µ) ∈ W r,+∞ (P; RNh ); then, from the findings of Section 4, since n ≪ N , we can infer that

wP OD+DN N = O(N ε−3/r log(ε−1 )) wP OD−DL−ROM = O(N ε−5/(s−1) log(ε−1 )).

Thus, owing to the fact that in the POD-DL-ROMs approach the perfect embedding

Assumption with coefficients s, s′ is enforced thanks to their peculiar loss formulation, we expect them yielding a less steep increase (when compared to POD+DNNs) in the model complexity as the accuracy level decreases whenever the decoder map is suitably regular, which is equivalent to require s > 5 r + 1. Figure 1 demonstrates that the latter behavior is more likely to happen as the regularity of the parameter-to-solution map r decreases.

We then compare POD-DL-ROMs against the lin+ResNets approach; for the lat-

ter, we limit the analysis to the case where the basis functions are yielded by POD for the sake of consistency. We thus fix the latent space dimension of the residual layers as k = 5 and, from the estimates obtained in Section 4, we recall that the complexity bound of lin+ResNets in terms of number of active weights is in general independent of the regularity of the parameter-to-solution map, namely: wlin+ResN ets = O(N kε−1 ).

We thus remark that the lin+ResNets approach does not take advantage of any regu-

larity assumption on the parameter-to-solution map: we then expect a similar trend

Figure 1: Benchmark test case: model complexity comparison between POD-DL-

ROMs and POD+DNNs as the parameter-to-solution regularity r varies in {1, 2, 3}. The trends are displayed through solid lines, which fit the collected results in the least squares sense.

as r varies in {1, 2, 3}. Nonetheless, if the trained POD-DL-ROM architecture are able to find an adequate representation in the latent space which induces a very reg- ular decoder, that is s > 6, we can ensure that the POD-DL-ROM outperform the lin+ResNets approach in terms of complexity: this behavior is indeed observed in

Figure 2.

Figure 2: Benchmark test case: model complexity trend of POD-DL-ROMs and the

lin+ResNets approach for different values regularity of the parameter solution map r. Finally, we consider the comparison against the CNN approach considered in : if the decoder map is sufficiently regular (from the theoretical analysis we derive the condition s ≥ 5 r + 1), POD-DL-ROMs take advantage of the basis optimality to achieve a less steep increase of complexity as the error bound ER < ǫ decreases: the behavior is indeed observed in Figure 3, in the cases when the regularity of the

parameter-to-solution map is low (r = 1, 2). Moreover, differently from the CNN- based technique, we remark that the POD-DL-ROMs’ algorithm does not require to learn the basis functions, thus not affecting the overall complexity of the underlying network.

Figure 3: Benchmark test case: comparison between POD-DL-ROMs and CNNs in

terms of number of active weights, varying the regularity r ∈ {1, 2, 3}.

5.2 1D Initial Boundary Value Problem

The present test case is designed to highlight the advantages of POD-DL-ROMs when

compared to other considered approaches even when dealing with time-dependent parametrized problems. Moreover, before starting the training process, we show a priori how to choose the hyperparameters N, Ns , Nt , based on the analysis of ES and EP OD . In particular, we consider the following IBVP:

∂u ∂ 2 u  − 2 = u + 1 cos(x) sin(2πt), in (0, π) × (0, T ]   ∂t ∂x     u = 10(2µ − 3µ + µ),  at {x = 0} × (0, T ]  ∂u   = 2|1 − 2µ| − 1, at {x = π} × (0, T ] ∂x     u(x, 0) = u (µ), in (0, π), 

u = u (µ) = 10(2µ − 3µ + µ) cos(x) + (2|1 − 2µ| − 1) sin(x),

while µ ∈ P = [0, 1] and T = 1. Thus, p = 1 and we can fix n = 5 = 2p + 3 to ensure an adequate representation in the latent space, according to the framework presented in the present paper. We collected synthetic data generated with an high- fidelity model solved on a uniform grid of Nh = 1 points: we generate a test set of Nstest = 1 samples of Nttest = 2 snapshots each with a Matlab-based PDE solver, sampling µ ∼ U(P) iid and t from a uniform grid of step ∆ttest = T /Nttest .

We start by analyzing the dependence of ES on Ns , Nt and N ; for the sake of clarity, we specify that the sampling criterion employed in the a priori analysis below is based on the theoretical analysis of the entire work: thus, we assume µ ∼ U(P) iid and that t is sampled from a uniform grid of step ∆t = 1/Nt . To analyze the effect of Ns on the sampling error, we fix Nt = 10 and we generate a group of datasets depending on Ns ∈ {l = 2k : k = 1, . . . , 7}: as shown in Figure 4, the decay has slope

−1/4 and it is independent of the chosen value of N . Conversely, we fix Ns = 1 and vary Nt ∈ {l = 2k : k = 3, . . . , 9}, validating experimentally in Figure 4 that −1/2 ES ∼ N t , independently of N . We then move to the analysis of the projection error, showing in Figure 5 how EP OD decays with N and is mostly independent of Ns

and Nt respectively. We notice that the present analysis is done before the training of the underlying neural network and allow us to know a priori how much variance is not accounted for due to the sampling (ES ) and the initial dimensionality reduction (EP OD ), allowing us to calibrate the values N, Ns , Nt before we start the expensive training procedure. The idea is to choose N, Ns , Nt to guarantee that EP OD and ES are suitably small, so that we can control the relative error ER with a strict bound,

which is provided by the error decomposition of Theorem 3.1. Thus, based on the

results of the present a priori analysis, we choose Ns = 50, Nt = 20, N = 20.

We then move our focus to the comparison of the POD-DL-ROM technique against

other approaches in terms of complexity, showing the relation between the relative error ER and the number of active weights employed in the underlying neural network. Notice that, since the analytical solution of the IBVP is not available, here we are not provided with any information on the regularity of the parameter-to-solution map.

Anyway, experimental results on the complexity analysis confirm our theoretical ex-

pectations: when dealing with parameter-to-solution maps arising from parametric

PDEs, POD-DL-ROMs’ complexity increases slower than POD+DNNs’ one as the

relative error decreases. Indeed, the latent representation of the POD-DL-ROM ap- proach induces a decoder that is extremely regular, that is s ≫ 2, which enables a slow increase in network complexity, as suggested by the theoretical approximation bounds of Theorem 3.3 and validated in Figure 7. Similarly, we notice that the results relative to the comparison between POD-DL-ROMs and lin+ResNets are in agree- ment with the theory, demonstrating again how, lin+ResNets are outperformed in

terms of complexity by POD-DL-ROMs, when it is possible for the latter to achieve an extremely regular decoder map due to an adequate latent representation. Finally, when compared to the Fourier-inspired CNN technique POD-DL-ROMs’ number of active weights show a slower increase as the relative error ER decreases, as shown in

Figure 7; as proved theoretically in Section 4, the magnitude of the slope is strongly

linked to the optimality of the basis functions. Moreover we validate how the burden of learning the set of basis function impacts heavily on the underlying CNN complex- ity, which shows a remarkable difference when compared the POD-DL-ROM approach in terms of number of active weights, not only regarding the slope magnitude but also in the absolute sense. The observed behavior highlights how crucial it is in terms of complexity to consider a fixed set of optimal basis functions instead of a learnable set

Figure 4: 1D IBVP test case: decay of the sampling error ES with respect to Ns , Nt

Figure 5: 1D IBVP test case: decay of the projection error EP OD varying Ns , Nt and

Figure 6: 1D IBVP test case: comparison between the ”true” solution (solid black line)

and the most accurate POD-DL-ROM prediction (dashed red line) to demonstrate that the variability of the solution manifold is correctly reproduced.

Thus, this validates the theoretical considerations and concludes our comparison

based on model complexity, demonstrating how POD-DL-ROMs outperform any of the considered techniques when tackling more complex problems, for which the reg- ularity of the parameter-to-solution map is low or unknown a priori.

5.3 2D nonlinear Initial Boundary Value Problem

The last test case involves a nonlinear version of a time-dependent nonlinear parametrized

diffusion equation with a non-affine source term in an unconventional domain; the

strong formulation of the problem at hand takes the form

∂u  Å ã   − ∇ · 0.001(1 + u )∇u = 0, in Ω × (0, T ]     ∂t u = 1 − e−100t + h(x, y, µ)e−100t ,  on ΓD × (0, T ]   ∂u = 0, on ΓN × (0, T ]   ∂n     u = h(x, y, µ), in Ω, 

where T = 0.0 and • h(x, y, µ) = 0.1 + 10y sin(µπx) represents a non-affine term, being µ ∈ P = [5, 7] the parameter that regulates the spatial frequency of h = h(x, y, µ); • letting Ea,b (x, y) be the ellipse of axes a and b and center (x, y), we set D = E0.2,0.2 (0.5, 0.4) and D = E0.3,0.1 (1.0, 0.2); then, we can define the domain as

Ω = (0, 1) × (0, 0.4) \ (D ∪ D );

• the Dirichlet and the Neumann boundary are ΓD = ∂D ∪ ∂D and ΓN = ∂Ω \ (∂D ∪ ∂D ), respectively.

Through this numerical experiment we aim at verifying the upper bound and lower

bound results presented in Section 3. To do so, we generate the training set and the test set input-output pairs through the numerical solution of the discretized problem on a mesh of Nh = 16 dofs by means of P1-FEM, employing a Forward Euler time- advancing scheme and the Newton method to handle nonlinearities. The training set is made by Ns = 2 samples relative to µ ∼ U(P) iid of Nt = 3 snapshots each, sampling t from a uniformly space time grid of step T /Nt . The test set data consist

of Nstest = 3 samples, evaluated on the same time grid employed in the training set. Then, for each N ∈ {2k , k = 0, . . . , 5} we train a POD-DL-ROM of latent dimen- sion n = 2p + 1 = 5, which is composed of: • a reduced network of 3 hidden dense layers of 1 units each; • an encoder and a decoder with 5 hidden dense layers of 2 units each.

Figure 7: 1D IBVP test case: comparison between POD-DL-ROMs and other tech-

niques in terms of number of active weights. The solid line represents the least squares fitting of the log-log data.

Figure 8: 2D IBVP test case: domain and boundary specifics (upper left), comparison

between ”true” solution (upper right) and POD-DL-ROM’s predicted solution (lower left) and visualization of the absolute error (lower right), in the case of N = 32.

m We then evaluate the lower bound M ẼP OD , the upper bound due to the error de- composition formula EN N + ES + EP OD , the value relative error ER , according to the theoretical framework of Section 3. We show both the lower bound and the upper bound results in Figure 9, displaying as well the error contributions EN N , ES , EP OD to assess the way they affect the relative error ER . We then remark again that it is crucial for POD-DL-ROMs to provide both an adequate neural network approximation of the parameter-to-solution map and

a suitably large POD dimension. Indeed, we notice that in the present test case, especially for low values of N , EN N shows a marginal contribution to the upper bound value when compared to the sampling error ES and the projection error EP OD .

Furthermore, as expected, we observe the strong dependence of the lower bound

M ẼP OD on the POD dimension, demonstrating again the importance of choosing an

adequate value for N . Finally, we assess a posteriori that the number of samples in the training set is suitable since the sampling error ES does not heavily influence the upper bound of the relative error.

Conclusions The main goal of this work is to suggest effective and practical strategies to set a

POD-DL-ROM stemming from a rigorous analysis of the technique, to control the

approximation accuracy, measured in terms of the relative error ER , which is linked to relevant features and hyperparameters that can be effectively regulated. To ac- complish the task, we analyze the error ER , providing a lower bound that depends only on the projection-based nature of the method. Then, by the error decomposition formula and the upper bound result, we highlight the contribution of sampling, POD projection and neural network approximation; in particular: (i) on the basis of the analysis of the sampling error ES we propose a family of

strategies to adopt in the data collection phase in order to ensure the conver- gence of ES → 0 in the limit of infinite data, providing also a decay estimate through Monte Carlo analysis in terms of the number of sampled snapshots

Figure 9: 2D IBVP test case: error bounds analysis varying the POD dimension N

Ndata ; (ii) we determine a practical criterion based on the eigenvalue decay to control EP OD in terms of the reduced dimension N ; (iii) starting from the approximation results proposed in , we estimate the com- plexity of the underlying neural network that is required to reach a given accu- racy.

Then, relying on the aforementioned findings, we compare the POD-DL-ROM

paradigm to other architectures that are widely used in the literature, namely DL-

ROMs [9, 12, 17], POD+DNNs [6, 21, 36], POD-DeepONets , lin+ResNets as

well as CNNs , showing the strengths of the POD-DL-ROM strategy, especially when dealing with low-regularity maps. Ultimately, we demonstrate the outstanding approximation properties of POD-DL-ROMs, which motivate the excellent perfor- mance already encountered in a variety of test cases analyzed in the recent literature [14, 15] and in the present work. Several working directions could stem from the present paper; for instance, more efficient sampling criteria arising from Monte Carlo

analysis could be implemented: we mention variance reduction techniques and Quasi

Monte Carlo methods , among others. On the other hand, one could consider

ad hoc layers to be employed in the reconstruction of parameter-to-POD-coefficients maps instead of relying purely on dense layers; however, this latter option would re- quire novel and precise approximation results for the considered layers. Moreover, an alternative formulation could split the time- and the parameter-dependence, avoiding to treat time as an additional parameter, similarly to what has been proposed in , in order to further enhance the approximation bounds proposed in this paper.

Acknowledgments

We acknowledge the support of Fondazione Cariplo, Italy, Grant n. 2019-4608, of the

PNRR-PE-AI FAIR project funded by the NextGeneration EU program, as well as of

the National Group of Scientific Computing (GNCS) of INDAM - Istituto Nazionale di

Alta Matematica. SF also acknowledges the Isaac Newton Institute for Mathematical

Sciences, Cambridge, UK, for support and hospitality during the programme “The

mathematical and statistical foundation of future data-driven engineering”, EPSRC grant no EP/R014604, where part of this work was undertaken.

Appendix A Additional proofs

A.1 Proof of Proposition 2

We notice immediately that the integral is well defined ∀v ∈ L (P × T ; RNh ) thanks to the boundedness assumptions on the solution u ∈ L (P × T ; RNh ). We also remark that the boundedness hypotheses may be relaxed: our choice was aimed at consistency with the other theoretical results of the present work. In order to prove that k · kL2w is a norm, we have to show that: (i) It satisfies the triangle inequality. Given v, z ∈ L (P × T ; RNh ), by means of the triangular inequality, it is trivial to show that

kv + zk2L2w = Z = kv(µ, t) + z(µ, t)k w(µ, t)d(µ, t) ≤ P×T Z ≤ (kv(µ, t)k + kz(µ, t)k)2 w(µ, t)d(µ, t) = P×T Z = (kv(µ, t)k + kz(µ, t)k + 2kv(µ, t)kkz(µ, t)k)w(µ, t)d(µ, t). P×T

Moreover, by the Cauchy-Schwarz inequality, the following inequality holds, Z kv(µ, t)kkz(µ, t)kw(µ, t)d(µ, t) ≤ P×T

Z Z

≤ kv(µ, t)k w(µ, t)d(µ, t) kz(µ, t)k w(µ, t)d(µ, t).

P×T P×T

Thus, we can infer

kv + zk2L2w ≤ kvk2L2w + kzk2L2w + 2kvkL2w kzkL2w = (kvkL2w + kzkL2w )2

(ii) k · kL2w is homogeneous thanks to the linearity of the integral;

(iii) If v ∈ L (P × T ; RNh ), kvkL2w = 0 implies that v = 0 a.e. by trivial arguments.

A.2 Proof of Proposition 1

Thanks to Assumption 1, trivially we obtain ∆t = T Nt−1 = O(Nt−1 ) and we set ti = i∆t. Letting f = f (µ, t) be the (sufficiently regular) integrand of the integral that we want to approximate, we obtain t XsN N ∆t|P| X Z E f (µ, t)d(µ, t) − f (µj , ti ) ≤ I + I ,

P×T Ns i=1 j=1

Z Nt Z

X I = f (µ, t)d(µ, t) − ∆t f (µ, ti )dµ = O(Nt−1 )

P×T i=1 P

Nt N

|P| X

X Z

I = ∆t E f (µ, ti )dµ − f (µj , ti )

i=1 P Ns j=1

Å Nt

X ã

= O Ns−1/2 ∆t (Var(f (µ, ti )))1/2 i=1

Å Å Z ãã

= O Ns−1/2 O(Nt−1 ) + Var(f (µ, t))dt = O(Ns1/2 ) T

Notice that Z

Var(f (µ, t)) < +∞

Z ÅZ ã1/2 ÅZ ã1/2

f (µ, t)2 dµ dt ≤ T 1/2 f (µ, t)2 d(µ, t) < +∞,

T P T ×P

since f ∈ L (P × T ). Thus, the error we commit in approximating the integral goes to zero upon requiring Ns , Nt → ∞. Finally, notice that ∆t|P| T |P| |P × T | = = ,

Ns Ndata Ndata

Nt X

|P × T | X

Z E f (µ, t)d(µ, t) − f (µj , ti ) ≤ O(Ns−1/2 + Nt−1 ).

P×T N data

A.3 Quasi-optimality of the discrete formulation of the POD

decomposition We base the following analysis on the results of the (P × T )-continuous problem proposed in . We first recall that by definition V∞ ∈ RNh ×N (where N is the

POD dimension) is optimal for the (P × T )-continuous formulation, that is with

respect to the L (P × T ; RNh ) norm. Formally, we set δ, ε > 0 and, by assuming u(µ, t) ∈ L (P × T , RNh ), we define T : L (P × T ) → RNh as Z T g := u(µ, t)g(µ, t)d(µ, t) ∀g ∈ L (P × T ). P×T

The adjoint operator of T , namely T ∗ , enjoys the property

T ∗ w = (u(µ, t), w)2 ∀w ∈ RNh .

Moreover, recall the definition of the (continuous) correlation matrix (3) and denote

by (σk,∞ , ζk ) its eigenpairs (where {ζk }k denotes an orthonormal basis). We thus define the HS-norm of T as s X kT kHS = 2 σk,∞ . k≤rank(T )

T ∗ ζk

ξk = ∀k = 1, . . . , Nh , σk,∞ we denote by TN,∞ the rank-N Schmidt approximation, with N X T TN,∞ = σk,∞ ζk (ξk (µ, t), ·)L (P×T ) = V∞ V∞ T. k=1 T and by TN = VV T its approximation by means of the discrete POD formulation.

Theorem 6.2 and Proposition 6.3 in show that the rank-N Schmidt operator and

therefore the set of basis V∞ are optimal with respect to the HS-norm, namely they retain the most variability. Formally: kTN,∞ − T kHS = min kB − T kHS

B∈BN

ÅZ ã1/2

T 2

= min ku(µ, t) − WW q(µ, t)k d(µ, t)

W∈RNh ×N :WT W=I P×T (11)

= mEP OD,∞ , where BN = {B ∈ L(L

FAQ

Typical stacks include Git, a CI engine (Jenkins/GitHub Actions/GitLab CI), Docker, Kubernetes or a cloud PaaS, infrastructure-as-code (Terraform/Ansible), and monitoring (Prometheus/Grafana). Exact tools for Devops Failover Automation Project depend on the chosen cloud and delivery model.

It can be delivered as a fully documented simulation on local/kind clusters or as a guided deployment on a cloud free tier. Both approaches are acceptable for academic evaluation when metrics, logs and diagrams are captured.

Pipeline screenshots, successful/failed run history, deployment frequency charts, rollback demos, monitoring dashboards, IaC plans/applies, and a short viva demo script are commonly included.