In recent years, machine learning techniques have become very popular to solve inverse problems. Inverse problems are ubiquitous in science and engineering, and appear whenever a physical quantity has to be reconstructed from indirect measurements. Inverse problems are typically ill-posed, meaning that small errors in the measurements may have large effects in the reconstruction. Classically, regularisation is used to overcome ill-posedness, but prior knowledge on the unknown is needed for the choice of regularisation. Machine learning comes into play as a way to make regularisation, and more generally the inversion process, data-driven, thanks to the use of a training set. 

Many inverse problems are modelled by integral or differential operators and, consequently, are intrinsically infinite-dimensional. In this talk, I will discuss how machine learning can be rigorously used in infinite-dimensional inverse problems, in the context of learning the optimal regulariser or by using generative models in function spaces.

This is based on joint works with E. De Vito, T. Helin, J. Hertrich, M. Lassas, L. Ratti, M. Santacesaria and S. Sciutto.

This work describes how machine learning may be used to develop accurate and efficient nonlinear dynamical systems models for complex natural and engineered systems.  We explore the sparse identification of nonlinear dynamics (SINDy) algorithm, which identifies a minimal dynamical system model that balances model complexity with accuracy, avoiding overfitting.  This approach tends to promote models that are interpretable and generalizable, capturing the essential “physics” of the system. 

We also discuss the importance of learning effective coordinate systems in which the dynamics may be expected to be sparse.  This sparse modeling approach will be demonstrated on a range of challenging modeling problems in fluid dynamics, and we will discuss how to incorporate these models into existing model-based control efforts.  Because fluid dynamics is central to transportation, health, and defense systems, we will emphasize the importance of machine learning solutions that are interpretable, explainable, generalizable, and that respect known physics.

The (discrete) optimal control point of view to neural networks offers an interpretation of deep learning from a dynamical systems and numerical analysis perspective and opens the way to mathematical insight. In this talk we discuss topics of structure preservation for supervised and unsupervised deep learning. Some deep neural networks  can be designed to have desirable properties such as invertibility and group equivariance or can be adapted to problems of manifold value data. Contractivity is identified as a desirable property for stability and robustness of neural networks.  We discuss classical results of contractivity of numerical ODE integrators, applications to  neural networks and recent extensions to Riemannian manifolds.

In this work we will review the connection between P/ODE’s and Neural networks. We start by a discussion of how Deep Networks can be thought of as a discretization of time dependent O/PDE’s and extend this to graphs.  This will motivate a discussion on new architectures that enhance our understanding on the behaviour of Deep Networks and can break the State Of The Art for many different problems.

Dropout has emerged as an algorithmic regularization technique that randomly drops neurons during training, exhibiting effectiveness across various applications. However, despite its empirical success, a comprehensive theoretical understanding of how dropout achieves regularization is still somewhat limited.

In the case of a linear model, it was shown that under an averaged form of dropout the least squares minimizer performs a weighted variant of l2-penalization. In turn, the heuristic “dropout performs l2-penalization” has even made it in popular textbooks. We challenge this relation by investigating the statistical behavior of iterates generated by gradient descent with dropout. In particular, non-asymptotic convergence rates for the expectation and covariance matrices of the iterates are derived. While in expectation the connection between dropout and l2-penalization can be verified, we show sub-optimality of the asymptotic variance compared to the estimator resulting from direct minimization of averaged dropout. As an illustrative example, we also discuss a simplified variant of dropout, which features much simpler interactions.

The combination of scientific models into deep learning structures, commonly referred to as scientific machine learning (SciML), has made great strides in the last few years in incorporating models such as ODEs and PDEs into deep learning through differentiable simulation. However, the vast space of scientific simulation also includes models like jump diffusions, agent-based models, and more. Is SciML constrained to the simple continuous cases or is there a way to generalize to more advanced model forms? This talk will dive into the mathematical aspects of generalizing differentiable simulation to discuss cases like chaotic simulations, differentiating stochastic simulations like particle filters and agent-based models, and solving inverse problems of Bayesian inverse problems (i.e. differentiation of Markov Chain Monte Carlo methods). We will then discuss the evolving numerical stability issues, implementation issues, and other interesting mathematical tidbits that are coming to light as these differentiable programming capabilities are being adopted. 

Joint work with Bolin Pan

In photoacoustic tomography (PAT) with flat sensor, we routinely encounter two types of limited data. The first is due to using a finite sensor and is especially perceptible if the region of interest is large relatively to the sensor or located farther away from the sensor. In this talk we focus on the second type caused by a varying sensitivity of the sensor to the incoming wavefront direction which can be modelled as binary i.e. by a cone of sensitivity. Such visibility conditions result, in Fourier domain, in a restriction of the data to a bowtie, akin to the one corresponding to the range of the forward operator but further narrowed according to the angle of sensitivity. Via the wavefront mapping we obtain the corresponding visible and invisible wavefront directions in the PAT image. We adapt the wedge restricted Curvelet decomposition, we previously proposed for the representation of the full PAT data, to separate the visible and invisible wavefronts in the data as well as in the image. We optimally combine fast approximate operators with tailored deep neural network architectures into efficient learnt reconstruction methods which perform sparse reconstruction of the visible coefficients and the invisible coefficients are learnt from a training set of similar data.

PDE learning is an emerging field at the intersection of machine learning, physics, and mathematics, that aims to discover properties of unknown physical systems from experimental data. Popular techniques exploit the approximation power of deep learning to learn solution operators, which map source terms to solutions of the underlying PDE. Solution operators can then produce surrogate data for data-intensive machine learning approaches such as learning reduced order models for design optimization in engineering and PDE recovery. In most deep learning applications, a large amount of training data is needed, which is often unrealistic in engineering and biology. However, PDE learning is shockingly data-efficient in practice. We provide a theoretical explanation for this behaviour by constructing an algorithm that recovers solution operators associated with elliptic PDEs and achieves an exponential convergence rate with respect to the size of the training dataset. The proof technique combines prior knowledge of PDE theory and randomized numerical linear algebra techniques and may lead to practical benefits such as improving dataset and neural network architecture designs.

Limited angle CT is a challenging testing ground, where both variational regularisation and data-driven techniques have been investigated extensively in the last years. In this talk, I will present a hybrid reconstruction framework where the proximal operator of an accelerated unrolled scheme is learned to ensure suitable theoretical guarantees. The recipe relays on the interplay between sparse regularization theory, harmonic analysis, microlocal analysis and Plug and Play methods. 

The numerical results show that these approaches significantly surpasses both pure model- and more data-based reconstruction methods.

Recently, lots of research on finding suitable Deep Learning strategies for solving differential equations has been done, having led to many promising methods. Some examples are Physics Informed Neural Networks (PINNs), the Deep Ritz Method or operator-learning methods like (Physics-Informed) Deep Operator Networks (DeepONets).

In this talk the TorchPhysics library will be presented which has been developed at the University of Bremen in cooperation with the Robert Bosch GmbH. It provides a user-friendly framework for implementing several Deep Learning methods – like those mentioned above – for solving differential equations, but also parameter-identification problems can be tackled.

Among TorchPhysics‘ main features is the simplicity of translating differential equations into readable code. In combination with a clean documentation and detailed tutorials, this enables a quick start for the user. Furthermore, complex or time-dependent domains can easily be realized and sampling within these domains can be done in various ways.

After a broad overview about the functionalities of TorchPhysics, we will show an illustrative example and finally draw a comparison to other existing libraries.

Learned iterative reconstructions hold great promise to accelerate tomographic imaging with empirical robustness to model perturbations. Nevertheless, an adoption for photoacoustic tomography is hindered by the need to repeatedly evaluate the computational expensive forward model. Computational feasibility can be obtained by the use of fast approximate models, but a need to compensate model errors arises.

In this work we discuss a methodological and theoretical basis for model corrections in learned image reconstructions by embedding the model correction in a learned primal-dual framework. The proposed formulation allows an extension to a primal-dual deep equilibrium model providing fixed-point convergence as well as reduced memory requirements for training.

The talk focuses on two distinct aspects of artificial neural network (NN) based optimization problems: (i) Learning-informed (hybrid) PDE models as constraints in optimal control and inverse problems are discussed in view of approximation quality, first-order conditions and solution algorithms. In this context, learning a time-discrete version of the solution map of the Bloch equations in quantitative magnetic resonance imaging serves as an application example. Further, particular attention is given to ReLU-based NN-models concerning the proper handling and consequences of the induced non-smoothness in both, theory and algorithms.

The second part of the talk highlights a PINN-type homogenization technique in multi scale material modeling which leads to minimization task reminiscent of PDE-constrained optimization.

We’re building an open-source ecosystem for scientific computing and machine learning in JAX! Differential equation solvers, neural networks, root finding, optimisation, etc.

Many scientific problems require computational modelling, and these can now take advantage of the ubiquitous autodifferentiation, autoparallelism, and GPU/TPU acceleration, offered by modern computational frameworks. (Going beyond that offered by older tools like SciPy, MATLAB, or Julia.) My talk will offer an overview of this work, and which I hope will provide new tools for you to solve the problems you are tackling.

Many problems in the physical sciences, and beyond, require the analysis of spherical data.  For example, the cosmic microwave background (CMB) relic radiation from the Big Bang is observed on the celestial sphere.  To leverage the potential of deep learning in these areas, geometric deep learning techniques defined natively on the sphere are required.  I will review our recent developments in spherical CNNs and spherical scattering networks that encode rotational equivariance accurately, while for the first time also providing highly scalable computation.  These developments rely on new differentiable and accelerated Fourier transforms on both the sphere and rotation group, which will also be presented. 

Finally, I will conclude with an example of a generative model built on these components to emulate anisotropies induced in the CMB by cosmic strings — line-like discontinuities in the fabric of the Universe.  Our spherical cosmic string emulation model reduces the computational time required to generate string-induced CMB signatures from hundreds of thousands of CPU hours on a supercomputer, to less than an hour on a laptop, opening up many new types of analyses to search for evidence of cosmic strings.

Physics-informed neural networks (PINNs) are a powerful approach for solving problems related to differential equations. However, their performance often declines rapidly when solving problems with high frequency and/or multi-scale solutions. This is typically due to the spectral bias of neural networks, as well as the fact that, as the problem complexity grows, significantly more free parameters and collocation points are required to model the solution, which leads to an increasingly complex optimisation problem.

In this talk, we will present finite basis physics-informed neural networks (FBPINNs), which attempt to improve the performance of PINNs in this regime by combining them with domain decomposition. We show that FBPINNs can significantly outperform PINNs when solving problems with high frequency and multi-scale solutions, whilst being an order of magnitude faster to train. Furthermore, we present multilevel FBPINNs, which extend FBPINNs by incorporating ideas from classical multilevel domain decomposition methods and show that they can improve the accuracy of FBPINNs further. Finally, we present the limitations and future directions of our work.

Motivated by the paradigm of reservoir computing, I will consider randomly initialized neural controlled differential equations and show that in the infinite-width limit and under proper rescaling of the vector fields, these neural architectures converge weakly to Gaussian processes indexed on path-space and with covariances satisfying certain PDEs varying according to the choice of activation function. In the special case where the activation function is the identity, the equation reduces to a linear PDE and the limiting kernel agrees with the original signature kernel.

We leverage recent advances in operator learning to accelerate multiscale solvers for laminar fluid flow over a rough boundary. We focus on the HMM method, which involves formulating the problem through a coupled system of microscopic and macroscopic subproblems. We combine adjoint calculus and classical boundary integral methods for Stokes flow to formulate the microscopic problems as a nonlinear operator mapping from the space of micro domains to functions on their boundary. Our main contribution is to use an FNO-type architecture to learn this mapping.

Most computational imaging algorithms rely either on hand-crafted prior models (total variation, wavelets) or on supervised learning (deep neural networks) with a ground truth dataset of references. The first approach generally obtains suboptimal reconstructions, whereas the latter is impractical in many scientific and medical imaging applications, where ground-truth data is expensive or even impossible to obtain. In this talk, I will present recent algorithmic and theoretical advances in unsupervised learning for imaging inverse problems that overcome these limitations, by learning from noisy and incomplete measurement data alone. I will show how weak prior knowledge on the reconstructed image distribution, such as invariance to groups of transformations (rotations, translations, etc.) and low-dimensionality, play a key role in learning from measurement data alone.

The focus of this talk is on the numerical methods used to identify the conductivity in an elliptic equation. Commonly, a regularized formulation consists of a data fidelity and a regularizer is employed, and then it is discretized using finite difference method, finite element methods or deep neural networks. One key issue is to establish a priori error estimates for the recovered conductivity distribution. In this talk, we discuss our recent findings on using deep neural networks for this class of problems, by effectively utilizing relevant stability estimates.