The Maths4DL conference on Deep Learning for Computational Physics was hosted by University College London from 4-6 July 2023. The conference was open to the public and approximately 90 people attended in total. During the conference, participants heard from keynote and invited speakers, as well as short lightning talks from early career researchers and spent time networking during breaks. Slides from the talks will be available to download from these pages (programme and lightning talk sections) from speakers who are happy to share them.
The first day was opened by Prof Simon Arridge, UCL and after several talks from speakers, a poster session and drinks reception was held. Prizes (kindly donated by World Scientific Publishing and Springer) for the best posters were awarded to Pablo Arratia, Amir Ehsan Khorashadizadeh, Agnese Pacifico, and Victor Wang. Congratulations to you all!
The second day included further talks and discussion followed by the popular conference dinner at Drake & Morgan, Kings Cross. The final day’s talks concluded with a closing speech from Prof Chris Budd, University of Bath, Principal Investigator on the Maths4DL grant.
The overall feedback has been very positive, with over 90% of respondents rating the conference and the organisation of the conference as excellent or very good. Participants included PhD students, early career researchers, academics, plus some representatives from industry and truly represented the international nature of the work being carried out in the Maths4DL programme. Attendees came from towns and cities in Canada, England, Estonia, Finland, Germany, Italy, Japan, Netherlands, Norway, Saudi Arabia, Scotland, Sweden, Switzerland, USA and Wales.
We are already looking forward to our next conference, so watch this space!
Deep learning in physics represents a very active and rapidly growing field of research. This shift in approach has already brought with it many advances, which this meeting aims to highlight. Recent examples include PINNs, SINDy, symbolic regression, Fourier neural operators, meta-learning, and neural ODEs to name a few. The applications also embrace many disciplines across the scientific spectrum, from medical sciences, to computer vision, to the physical sciences. We believe that the next steps for machine learning require a firm theoretical understanding and have organised this conference, taking place at UCL in central London, to bring together like-minded individuals to discuss current and future research in this area.
Places at the conference are now full but we are looking at ways to increase capacity. If you are still interested in registering, please email email@example.com and we will keep you informed of our plans.
Please note, there are no longer places available at the conference dinner.
|Tuesday 4th July||Wednesday 5th July||Thursday 6th July|
|Arrival, registration, and refreshments|
|Welcome and introduction – Prof. Simon Arridge||Arrival, registration, and refreshments||Arrival, registration, and refreshments|
|Asst. Prof. Sophie Langer
Understanding dropout in the linear world
|Prof. Giovanni S. Alberti
Machine learning for infinite-dimensional inverse problems
|Prof. Elena Celledoni
A dynamical systems view to Deep learning: contractivity and structure preservation.
|Coffee break||Coffee break||Dr Marta Betcke
Complementary learning in photoacoustic tomography
|Prof. Jason McEwen
Geometric deep learning on the sphere for the physical sciences
|Dr Julián Tachella
Learning to reconstruct images without ground-truth
|Prof Andreas Hauptmann
Model-corrected learned primal-dual models for fast limited-view photoacoustic tomography
|Dr Benjamin Moseley
Scaling physics-informed neural networks to high frequency and multiscale problems using domain decomposition
|Dr Cristopher Salvi
Large-width limits of neural ODE-type methods
|Dr Steve Brunton
Machine Learning for Scientific Discovery, with Examples in Fluid Mechanics
|Prof Eldad Haber
PDE’s, ODE’s Graphs and Neural Networks
|Dr Chris Rackauckas
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond Continuous Models
|Dr Zhi Zhou
Identification of Conductivity in Elliptic equations using Deep Neural Networks
|Dr Nicolas Boulle
Data-efficient PDE learning
Acceleration of multiscale solvers via adjoint operator learning
|Coffee break||Coffee break||Coffee break|
|Lightning talks||Janek Gödeke
TorchPhysics: A Deep Learning Library for Solving Differential Equations
|Dr Patrick Kidger
Scientific machine learning in JAX
|Dr Tatiana Bubba
Integrating data-driven techniques and theoretical guarantees for limited angle tomography
|Prof. Michael Hintermüller
Learning-informed and PINN-based multi scale PDE models in optimization
|16:30-18:00, Poster session and drinks reception||16:30-16:40, Summary and close, Prof. Chris Budd|
|18:30-late, Conference dinner at Drake & Morgan, Kings Cross.|
University of Genoa
In recent years, machine learning techniques have become very popular to solve inverse problems. Inverse problems are ubiquitous in science and engineering, and appear whenever a physical quantity has to be reconstructed from indirect measurements. Inverse problems are typically ill-posed, meaning that small errors in the measurements may have large effects in the reconstruction. Classically, regularisation is used to overcome ill-posedness, but prior knowledge on the unknown is needed for the choice of regularisation. Machine learning comes into play as a way to make regularisation, and more generally the inversion process, data-driven, thanks to the use of a training set.
Many inverse problems are modelled by integral or differential operators and, consequently, are intrinsically infinite-dimensional. In this talk, I will discuss how machine learning can be rigorously used in infinite-dimensional inverse problems, in the context of learning the optimal regulariser or by using generative models in function spaces.
This is based on joint works with E. De Vito, T. Helin, J. Hertrich, M. Lassas, L. Ratti, M. Santacesaria and S. Sciutto.
University of Washington
This work describes how machine learning may be used to develop accurate and efficient nonlinear dynamical systems models for complex natural and engineered systems. We explore the sparse identification of nonlinear dynamics (SINDy) algorithm, which identifies a minimal dynamical system model that balances model complexity with accuracy, avoiding overfitting. This approach tends to promote models that are interpretable and generalizable, capturing the essential “physics” of the system.
We also discuss the importance of learning effective coordinate systems in which the dynamics may be expected to be sparse. This sparse modeling approach will be demonstrated on a range of challenging modeling problems in fluid dynamics, and we will discuss how to incorporate these models into existing model-based control efforts. Because fluid dynamics is central to transportation, health, and defense systems, we will emphasize the importance of machine learning solutions that are interpretable, explainable, generalizable, and that respect known physics.
Norwegian University of Science and Technology (NTNU)
The (discrete) optimal control point of view to neural networks offers an interpretation of deep learning from a dynamical systems and numerical analysis perspective and opens the way to mathematical insight. In this talk we discuss topics of structure preservation for supervised and unsupervised deep learning. Some deep neural networks can be designed to have desirable properties such as invertibility and group equivariance or can be adapted to problems of manifold value data. Contractivity is identified as a desirable property for stability and robustness of neural networks. We discuss classical results of contractivity of numerical ODE integrators, applications to neural networks and recent extensions to Riemannian manifolds.
University of British Columbia
In this work we will review the connection between P/ODE’s and Neural networks. We start by a discussion of how Deep Networks can be thought of as a discretization of time dependent O/PDE’s and extend this to graphs. This will motivate a discussion on new architectures that enhance our understanding on the behaviour of Deep Networks and can break the State Of The Art for many different problems.
University of Twente
Dropout has emerged as an algorithmic regularization technique that randomly drops neurons during training, exhibiting effectiveness across various applications. However, despite its empirical success, a comprehensive theoretical understanding of how dropout achieves regularization is still somewhat limited.
In the case of a linear model, it was shown that under an averaged form of dropout the least squares minimizer performs a weighted variant of l2-penalization. In turn, the heuristic “dropout performs l2-penalization” has even made it in popular textbooks. We challenge this relation by investigating the statistical behavior of iterates generated by gradient descent with dropout. In particular, non-asymptotic convergence rates for the expectation and covariance matrices of the iterates are derived. While in expectation the connection between dropout and l2-penalization can be verified, we show sub-optimality of the asymptotic variance compared to the estimator resulting from direct minimization of averaged dropout. As an illustrative example, we also discuss a simplified variant of dropout, which features much simpler interactions.
Massachusetts Institute of Technology (MIT)
The combination of scientific models into deep learning structures, commonly referred to as scientific machine learning (SciML), has made great strides in the last few years in incorporating models such as ODEs and PDEs into deep learning through differentiable simulation. However, the vast space of scientific simulation also includes models like jump diffusions, agent-based models, and more. Is SciML constrained to the simple continuous cases or is there a way to generalize to more advanced model forms? This talk will dive into the mathematical aspects of generalizing differentiable simulation to discuss cases like chaotic simulations, differentiating stochastic simulations like particle filters and agent-based models, and solving inverse problems of Bayesian inverse problems (i.e. differentiation of Markov Chain Monte Carlo methods). We will then discuss the evolving numerical stability issues, implementation issues, and other interesting mathematical tidbits that are coming to light as these differentiable programming capabilities are being adopted.
University College London
Joint work with Bolin Pan
In photoacoustic tomography (PAT) with flat sensor, we routinely encounter two types of limited data. The first is due to using a finite sensor and is especially perceptible if the region of interest is large relatively to the sensor or located farther away from the sensor. In this talk we focus on the second type caused by a varying sensitivity of the sensor to the incoming wavefront direction which can be modelled as binary i.e. by a cone of sensitivity. Such visibility conditions result, in Fourier domain, in a restriction of the data to a bowtie, akin to the one corresponding to the range of the forward operator but further narrowed according to the angle of sensitivity. Via the wavefront mapping we obtain the corresponding visible and invisible wavefront directions in the PAT image. We adapt the wedge restricted Curvelet decomposition, we previously proposed for the representation of the full PAT data, to separate the visible and invisible wavefronts in the data as well as in the image. We optimally combine fast approximate operators with tailored deep neural network architectures into efficient learnt reconstruction methods which perform sparse reconstruction of the visible coefficients and the invisible coefficients are learnt from a training set of similar data.
University of Cambridge
PDE learning is an emerging field at the intersection of machine learning, physics, and mathematics, that aims to discover properties of unknown physical systems from experimental data. Popular techniques exploit the approximation power of deep learning to learn solution operators, which map source terms to solutions of the underlying PDE. Solution operators can then produce surrogate data for data-intensive machine learning approaches such as learning reduced order models for design optimization in engineering and PDE recovery. In most deep learning applications, a large amount of training data is needed, which is often unrealistic in engineering and biology. However, PDE learning is shockingly data-efficient in practice. We provide a theoretical explanation for this behaviour by constructing an algorithm that recovers solution operators associated with elliptic PDEs and achieves an exponential convergence rate with respect to the size of the training dataset. The proof technique combines prior knowledge of PDE theory and randomized numerical linear algebra techniques and may lead to practical benefits such as improving dataset and neural network architecture designs.
University of Bath
Limited angle CT is a challenging testing ground, where both variational regularisation and data-driven techniques have been investigated extensively in the last years. In this talk, I will present a hybrid reconstruction framework where the proximal operator of an accelerated unrolled scheme is learned to ensure suitable theoretical guarantees. The recipe relays on the interplay between sparse regularization theory, harmonic analysis, microlocal analysis and Plug and Play methods.
The numerical results show that these approaches significantly surpasses both pure model- and more data-based reconstruction methods.
University of Bremen
Recently, lots of research on finding suitable Deep Learning strategies for solving differential equations has been done, having led to many promising methods. Some examples are Physics Informed Neural Networks (PINNs), the Deep Ritz Method or operator-learning methods like (Physics-Informed) Deep Operator Networks (DeepONets).
In this talk the TorchPhysics library will be presented which has been developed at the University of Bremen in cooperation with the Robert Bosch GmbH. It provides a user-friendly framework for implementing several Deep Learning methods – like those mentioned above – for solving differential equations, but also parameter-identification problems can be tackled.
Among TorchPhysics‘ main features is the simplicity of translating differential equations into readable code. In combination with a clean documentation and detailed tutorials, this enables a quick start for the user. Furthermore, complex or time-dependent domains can easily be realized and sampling within these domains can be done in various ways.
After a broad overview about the functionalities of TorchPhysics, we will show an illustrative example and finally draw a comparison to other existing libraries.
University of Oulu
Learned iterative reconstructions hold great promise to accelerate tomographic imaging with empirical robustness to model perturbations. Nevertheless, an adoption for photoacoustic tomography is hindered by the need to repeatedly evaluate the computational expensive forward model. Computational feasibility can be obtained by the use of fast approximate models, but a need to compensate model errors arises.
In this work we discuss a methodological and theoretical basis for model corrections in learned image reconstructions by embedding the model correction in a learned primal-dual framework. The proposed formulation allows an extension to a primal-dual deep equilibrium model providing fixed-point convergence as well as reduced memory requirements for training.
Weierstrass Institute and Humboldt-Universität zu Berlin
The talk focuses on two distinct aspects of artificial neural network (NN) based optimization problems: (i) Learning-informed (hybrid) PDE models as constraints in optimal control and inverse problems are discussed in view of approximation quality, first-order conditions and solution algorithms. In this context, learning a time-discrete version of the solution map of the Bloch equations in quantitative magnetic resonance imaging serves as an application example. Further, particular attention is given to ReLU-based NN-models concerning the proper handling and consequences of the induced non-smoothness in both, theory and algorithms.
The second part of the talk highlights a PINN-type homogenization technique in multi scale material modeling which leads to minimization task reminiscent of PDE-constrained optimization.
We’re building an open-source ecosystem for scientific computing and machine learning in JAX! Differential equation solvers, neural networks, root finding, optimisation, etc.
Many scientific problems require computational modelling, and these can now take advantage of the ubiquitous autodifferentiation, autoparallelism, and GPU/TPU acceleration, offered by modern computational frameworks. (Going beyond that offered by older tools like SciPy, MATLAB, or Julia.) My talk will offer an overview of this work, and which I hope will provide new tools for you to solve the problems you are tackling.
Mullard Space Science Laboratory, UCL
Many problems in the physical sciences, and beyond, require the analysis of spherical data. For example, the cosmic microwave background (CMB) relic radiation from the Big Bang is observed on the celestial sphere. To leverage the potential of deep learning in these areas, geometric deep learning techniques defined natively on the sphere are required. I will review our recent developments in spherical CNNs and spherical scattering networks that encode rotational equivariance accurately, while for the first time also providing highly scalable computation. These developments rely on new differentiable and accelerated Fourier transforms on both the sphere and rotation group, which will also be presented.
Finally, I will conclude with an example of a generative model built on these components to emulate anisotropies induced in the CMB by cosmic strings — line-like discontinuities in the fabric of the Universe. Our spherical cosmic string emulation model reduces the computational time required to generate string-induced CMB signatures from hundreds of thousands of CPU hours on a supercomputer, to less than an hour on a laptop, opening up many new types of analyses to search for evidence of cosmic strings.
ETH Zürich AI Center
Physics-informed neural networks (PINNs) are a powerful approach for solving problems related to differential equations. However, their performance often declines rapidly when solving problems with high frequency and/or multi-scale solutions. This is typically due to the spectral bias of neural networks, as well as the fact that, as the problem complexity grows, significantly more free parameters and collocation points are required to model the solution, which leads to an increasingly complex optimisation problem.
In this talk, we will present finite basis physics-informed neural networks (FBPINNs), which attempt to improve the performance of PINNs in this regime by combining them with domain decomposition. We show that FBPINNs can significantly outperform PINNs when solving problems with high frequency and multi-scale solutions, whilst being an order of magnitude faster to train. Furthermore, we present multilevel FBPINNs, which extend FBPINNs by incorporating ideas from classical multilevel domain decomposition methods and show that they can improve the accuracy of FBPINNs further. Finally, we present the limitations and future directions of our work.
Imperial College London
Motivated by the paradigm of reservoir computing, I will consider randomly initialized neural controlled differential equations and show that in the infinite-width limit and under proper rescaling of the vector fields, these neural architectures converge weakly to Gaussian processes indexed on path-space and with covariances satisfying certain PDEs varying according to the choice of activation function. In the special case where the activation function is the identity, the equation reduces to a linear PDE and the limiting kernel agrees with the original signature kernel.
We leverage recent advances in operator learning to accelerate multiscale solvers for laminar fluid flow over a rough boundary. We focus on the HMM method, which involves formulating the problem through a coupled system of microscopic and macroscopic subproblems. We combine adjoint calculus and classical boundary integral methods for Stokes flow to formulate the microscopic problems as a nonlinear operator mapping from the space of micro domains to functions on their boundary. Our main contribution is to use an FNO-type architecture to learn this mapping.
French National Centre for Scientific Research CNRS
Most computational imaging algorithms rely either on hand-crafted prior models (total variation, wavelets) or on supervised learning (deep neural networks) with a ground truth dataset of references. The first approach generally obtains suboptimal reconstructions, whereas the latter is impractical in many scientific and medical imaging applications, where ground-truth data is expensive or even impossible to obtain. In this talk, I will present recent algorithmic and theoretical advances in unsupervised learning for imaging inverse problems that overcome these limitations, by learning from noisy and incomplete measurement data alone. I will show how weak prior knowledge on the reconstructed image distribution, such as invariance to groups of transformations (rotations, translations, etc.) and low-dimensionality, play a key role in learning from measurement data alone.
Hong Kong Polytechnic University
The focus of this talk is on the numerical methods used to identify the conductivity in an elliptic equation. Commonly, a regularized formulation consists of a data fidelity and a regularizer is employed, and then it is discretized using finite difference method, finite element methods or deep neural networks. One key issue is to establish a priori error estimates for the recovered conductivity distribution. In this talk, we discuss our recent findings on using deep neural networks for this class of problems, by effectively utilizing relevant stability estimates.
The conference dinner will take place on Wednesday 5 July. The venue is Drake and Morgan, King Cross. You need to sign up and pay to attend the dinner when you register. You will be contacted nearer the time regarding your menu choices.
UCL is located in the Bloomsbury district at the very centre of London. There are easy connections to UCL from London’s global hub airports at Heathrow, Gatwick and Stansted and you will find that London’s extensive public transport system is convenient and easy to use.
London has a wide variety of accommodation to suit all tastes and budgets. There are lots of options in and around Bloomsbury, close to the conference.
Accommodation is not included in the registration fee, delegates are required to book their own accommodation. We encourage delegates to book accommodation as early as possible.
Posters were presented by the following people. Click on their name for their slides and poster title for the poster pdf (where available).
|Jonathan Chirinos Rodriguez||A Supervised Learning Approach to Regularization of Inverse Problems|
|Alexander Denker||Invertible residual networks in the context of regularization theory for linear inverse problems|
|Simon Driscoll||Sensitivity Analysis and Machine Learning of Sea Ice Thermodynamics|
|Amir Ehsan Khorashadizadeh||FunkNN: Neural Interpolation for Functional Generation|
|Takashi Matsubara||Deep Learning for Discrete-Time Physics|
|Derick Nganyu Tanyu||Deep Learning Methods for PDEs and Related Parameter Identification Problems|
|Agnese Pacifico||Online identification and control of PDEs via Reinforcement Learning methods|
|Danilo Riccio||Regularization of Inverse Problems: Deep Equilibrium Models versus Bilevel Learning|
|Ivan Sudakow||Statistical mechanics in climate emulation: Challenges and perspectives|
|Xiaoyu (Victor) Wang||Lifted Bregman Training of Neural Networks|
|Takaharu Yaguchi||Neural symplectic form and its variational principle|