The identification of models of dynamical systems from observational data is an important task in deep learning. Taking on the one hand physical and geometric properties of the dynamical systems (such as symmetries or variational structure) in the learning process into account (“physics-informed learning”), improves qualitative aspects of data-driven models leading to more reliable and accurate predictions. On the other hand, the choice of network architectures in deep learning influences the numerical stability and efficiency of the learning process. In this talk we will show how symplectic methods can be exploited for first, learning more reliable and robust models for physical systems based on variational principles in particularly for longterm simulations, and second, to construct network architectures with specific theoretical guarantees.

High dimensional data often consist of parts with different intrinsic dimension. We study how spectral methods on graphs adapt to data containing intersecting pieces of different dimensions. We show that unnormalized Laplacian on random geometric (and similar) graphs  only sees the variations in the highest dimension, while appropriately normalized Laplacian converges to Laplace-Beltrami operator in all dimensions simultaneously. Somewhat surprisingly, we show that graph Laplacians based on kNN graphs only see the variations in the lowest dimension. We thus suggest new normalizations that allow for variations in all dimensions to be captured. For intersecting manifolds we identify when and how is the information transferred between manifolds.

The dynamics of information diffusion in graph message passing is a key issue that heavily influences graph representation learning, especially when long-range propagation is needed. We vouch for principled approaches that control and regulate the degree of propagation, conservation and dissipation of information throughout the neural flow. In the talk we explore some approaches stemming from a dynamical systems’ view of graph message passing, with particular focus on wide applicability of the concepts and on the theoretical guarantees on information conservation.

Based on the theory of homogeneous spaces we derive geometrically optimal edge attributes to be used within the flexible message-passing framework. We formalize the notion of weight sharing in convolutional networks as the sharing of message functions over point-pairs that should be treated equally. We define equivalence classes of point-pairs that are identical up to a transformation in the group and derive attributes that uniquely identify these classes. Weight sharing is then obtained by conditioning message functions on these attributes. As an application of the theory, we develop an efficient equivariant group convolutional network for processing 3D point clouds. The theory of homogeneous spaces tells us how to do group convolutions with feature maps over the homogeneous space of positions 

$\mathbb{R}^n$, position and orientations $\mathbb{R}^n \times S^{n-1}$, and the group $SE(n)$ itself. Among these, $\mathbb{R}^n \times S^{n=1}$ is an optimal choice due to the ability to represent directional information, which $\mathbb{R}^n$ methods cannot, and it significantly enhances computational efficiency compared to indexing features on the full $SE(n)$ group. We support this claim with state-of-the-art results —in accuracy and speed— on five different benchmarks in 2D and 3D, including interatomic potential energy prediction, trajectory forecasting in N-body systems, and generating molecules via equivariant diffusion models.

Today modern machine learning offers state-of-the art solutions for most computer vision and computational imaging tasks, exploiting the sophisticated statistical dependencies within images. However, such solutions require access to a large quantity of ground truth images for training, such as in the latest diffusion models. While these create very powerful models it suggests that we need to have “solved” the imaging problem somehow beforehand in order to have access the ground truth. This has led to a big drive to try to train such systems without ground truth data.  In the talk, I will discuss a new geometric framework for learning solutions to such ill-posed inverse problems using only access to the incomplete measurement data for training, presenting initial results for necessary and sufficient conditions for model identification, in particular highlighting the role of symmetry in the signal model. I will also discuss a new class of self-supervised learning, called Equivariant Imaging based on this theory, and demonstrate its performance on various imaging problems.

Deep learning has emerged as a dominant approach in machine learning and has achieved remarkable success in various domains such as computer vision and natural language processing. Its influence has progressively extended to numerous research areas within the fields of science and engineering. In this presentation, I will outline our work on the design and training of a foundation model, named PDEformer, which aims to serve as a flexible and efficient solver across a spectrum of parametric PDEs. PDEformer is specifically engineered to facilitate a range of downstream tasks, including but not limited to parameter estimation and system identification. Its design is tailored to accommodate applications necessitating repetitive solving of PDEs, where a balance between efficiency and accuracy is sought.

Neural networks, particularly LLMs, are notoriously poor at algorithmic tasks, such as sorting,
shortest path, and even basic arithmetic. Across three papers, we explored the problem of
“aligning” architectures to classical computer programs, and showed that this question relates
to well-established mathematical concepts: polynomial functors, cohomology, and higher categories.

We consider PDE-based Group Convolutional Neural Networks (PDE-G-CNNs) that generalize Group equivariant Convolutional Neural Networks (G-CNNs). In PDE-G-CNNs a network layer is a set of PDE-solvers. 

The underlying (non)linear PDEs are defined on the homogeneous space M(d) of positions and orientations within the roto-translation group SE(d) and provide a geometric design of the roto-translation equivariant neural network. 

The network consists of morphological convolutions with (approximative) kernels solving nonlinear PDEs (HJB equations for max-pooling over Riemannian balls), and linear convolutions solving linear PDEs (convection, fractional diffusion).

Our analytic approximation kernels are accurate in comparison to our recent exact PDE-kernels.  Common mystifying (ReLU) nonlinearities are now obsolete and excluded. 

We achieve network interpretability as we train sparse association fields (modeling contour perception in our own visual system).

We present several medical imaging applications (tracking and segmentation of complex vasculature) and industrial image analysis applications

where each time we show benefits of PDE-G-CNNs compared to state-of-the-art G-CNNs: increase of performance along with a large reduction in network parameters and less training data. 

When solving discrete problems with deep learning, making the problem differentiable is one of the core challenges. This talk will look at different problems sets with complex discrete aspects, for example permutation, and how a change in representation can make these problems feasible for optimisation with gradient descent.

Natural policy gradient methods are among the most popular policy optimization techniques in modern reinforcement learning. We show that several of these methods can interpreted via gradient flows of linear programs with respect to Hessian Riemannian geometries or equivalently the solutions of regularized linear programs. This motivates us to study Fisher-Rao gradient flows of linear programs where we prove linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. Leveraging the equivalence of Hessian gradient flows of linear programs and the solutions of regularized linear programs we give essentially sharp estimates on the entropy regularization error in Markov decision processes and describe the implicit bias of several natural policy gradient methods. 

In this talk, we explore how the imposition of different types of invariances within neural networks forces them to learn domain-transferable, invariant patterns that significantly bolster zero-shot domain and out-of-distribution (OOD) generalizations. We will start discussing the difference between sequences, sets, and graphs. Building on these foundations, I will introduce a novel graph invariance, specifically designed to improve the abstract reasoning capabilities of GNNs for zero-shot domain generalization in knowledge graphs. Then, we extend the exploration of this invariance-centric design principle to a diverse array of OOD generalization scenarios, ranging from causal link prediction and computer networking to emerging frontiers in Physics-Informed Machine Learning. This talk aims to discuss the transformative potential of causality and invariance in improving robustness and domain transferability in neural networks.

The human cerebral cortex is a convoluted sheet whose shape and patterns of functional organisation vary dramatically across individuals making it challenging to detect subtle signatures of cognition and disease. In this talk I will discuss how we can use geometric deep learning to better encode this variation within individual brains – in order to map cortical organisation and model the spatio-temporal dynamics of human brain activity.

Recent engineering achievements of deep-learning machines have both impressed and intrigued the scientific community due to our limited theoretical understanding of the underlying reasons for their success. I will briefly review some of the challenges to be addressed and then focus on properties of the function space of different types of deep-learning machines, based on the generating functional analysis. This approach facilitates studying the number of solution networks of a given error around a reference multi-layer network. Exploring the function landscape of densely-connected networks, we uncover a general layer-by-layer learning behaviour, while the study of sparsely-connected networks indicates the advantage in having more layers for increasing generalization ability in such models. This framework accommodates other network architectures and computing elements, including networks with correlated weights, convolutional networks and discretised variables. A similar approach also facilitates studying the distribution of Boolean functions computed by recurrent and layer-dependent architectures, which we find to be the same. Depending on the initial conditions and computing elements used, we characterize the space of functions computed at the large depth limit and show that the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with the growing depth, depending on the activation functions or Boolean gates used. 

1. Li and D. Saad, Phys. Rev. Lett. 120, 248301 (2018)
2. Li and D. Saad, Jour. Phys. A, 53, 104002 (2020)
3. A. Mozeika, B. Li, and D. Saad, Phys. Rev. Lett. 125, 168301, (2020).

In the last few years ideas from dynamical systems have fed into the design of neural networks. In this talk I will discuss the connection from neural networks to dynamical systems via deep layer limits. Further, I will bring in ideas from graph-based semi-supervised learning to graph neural networks to design new architectures which do not suffer from the over-smoothing phenomena.