Trustworthy learning of mechanical systems, and Stiefel optimization with applications to transformer and optimal transport
Georgia Institute of Technology
The interaction of machine learning and dynamics can lead to both new methodology for dynamics, and deepened understanding and/or efficacious algorithms for machine learning. This talk will focus on such interactions centered around structured dynamics.
Specifically, I will first discuss data-driven learning and prediction of mechanical dynamics, for which I will demonstrate one strong benefit of having physics hard-wired into deep learning models; more precisely, how to make symplectic predictions, and how that provably improves the accuracy of long-time predictions.
Then I will report a construction of momentum-accelerated gradient descent algorithms on Riemannian manifolds, focusing on a particular case known as Stiefel manifold. The treatment will be based on, firstly, the design of continuous-time optimization dynamics on the manifold, and then a thoughtful time-discretization that preserves all geometric structures. Since Stiefel manifold corresponds to matrices that satisfy orthogonality constraint, two practical applications will also be described: (1) we markedly improved the performance of trained-from-scratch Vision Transformer by appropriately placing orthogonality into its self-attention mechanism, and (2) our optimizer also makes the useful notion of Projection Robust Wasserstein Distance for high-dim. optimal transport even more effective.
Tuesday, September 27, 2022
11:00AM AP&M 2402 and Zoom: https://ucsd.zoom.us/my/mleok
Center for Computational Mathematics9500 Gilman Dr. #0112La Jolla, CA 92093-0112Tel: (858)534-9056