White Paper: Computational Microscopy
This white paper is an outcome of IPAM’s fall 2022 long program, Computational Microscopy.
This whitepaper summarizes the activities and outcomes of the long program on “Computational Microscopy” at the Institute of Pure and Applied Mathematics (IPAM) from September 12 to December 16, 2022.
For more than three centuries, lens-based microscopy, such as optical, phase-contrast,
fluorescence, confocal, and electron microscopy, has played an important role in the evolution of
modern science and technology. In 1999, a novel form of microscopy, known as coherent
diffractive imaging (CDI), was developed and transformed our traditional view of microscopy, as
the diffraction pattern of a noncrystalline object was first measured and then directly phased to
obtain a high-resolution image. The well-known phase problem—the usually unavoidable loss of
phase information in the diffraction intensity—was solved by a combination of coherent
illumination and computational algorithms. Over the years, various CDI methods including
plane-wave CDI, ptychography (i.e. scanning CDI), Bragg CDI and Fourier ptychography have
been broadly implemented using synchrotron radiation, X-ray free electron lasers, high harmonic
generation, and optical and electron microscopy. Furthermore, the 2017 Nobel Prize in chemistry
was awarded to Richard Henderson, Joachim Frank, and Jacques Dubochet for developing
cryo-electron microscopy (cryo-EM) for the high-resolution structure determination of
biomolecules in solution. All these groundbreaking developments require the use of advanced
computational algorithms and mathematical tools. This IPAM long program brought together
senior and junior applied mathematicians, physicists, chemists, materials scientists, engineers,
and biologists to discuss and debate on the current status and future perspectives of modern
microscopy using computation, mathematics, and modeling. The program hosted four workshops
focusing on different aspects of computational microscopy:
● Workshop I: “Diffractive Imaging with Phase Retrieval” focused on advanced
computational methods to solve the phase problem using iterative algorithms and deep
● Workshop II: “Mathematical Advances for Multi-Dimensional Microscopy” focused on
the incorporation of state-of-the-art mathematical and computational methods into
multi-dimensional electron microscopy.
● Workshop III: “Cryo-Electron Microscopy and Beyond” focused on the current
challenges and future perspectives of the cryo-EM field.
● Workshop IV: “Multi-Modal Imaging with Deep Learning and Modeling” focused on the
integration of data acquisition, mathematical modeling, and deep learning in multimodal
In addition to these four workshops, we formed seven working groups, including 1) Simulation
for electron and optical microscopy, 2) Inverse problems in cryo-EM and phase retrieval, 3) AI
& learning theory, 4) Data-driven information extraction from microscopy data, 5) Multimodal
data processing and acquisition, 6) Space-time models, and 7) Geometry in data processing for
microscopy. The working groups met regularly during the program and tackled a number of
outstanding problems in the field. Below we provide the open challenges that we identified in
computational microscopy, the progress that we made at IPAM, and the research directions that
we will continue to investigate in the future.rections in the field of electronic structure theory and computational chemistry as well as related fields that were discussed during the program.
Field Theory Approaches: One challenge in quantum mechanics for large systems is connecting established methods across scales. Quantum Field Theory (QFT) provides a natural framework to derive existing methods from first principles, highlighting common approximations and a pathway to more accurate ones. This framework allows the description of collective and long-range effects in large complexes. In this context, the development of a rigorous effective field theory approach and the analysis of the light-matter interaction beyond the semiclassical approximations in large molecular systems have been identified as key open questions.
Excited and open quantum systems: Multiscale approaches to quantum mechanics need to describe collective degrees of freedom arising from large time and length scales. Open quantum system theory offers a productive way to describe excited electronic states, especially in the context of light-matter interactions. This accurate description is a key aspect both for cavity molecular quantum electrodynamics and large biomolecular complexes. The development of rigorous, accurate, and efficient methodologies for multiscale modeling across optical, electronic, and vibrational degrees of freedom has been envisaged as one of the main directions for future research.
Embedding theories: Embedding theories provide an efficient way to combine computational methods of different scales to treat large quantum systems. Within this program, a variety of embedding methods were discussed and scrutinized. Particular attention has been given to density matrix embedding theory, for which a numerical remedy for the chemical potential fitting problem has been developed. The implementation thereof is important as it extends the numerical applicability of embedding methods to larger and more complicated systems.
Response functions in molecules and solids: The number of materials and molecules grows combinatorially with the elements and building blocks considered. Dominating quantum effects can often be recovered with response functions describing how electronic structure changes under external influences. Participants devised new methods to obtain some of these response functions relevant to interacting systems and many-body interactions. Open challenges include obtaining high-quality response functions and the systematic improvement over current limitations of quantum mechanical calculations.
Non-covalent Interactions: Macroscopic properties of matter are chiefly determined by intermolecular interactions. In large systems, non-covalent interactions are highly non-local with van der Waals (vdW) dispersion being of particular importance. Due to its long-range nature and complex quantum many-body character, an accurate treatment of the vdW dispersion is currently challenging for large systems. During the program an interdisciplinary effort addressed the current lack of accurate and efficient methods capturing vdW dispersion interactions. In addition to the development of novel numerical approaches, this multidisciplinary perspective allows for a better understanding of the physical origins of non-covalent interactions.
Machine Learning for Quantum Mechanics: Quantum mechanical problems are high-dimensional and often nonlinear. If machine learning methods can yield models of relevant accuracy, more applications and systems become feasible. The program explored multiple complementary methods that aim to describe long-range interactions and improve transferability by introducing physical constraints directly. The participants identified interpretability of models, inclusion of functional derivatives, and treatment of large-scale many-body systems as some of the key open questions.
Additional Long Program Activity – 2D Moire Materials Workshop: Since its isolation by Geim, Novoselov, and collaborators in 2004, single layer graphene has drawn intense interest for its remarkable physical and theoretical properties. Graphene has the highest known tensile strength, and conductivity. More recently, theoretical studies have focused on the mathematical modeling of twisted bilayer graphene (TBG), a metamaterial constructed by stacking two layers of graphene and then rotating one layer with respect to the other. This has led to the development of models such as the Bistritzer-Mcdonald model (2011) and the Chiral model proposed by Tarnopolsky and coworkers (2019). The working group on 2D materials focused on reading and understanding the most recent mathematical results on the spectral properties of TBG and strained graphene at magic angles.