DL4OpticalEngineering

Curated academic resources on deep learning for optical system design and assembly.

Deep Learning for Optical Engineering

This repository accompanies the topical review “Deep Learning across Optical System Workflow: A Comprehensive Review from Design to Assembly”. It provides a curated list of references and links to publicly available resources relevant to deep learning for optical system design and assembly.

Notes on links and updates

Links are provided when a public resource is available. This repository is intended as a living academic resource and will be updated periodically as new works appear. If you notice missing or outdated links, please open an issue or submit a pull request.

Method summaries

Starting Point Generation (SPG)

Starting Point Generation methods aim to assist optical designers by learning mappings from system specifications to initial optical design structures or prescriptions. These approaches are primarily used to accelerate early stage optical design, where the selection of a suitable starting point strongly influences convergence and final performance. Existing SPG methods cover spherical, aspheric, and freeform optical surfaces, and typically rely on supervised learning using reference designs optimized with commercial optical design software. More recent works incorporate unsupervised or hybrid learning strategies based on differentiable ray tracing and optics aware loss functions, reducing the reliance on labeled datasets. Applications of SPG networks include freeform imaging systems, microscope objectives, illumination optics, and compact off axis imaging configurations.

End to End co design (E2E)

End to End co design methods jointly optimize optical elements and computational processing within a single learning framework. These approaches embed differentiable imaging models into neural network training loops, enabling gradients to propagate from image based or task driven objectives back to optical parameters. Learnable optical variables include lens curvatures, thicknesses, aspheric coefficients, diffractive phase profiles, and metasurface patterns. Forward models commonly rely on differentiable point spread function computation, wave optics simulation, or rendering based image synthesis. E2E methods are widely applied in computational imaging tasks such as depth sensing, extended depth of field imaging, hyperspectral imaging, super resolution, and task specific vision problems, where optical and algorithmic components are optimized together to meet system level performance goals.

Optical Active Alignment (OAA)

Optical Active Alignment methods focus on bridging simulation and hardware realization by learning to infer fabrication or alignment errors from optical measurements acquired during assembly. These methods use neural networks to map measured features, such as wavefront error coefficients, point spread function images, or beam patterns, to misalignment states or correction commands. Depending on the feature modality, multilayer perceptrons or convolutional neural networks are commonly employed. OAA approaches are applied to a wide range of systems, including telescopes, camera lenses, micro optics, and multi element imaging assemblies, enabling faster and more accurate alignment compared to traditional iterative procedures. By integrating learning based prediction with physical alignment processes, OAA methods support improved manufacturing efficiency and robustness.

Paper lists

Starting Point Generation (SPG) methods

• Gannon C, Liang R. Using machine learning to create high-efficiency freeform illumination design tools. arXiv preprint arXiv:1903.11166, 2018. [Paper]
• Côté G, Lalonde J F, Thibault S. Toward Training a Deep Neural Network to Optimize Lens Designs. Frontiers in Optics. Optica Publishing Group, 2018. [Paper]
• Yang T, Cheng D, Wang Y. Direct generation of starting points for freeform off-axis three-mirror imaging system design using neural network based deep-learning. Optics express, 2019. [paper]
• Côté G, Lalonde J F, Thibault S. Deep learning-enabled framework for automatic lens design starting point generation. Optics express, 2021.[paper]
• Chen W, Yang T, Cheng D, et al. Generating starting points for designing freeform imaging optical systems based on deep learning. Optics Express, 2021.[paper]
• Côté G, Zhang Y, Menke C, et al. Inferring the solution space of microscope objective lenses using deep learning. Optics Express, 2022.[paper]
• Mao B, Yang T, Xu H, et al. FreeformNet: fast and automatic generation of multiple-solution freeform imaging systems enabled by deep learning. Photonics Research, 2023.[paper]
• Nie Y, Zhang J, Su R, et al. Freeform optical system design with differentiable three-dimensional ray tracing and unsupervised learning. Optics Express, 2023. [paper]
• Yang T, Cheng D, Wang Y. Designing freeform imaging systems based on reinforcement learning. Optics Express, 2020. [paper]

End to End co design (E2E) methods

• Elmalem S, Giryes R, Marom E. Learned phase coded aperture for the benefit of depth of field extension. Optics express, 2018.[paper]
• Chang J, Wetzstein G. Deep optics for monocular depth estimation and 3d object detection. In ICCV, 2019.[paper]
• Wu Y, Boominathan V, Chen H, et al. Phasecam3d—learning phase masks for passive single view depth estimation. In ICCP, 2019.[paper]
• Sun Q, Tseng E, Fu Q, et al. Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In CVPR, 2020.[paper]
• Sun Q, Zhang J, Dun X, et al. End-to-end learned, optically coded super-resolution SPAD camera. Acm Transactions on Graphics (TOG), 2020.[paper]
• Dun X, Ikoma H, Wetzstein G, et al. Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. Optica, 2020.[paper]
• Nehme E, Freedman D, Gordon R, et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nature methods, 2020.[paper][code]
• Tseng E, Mosleh A, Mannan F, et al. Differentiable compound optics and processing pipeline optimization for end-to-end camera design. ACM Transactions on Graphics (TOG), 2021.[paper]
• Sun Q, Wang C, Qiang F, et al. End-to-end complex lens design with differentiable ray tracing. ACM Transactions on Graphics (TOG), 2021.[paper]
• Li Z, Hou Q, Wang Z, et al. End-to-end learned single lens design using fast differentiable ray tracing. Optics Letters, 2021.[paper]
• Baek S H, Ikoma H, Jeon D S, et al. Single-shot hyperspectral-depth imaging with learned diffractive optics. In ICCV, 2021.[paper]
• Ikoma H, Nguyen C M, Metzler C A, et al. Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In ICCP, 2021.[paper][code]
• Burgos C M V, Yang T, Zhu Y, et al. Design framework for metasurface optics-based convolutional neural networks. Applied Optics, 2021.[paper]
• Tseng E, Colburn S, Whitehead J, et al. Neural nano-optics for high-quality thin lens imaging. Nature communications, 2021.[paper][code]
• Hinojosa C, Niebles J C, Arguello H. Learning privacy-preserving optics for human pose estimation. In ICCV, 2021.[paper][code]
• Arguello H, Pinilla S, Peng Y, et al. Shift-variant color-coded diffractive spectral imaging system. Optica, 2021.[paper]
• Wang C, Chen N, Heidrich W. do: A differentiable engine for deep lens design of computational imaging systems. IEEE Transactions on Computational Imaging, 2022. [paper][code]
• Shi Z, Bahat Y, Baek S H, et al. Seeing through obstructions with diffractive cloaking. ACM Transactions on Graphics (TOG), 2022.[paper][code]
• Li L, Wang L, Song W, et al. Quantization-aware deep optics for diffractive snapshot hyperspectral imaging. In CVPR, 2022.[paper][code]
• Zhang B, Yuan X, Deng C, et al. End-to-end snapshot compressed super-resolution imaging with deep optics. Optica, 2022.[paper]
• Yang T, Xu H, Cheng D, et al. Design of compact off-axis freeform imaging systems based on optical-digital joint optimization. Optics Express, 2023.[paper]
• Côté G, Mannan F, Thibault S, et al. The differentiable lens: Compound lens search over glass surfaces and materials for object detection. In CVPR, 2023.[paper][code]
• Zhang Q, Yu Z, Liu X, et al. End-to-end joint optimization of metasurface and image processing for compact snapshot hyperspectral imaging. Optics Communications, 2023. [paper]
• Yang X, Fu Q, Nie Y, et al. Image quality is not all you want: Task-driven lens design for image classification. arXiv preprint arXiv:2305.17185, 2023.[paper]
• Yang X, Fu Q, Heidrich W. Curriculum learning for ab initio deep learned refractive optics. Nature communications, 2024. [paper][code]

Optical Active Alignment (OAA) methods

• Esther Oteo and Josep Arasa. New strategy for misalignment calculation in optical systems using artificial neural networks. Optical Engineering, 2013.[paper]
• Keith M Hinrichs and John J Piotrowski. Neural networks for faster optical alignment. Optical Engineering, 2020.[paper]
• Alexander Khachikyan, Giulia Pippione, Mehmet Inanc Seng¨unes, Roberto Paoletti, and Moritz Seyfried. Micro-optics assembly for fast axis collimation by means of convolutional neural network. Optics Express, 2021.[paper]
• Zhixu Wu, Yiming Zhang, Rongxin Tang, Zhengyang Li, Xiangyan Yuan, Yong Xia, Hua Bai, Bo Li, Zhou Chen, Xiangqun Cui, et al. Machine learning for improving stellar image-based alignment in wide-field telescopes. Research in Astronomy and Astrophysics, 2022.[paper]
• Miao Zhang, Peng Jia, Zhengyang Li, Wennan Xiang, Jiameng Lv, and Rui Sun. Perception of misalignment states for sky survey telescopes with the digital twin and the deep neural networks. Optics Express, 2023.[paper]
• Haibin Liu, Wenyong Li, Shaohua Gao, Qi Jiang, Lei Sun, Benhao Zhang, Liefeng Zhao, Jiahuang Zhang, and Kaiwei Wang. Application of deep learning in active alignment leads to high-efficiency and accurate camera lens assembly. Optics Express, 2024.[paper]
• Ismail Baslar and Mahir Dursun. Improving neural-network-based prediction models for misalignment in off-axis three-mirror anastigmat telescopes. Applied Optics, 2024.[paper]
• Ryo Hashimoto, Shuji Matsuura, and Yusuke Iida. Method for optical adjustment with deep learning to quantitatively predict misalignment in optics. Applied Optics, 2024.[paper]