All-optical chips for large-scale intelligent vision generation

All-optical chip for large-scale intelligent vision generation

On December 19, 2025, a team led by Chen Yitong from Shanghai Jiao Tong University published a paper in Science entitled "All-optical synthesis chip for large-scale intelligent semantic vision generation". This research developed an all-optical chip for large-scale intelligent vision generation, named "LightGen". LightGen integrates over 2 million photonic neurons. By altering the network dimension through a proposed Optical Latent Space (OLS) composed of a single-mode fiber array, and combined with an unsupervised Bayesian training algorithm, high-resolution semantic image generation, denoising, style transfer, 3D generation, and manipulation were experimentally achieved. The measured end-to-end energy efficiency is more than two orders of magnitude higher than that of state-of-the-art electrical chips. When achieving performance comparable to Stable Diffusion, StyleGAN, NeRF, and VGG-19 in specific tasks, LightGen achieved end-to-end computing speed, energy efficiency, and computing density of 3.57 × 10⁴ TOPS, 6.64 × 10² TOPS/W, and 2.62 × 10² TOPS/mm², respectively.

LightGen's generation tasks

Background:

Photonic computing is considered one of the most promising avenues for reshaping traditional computing paradigms, already offering orders-of-magnitude advantages in computational speed and energy efficiency compared to state-of-the-art GPUs. However, most current end-to-end research breakthroughs are limited to decision-making tasks, such as image and vowel classification. The main reasons for this predicament are:

1. Scale Limitation: Existing integrated photonic computing chips have a limited number of neurons, failing to meet the scale requirements of advanced generative models.

2. Dimensionality Limitation: Simulated light signals lack dimensional variation, making it difficult to meet the dimensionality requirements of generative models.

3. Training Algorithms: Generative AI requires unsupervised algorithms that do not rely on predefined real labels.

To address these issues, Chen Yitong's team proposed an all-optical generative chip called LightGen.

Size comparison and micrographs of LightGen; schematic diagram of LightGen with photon encoder, OLS, and photon generator.

LightGen's basic architecture and principles:

LightGen completely migrates the "encoding-latent space-generation" process to the optical domain. The photonic encoder, composed of an integrated diffractive metasurface with an area less than 35 mm², is used for image feature extraction. The encoded light field is coupled into a single-mode fiber array for feature subsampling, while simultaneously using the amplitude and phase of the light for encoding.

LightGen的结构

 Thanks to the characteristic that single-mode fiber only allows fundamental mode transmission, OLS can perform the transformation of simulated spot dimensions under all-optical conditions. LightGen uses the complex information carried by the spot to characterize high-dimensional latent variables. Each single-mode fiber in the fiber array corresponds to one spot, forming a Gaussian spot array with different amplitudes and phases.

In LightGen, mode conversion is achieved through a single-mode fiber array, thereby realizing dimensional change.

Experiments demonstrated that the functionality of OLS is comparable to that of electronic variational autoencoders (VAEs) and β-VAEs. In the experiment of clustering dogs of different coat colors, the photonic encoder embedded the original image into a 100-dimensional optical latent space and visualized it using t-SNE for dimensionality reduction. The experimental results showed that dogs of different coat colors were successfully clustered, indicating that OLS can effectively embed fine-grained features in images.

For training, LightGen employs a Bayesian-based unsupervised training method (BOGT), which improves the generation quality by introducing modeling of the probability distribution of semantic data.

An OLS illustration of subsampling using t-SLE. Images with different foreground and background features are clustered in the proposed OLS in an unsupervised manner.

Semantic image generation, denoising, and style transfer results:

LightGen can perform various tasks such as semantic image generation, denoising, and style transfer on high-resolution color images.

Generation for animal categories

  LightGen can generate animal images at a resolution of 512×512, covering different categories, colors, expressions, and backgrounds. Zoomed-in areas reveal rich details such as fur texture and light reflections in the eyes.

Experimental results show that the generated images achieve classification accuracy comparable to the real test data, indicating that the generated data closely resembles natural data in both overall and detailed features.

2. Semantic Denoising

LightGen benefits from the information capacity difference between free-space mode and single-mode fiber fundamental mode, enabling effective noise reduction even with a high proportion of bad-matched pixels (PBMP).

3. Style Transfer Task

LightGen successfully achieved multiple transformations in style, including Van Gogh, Malevich, and mosaic, with qualitative and quantitative performance comparable to VGG-19, StyleGAN, and StyleID.

4. Continuous Structure Generation Results

LightGen has significant advantages in handling global structures and continuous curves, avoiding issues such as inter-block discontinuities and structural damage.

5. 3D Generation and Semantic Manipulation

LightGen generated 256×256 resolution images containing various furniture pieces in an indoor scene. Through 2D projection training, it unsupervisedly learned the 3D structure of objects and their representation in an OLS, achieving performance comparable to NeRF. By manipulating the OLS, it can generate 3D furniture with different styles and perspectives, with qualitative and PSNR metrics comparable to NeRF.

Performance Comparison:

Under conservative estimates, LightGen achieved a computation speed of 3.57×10⁴ TOPS, energy efficiency of 6.64×10² TOPS/W, and computational density of 2.62×10² TOPS/mm² in experiments, all exceeding the NVIDIA A100 by more than two orders of magnitude. With a faster spatial light modulator, its theoretical computation speed can reach 5.69×10⁹ TOPS.

Original Link: https://www.science.org/doi/10.1126/science.adv7434