Method

Overview of our LED.
Unlike typical calibration-based algorithms, our method, as shown in the figure, consists of four steps in total:

1. Synthetic Noise Generation.
2. Pre-training the denoiser.
3. Collecting few-shot paired data using the target camera.
4. Fine-tuning the denoiser using the data collected in Step 3.

Compared to calibration-based algorithms, LED uses randomly sampled virtual camera parameters during synthetic noise generation, thereby avoiding the calibration process.
During each iteration of pre-training, a random virtual camera is first selected, and the training is performed using the paired data synthesized with that virtual camera. When the k-th virtual camera is selected, only the k-th CSA (Camera-Specific Alignment) is trained. This approach aims to decouple camera-specific information from noise information.
During Finetuning, the first step is to average all CSAs to enhance generalization capability. After that, an additional branch is added to handle out-of-model noise. It's important to note that the blue 3x3 convolution is frozen during this process.

Note: The network architecture of LED is exactly the same as other methods during deployment! This is possible thanks to the reparameterization technique. Below are the details regarding these aspects.
More detail can be found in our main paper.

Details when deploy

LED's another highlight lies in its final deployment, where the neural network remains completely consistent with other methods! No additional computational overhead is added, thanks to the following reparameterization process.

LED Pre-training Could Boost Existing Methods!

Experiments on Pre-Traning

LED pre-training could boost the performance of other methods. By integrating LED pre-training into various existing calibration-based or paired data-based methods, ELD and SID, our approach facilitates notable enhancements in performance. These improvements are not uniform but rather depend on the difference of the pre-training strategies employed. This proves particularly effective in the industrial applications, where the demands for efficiency are paramount. The strategic application of LED pre-training not only boost the performance of the denoiser but also paves the way for more advanced, adaptable, and efficient denoising.

Discussion on "Why Two Pairs"?

The left figure shows the performance of LED as it varies with the number of few-shot data pairs. Notice that when using only one data pair for fine-tuning, LED's performance is not as high as ELD. However, with two data pairs, LED's performance significantly surpasses that of ELD.

Indeed, this is because the camera's gain and noise variance have a linear relationship, as illustrated in the right graph. With just two pairs of images, LED can effectively learn this linear relationship, as two points are enough to determine a straight line.

However, due to the presence of errors, the horizontal coordinates of these two points need to be sufficiently different. In other words, the two pairs of images used by LED should have a significant difference in their ISO settings at the time of capture (ISO < 500 and ISO > 5000).

More details (including validation experiments) can be found in our main paper.

BibTex

@inproceedings{jin2023lighting,
    title={Make Explict Calibration Implicit: "Calibrate" Denoiser Instead of The Noise Model},
    author={Jin, Xin and Xiao, Jia-Wen and Han, Ling-Hao and Guo, Chunle and Liu, Xialei and Li, Chongyi and Cheng Ming-Ming},
    journal={Arxiv},
    year={2023}
}
                

Contact

Feel free to contact us at xjin[AT]mail.nankai.edu.cn!