Quanta Image Sensor (QIS) is a single photon image sensor with extremely small full well capacity. Since its introduction in 2005, the feasibility of QIS has been reported in many occasions, e.g., [1,2]. The latest QIS prototype in [2] can achieve a read noise below 0.25e - rms at room temperature, and frame rate beyond 1000 frames per second. This high level of photon sensitivity, low read noise, and high speed has made QIS an ideal sensor for low-light applications.
The subject of this paper is to use QIS for high dynamic range imaging. Compared to conventional CMOS image sensors (CIS) which acquire multiple frames and fuse them using linear averaging, QIS acquires a space-time data cubicle. To reconstruct the high dynamic range image, the algorithm has to first sum the frames, denoise, and then form an average. However, because of the quantized Poisson statistics of QIS, reconstruction methods for conventional CIS are not applicable. In particular, weighted averages based on estimating the pixel variance are extremely sensitive to noise which are not suitable for QIS.
The contribution of this paper is to propose a high dynamic range image reconstruction algorithm for QIS. The paper consists of two parts. First, we theoretically derive the dynamic range offered by QIS and compare it with CIS. This provides a foundation of how much dynamic range we can expect from QIS. Second, we present a new reconstruction pipeline as illustrated in Figure 1. The reconstruction uses the theoretical re- sults to predict the optimal combination weights. We compare the new algorithm with other state-of-the-art image reconstruction methods.
We start by reviewing the concept of dynamic range. Dynamic range of a sensor is the range of exposures that the signal-to-noise ratio is sustained before it drops below 1 (i.e., signal = noise). The signal-to-noise ratio considered in this paper follows from [3,4,5], which is known as the exposure referred signal-to-noise ratio SNR H . Our goal is to analytically derive SNR H , and use the results to design the reconstruction algorithm.
The mathematical model of the sensor is as follows. Denote c as the exposure (photons per second) of a particular pixel in the scene, and let τ be the integration time. The number of photons X reaching the sensor is a Poisson random variable X ∼ Poisson(τ c). The full-well capacity is assumed to be . Thus, the observed signal is a random variable B such that B = X, if X < , and B = , if X ≥ .
The exposure referred signal-to-noise SNR H (for both CIS and QIS) is defined as
where
is the variance, and θ def = τ c is the average number of photons seen in time τ . The exact expressions of the quantities in (1) are given by the theorem below, where the proof is skipped due to space limit.
The quantities µ B and σ B are
where
the incomplete gamma function. Consequently, the partial derivative ∂µ B /∂θ is
This exact analytic expression for arbitrary is a new result compared to previous work, e.g., [3,4]. Using the expressions in the theorem, we can plot SNR H as a function of the exposure. Figure 2 illustrates the behavior of SNR H for CIS and QIS. For CIS, we set = 4000, and for QIS we set to either = 1 or = 3, which are typical values of the sensors. The results in Figure 2 show that a single-exposure of QIS (87dB) is already larger than that of CIS (72dB). The combinedexposure is 148dB compared to 133dB. Therefore, regardless if we use single or multiple integration times, QIS offers a greater dynamic range. The result also indicates that 1-bit QIS has a larger dynamic range than a 2-bit QIS. This is a direct consequence of the decrease in over-exposure latitude as the number of bits increases [3].
To visually compare the images captured by a CIS and a QIS, we show in Figure 3 a simulated experiment. The images are simulated by setting the maximum illumination to 6×10 6 photons per pixel per second, and using CIS and QIS to measure the photons. For evaluation, we report the peak-signal-to-noise ratio (PSNR) which is proportional to the negative log of the mean squared error between the observed and the ground truth. The result again shows that CIS has a smaller contrast due to the low dynamic range.
Using the analytic expressions derived in Theorem 1, we now present a new HDR reconstruction algorithm for QIS. Consider a static scene captured by a QIS usings N frames of 1-bit or multi-bit measurements. To construct a HDR image, the N frames are grouped into M groups where each group corresponds to a different integration time. Within each group, the frames are summed to generate a single output, thus giving S m , m = 1, . . . , M . Since each S m is a simple sum, there will be Poisson noise. To denoise, we follow [4] Given the low dynamic range images {S 1 , . . . , S M }, we construct the high dynamic range image c by formulating a weighted average:
Here, τ m is the equivalent exposure time used when constructing S m , and w m (i, j) is the weight of the mth exposure image. The running index (i, j) denotes the pixel. Thus, ( 5) is a per-pixel weighted average. Readers familiar with the HDR literature will notice that ( 5) is just the classical weighted averaging. However, in the classical CIS based HDR reconstruction, the combination weight is inversely proportional to the local image variances [6]. For QIS, such variance-based weight can cause serious problems, because in bright regions the noise is squeezed and so the variance is close to zero. The following new theorem gives the optimal combination weights for QIS.
The optimal weights w m (i, j) (for pixel (i, j)) which maximize the SNR H of the signal c(i, j) in ( 5) is given by
where SNR H,m is m-th SNR H corresponding to the mth exposure (curve) in Figure 2. To evaluate the effectiveness of the new algorithm we compare it with two existing CIS-based image reconstruction methods: A built-in function in MATLAB, and linear sum used in [7]. Figure 4 shows the simulation result on an existing synthetic HDR dataset by Stanford. Since the inputs are simulated according to QIS statistics, both existing methods fail to reconstruct the image. In Figure 5, we show a set of real images captured using the Gigajot QIS pathfinder camera, at 1-bit mode and 3-bit mode. The result shows that the method can effectively recover the dynamic range while smoothing out the noise.
A new image reconstruction method is proposed for high dynamic range imaging using QIS. The algorithm is based on a linear weighted averaging method, where the weights are determined according to a new theoretical study on how the exposure referred signal to noise ratio changes with the exposure. Experimental results demonstrate the effectiveness of the new method on both synthetic and real QIS datasets.
Linear sum Proposed