One of the primary challenges for image sensing and processing comes from the large size of the raw data from image sensors [1]. A high-resolution image sensor produces a large amount of image data, increasing the processing units' computing overhead. The high computing overhead is also linked to the high power consumption of the device. The recent efforts to alleviate this problem can be categorized into hardware methods and software methods. The hardware of image sensing and processing contains pixel circuits, analog-to-digital converters (ADC), and the image signal processor (ISP). Most of the power in an image sensor is consumed by the column-parallel ADCs and digital circuits [2]. Various methods have been proposed to reduce the ADC power such as reducing supply voltage [3], using multi-stage ADCs [4], and applying data compression [5]. However, reducing supply voltage may affect the dynamic range of the sensor. Although Multi-stage ADCs can reduce the power consumption of the image sensor, they still suffer from large data throughput which requires high computing overhead. Data compression needs additional circuits for image compression and reconstruction. On the software side, specialized embedded machine-learning models have been applied to reduce model size for limited hardware resources. For example, [6] proposed a hardware-friendly user-specific machine learning algorithm for edge devices that leverage transfer learning. Standard embedded machine learning solutions have been proposed, such as Tiny Machine Learning (TinyML) [7], [8]. However, these methods do not solve the high computing overhead problem since the image sensor still generates much redundant data for the model.
In this paper, we applied a predictive sampling method for image sensing and processing, which reduces the data throughput and computing overhead for low-power applications. The predictive sampling method was recently proposed for time domain signal sensing and processing systems [9], [10]. It uses the digital value of prior samplings to predict values of the incoming data and only performs digitization when the prediction fails. One advantage of this method is that it can save power consumption by avoiding unnecessary comparison in quantization if the prediction is correct so that the data conversion can be skipped. A more important advantage is that it greatly reduces the output digital data throughput, which can save computing overhead and power consumption of the following digital processing unit. Unlike a more established solution such as compressed sensing [15], dynamic sampling approaches the problem in the pre-processing phase to reduce the data throughput. Compared to other sampling methods such as the adaptive sparse image sampling [16], our sampling method allows for more effective real-time implementation on edge devices.
When applying the predictive sampling method in image sensing and processing, the proposed system scans the entire image and uses the digital value of the neighboring pixels to predict the current pixel value. The scan includes both horizontal row pixel scan and vertical column pixel scan. The goal of these scans is to identify the location and digital values of the pixels that are turning points in each row and column. These pixels are the critical pixels in the image while other pixels may be removed without introducing large distortion of the original image. By doing so the whole image can be compressed to save data throughput. Moreover, the critical pixel locations and values can be directly processed using a low-computing overhead image processing algorithm to save power and memory. While this method can be implemented at the pixel level using a similar method of prediction ADC architecture [2], in this paper, we perform analysis using an existing full digital image to evaluate the performance of the proposed method.
The one-dimensional dynamic predictive sampling method can be
1949-307X © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications\_standards/publications/rights/index.html for more information. Row Index 0 50 150 250 350 Pixel Value Dynamic Sampling for Column 2 Actual Values Sampled Points Dynamic Sampling for Row14 Example Image 2 6 10 14 Selected Pixels 0 50 150 250 350 Pixel Value Column Index 2 6 10 14 Actual Values Sampled Points implemented in two-dimensional image sensing and processing by considering each row or column as a one-dimensional signal. The overall goal is to locate and record the turning points in each row and column, and then represent each row and column using these points as piece-wise linear wave. If the image is sparse, most of the pixel values are not necessary to be sampled or quantized. only the locations and values of critical pixels need to be recorded. This saves much digital storage space and signal processing overhead, depending on the sparsity of the incoming images. The accuracy of the processing outcome depends on the specific application, which can be adjusted by the error tolerance threshold. Moreover, since the sampling points can represent the information of the image, a low-overhead computing model can use only these sampling points to perform image processing, such as classification.
For example, a grayscale image can be represented by a threedimensional shape with the and directions referring to the pixel locations and the direction for the amplitude value of the pixel. The ranges of the and directions limit the size of the image while the range of the direction represents the amplitude. To apply the dynamic predictive sampling method in image sensing, we divide the image into a series of pixel rows and columns. Each row and column is then a vector of pixel amplitude. After applying the dynamic predictive sampling method to each vector, the pixels that represent the amplitude turning points are selected. This can also be done during the sensing phase using prediction-based ADC architecture [2]. The collection of the values of these pixels ( , , and ) is then recorded to represent the image.
Fig. 1 shows an example of the above-mentioned process. The prediction starts at the edge pixels, two consecutive pixels are applied to form a prediction. An error threshold is applied to the process so that a small variation of the amplitude is ignored. If a higher accuracy of the image is required during reconstruction, a smaller value should be applied. For example, a zero keeps all the pixel values. A pixel is marked as a turning point if its value is outside the window formed by the prediction value and the error threshold. In the example image, some pixels are identified as horizontal turning points while some other pixels are identified as vertical turning points. The combination of these two groups is selected as the collection of turning points, which are marked white. The proposed method is different from conventional edge detection since the error threshold is applied to the slope of the prediction instead of the absolute amplitude. Set [1] and [2] to the values of the first two pixels in row for each column from 3 to end in row do end if end for end for {Similar Process for vertical pixel selection} for each row in image do Initialize prediction array of zeros, length equal to number of columns in
Set [1] and [2] to the values of the first two pixels in row for each column from 3 to end in row do
and [ ] to the latest values based on [ , -1] and [ , ] end if end for end for Combine matrices and to get matrix
Depending on the application specification and the sparsity of the image, the value of pixels that are not edge or turning points may not be necessary to be stored. While this saves the size of the image file, it is necessary to reconstruct the image using the values of the selected pixels. This can be achieved in two-dimensional (2-D) interpolation, in which the image is reconstructed by neighboring key sampling pixels and forms triangle meshes. The algorithm performs interpolation of scattered data points onto a regular grid and fills the missing data. Depending on the interpolation method, the algorithm creates a series of triangular or sinusoidal meshes that approximate the original image. The error threshold determines the trade-off between the data throughput and the quality of reconstructed image. Figure 3 compares the selected pixels (top row), reconstructed image (middle row), and difference between the original image and the reconstructed image (bottom row) of four values. From the left to the right of Fig. 3, the values are "128", "64", "32", and "16", respectively. The original image is a gray-scale "Camera Man" image with a 8bit resolution. The value of the pixels are between 0 and 255. As shown in Fig. 3, a higher value results in a lower quality of reconstructed image. The quality of the reconstructed images are quantitatively evaluated using structural similarity index measure (SSIM) values. SSIM is an image quality computation method and metric-based, which assesses reconstruction quality based on the luminance, contrast, and structure of image objects [14], making it much more akin to the human perception and evaluation of image quality. A higher SSIM indicates more similarity between the original image and the reconstructed image. From the left to the right of Fig. 3, the SSIM values are 0.7754, 0.8624, 0.9236, and 0.9560, respectively. The acceptable SSIM of images depends on specific applications.
As predictive sampling produces a less data-intensive copy of images containing only the essential pixels within the original, it is possible to perform image processing only using these samples to save computing overhead. To validate this idea, we trained multiple image classification convolutional neural network (CNN) models using datasets of sampled "dog" and "cat" images, a typical image classification problem for model performance assessment. In our testing, We trained three models of varying complexity levels on three different "cat vs dog" datasets, in total nine model for comparison. The datasets selected for the training was a modified version of the Kaggle "Dogs vs. Cats" dataset [11]. The original dataset contains 25,000 images (12,500 "dog" and 12,500 "cat" images). We selected a subset of images from this dataset with 452 'dog" and 452 "cat" images. The selected images are pre-processed to grayscale images with 8-bit resolution for pixel amplitude (0-255). These images are then processed using predictive sampling with a of 126. Therefore, we obtained three datasets as colored images, grayscale images, and predictive sampling images. The files sizes of these datasets are 31.4 MB, 20.8 MB, and 8.51 MB, respectively. Three sequential CNN models were trained using each of the datasets; Small, Medium, and Large CNNs. All of the models were trained for 20 epochs. The architectural frameworks of these models are 1) Small and Medium CNNs: The small and Medium CNNs have similar structures. The only difference between them is the number of kernels in the convolution layers. The small CNNs have half of the number of kernels per convolution layer as the Medium CNNs. They each consist of one input layer, three convolutional layers, a pooling layer, a flattening layer, and a final dense layer with a single neuron for binary classification and 'Sigmoid' activation. The models are then compiled with the 'Adam' optimizer. 2) Large CNN: The structure of the large CNN models includes a larger number of kernels and no zero padding in the convolution layers, as well as two dense layers prior to the final binary classification. It consists of one input layer, four convolutional layers, a pooling layer, two flattening layer, and a final dense layer with a single neuron for binary classification and 'Sigmoid' activation. The model is then compiled with the 'Adam' optimizer.
The performances of the models are evaluated using the validation datasets. During the training process, the evaluation metrics for the validations are and . After the training process, the evaluation metrics are , , and . These results are shown in Table 1. The models trained on the sample images are comparable to the gray-scale and color image models in terms of accuracy and inference. In terms of speed, the models trained on the sample images generated using the predictive sampling method had a better inference rate than those of gray-scale and color image models. In terms of accuracy, the predictive sampling models were comparable in the case of the large model and better in the grayscale and color image models. The main advantage of the predictive sampling method can be seen in the file size of the dataset for training. Compared to other implementations of embedded machine learning for image processing, the proposed predictive sampling method and its models offer a good mix of accuracy and inference speed, especially in the small sample model. We compared our results to two other implementations of embedded machine learning as shown in Table 2. While having a slower performance in speed, the predictive sampling models have a higher accuracy. The predictive sampling method has advantages when the input image is sparse. For example, for a 511 by 511 black and white image, the proposed method can reduce the model size only when the number of the selected pixels is under 2 8 , which is about 0.1%, as shown in Figure 4. Table 2: Performance Comparison to Similar Solutions in terms of accuracy and inference time.
Accuracy Inf. Time SVM Classifier [12] 92.47% 0.28(ms) EyeLearn (with SVM) [13] 81.44% 0.16B (FLOPs)
EyeLearn (with 2-Layer MLP) [13] 81.5% 0.16B (FLOPs)
This work: Small CNN (Predictive Sampling) 96.9% 34.52(ms) This work: Medium CNN (Predictive Sampling) 98.4% 37.23(ms)
In this work, we presented a predictive sampling method for selecting key pixels in image sensing and demonstrated the feasibility of using the key pixels for image classification. The proposed method reduces the image size while allowing for effective image reconstruction using only 27.1% of the input data size of the original image. This reduction in image size allows for the reduction in the file sizes of image datasets and the faster training of image classification, which would benefit computer vision implementation of edge devices without cloud connectivity. We used our sample models alongside their color and gray-scale equivalents to train image classification CNNs of different complexity levels to test the validity of the proposed methods. Our results showed that despite the far smaller dataset file size, the sample models had comparable accuracy ratings to that of the color and gray-scale models.