Paper documents carry valuable information and serve an essential role in our daily work and life. Digitized documents can be archived, retrieved, and shared in a convenient, safe, and efficient manner. With the increasing popularity of portable cameras and smartphones, document digitization becomes more accessible to users through picture taking. Once captured, the document images can be converted into electronic formats, for example, a PDF file, for further processing, exchange, information extraction, and content analysis. While capturing images, it is desirable to preserve the information on the document with the best possible accuracy -with a minimal difference from a flatbedscanned version. However, casual photos captured with mobile devices often suffer from different levels of distortions due to uncontrollable factors such as physical deformation of the paper, varying camera positions, and unconstrained illumination conditions. As a result, these raw images are often unsuitable for automatic information extraction and content analysis.
Previous literature has studied the document-unwarping problem using various approaches. Traditional approaches [26,46] usually rely on the geometric properties of the paper to recover the unwarping. These methods first estimate the 3D shape of the paper, represented by either some parametric shape representations [9,47] or some non-parametric shape representations [35,45]. After that, they compute the flattened image from the warped image and the estimated shape using optimization techniques. A common drawback of these methods is that they are usually computationally expensive and slow due to the optimization process. Recent work by Ma et al. [23] proposed a deep learning system that directly regresses the unwarping operation from the deformed document image. Their method significantly improved the speed of document unwarping system. However, their method did not follow the 3D geometric properties of the paper warping -training data was created with a set of 2D deformations -and therefore often generate unrealistic results in testing. Paper folds happen in 3D: papers with different textures but the same 3D shape can be unwarped with the same deformation field. Hence, 3D shape is arguably the most critical cue for recovering the unwarped paper. Based on this idea, we propose DewarpNet, a novel data-driven unwarping framework that utilizes an explicit 3D shape representation for learning the unwarping operation. DewarpNet works in two-stages with two sub-networks: i) The "shape network" consumes an image of a deformed document and outputs a 3D-coordinate map which has shown to be sufficient for the unwarping task [45]. ii) The "texture mapping network" backward maps the deformed document image to a flattened document image. We train both sub-networks jointly with regression losses on the intermediate 3D shape and final unwarping result (Fig. 1). After that, we provide a "refinement network" that removes the shading effect from the rectified image, further improving the perceptual quality of the result.
To enable the training of this unwarping network with explicit intermediate 3D representation, we create the Doc3D dataset -the largest and most comprehensive dataset for document image unwarping to date. We collect Doc3D in a hybrid manner, combining (1) captured 3D shapes (meshes) from naturally warped papers with (2) photorealistic rendering of an extensive collection of document content. Each data point comes with rich annotations, including 3D coordinate maps, surface normals, UV texture maps, and albedo maps. In total, Doc3D contains approximately 100,000 richly annotated photorealistic images.
We summarize our contributions as follows: First, we contribute the Doc3D dataset. To the best of our knowledge, this is the first and largest document image dataset with multiple ground-truth annotations in both 3D and 2D domain.
Second, we propose DewarpNet, a novel end-to-end deep learning architecture for document unwarping. This network enables high-quality document image unwarping in real-time.
Third, trained with the rich annotations in the Doc3D dataset, DewarpNet shows superior performance compared to recent state-of-the-art [23]. Evaluating with perceptual similarity to real document scans, we improve the Multi-Scale Structural Similarity (MS-SSIM) by 15% and reduce the Local Distortion by 36%. Furthermore, we demonstrate the practical significance of our method by a 42% decrease in OCR character error rate.
Based on how deformation is modeled, the two groups of prior work on document unwarping are: parametric shapebased models and non-parametric shape-based models:
Parametric shape-based methods assume that document deformation is represented by low dimensional parametric models and the parameters of these models can be inferred using visual cues. Cylindrical surfaces are the most prevalent parametric models [8,16,19,26,41,46]. Other models include Non-Uniform Rational B-Splines (NURBS) [10,44], piece-wise Natural Cubic Splines (NCS) [36], Coon patches [9], etc. Visual cues used for estimating model parameters include text lines [25], document boundaries [5], or laser beams from an external device [27]. Shafait and Breuel [33] reported several parametric shape based methods on a small dataset with only perspective and curl distortions. However, it is difficult for such low dimensional models to model complex surface deformations.
Non-parametric shape-based methods, in contrast, do not rely on low-dimensional parametric models. Such methods usually assume a mesh representation for the de- formed document paper, and directly estimate the position of each vertex on the mesh. Approaches used to estimate the vertex positions, include reference images [29], text lines [21,35,39], and Convolutional Neural Networks (CNNs) [30]. Many approaches reconstruct the mesh from estimated or captured 3D paper shape information. Notable examples are point clouds estimated from stereo vision [38], multi-view images [45], structured light [4], laser range scanners [47], etc. There is also work on directly using texture information for this task [11,24,43]. However, resorting to external devices or multi-view images makes the methods less practical. Local text line features cannot handle documents that mix text with figures. Moreover, these methods often involve complicated and time-consuming optimization. Recently, Ma et al. [23] proposed "DocUNet", which is the first data-driven method to tackle document unwarping with deep learning. Compared to prior approaches, DocUNet is faster during inference but does not always perform well on real-world images, mainly because the synthetic training dataset only used 2D deformations.
We created the Doc3D dataset in a hybrid manner, using both real document data and rendering software. We first captured the 3D shape (mesh) of naturally deformed real document paper. After that, we rendered the images with real document texture in Blender [1] using path tracing [40]. We used diverse camera positions and varying illumination conditions in rendering.
A significant benefit of our approach is that the dataset is created in large scale with photorealistic rendering. Meanwhile, our method generates multiple types of pixel-wise document image ground truth, including 3D coordinate maps, albedo maps, normals, depth maps, and UV maps. Such image formation variations are useful for our task, but usually harder to obtain in real-world acquisition scenarios.
Compared with the dataset in [23] where 3D deformation was modeled in 2D only [28], our dataset simulates document deformation in a physically-grounded manner. Thus, it is reasonable to expect that deep-learning models trained on our dataset will generalize better when testing on realworld images, compared to models trained on the dataset of [23]. We visually compare dataset samples in Fig. 2.
3D point cloud capture. Our workstation (Fig. 3 (I)) for deformed document shape capture consists of a tabletop, a gantry, a depth camera, and a relief stand. The gantry holds the depth camera level, facing towards the tabletop, at the height of 58 cm. At this height, the depth camera captures the whole document while still preserving deformation details. The relief stand has 64 individually controlled pins, raising the height of the document to isolate it from the tabletop. The height differences make it easier to extract the document from the background in the depth map. The stand simulates complex resting surfaces for the document and also supports the deformed document to maintain curls or creases.
We used a calibrated Intel RealSense D415 depth camera to capture the depth map. Assuming no occlusion, the point cloud of the document was obtained via X (3D) = K -1 [i, j, d ij ] T , where d ij is the depth value at the pixel position i, j in the depth map. The intrinsic matrix K was read from the camera. We averaged 6 frames to reduce zeromean noise, and applied Moving Least Squares (MLS) [32] with a Gaussian kernel to smooth the point cloud.
Mesh creation. We extracted a mesh from the captured point cloud using the ball pivoting algorithm [3]. The mesh has ∼130,000 vertices and 270,000 faces covering all vertices. We then subsampled each mesh to a 100 × 100 uniform mesh grid to facilitate mesh augmentation, alignment, and rendering. Due to the accuracy limits of our inexpensive sensor, even a higher resolution mesh grid cannot provide finer details like subtle creases. Each vertex has a UV position, to indicate texture coordinates, used for texture mapping in the rendering step. Assigning (u, v) = {(0, 0), (0, 1), (1, 0), ( reconstruction quality. For joint training we used α = β = 0.5 (Eq. 3). We use the Adam solver [15] with a batch size of 40, and weight decay of 5 × 10 -4 . The learning rate is initially set at 1 × 10 -4 , and reduced by a factor of 0.5 if the loss does not reduce for 5 epochs.
We evaluate our method with multiple experiments on the 130-image benchmark from [23], and also show qualitative results on real images from [45]. As a baseline, we train the DocUNet [23] unwarping method on our new Doc3D dataset. Furthermore, we evaluate OCR performance of our method from a document analysis perspective. Finally, we provide a detailed ablation study to show how the use of the Coordinate Convolutions [22], and the loss L D affect unwarping performance. Qualitative evaluations are shown in Fig. 7.
Benchmark. For quantitative evaluation, we classify the 130-image benchmark [23] into six classes indicating six different levels of deformation complexity (see Table 1). The benchmark dataset contains various kinds of documents, including images, graphics, and multi-lingual text.
Evaluation Metrics. We use two different evaluation schemes based on (a) Image similarity and (b) Optical Character Recognition (OCR) performance.
We use two image similarity metrics: Multi-Scale Structural Similarity (MS-SSIM) [42] and Local Distortion (LD) [45], as quantitative evaluation criteria, following [23]. SSIM computes the similarity of the mean pixel value and variance within each image patch and averages over all the patches in an image. MS-SSIM applies SSIM at multiple scales using a Gaussian pyramid, better suited for the evaluation of global similarity between the result and groundtruth. LD computes a dense SIFT flow [20] from the unwarped document to the corresponding document scan, thus focusing on the rectification of local details. The parameters of LD are set to the default values of the implementation provided by [23]. For a fair comparison, all the unwarped output and target flatbed-scanned images are resized to a 598400 pixel area, as recommended in [23].
OCR accuracy is calculated in terms of Character Error Rate (CER). CER is evaluated by calculating the Edit Distance (ED) [17] between the reference and recognized text. ED is the total number of substitutions (s), insertions (i) and deletions (d) to obtain the reference text, given the recognized text. CER = (s+i+d)/N , where N is the number of characters in the reference text, which is obtained from the flatbed scanned document images.
We present a baseline validation of the proposed Doc3D dataset by training the network architecture in Do-cUNet [23] on our dataset -Doc3D. DocUNet is a 3Dagnostic model. The architecture consists of two stacked UNets. DocUNet takes a 2D image as input and outputs a forward mapping (each pixel represents the coordinates in the texture image). The supervisory signal is solely based on the ground truth forward mapping. Unlike the proposed DewarpNet which can directly output the unwarped image, DocUNet needs several post-processing steps to convert the forward mapping to the backward mapping (each pixel represents the coordinates in the warped input image) and then sample the input image to get the unwarped result. 2 show significant improvement when we train DocUNet on Doc3D instead of the 2D synthetic dataset from [23]. The significant reduction of LD (14.08 to 10.85) signals a better local detail rectification. This improvement is the result of both (1) the Dewarp-Net architecture and ( 2) training with a more physically grounded Doc3D dataset, compared to the 2D synthetic dataset in [23].
We evaluate both DewarpNet and DewarpNet(ref ) (i.e., DewarpNet augmented with the post-processing refinement network) on the DocUNet Benchmark dataset. We provide comparisons on both (1) the overall benchmark dataset (Table 2) and ( 2) each class in the benchmark (Fig. 6). The latter provides detailed insight into the improvements of our approach over previous methods. From class (a) to (e), our model consistently improves MM-SSIM and LD over the previous state-of-the-art. In the most challenging class (f), where the images usually exhibit multiple crumples and random deformations, our method achieves comparable and slightly better results. Time Efficiency of DewarpNet. Our model takes 32ms on average to process a 4K resolution image. Compared to DocUNet [23] this represents a 125x speed up. Dewarp-Net directly outputs the unwarped image whereas DocUNet requires an expensive separate post-processing step.
We use PyTesseract (v0.2.6) [34] as the OCR engine to evaluate the utility of our work on text recognition from images. The text ground-truth (reference) is generated from 25 images from DocUNet [23]. In all these images, more than 90% of the content is text. The supplementary material contains some samples from our OCR test-set. OCR performance comparison, presented in Table 3, shows our method outperforms [23] with a large margin in all metrics. In particular, DewarpNet reduces CER by 33% compared to DocUNet, and the refinement network gives a reduction of 42%.
Coordinate Convolution (CoordConv). We investigate the effects of CoordConv on texture mapping network performance. The experiment (Table 4) on Doc3D validation set demonstrates that using CoordConv leads to a 16% ℓ2error reduction on B and a slight improvement of SSIM on D from 0.9260 to 0.9281.
Loss L D . The texture mapping network benefits greatly from using L D (unwarped visual quality loss). As shown in Table 4 compared to using the absolute pixel coordinate loss L B only, using L B + L D significantly reduces the ℓ2 error on B by 71% and improve the SSIM on D by 9%.
For qualitative evaluation, we compare DewarpNet with DocUNet in Fig. 7 and You et al. [45] in Fig. 8. The method by [45] utilizes multi-view images to unwarp a deformed document. Even with a single image, DewarpNet shows competitive unwarping results.
Additionally, we show that the proposed method is robust to illumination variation and camera viewpoint changes in Fig. 9. To evaluate the illumination robustness, we test on multiple images with a fixed camera viewpoint but different directional lighting from front, back, left, right of the document, and environment lighting. We also test DewarpNet robustness to multiple camera viewpoints, on a sequence of multi-view images provided by [45]. Results show that DewarpNet yields almost the same unwarped image in all cases.
In this work, we present DewarpNet, a novel deep learning architecture for document paper unwarping. Our method is robust to document content, lighting, shading, or background. Through the explicit modeling of 3D shape, DewarpNet shows superior performance over previous state-of-the-art. Additionally, we contribute the Doc3D dataset -the largest and most comprehensive dataset for document image unwarping, which comes with multiple 2D and 3D ground truth annotations. Some limitations exist in our work: First, the inexpensive depth sensor cannot capture fine details of deformation like subtle creases on a paper crumple. Thus our data lacks samples with highly complex paper crumple. In future work, we plan to construct a dataset with better details and more complex structures. Second, DewarpNet is rela- tively sensitive to occlusion: results degrade when parts of the imaged document are occluded. In future work, we plan to address this difficulty via data augmentation and adversarial training.