There is a significant body of prior work which has focused on developing BIM integrated computer vision systems for construction robotic applications, however, most of this work focuses -2 -on stacking tasks rather than framing, and uses an eye in hand camera rather than a fixed camera configuration (Dawod and Hanna 2019;Feng et al. 2015;Tish et al. 2020). Due to the relatively small size of building elements assembled in these types of tasks, computer vision sensors can be placed very close to the material being measured, limiting challenges associated with sensor noise and accuracy. Computer vision systems intended for other construction applications, such as progress monitoring, have also been demonstrated (Deng et al. 2020;Han et al. 2018;Troncoso-Pastoriza et al. 2018;Vähä et al. 2013).
The more general problem of object detection using computer vision sensors and image segmentation has been the subject of much research, with work by Schwarz et al. (2017), Li et al. (2017), Feng et al. (2014), and Wang et al. (2015) providing a good template for solving this problem in the construction domain. Prior work by Zhang (2018), Zhang et al. (2015) and Akanbi et al (2020) explores automated extraction of geometric information from BIM documents, providing approaches which can be readily applied to BIM integrated computer vision.
This paper presents a novel system for the detection and measurement of framing elements using a depth sensing computer vision (RGB-D) camera. The system described in this paper is intended for use with construction robots designed to assist in framing operations such as assembly or sheathing. The output of the system is a set of vertices representing the geometry of a framing structure and its position relative to the RGB-D camera. The authors additionally propose a framework for cross referencing the detected geometry with building information modeling (BIM) data in order to allow for BIM directed construction robot/machine control.
The stacking tasks investigated by Dawod and Hanna (2019), Feng et al. (2015), and Tish et al. (2020) differ significantly in their requirements from the framing tasks that are the focus of this paper. All three of these prior papers implemented computer vision algorithms which measured the 3D geometry of small building elements placed very close to a depth sensor. A computer vision system for robotic automation of framing tasks must be able to measure the geometry of larger framing structures that are at least several meters from the sensor. A different approach to sensor configuration and computer vision algorithms is necessary to meet these requirements. Currently, there is very little research which addresses computer vision for framing or related tasks, and the authors seek to address this gap to support longer term research goals in construction robotics.
The goal of developing this system was to create a computer vision system capable of precisely measuring framing elements using low-cost sensors. The system was designed to function in a variety of lighting and background conditions, and to be compatible with modern BIM formats such that it can be applied easily to construction robots.
The computer vision system proposed by the authors was tested in an indoor environment using a set of three test frames constructed using standard 2x4 pine boards. The algorithm was able to detect the geometry of these frames with an average accuracy of +/-5.386 cm. Due to increased -3 -noise in the data provided by the RGB-D sensor, the computer vision system was not able to function successfully in an outdoor environment.
The proposed computer vision system measures the geometry of a single surface of a framing structure. Since a frame is normally constructed from uniformly shaped beams, this provides enough information to fully define the geometry of the structure. This information is extracted from the depth image produced by an RGB-D camera. In order to extract the geometry of the surface of a frame, the computer vision system grows a plane outward from a user selected point in a depth image, and then approximates the extracted plane as a set of polygons.
An RGB-D camera provides two separate datasets: an RGB image containing the color of each pixel in the image, and a depth image containing the distance of each pixel from the camera in the z (depth of view) direction. The coordinates of each depth pixel in three-dimensional space relative to the camera can be readily extracted from the depth image (Intel Corporation 2021). Data from the RGB image produced by the camera is not used by the proposed algorithm, except for facilitating the display of outputs. The RGB data is more sensitive to changes in lighting conditions, and is less useful for object identification than depth data due to the uniform color of most framing elements.
Depth data from the RGB-D camera is processed in several steps to extract the geometry of a framing structure. The first step is to preprocess the depth data by applying spatial and temporal filtering. Multiple depth images taken over a short span of time are combined together and then smoothed spatially to reduce noise. Raw depth data received from the camera contains a significant amount of noise, especially in the time domain, so this filtering is necessary to create a smooth and accurate image for further processing steps.
The second step is to extract the plane of the framing structure from the depth image. The plane extraction algorithm starts by computing the normal vector of the selected point and its nearest neighbors. It then grows a surface outward from that point, adding additional points to the surface if the normal vector of the plane formed by those points and their nearest neighbors is sufficiently close to the normal vector of the surface. The algorithm halts when it can no longer find additional valid points to add to the surface.
The plane extraction algorithm proposed by the authors is conceptually similar to approaches presented in prior literature. However, this algorithm differs from those presented by Li et al. (2017), Feng et al. (2014), and Wang et al. (2015) because it operates on the depth image directly, rather than a point cloud generated from the depth image, allowing for a simpler codebase and reduced computational complexity. This approach may provide slightly reduced performance ,
• Suggested Citation: Lacny, C., and Zhang, J. (2022) -4 -but the error tolerance in construction jobs may also be higher than other more precise industries (e.g., integrated circuit manufacturing).
The final step required is to measure the geometry of the extracted plane. Once the plane of interest is extracted from the depth image, the geometry is smoothed out using a filter in order to provide additional tolerance for noise in the input dataset. The shape of the plane of interest is then approximated as a polygon, and the positions of the vertices of that polygon are found in three-dimensional space. These points are then projected into a local camera coordinate system to facilitate a position independent comparison with the expected geometry. The algorithm outputs the shape of the framing structure as a list of connected vertex positions in three dimensional space. A visualization of the major steps in this algorithm is shown in Figure 1.
All of the code developed to implement this algorithm was written using OpenCV, Python, and the Intel RealSense libraries (Python Software Foundation 2021a,b; Intel Corporation 2021). This set of open-source libraries allowed for much more efficient development of the computer vision codebase by conveniently implementing basic functions such as spatial filtering or extracting a 3D coordinate from a depth image. Python was chosen as a programming language to facilitate rapid prototyping and easy modification of code due to its simplified syntax.
A computer vision system utilizing the algorithm described above was subjected to a series of realworld tests using an Intel RealSense D455 RGB-D camera and three test frames constructed using standard pine 2x4 boards. An image showing the geometry of each of the test frames is contained in Figure 2. Testing was performed by placing the computer vision camera 1 meter away from each text frame, perpendicular to the surface of the frame. A point on the surface of interest was then selected by a human operator, and the geometry detected by the computer vision system was compared to the known geometry of the frame. The results from this testing are summarized in Figure 3. Error was computed by measuring the Hausdorff distance between the sets of vertices produced by the computer vision algorithm and the known measurements of the frames. The Hausdorff distance essentially represents the worst-case error between the expected position of a point and its measured position in the output dataset. This method of computing error was used because it has been shown to be effective in assessing object detection accuracy (Dubuisson and Jain 1994). The standard Hausdorff distance was used, rather than the modified form proposed by Dubuisson and Jain (1994), as the computer vision algorithm produces a set of points that does not contain significant noise or outliers. The average error between the expected and measured -6 -location of vertices for each frame is also provided, however, when the authors refer to the 'accuracy' of the algorithm, they are referring to the Hausdorff distance or 'worst case' error.
In an indoor environment, the computer vision system was able to measure the geometry of the 0.8x0.8 m rectangular frame with an accuracy of +/-2.844 cm, the 1.6x0.8 m frame with an accuracy of +/-7.930 cm and the 0.525 m pentagonal frame with an accuracy of +/-5.385 cm. This level of accuracy for our initial testing is fairly promising, and represents less than 10% error for the positions of points in both the local coordinate system and in the camera's coordinate system. Between the three frames, this represents an average accuracy of +/-5.386 cm.
The outputs shown in Figure 3 demonstrate that the algorithm was able to successfully extract the shape of each of the test frames and extract the positions of each of the frames' vertices with a reasonably high level of accuracy. This level of performance is likely acceptable for object detection and matching applications, but falls short of what is necessary for real-time robotic control. However, adding redundancy could further increase the level of accuracy which may still allow sensing systems using such technology to outbid classic LiDAR systems. In addition, this technology could be used in conjunction with other sensing data to cross validate.
Limited testing was also performed in an outdoor environment; however, the depth camera was unable to produce a sufficiently accurate depth map in bright sunlight to run the plane extraction algorithm under these conditions. This is in contrast to what the documentation for the device suggested (the performance of this type of depth camera should be comparable or better in an outdoor environment relative to an indoor one) (Grunnet-Jepsen et al. 2020).
The primary issue in outdoor testing seemed to be sensor calibration. Since the algorithm proposed by the authors relies primarily on depth data from the sensor, rather than RGB data, lighting conditions do not directly affect the output. It was observed, however, that the quality of the depth image produced by the RealSense D455 camera varied significantly based on lighting conditions and camera calibration settings. In order to yield a high-quality depth map, RGB-D sensor settings such as the camera exposure and IR emitter intensity required adjustment based on lighting conditions. Although the RealSense D455 camera has built-in features to automatically adjust these settings, the automatic settings often did not yield an acceptable depth map, and manual tuning was required. This was especially apparent in outdoor environments, where lighting conditions were more variable, and the overall intensity of ambient light was higher. In an indoor environment, the camera's automatic calibration settings were sufficient to produce a high-quality depth image under various levels of illumination. In bright sunlight, the camera's automatic calibrations produced an extremely noisy depth image. A better result was achieved by manually tuning the camera's settings; however, the resulting performance still fell short of the consistency required to run the proposed algorithm.
It is likely that acceptable performance could be achieved in an outdoor environment with improved sensor calibration, however, further testing must be conducted to determine if this is the case, and to ascertain the performance of the sensor and algorithm in an outdoor environment.
For the prior dataset, the output of the computer vision system was compared to the measured parameters of the test frames physically. Data from this system could also be compared to an expected geometry extracted from a BIM file, allowing this system to match the assembly to an object in the BIM plan and verify that an assembly has been constructed within its designed tolerances. Prior work from the Automation and Intelligent Construction Lab that focused on extracting generalized geometric information from BIM data provides a useful template for implementing this functionality (Akanbi et al. 2020;Zhang 2018;Zhang and El-Gohary 2015), and this is an important long-term focus of work by the authors on this subject. For example, one possible workflow is proposed as follows (Figure 4):
The coordinate system in an input BIM that depicts the plan and design could be used as the basis. In other words, the origin, x-axis, y-axis, and z-axis used by the input BIM will serve as the origin, x-axis, y-axis, and z-axis of the operational space of the construction system (e.g., a construction robot). The algorithm demonstrated above processes sensed physical building components into camera coordinates (coordinates with respect to the camera used), which will be further converted into coordinates in the as-planned BIM environment. This could be achieved through classic scaling, translation, and rotation transformations through 4x4 matrices operations (Golparvar-Fard et al. 2011). The transformations could be conducted indirectly through referencing known points that are common both on site and in BIM (e.g., staked reference points). We propose to use the industry foundation classes (IFC) format to represent the BIM, which can be exported/generated from almost all major BIM authoring tools such as Autodesk Revit, Bentley AECOSim, Trimble SketchUp, and GraphiSoft ArchiCAD. Furthermore, the data-driven reverse Akanbi et al. (2020) will be used to generate robust extraction algorithms for extracting cartesian points that represent the corresponding building elements in BIM (note that the IFC model in this case needs to be at a high level of development/detail (LOD), usually at LOD 400 and above). Figure 5 shows one example trace that could be identified following the D-READ method to extract cartesian points, for 3D geometries represented using the Boundary Representation (Brep) method. Through comparing the as-planned cartesian points and the as-constructing (i.e., being built) coordinates of the building elements, path planning for construction robot or other construction machines could be performed.
Although the performance of the computer vision system presented in this paper is promising as shown by the initial experimental results, the use cases of the system are currently limited. The vision system is specifically designed to measure framing structures, and may need further deliberation to adapt to other use cases. Additionally, the computer vision camera requires calibration for different lighting conditions, so it could only be deployed for a very specific purpose in its current form (i.e., interior wall studs and panel construction).
The most significant limiting factor on the performance of this computer vision algorithm is the performance of the current generation of inexpensive RGB-D cameras. Although the Intel RealSense D455 camera used for testing the system has a reasonably high depth resolution, the depth image produced by the camera is prone to noise and holes, especially at large distances, limiting its utility for object detection and measurement. Measuring the geometry of a frame to facilitate a robot placing material on it requires a very high level of precision relative to other tasks that these types of cameras have regularly been used for, such as robot localization. Higher performance depth sensors based on LiDAR could likely achieve significantly improved accuracy using a similar algorithm and reduce the need for recalibration based on lighting conditions.
Another possible approach to improving the performance of the system would be combining data from multiple sensors. Multiple depth sensing cameras could be placed in different positions relative to a framing structure with data from each camera combined into a single point cloud. This would provide a significantly enhanced field of view, allowing the system to measure larger structures, as well as providing a higher density of data in areas where the field of view of multiple cameras overlap. Pérez et at (2016) additionally suggest that data from a depth sensing camera can be combined with data from a high-definition camera to produce a denser depth map. The performance of the plane extraction algorithm used by the computer vision system could likely be improved by implementing a more advanced mathematical approach, such as the methods proposed by Feng et al. (2014) and Li et al. (2017). An approach based around neural networks may also yield superior performance, however, more research would be required to determine if this is the case. Schwarz et al. (2017) suggest that machine learning techniques have been shown to achieve a higher level of performance than procedural approaches for the general problem of image segmentation and object detection.
Current implementations of the proposed computer vision system are only capable of accurately measuring the geometry of relatively small test frames at distances on the order of a few meters. At larger distances, noise in the depth image may overwhelm the ability of the plane extraction algorithm to successfully extract the plane of interest, and error in the depth data from the RGB-D camera used scales approximately with distance squared (Grunnet-Jepsen et al. 2020) .
• Suggested Citation: Lacny, C., and Zhang, J. (2022) -10 -Developing a computer vision system for measuring framing elements on the scale of a construction site would likely require further improved algorithms, better sensor and a system with multiple sensors.
In this paper a novel computer vision system for mapping the geometry of framing structures was presented. The algorithm was tested with a series of test frames in an indoor environment, yielding an average measurement accuracy of +/-5.386 cm. The performance demonstrated by the algorithm is likely sufficient for object recognition and matching with BIM data, however it falls short of the accuracy required for most real-time robotic control applications. In order to fulfill its desired role in construction robotics/machine guidance, the performance of the computer vision system will likely need to be further improved using enhanced sensors and more advanced algorithms and configurations.