<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Point-based Instance Completion with Scene Constraints</title></titleStmt>
			<publicationStmt>
				<publisher>OpenReview</publisher>
				<date>01/02/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10695689</idno>
					<idno type="doi"></idno>
					
					<author>W Khademi</author><author>F Li</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Recent point-based object completion methods have demonstrated the ability to accurately recover the missing geometry of partially observed objects. However, these approaches are not well-suited for completing objects within a scene, as they do not consider known scene constraints (e.g., other observed surfaces) in their completions and further expect the partial input to be in a canonical coordinate system, which does not hold for objects within scenes. While instance scene completion methods have been proposed for completing objects within a scene, they lag behind point-based object completion methods in terms of object completion quality and still do not consider known scene constraints during completion. To overcome these limitations, we propose a point cloud-based instance completion model that can robustly complete objects at arbitrary scales and pose in the scene.To enable reasoning at the scene level, we introduce a sparse set of scene constraints represented as point clouds and integrate them into our completion model via a cross-attention mechanism. To evaluate the instance scene completion task on indoor scenes, we further build a new dataset called ScanWCF, which contains labeled partial scans as well as aligned ground truth scene completions that are watertight and collision-free. Through several experiments, we demonstrate that our method achieves improved fidelity to partial scans, higher completion quality, and greater plausibility over existing state-of-the-art methods.RfD-Net DIMR Ours]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>Figure <ref type="figure">1</ref>: Visual comparison of completion results. Our approach is better at recovering missing geometry, avoiding collisions, and preserving observed surfaces and known free space.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">INTRODUCTION</head><p>Object modeling from sensor observations is an increasingly prominent problem for robot interaction within real environments. However, sensors such as LiDAR and depth cameras provide us with only partial observations of objects in a scene. Even with dense scanning and reconstruction, many objects in a reconstructed scene are left incomplete, as parts of their geometry were completely occluded during the scanning process. With a lack of knowledge about the full 3D geometry of objects, the performance of a robot on downstream tasks such as navigation and grasp planning may be hindered because of limited capabilities to infer the center of gravity and other physical properties of objects.</p><p>Point-based 3D object completion has been researched for several years starting from <ref type="bibr">Yuan et al. (2018)</ref>. In recent years, the performance on this task has improved significantly by leverag-ing encoder-decoder architectures, which first produce a coarse completion, commonly known as seeds <ref type="bibr">(Xiang et al., 2021;</ref><ref type="bibr">Zhou et al., 2022)</ref> and then upsample it hierarchically. However, most of these work operate in canonically aligned coordinate systems -i.e. setting the center of the complete object as having 0 coordinates and a scale of 1, and rotated so that a designated "front" side will always be along the z direction. This is also applied to the partial inputs so that they match the normalization and pose of the completed object. These are unrealistic assumptions when objects are presented in scenes, but they indeed simplify the problem so that, in some sense, algorithms only have to match the partial object to complete objects in the training set, then directly generate the coordinates for the completion, as the canonical coordinates of the complete object will always be the same no matter which part is used as input.</p><p>On the other hand, semantic scene completion <ref type="bibr">(Song et al., 2017;</ref><ref type="bibr">Liu et al., 2018;</ref><ref type="bibr">Li et al., 2019;</ref><ref type="bibr">Dong et al., 2023)</ref> has been studied heavily, but they only provide voxel labels without identifying objects. <ref type="bibr">Hou et al. (2020)</ref> proposed the instance scene completion task, which involves detecting individual object instances in the scene and completing the missing regions of these objects. There has been some work on this front <ref type="bibr">(Nie et al., 2021;</ref><ref type="bibr">Tang et al., 2022;</ref><ref type="bibr">Li et al., 2023a)</ref>, but the network structures have not been explored as much as the point-based object completion task.</p><p>In this paper, we explore whether the more refined network structures of point-based object completion models can be adapted to the instance scene completion task. We note two hurdles to this goal, the first is to lift the canonical coordinates assumption so that the completion model no longer has to know the scale and pose of the full object. The second is to make the instance completion to be aware of potential scene constraints that can be deduced from visibility. For example, if one sees a surface, then the network should not add points on this viewing ray. Besides, if there is another object already present at some location, then the completion should not collide with those areas.</p><p>To solve these two difficult problems, we significantly improve the point-based completion framework mainly by adopting a more sophisticated seed generator. First, we change the seed prediction to two parts, predicting an object center location and seed offsets from the center. We made several architectural improvements to improve performance in this setup to match the completion performance when using canonical coordinates. Next, to provide our completion model with scene context, we introduce a set of sparse constraints which encode known information about the scene. Unlike dense TSDFs, our constraints are represented as two bounding shells of the underlying surface, indicating the transition boundary from surface to known free space and surface to known occluded space. We integrate these constraints into our seed generator, enabling our model to reason about completions which are plausible in the context of the observed scene. Using this additional scene information, we find that our approach produces fewer collisions between predicted completions. Furthermore, existing datasets for the instance scene completion task suffer from errors in the ground truth data, making evaluation on them unreliable. Scan2CAD <ref type="bibr">(Avetisyan et al., 2019a)</ref> lacks alignment between real partial scans from ScanNet and synthetic ground truth meshes from ShapeNet, causing a trade-off between respecting the partial scan and matching the ground truth mesh. Scan-ARCW <ref type="bibr">(Li et al., 2023a)</ref> contains collisions in their ground truth data, making it difficult to measure the plausibility of scene completion with collision metrics. We introduce a new dataset called Scan-WCF addressing the issues present in these existing datasets. On our newly proposed dataset we demonstrate that our approach outperforms existing works in terms of both partial reconstruction quality and completion quality while producing less collisions between predictions.</p><p>In summary, our main contributions are as follows:</p><p>&#8226; We propose a novel object-level completion model which is robust to the scale and pose of objects found within scenes. &#8226; We integrate a sparse set of scene constraints into our model to provide our object completions with scene context (e.g., other observed surfaces, free space, occluded space). &#8226; We build a new dataset for instance scene completion on indoor scenes, containing partial scans and watertight ground truth meshes that are aligned, labeled, and collision free.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">RELATED WORK</head><p>The 3D completion literature can be roughly broken down into two main categories: (1) object-level completion and (2) scene-level completion.</p><p>Object-level Completion 3D object completion aims at recovering the complete geometry of an object given some partial observation. With the introduction of point cloud architectures, <ref type="bibr">Yuan et al. (2018)</ref> was the first to propose a fully point cloud based completion architecture. Since then, there have been many follow-up works which leveraged increasingly better encoder-decoder architectures <ref type="bibr">(Tchapmi et al., 2019;</ref><ref type="bibr">Liu et al., 2020;</ref><ref type="bibr">Wen et al., 2020;</ref><ref type="bibr">Huang et al., 2020;</ref><ref type="bibr">Wen et al., 2021;</ref><ref type="bibr">Yu et al., 2021)</ref>. In recent years, a popular choice has been to first produce a coarse completion, often referred to as seeds, and then upsample the completion in a hierarchical fashion <ref type="bibr">(Xiang et al., 2021;</ref><ref type="bibr">Zhou et al., 2022;</ref><ref type="bibr">Chen et al., 2023;</ref><ref type="bibr">Khademi &amp; Fuxin, 2024)</ref>. However, all these approaches operate in canonically aligned coordinate systems and thus are not suited for completing objects in the context of scenes where pose and scale are arbitrary. SCARP <ref type="bibr">(Sen et al., 2023)</ref> is a point cloud completion method that aims to be robust to arbitrary pose by predicting the completion in a canonical frame along with the pose needed to transform the completion back to its true coordinate system. Unlike SCARP, our method considers the scene context around the object as we are interested in the scene completion task. Additionally, our approach does not require estimating the pose of the object, avoiding possible misalignment with the partial scan due to inaccuracies in pose estimation.</p><p>Scene-level Completion Scene-level completion has heavily been studied as a joint task of semantically labeling the scene while recovering the geometric structures that are missing from it. For indoor environments, this task has been studied almost exclusively as predicting the semantic label and occupancy of each voxel in a dense voxel grid <ref type="bibr">(Song et al., 2017;</ref><ref type="bibr">Liu et al., 2018;</ref><ref type="bibr">Li et al., 2019;</ref><ref type="bibr">Chen et al., 2020;</ref><ref type="bibr">Cai et al., 2021;</ref><ref type="bibr">Dong et al., 2023;</ref><ref type="bibr">Wang et al., 2024)</ref>. These methods do not assign voxels to object instances in their completion, and use dense 3D convolutions to produce predictions on low-resolution voxel grids, prohibiting the representation of fine-grained geometry typically present in indoor environments.</p><p>Avoiding the need for dense occupancy and semantic prediction, some methods produce completions of partial scans by performing CAD model retrieval and alignment <ref type="bibr">(Avetisyan et al., 2019b;</ref><ref type="bibr">Ishimtsev et al., 2020)</ref>. However, these approaches require the existence of a CAD pool, have to search for the nearest CAD model for each object in the partial scan, and potentially require an additional optimization for roughly aligning the models to the scan.</p><p>To improve the level of scene understanding required for object interaction within environments, <ref type="bibr">Hou et al. (2020)</ref> proposed the instance scene completion task, which involves detecting individual object instances in the scene and completing the missing regions of these objects. Their approach relies on producing occupancy predictions and semantic labels for a dense voxel grid using 3D convolutions, limiting the resolution of scenes due to memory footprint. More recent instance scene completion methods operate directly on point clouds. RfD-Net <ref type="bibr">(Nie et al., 2021)</ref> generates instance proposals via 3D object detection and completions via an implicit function. DIMR <ref type="bibr">(Tang et al., 2022)</ref> trains a 3D instance segmentation model for proposal generation and produces completions from latent codes with a pre-trained shape generator. DDIT <ref type="bibr">(Li et al., 2023a)</ref> performs 3D instance segmentation and deforms deep implicit shape templates into completions conditioned on shape latent codes extracted from the segmented partial objects. However, DDIT requires an iterative procedure for estimating object pose and a per-scene optimization step for respecting the partial input which is slow. PaSCo <ref type="bibr">(Cao et al., 2024)</ref> performs panoptic scene completion on LiDAR scans of outdoor scenes. However, they employ a dense 3D CNN in part of their network, making their work infeasible for our indoor scenes due to memory constraints (our 2cm resolution indoor scans would require &#8764; 4 -5&#215; more voxels than the outdoor scans with 20cm voxels used in their work).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">METHOD</head><p>We present an overview of our architecture in Figure <ref type="figure">2</ref>. Given a partial scan of a scene, we run a state-of-the-art 3D instance segmentation method Mask3D <ref type="bibr">(Schult et al., 2023)</ref> to decompose our scene into a set of objects. The partial information of each object is first encoded into a set of local features and a global shape descriptor. We then introduce a novel seed generator which generates a coarse representation of the complete shape, called Patch Seeds, from our partial encoding. Our generator produces Patch Seeds as offsets of the object's predicted center, which we find to be more robust to change in pose compared to existing Patch Seed generators. To provide our object completions with scene context, we additionally integrate a set of scene constraints into our seed generator via cross-attention. Generated Patch Seeds are then decoded into a dense completion in a</p><p>Stage I: Instance Segmentation Stage II: Object Completion Stage III: Mesh Reconstruction Partial Encoder Seed Generator Scene Constraints Segmented Scene Object Instances Free Space Points Occluded Space Points Normals Reconstructed Meshes Instance Completions Downsampled Points Local Features Seed Coordinates Seed Features Completion &amp; Normals Partial Input Coarse-to-Fine Decoder Upsample Upsample Upsample hierarchical fashion by applying a series of upsampling layers. We design our upsampling layer to contain both local and global attention, which helps with producing a globally coherent completion capable of representing fine-grained geometry. At the densest completion level, we additionally leverage a local transformer to predict surface normals, allowing us to reconstruct meshes of our completions using an off-the-shelf surface reconstruction method. Finally, we place each mesh back into the world frame, producing a scene of labeled object instances with complete geometry. In the following sections, we describe the key components of our completion model in more detail.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">PARTIAL ENCODER</head><p>Our partial encoder takes as input the partial object instance P &#8712; R M &#215;3 and estimated surface normals N &#8712; R M &#215;3 to produce a downsampled set of points P l &#8712; R M l &#215;3 with local features F l p &#8712; R M l &#215;C local and a global shape descriptor f p &#8712; R C global from the object. We base our encoder on the design proposed by <ref type="bibr">Khademi &amp; Fuxin (2024)</ref>, which is in turn based on <ref type="bibr">Zhou et al. (2022)</ref>. The encoder consists of l downsampling blocks, where at each block the point set is first downsampled, followed by a series of point convolutions for interpolating and aggregating features for the downsampled point set. The final downsampled set of features are then passed through an MLP followed by a max-pooling operation to produce global shape descriptor f p .</p><p>The main improvement we made to the partial encoder is that we replace the PointConv <ref type="bibr">(Wu et al., 2019)</ref> layers in <ref type="bibr">Khademi &amp; Fuxin (2024)</ref> with VI-PointConv <ref type="bibr">(Li et al., 2023b</ref>) layers and additionally use estimated normals as input. The convolution filters generated in PointConv are only translation invariant. To be robust to rotation and scale, the network would have to be shown the same object under many different rotations and scales and have enough capacity to encode appropriate filter weights for each of them. On the other hand, the convolution filters in VI-PointConv are generated from a mix of non-invariant, scale-invariant, and rotation-invariant position embeddings. In this way, the MLP generating filter weights can potentially learn to ignore the non-invariant position embeddings to share filter weights across neighborhoods of different scales and rotations, increasing robustness of the network. Furthermore, using both XYZ coordinates and surface normals provide a better description of local surface geometry than XYZ coordinates alone.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">SEED GENERATOR</head><p>Given the downsampled point set P l , locally extracted features F l p , and global shape descriptor f p , our seed generator is tasked with producing a set of Patch Seed coordinates S &#8712; R M seed &#215;3 and features F seed &#8712; R M seed &#215;C seed which represent a coarse encoding of the complete shape.</p><p>SeedFormer <ref type="bibr">(Zhou et al., 2022)</ref> produces Patch Seeds through a local attention-based upsampler followed by an MLP to regress the seed coordinates. In Table <ref type="table">3</ref> we show that such an approach suffers a significant drop in completion quality if we attempt to use it to complete objects not canonically aligned along an axis. Instead, we design a global attention-based seed generator, which allows the generator to use information from the entire scene to predict the center of the object before producing seed coordinates as offsets of the object center. Our full architecture is shown in Figure <ref type="figure">3</ref>.</p><p>We introduce a learnable token o token &#8712; R C which is to be decoded into the center of an object O &#8712; R 3 . Our learnable token o token along with downsampled partial information {P l , F l p } is first passed through a set of transformer blocks performing multi-head self-attention to produce an updated object center embedding o obj &#8712; R C and updated partial features F obj &#8712; R M l &#215;C . Having  <ref type="formula">2022</ref>). Our object completions additionally consider other objects in the scene through cross-attention with our known free and occluded space constraints. aggregated information from the partial input via attention, o obj is then used to regress the object center. We concatenate o obj with the global shape descriptor f p along the feature dimension before predicting the object center O with an MLP &#952;:</p><p>To reliably cover the entire object in our coarse representation, we increase the number of points present in our Patch Seeds compared to the partial points and features from which they are generated. Specifically, we "split" our output partial features F obj using a transposed convolution with a stride and kernel size of 2. Upsampled features are then passed through a multi-head self-attention block followed by an MLP &#969; to produce Patch Seed features F seed :</p><p>Finally, Patch Seed coordinates S are predicted as offsets from object center O. We use a MLP &#947; which predicts per point offsets from Patch Seed features F seed concatenated with object center token o obj and global shape descriptor f p along the feature dimension:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">SCENE-AWARE OBJECT COMPLETION</head><p>As mentioned earlier, correctly completing objects in a scene requires satisfying visibility constraints from the scene, i.e. not creating object parts that cut into other objects or grow towards the camera (violating the free space that has been seen). TSDFs are commonly used for representing known information in a scene, such as the surfaces of objects as well as known free or occluded space due to the viewpoints at which the scene was captured. However, TSDFs are typically stored as dense voxel grids, making them computationally expensive to process and not compatible with point clouds. To our knowledge, no one has attempted to address the difficult task of incorporating scene constraints to point-based completion. We propose to represent the constraint information in the scene as sparse sets of points representing known occupied and free spaces. Our goal is to input these constraint points to the network so that it can learn to avoid generating parts in those areas.</p><p>We generate our scene constraints as two bounding shells of the surface of the partial scan defined as P in &#177; &#948;N in , where P in &#8712; R Min&#215;3 is the partial scan with surface normals N in &#8712; R Min&#215;3 and &#948; &#8712; R. We further resample these surfaces to 10 cm resolution, leaving us with a sparse set of free space points P f ree &#8712; R M f ree &#215;3 and occluded space points P occ &#8712; R Mocc&#215;3 . Along with these point sets, we learn additional embeddings e f ree &#8712; R C and e occ &#8712; R C , which are shared across all free space points and occluded space points, respectively.</p><p>We provide our constraints as information to our seed generator through a set of transformer blocks containing multi-head cross-attention as shown in Figure <ref type="figure">3</ref>. Here the object's partial information is treated as the query tokens while the scene constraints represent the key-value pairs, allowing each partial object to decide what constraints to focus on when completing the object. The transformer blocks output a set of features F scene &#8712; R M l &#215;C and o scene &#8712; R C , which are directly added to the outputs F obj and o obj , respectively, before being used to predict the object's center and Patch Seeds. 3.4 COARSE-TO-FINE DECODER For upsampling, we use the upsampling layer proposed by Zhou et al. (2022). In each upsampling layer, we further introduce a series of global attention layers before their local attention-based Upsample Transformer. With our added global attention, we provide the refinement and upsampling layers with information which can encourage global coherence across the completion. Additionally, global attention allows us to potentially learn fine-grained structure for the missing geometry from the existing geometry which may not be present in the local neighborhood (e.g., through a symmetric part on the opposite side of the object).</p><p>Starting from our Patch Seed coordinates S and features F seed , we repeatedly apply our upsampling layer to generate a dense completion in a coarse-to-fine manner. At each upsampling layer j, we produce a completion C j and the corresponding upsampled features F j up with double the resolution of the previous layer. We apply 3 upsampling layers to upsample our completion from a coarse resolution of M coarse = 256 points to M dense = 2, 048 points. The completion produced at each layer is supervised by the ground truth completion C j gt which has been subsampled to the same resolution as the output resolution at layer j (we additionally treat the Patch Seed coordinates S as our coarsest resolution C 0 during training).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.5">MESH RECONSTRUCTION</head><p>We used NKSR <ref type="bibr">(Huang et al., 2023)</ref> to reconstruct meshes from point clouds. NKSR requires both point clouds and surface normals, where normals are typically estimated from the point cloud using PCA-based plane fitting. In practice, we found these normals to be overly noisy, leading to poor reconstruction quality on our completions. To address this, we introduce a normal estimation module which is jointly trained with our completion network. We first process our final completion C 3 &#8712; R M dense &#215;3 and corresponding upsampled features F 3 up &#8712; R M dense &#215;C using a modified version of the Upsample Transformer from SeedFormer <ref type="bibr">(Zhou et al., 2022)</ref>, replacing the transposed convolution used for upsampling with a regular convolution. The features output from this layer encode local surface information of the object and are directly used as input to a small MLP that regresses surface normals N 3 &#8712; R M dense &#215;3 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.6">LOSS</head><p>We use the same loss function for both pre-training the object completion model and training our scene completion model. Our overall loss objective is defined as:</p><p>where L CD is the Chamfer Distance between predicted and ground truth completions at each upsample layer j, L OCD is the One-sided Chamfer Distance between partial input and completion at each upsample layer j, L M SE is the mean squared error between predicted object center and ground truth object center O gt &#8712; R 3 , and L CS is a cosine similarity based loss between the predicted normals N 3 and ground truth normals N 3 gt . We set &#955; c = 1, &#955; p = 1, &#955; o = 1, &#955; n = 10 -2 for experiments.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">SCANWCF DATASET</head><p>Previous works evaluate the instance scene completion task on the Scan2CAD dataset <ref type="bibr">(Avetisyan et al., 2019a)</ref>. The Scan2CAD dataset is derived from both the ScanNet <ref type="bibr">(Dai et al., 2017)</ref> and <ref type="bibr">ShapeNet (Chang et al., 2015)</ref> datasets, where for each detected partial object in a ScanNet scene, the closest synthetic mesh from ShapeNet has been selected and fit to the scan to serve as the ground truth completion. The synthetic meshes are not the true completion of the partial object and their fit to the partial scan is imperfect as shown in Figure <ref type="figure">4</ref>. The lack of alignment between the input and ground truth makes the evaluation of metrics unreliable on this data. To address this, <ref type="bibr">Li et al. (2023a)</ref> proposed ScanARCW which regenerates new partial scans by rendering depth maps and semantic labels of the ground truth meshes and then backprojects this information back into 3D to generate an aligned scan. While ScanARCW addresses the alignment issue, the ground truth scenes in their dataset contain collisions. This makes it unreliable to measure scene completion plausibility with collision metrics, as the collision may be an artifact of the ground truth data itself.</p><p>To address the limitations of existing datasets, we introduce a new dataset called ScanWCF, where "WCF" refers to our ground truth scenes being "Watertight and Collision Free". Our dataset contains scenes with partial scans and ground truth complete meshes that are aligned and labeled, with each ground truth scene being watertight and collision free. We generate our scenes from data included in both the Scan2CAD and ScanARCW datasets. We use the background meshes from ScanARCW as scene boundaries (e.g., walls, floor, ceiling), filling holes in the mesh to make it completely enclosed. For selecting which objects to place in the scene, we use the Scan2CAD object matchings. Each ground truth mesh is processed to be watertight and initialized in the scene using the pose and scale from Scan2CAD. We then optimize the pose and scale of each object in the scene such that they are:</p><p>(1) closely aligned with the partial scan from ScanNet, (2) do not contain any collisions, and (3) are not floating in air. After optimization, we manually verify the scene is collision-free before including it in our dataset. If there exist minor implausibilities after optimization, we manually correct the scene; otherwise we completely discard it if too many collisions exist. After our optimization and manual verification, we find that on average only 0.14% of the points in the scene are in collision with another object, compared to an average of 2.5% of points per scene in the ScanARCW dataset. Finally, to generate aligned and labeled partial scans, we render depth maps and instance segmentation maps from a subset of camera poses in the ScanNet camera trajectory. This information is then backprojected into 3D to generate a partial scan labeled with instance information.</p><p>Our dataset contains 1202 indoor scenes based on ScanNet scenes, where 946 of the scenes are used for training and the other 246 scenes are reserved for testing. To increase the amount of training and test data, we generate 2 partial scans per scene using a different subset of camera poses from the ScanNet camera trajectory. We refer readers to our appendix for more details.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">EXPERIMENTS</head><p>In this section, we perform a variety of experiments to demonstrate the superiority of our method over existing approaches. We first evaluate our method on the task of instance scene completion, where the goal is to jointly predict object instances and their completions. Secondly, we evaluate completion quality in isolation by removing the instance prediction task and instead providing the ground truth instance information directly to each method. Across both tasks, we show that our method outperforms existing approaches both quantitatively and qualitatively. Finally, we conduct a set of ablations which validate the design choices of our architecture.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">IMPLEMENTATION DETAILS</head><p>Our object-level completion model is pre-trained on the 34 ShapeNet categories present in our dataset, excluding objects present only in the validation scenes. For augmentations, we perform random rotations about the up-axis. For optimization, we use Adam with an initial learning rate of 1 &#215; 10 -4 and linearly decay by a factor of 0.98 every 2 epochs. We train for 150 epochs with a batch size of 64 using 2 NVIDIA V100 GPUs, which takes approximately 4 days. Our scene completion model is then trained on the ScanWCF dataset, initialized from the weights of our pre-trained object completion model. For training, we use the same augmentation and optimization setup as used for pre-training. We train for 200 epochs on a single RTX 4090 GPU which takes about 3 days.</p><p>Table <ref type="table">1</ref>: Instance scene completion quality. Mean average precision (mAP) is reported for different metric@threshold.</p><p>IoU@0.25 IoU@0.5 CD@0.1 CD@0.047 LFD@5000 LFD@2500 PCR@0.5 PCR@0.</p><p>75 RfD-Net 38.78 7.25 61.68 31.32 39.18 14.65 59.56 42.36 DIMR 34.18 8.67 41.15 23.30 27.14 8.81 43.37 31.14 Ours (no pre-training) 61.99 45.13 61.48 54.67 52.03 24.57 63.71 62.69 Ours (w/ pre-training) 62.77 47.55 61.57 56.07 51.29 29.03 63.71 62.67</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">BASELINES</head><p>We evaluate our proposed approach against instance scene completion methods RfD-Net <ref type="bibr">(Nie et al., 2021)</ref> and DIMR <ref type="bibr">(Tang et al., 2022)</ref>. Both methods are trained on our ScanWCF dataset using the same hyperparameters that were used for their original experiments on the Scan2CAD dataset. We additionally retrain the pre-trained shape generator used in DIMR, but on the 34 categories from ShapeNet present in our dataset rather than the 8 ShapeNet categories it was originally trained on.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">METRICS</head><p>For the instance scene completion task, we follow <ref type="bibr">Tang et al. (2022)</ref> and compute the mean average precision (mAP) across several different metrics and at varying thresholds. Specifically, Intersection over Union (IoU), Chamfer Distance (CD), and Light Field Distance (LFD) are used for evaluating completion quality while the Point Coverage Ratio (PCR) metric is used for evaluating partial reconstruction quality. More details of those metrics are presented in the appendix. For IoU and PCR, higher thresholds are more challenging, while for CD and LFD lower thresholds are more difficult.</p><p>To remove the possibility that poor completions are caused by worse instance segmentation, we evaluate completion quality when each method uses the ground truth instance masks. For partial reconstruction quality, we compute Unidirectional Hausdorff Distance (UHD) and One-Sided Chamfer Distance between the partial scan and the predicted complete scene. To evaluate completion quality, we use the Chamfer Distance (CD) between points uniformly sampled from the scene mesh and the predicted completion. Finally, we evaluate the plausibility of scene completions by measuring collisions between predicted completions. For each object, we sample points from its completion and compute their signed distance to all the other predicted objects in the scene and the scene background mesh, penalizing points which fall inside another object mesh or outside the background mesh. We measure both the average distance of collisions (COL) and the percentage of points in collision (%COL). We scale UHD, One-Sided CD, and CD by 10 3 and scale COL by 10 4 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4">RESULTS</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Instance Scene Completion</head><p>We present results on the instance scene completion task in Table <ref type="table">1</ref>. Despite the fact that we do not jointly train our instance segmentation network together with our completion model, unlike RfD-Net and DIMR, we outperform both approaches in terms of mAP across almost all metrics and thresholds. Both RfD-Net and DIMR suffer large drop-offs in performance for all metrics in the transition from easier threshold to more difficult threshold. This suggests that these approaches can produce completions that share some similarities to the ground truth shape, but likely cannot represent fine geometric details while also suffering from inaccuracies in their prediction of an object's pose and scale. On the other hand, we see much smaller dropoffs in our method for both CD and PCR, suggesting better ability to represent the shape's surface accurately. Furthermore, the pre-training of our object completion model is not a strict requirement,</p><p>Partial Scan RfD-Net DIMR Ours Ground Truth  as we outperform both RfD-Net and DIMR without it; however, we do find that pre-training helps improve completion quality at the more challenging metric thresholds.</p><p>In Figure <ref type="figure">5</ref> we share an example of instance scene completions on a partial scan. RfD-Net struggles to represent thin geometric structures such as the base of a rolling chair, merging the wheel bases into almost a solid circular base. DIMR suffers from low-fidelity completions, with many objects appearing to be a composition of planar primitives and failing to produce thin structures such as the legs of the chairs. On the other hand, our approach can faithfully represent the fine-grained geometry present in the partial scans while producing plausible hallucinations of the missing regions (e.g., wheel bases of chairs). We share more qualitative results in Figures 9 and 10 of our appendix.</p><p>Scene Completion To isolate our evaluation of completion quality, we present results on scene completion when the ground truth instance information has been provided to each method. In this setting, poor completion quality cannot be attributed to incorrect instance predictions, enabling us to better understand the limitations of the completion network of each method. In Table <ref type="table">2</ref>, One-Sided CD and UHD measure the predicted completions average and maximal deviation from the surface of a partial scan, respectively. Both of these metrics indicate that RfD-Net and DIMR are significantly worse at respecting the partial input in comparison to our approach. Similarly, we find that our method is capable of producing higher quality completions than previous approaches, reflected by the significant gap in CD. Finally, our method does a much better job at avoiding collisions between predictions. In particular, RfD-Net and DIMR tend to produce completions that penetrate much further into other objects in the scene, producing an average collision distance (COL) which is 3 -4&#215; larger than our method. This is further demonstrated by RfD-Net and DIMR having 2 -3% more points from predicted completions either in collision or extending outside the scene boundaries compared to our approach. In Figures <ref type="figure">1</ref> and <ref type="figure">6</ref>, we qualitatively demonstrate that our approach achieves higher fidelity to the partial scan, better completion quality, and produces less collisions than previous methods. We refer readers to our appendix for more qualitative results.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.5">ABLATION STUDIES</head><p>In Table <ref type="table">3</ref>, we conduct an ablation of our object completion model on the chair category from ShapeNet to justify our proposed additions for completing objects under arbitrary pose and scale. Our baseline method is SeedFormer (Zhou et al., 2022) with their partial encoder replaced by the PointConv encoder proposed in <ref type="bibr">(Khademi &amp; Fuxin, 2024)</ref>. This replacement suffers no drop in performance and allows us to later update the PointConv layers for VI-PointConv layers in our work. When trained and tested under ideal conditions (i.e., normalizing the input by the ground truth complete shape and having the object canonically aligned along a shared axis) our baseline is capable of producing high quality completions. When our baseline is trained and evaluated on a more realistic setting (i.e., normalizing by the partial input and objects containing arbitrary pose and scale) we see that performance drops in both completion and partial reconstruction quality. We find that adding VI-PointConv <ref type="bibr">(Li et al., 2023b)</ref> to our partial encoder improves partial reconstruction quality, as the added rotation and scale invariant features likely enable better reasoning about the partial input.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>RfD-Net DIMR Ours</head><p>Figure <ref type="figure">6</ref>: Results on scene completion when ground truth instance information is provided.  Additionally, with the introduction of global attention in the seed generator and producing seed coordinates by first regressing the object center followed by predicting seeds as offsets from the center, we see the coarse and dense completion quality significantly improve over the old seed generator. Furthermore, incorporating global attention into the upsampling layers allows us to not only match our baseline's performance under ideal conditions, but actually beat it even though we are under non-ideal conditions. Finally, we find that our surface normal prediction module neither harms nor helps our completions; however, it provides us with a way to reconstruct object meshes.</p><p>In Table <ref type="table">4</ref>, we evaluate the importance of leveraging scene constraints. We observe that using scene constraints does not improve partial reconstruction quality (UHD and One-Sided CD) as the completion model does not need information about free space and occluded space to reconstruct the already observed portion of the object. However, we find that incorporating scene constraints into our model produces a 7% relative improvement in completion quality (CD) and 29% relative improvement in how far points in collision are penetrating into each other (COL). The scene constraints provide our model with information about the scene boundaries as well as other objects in the scene, which helps better constrain our completions to ones which are plausible within the scene as shown in Figure <ref type="figure">7</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">CONCLUSION</head><p>We present a novel scene completion framework which obtains state-of-the-art performance on the instance scene completion task for indoor scenes. Our proposed object completion model is robust to arbitrary pose and scale, enabling our method to produce high-quality completions with high fidelity to the partial input without having to rely on accurately estimated pose and scale parameters needed to transform the object to a canonically aligned axis. Furthermore, our proposed scene constraints enable our completion model to incorporate scene context in our completions, improving completion quality and reducing collisions between predicted completions. To evaluate our approach, we build a new dataset for the instance scene completion task on indoor scenes called ScanWCF, which contains collision-free scenes whose partial scans and ground truth meshes are aligned and labeled. Through several experiments, we demonstrate our method achieves higher completion quality, greater fidelity to the partial scan, and better plausibility over existing approaches.</p><p>Our completion framework is deterministic, meaning that our approach can only produce one completion of a partial scan. In the future, we plan to explore incorporating generative models into our completion framework in order to be able to produce multiple plausible completions of a scene.</p><p>Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmp-net: Point cloud completion by learning multi-step point moving paths. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7443-7452, 2021. Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 9621-9630, 2019. Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5499-5509, 2021. Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12498-12507, 2021. Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pcn: Point completion network. In 2018 international conference on 3D vision (3DV), pp. 728-737. IEEE, 2018. Haoran Zhou, Yun Cao, Wenqing Chu, Junwei Zhu, Tong Lu, Ying Tai, and Chengjie Wang. Seedformer: Patch seeds based point cloud completion with upsample transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III, pp. 416-432. Springer, 2022.</p><p>A APPENDIX An overview of our appendix is presented as follows:</p><p>&#8226; Model details (Section B): we provide more details of our scene completion model</p><p>&#8226; ScanWCF dataset (Section C): we provide a detailed description of our proposed dataset &#8226; Metrics (Section D): we formally define our evaluation metrics &#8226; Results (Section E): we share more results and ablations of our method B MODEL DETAILS B.1 PARTIAL ENCODER Our partial encoder consists of l = 4 downsampling blocks, where each block contains a downsampling operation on the point set followed by 2 VI-PointConv (Li et al., 2023b) layers. For point convolutions, we use a neighborhood size of 16. We design our encoder such that the final downsampled point set P 4 &#8712; R M4&#215;3 and corresponding local features F 4 p &#8712; R M4&#215;C local have M 4 = 128 points and local feature dimension size C local = 256. From downsampled points P 4 and local features F 4 p , we extract global shape descriptor f p &#8712; C global through a 2-layer MLP followed by max-pooling. We set the global feature dimension size to be C global = 512.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B.2 SEED GENERATOR</head><p>Our seed generator produces Patch Seeds with coordinates S &#8712; R M seed &#215;3 and features F seed &#8712; R M seed &#215;C seed . We define our Patch Seeds to have double the resolution of the downsampled partial information they are generated from (i.e., M seed = 256) and set the seed feature dimension size to be C seed = 256. All of the multi-head attention layers in our seed generator use the same hyperparameters, consisting of 8 heads where each head performs attention on features of dimension size 48. We define our learned embeddings o token &#8712; R C , e f ree &#8712; R C , and e occ &#8712; R C to have an embedding size of C = 384. We positionally encode downsampled partial points P 4 , free space points P f ree , and occluded space points P occ via a 2-layer MLP, mapping the points to features of the same dimension size as our learned embeddings. The object's center is predicted by a 3-layer MLP, and the seed offsets which are added to the object center are predicted via a separate 3-layer MLP.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B.3 COARSE-TO-FINE DECODER</head><p>Our decoder uses the upsampling layer proposed by SeedFormer Zhou et al. ( <ref type="formula">2022</ref>) for producing a dense completion of an object in a hierarchical fashion. We use a neighborhood size of 20 when computing local attention in the Upsample Transformer of their upsampling layer. For our added global attention layers, we once again use multi-head self-attention with 8 heads. We upsample our completion 3 times, doubling the number of points present in the completion at each upsampling layer, to produce a dense completion of M dense = 2048 points.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B.4 SCENE CONSTRAINTS</head><p>Scene constraints are generated from the input partial scan P in &#8712; R Min&#215;3 and its estimated surface normals N in &#8712; R Min&#215;3 . As described in Section C.4, our partial scans are produced by backprojecting depth maps from 10 different viewpoints and resampling the scene to a 2 cm resolution. As we are backprojecting a depth map into a point cloud, we also estimate surface normals, orienting them towards the camera the point cloud was generated from so that the normals are pointing towards "free space". With our partial points and estimated normals, we generate constraint points as P in &#177; &#948;N in , where we set &#948; = 2 cm.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B.5 3D INSTANCE SEGMENTATION</head><p>For 3D instance segmentation, we retrain Mask3D <ref type="bibr">(Schult et al., 2023)</ref> on our ScanWCF dataset, following the training procedure that they use for ScanNet. Training for 600 epochs with a batch size of 3 on scenes with 2 cm voxelization takes approximately 4 days on a single RTX 4090 GPU. We note that the instance segmentations produced by Mask3D are only used during inference. During the training phase of our completion model we use the ground truth partial object instances.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B.6 MESH RECONSTRUCTION</head><p>Meshes of our object completions are only reconstructed at inference time. For this, we directly use NKSR's <ref type="bibr">(Huang et al., 2023)</ref> pre-trained kitchen-sink model, which has been jointly trained across several object and scene scale datasets.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C SCANWCF DATASET</head><p>Our proposed ScanWCF dataset is composed of data gathered from the ShapeNet <ref type="bibr">(Chang et al., 2015)</ref>, ScanNet <ref type="bibr">(Dai et al., 2017)</ref>, Scan2CAD <ref type="bibr">(Avetisyan et al., 2019a)</ref>, and ScanARCW <ref type="bibr">(Li et al., 2023a)</ref> datasets. All data was obtained directly through publicly available download links on their websites and permission to use the data was granted for datasets which required it.</p><p>The ScanWCF dataset consists of 1202 scenes, which are based off scenes from the ScanNet dataset. Our dataset does not contain all 1513 scans from ScanNet as it relies on the background meshes from the ScanARCW dataset, which are only provided for 1274 of the scenes. Additionally, the scenes in our dataset must pass a manual verification process of which 72 out of the 1274 scenes failed. Despite our dataset not containing all the scenes from ScanNet, we reuse the scene id train/test split from the original dataset, resulting in 946 training scenes and 246 test scenes. Objects from 34 different categories from ShapeNet are present across all the scenes in our dataset. While we train each method with all the objects present in the training scenes, we only evaluate the instance scene completion task on the 13 categories which had more than 150 training examples present across all the scenes. We chose this as we observed poor instance segmentation performance across all methods on categories which contained a small number of training examples.</p><p>In the following sections, we describe the data generation process for our proposed ScanWCF dataset, which is both Watertight and Collision Free (hence the name ScanWCF).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.1 PRE-PROCESSING FOR WATERTIGHTNESS</head><p>The objects provided in the ShapeNet dataset are not guaranteed to be watertight, making it difficult to reason about collisions between objects. To address this, we process ShapeNet objects for wa-tertightness using the method proposed by <ref type="bibr">Wang et al. (2022)</ref>. This enables us to obtain the signed distance of any point with respect to the object mesh, making collision checking easy. To reason about collisions with scene boundaries (e.g., walls, floor, ceiling), we use the background meshes from the ScanARCW dataset. While the background mesh for each scene aligns with the corresponding scan from ScanNet, some of the background meshes have holes in them where window or door meshes would be placed. Unlike ScanARCW, we leave out the door and window meshes and instead fill the holes in the background meshes to make them watertight.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.2 OPTIMIZING SCENE LAYOUT</head><p>Our goal is to be able to produce a scene layout containing meshes which closely align with objects in a ScanNet scene while being free of collisions. In order to do so, we begin by initializing our scene as the empty watertight background mesh which is already aligned with a real scan from ScanNet. We then proceed to place one object into the scene at a time, optimizing for both the pose and scale of the object. We make use of the Scan2CAD dataset for deciding which ground truth meshes from ShapeNet to place in the scene, as the dataset has already detected objects in the real ScanNet scenes and matched them to their closest mesh in ShapeNet. Additionally, we initialize each object's pose and scale using the Scan2CAD annotations as they have roughly aligned the synthetic meshes to the real scans. Then we optimize the pose and scale of each object such that the following criteria are best met:</p><p>1. Alignment with partial scan. We minimize the Chamfer Distance between points sampled on the ground truth mesh and the partial object it was matched to within the real ScanNet scene. 2. Minimize amount of collisions. We uniformly sample points on the object mesh and compute their signed distance to the scene mesh (both objects and background). We penalize points which fall inside another object or outside the scene boundaries. 3. Minimize large changes in scale. We penalize large deviations in object scale from their initialized value. This prevents the optimization from shrinking the object scale by large amounts to reduce collisions. 4. Minimize floating objects. Minimizing the amount of collisions while penalizing changes in scale can lead to the optimization producing objects floating in space. To help prevent this, we sample points on the bottom of the object mesh and compute their signed distance to the scene mesh. We penalize values which deviate from 0, which signifies the object is not resting on a surface.</p><p>After an object's pose and scale has been optimized, it is added to the ground truth scene mesh to provide further constraints for the objects which will later be optimized.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.3 MANUAL VERIFICATION</head><p>Not all scene layouts are guaranteed to be free of collisions upon the optimization phase finishing. Therefore, after optimizing for the scene layout, we manually verify that the scene is plausible. If no objects are in collision with each other and the scene looks plausible (e.g., no floating objects), we include the scene in our dataset. If the scene includes some objects which are in collision or are implausible, we manually correct the issue and include the scene in our dataset. If too many collisions still exist after our optimization step, we simply disregard the scene from being included in the dataset.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.4 PRODUCING ALIGNED PARTIAL SCANS</head><p>After the manual verification phase, we have a set of scenes which serve as the ground truth instance completions for our dataset and now need to generate partial scans which are aligned to them. To generate partial scans of the scene, we render depth maps from different viewpoints and backproject them back into 3D. For a particular scene id, we make use of the camera extrinsics and intrinsics from the corresponding scene id in the ScanNet dataset to render 2D information with. Using PyTorch3D <ref type="bibr">(Ravi et al., 2020)</ref>, we render depth maps, 2D instance segmentations, and surface normal maps of the scene from each viewpoint. We share some example renderings of a scene in Figure <ref type="figure">8a</ref>. Upon rendering 2D information, we select 10 viewpoints per scene and backproject their 2D information into world coordinates, fusing together the different point clouds to produce a partial scan as shown in Figure <ref type="figure">8b</ref>. While backprojecting the depth map of each viewpoint into a point cloud, we also estimate surface normals and orient them such that they are correctly oriented towards the camera.</p><p>To keep the number of points in the scene reasonable, we resample each partial scan using grid subsampling with a 2 cm resolution. Finally, To increase the amount of training/test data, for each scene, we choose to generate 2 partial scans using a different subset of 10 ScanNet camera poses.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D METRICS D.1 SCENE COMPLETION METRICS</head><p>For the scene completion task, we have removed the need to predict instance proposals from each method. Instead we provide each method with the ground truth instance segmentations so that each network is only responsible for completing the partial object. In this setting, false positive detections/proposals are not possible and therefore we do not have to rely on the use of mean Average Precision (mAP) as a way to evaluate completion quality. Instead we can evaluate the completion quality of each approach using common metrics from the point cloud completion literature. Each metric used in the scene completion task is defined in the following sections.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.1 CHAMFER DISTANCE (CD)</head><p>The Chamfer Distance between two point clouds P &#8712; R N &#215;3 and Q &#8712; R M &#215;3 can be defined as:</p><p>(5)</p><p>We use Chamfer Distance as a measure of completion quality. For evaluation, we sample 2048 points per predicted object and concatenate each object's point set into a single scene point cloud.</p><p>We do the same procedure for the ground truth objects, uniformly sampling the 2048 points from each object mesh. Chamfer Distance is then measured between the predicted scene point cloud and ground truth scene point cloud using Equation <ref type="formula">5</ref>. Numbers reported in tables have been scaled by a factor of 10 3 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.2 ONE-SIDED CHAMFER DISTANCE</head><p>Equation 5 measures the bi-directional (or symmetric) Chamfer Distance. To evaluate how well the completion respects the partial input, we instead measure the One-sided Chamfer Distance. The One-sided Chamfer Distance between two point clouds P &#8712; R N &#215;3 and Q &#8712; R M &#215;3 can be defined as:</p><p>We measure the One-sided Chamfer Distance between the 2 cm resolution partial scan excluding the background (i.e., points on the walls, floor, ceiling) and our predicted scene point cloud. Numbers reported in tables have been scaled by a factor of 10 3 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.3 UNIDIRECTIONAL HAUSDORFF DISTANCE (UHD)</head><p>One-sided Chamfer Distance measures the average deviation of the partial reconstruction from the partial input. We additionally measure the maximum deviation of the partial reconstruction from the partial input using the Unidirectional Hausdorff Distance (UHD). The Unidirectional Hausdorff Distance between point clouds P &#8712; R N &#215;3 and Q &#8712; R M &#215;3 can be defined as:</p><p>Similar to One-Sided Chamfer Distance, we measure UHD between the 2 cm resolution partial scan with background removed and our predicted scene point cloud. Numbers reported in tables have been scaled by a factor of 10 3 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.4 COLLISION METRIC (COL)</head><p>We measure completion plausibility by how badly predicted completions collide with each other or the scene boundaries. Let the watertight background mesh (i.e., walls, floor, ceiling) be denoted as B and the set of predicted watertight object meshes be denoted as {M 1 , ..., M n }. We define SDF (M, p) to represent the signed distance of an arbitrary 3D point p to its closest point on the surface of a mesh M, where SDF (M, p) &lt; 0 implies the point p falls inside mesh M. Additionally, we denote our set of point clouds which we reconstructed meshes from as {C 1 , ..., C n } (for RfD-Net and DIMR we sample 2048 points uniformly from a mesh M i to produce point cloud completion C i ). Each object point cloud C i is composed of a set of 3D points, where we denote a point being in the point cloud as p &#8712; C i . Now we can define the collision metric for a scene as:</p><p>If no points in the scene completion are violating scene constraints (i.e., penetrating into other objects or the scene boundaries), the collision metric is 0. Otherwise, the collision metric is equal to the average of how far each point has penetrated into an object or extended outside the scene boundaries.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2 INSTANCE SCENE COMPLETION METRICS</head><p>To evaluate our proposed approach on the joint task of instance segmentation and object completion, we use the metrics and setup proposed by <ref type="bibr">Tang et al. (2022)</ref>. In particular, we compute the 3D detection mean Average Precision (mAP) for each metric defined in the following sections. For computing average precision, the PASCAL VOC 2007 11-point interpolation method is used.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.1 INTERSECTION OVER UNION (IOU)</head><p>Intersection over Union is a voxel-based approach for evaluating completion quality. It measures the predicted completion's voxel occupancy against the ground truth completion's voxel occupancy. To compute it, both predicted meshes and ground truth meshes are first voxelized with a fixed voxel size of 4.7 cm. Then Intersection over Union is computed as the Volume of overlap Volume of union with regards to voxel occupancy.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.2 CHAMFER DISTANCE (CD)</head><p>Chamfer Distance provides a point-based evaluation of completion quality. It measures the distance from points sampled on the surface of the predicted mesh to points sampled on the ground truth mesh. To generate point sets, we uniformly sample 4096 points from both the predicted mesh and the ground truth mesh. Chamfer Distance is then computed using Equation <ref type="formula">5</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.3 LIGHT FIELD DISTANCE (LFD)</head><p>Light Field Distance is a visual similarity metric for meshes. The main idea is that if the predicted complete mesh is similar to the ground truth mesh then it should look similar from all viewpoints. To compute LFD, each predicted mesh and ground truth mesh is rendered into 2D images from multiple viewpoints and encoded into a light field descriptor that is used for measuring distances between meshes. We refer readers to the work by <ref type="bibr">Chen et al. (2003)</ref> for a description of the light field descriptor.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.4 POINT COVERAGE RATIO (PCR)</head><p>Point Coverage Ratio measures how well the completed mesh aligns with the partial input. The Point Coverage Ratio between a point set P &#8712; R N &#215;3 and a mesh M can be defined as:</p><p>where 1 is the indicator function, dist(x, M) is the distance from the point p to the surface of mesh M, and &#964; pcr is a distance threshold (we set &#964; pcr = 0.047 for evaluation).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3 SURFACE NORMAL METRICS</head><p>In section E.8, we evaluate the quality of our surface normal predictions needed to reconstruct a mesh. Rather than directly evaluate accuracy of the predicted surface normals, we evaluate the quality of the reconstructed meshes produced using NKSR <ref type="bibr">(Huang et al., 2023)</ref> with our point cloud and surface normals. To evaluate object mesh quality, we use the metrics and setup proposed by <ref type="bibr">Park et al. (2019)</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3.1 CHAMFER DISTANCE (CD)</head><p>Chamfer Distance is used to evaluate overall shape quality. In particular, we sample 30, 000 points on both the predicted and ground truth meshes and compute the Chamfer Distance between the two point sets using Equation <ref type="formula">5</ref>. Here a lower value indicates a better match to the ground truth shape.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3.2 MESH COMPLETION</head><p>Mesh completion evaluates how well the ground truth surface is covered by the predicted mesh reconstruction. In particular, we compute the percent of points sampled from the ground truth surface whose distance to the predicted mesh reconstruction is within a threshold &#964; comp (we set &#964; comp = 0.01 following <ref type="bibr">Park et al. (2019)</ref>). We compute the mesh completion metric over 1000 points uniformly sampled from the ground truth mesh. A higher score indicates better coverage of the ground truth surface by the predicted mesh reconstruction.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3.3 MESH ACCURACY</head><p>Mesh accuracy evaluates how close points on the predicted mesh surface are to the surface of the ground truth mesh. In particular, we find the minimum distance threshold &#964; acc such that 90% of points sampled from the surface of the predicted mesh are within distance &#964; acc to the ground truth surface. To compute mesh accuracy, we sample 1000 points from the predicted mesh reconstruction and compute each points distance to the ground truth surface. A lower score indicates better accuracy to the ground mesh.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E RESULTS</head><p>In this section we share more results of our scene completion method.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E.1 RUNTIME</head><p>On an RTX 4090 GPU, our completion model takes an average of 0.104 seconds to complete a scene containing an average of 9 objects in it. In other words, our method can complete about 86 objects per second.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E.2 INSTANCE SCENE COMPLETION</head><p>In Table <ref type="table">5</ref> and Table <ref type="table">6</ref>, we share per class Average Precision (AP) scores for the Intersection over Union (IoU) metric at thresholds 0.25 and 0.5, respectively. Note that at both thresholds our method outperforms RfD-Net and DIMR across almost all categories. In particular, we observe a significant gap in the performance of our method compared to previous approaches when evaluating at the more difficult threshold (Table <ref type="table">6</ref>).</p><p>In Table <ref type="table">7</ref> and Table <ref type="table">8</ref>, we show per class AP scores for the Chamfer Distance (CD) metric at threshold 0.1 and 0.047, respectively. Both RfD-Net and DIMR suffer large drops in performances when evaluating on the more difficult threshold (Table <ref type="table">8</ref>). On the other hand, the drops in performance, if any, for our method are much smaller.</p><p>Along with CD, we evaluate completion quality using the Light Field Distance (LFD) metric which measures visual similarity between meshes. In Table <ref type="table">9</ref> and Table <ref type="table">10</ref>, we see that RfD-Net and DIMR behave similarly to how they do with CD and IoU, achieving decent performance on easier thresholds but very low scores at the more difficult thresholds. This trend occurring across all three metrics suggests that previous approaches can only produce completions which are somewhat similar in shape to the ground truth objects. On the other hand, we see our method obtains a much higher LFD score on the more challenging threshold, which is similar to how we perform on CD and IoU</p><p>Partial Scan RfD-Net DIMR Ours Ground Truth   RfD-Net DIMR Ours  </p></div></body>
		</text>
</TEI>
