Registration methods play a vital role during image-guided interventions in assisting surgeons to target the key regions of interest, such as tumors and ves-sels, that lie beneath the organ surface. Image-to-physical registration aligns preoperative Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) data, represented as a point cloud or mesh, to intra-operative data acquired using surgical tracking devices or cameras, often represented as a point cloud [ 4,17]. However, the registration [ 3,17] faces challenges caused by the partial intraoperative visibility that arises due to constrained camera viewpoints and occlusions [ 3,5,7]. Therefore, it is necessary to address the issue of partial visibility, which prevents registration methods from performing with sufficient accuracy.
The reconstruction of an intra-operative 3D liver surface from a partially visible surface has the potential to mitigate the partial visibility issue faced by registration methods. Toward this effort, Jia et al. [ 5] proposed a non-rigid registration framework that integrates a learning-based point completion network to generate a complete surface from sparse intra-operative data to guide non-rigid registration. However, the proposed method still requires a rigid registration as the initialization. Foti et al. [ 2] adopted a Variational Autoencoder on preoperative models to generate a full liver surface from a partial point cloud, which undergoes an iterative optimization procedure to generate an intra-operative surface. However, this approach also requires manually identified correspondences between the generated mesh and the intra-operative point cloud, which is nontrivial.
In this work, we propose a patient-specific pipeline to improve image-tophysical registration (and hence pre-to intra-operative registration) by completing partial intra-operative liver surfaces. The pipeline leverages a vector neuronbased occupancy network (VN-OccNet [ 1]) to recover a complete liver mesh from a partial intra-operative point cloud. VN-OccNet offers two key advantages for the registration: (1) rotation equivariance, which addresses the failure of conventional models under varying orientations of intra-operative data [ 12], and (2) the ability to generate watertight meshes rather than point clouds, enabling uniform surface sampling-a crucial feature for registration methods requiring consistent point density [ 15,17]. We adopt a patient-specific training strategy that synthesizes intra-operative liver surfaces by deforming a pre-operative patientspecific liver model, hence allowing the model to focus on patient-specific geometry and deformation patterns. We first evaluate VN-OccNet's ability to generate rotation-equivariant surfaces across diverse rotation settings. We then compare registration outcomes using completed surfaces against those using only partial point clouds. Results show that the reconstructed surfaces significantly reduce registration error, highlighting the benefit of surface completion towards enhancing registration accuracy.
We define the intra-operative partial liver point cloud as the target point cloud represents the learned parameters, x i represents the i th query point, and z is the latent representation of T . The solution is to utilize f θ (x i , z) to reconstruct a complete estimate of the target point cloud, denoted as � T , and subsequently perform registration with S.
Our proposed pipeline illustrated in Fig. 1 consists of three key stages: (1) Generation of occupancy points from a target point cloud using VN-OccNet, (2) Mesh construction from the predicted occupancy points, and (3) Registration, where the complete target point cloud (obtained by extracting vertices from the generated mesh) is aligned with the source point cloud using the Go-ICP registration method.
Vector Neuron Occupancy Network (VN-OccNet): To accomplish the rotation-equivariant point cloud completion task, VN-OccNet [ 1] leverages the original OccNet architecture [ 10], with a vector-based equivariant encoder. The decoder is invariant, where all the operations are non-vector. The target point cloud T = {t i } N i=1 ∈ R 3 is input to the encoder network, and outputs equivariant latent vector-list features z. The query set X = {x i } K i=1 ∈ R 3 , set of points that encloses the liver, and latent feature z are input to the decoder network to output the occupancy probabilities between 0 and 1. Points with probability greater than or equal to a threshold c correspond to occupancy points that implicitly represent the liver surface. Since VN-OccNet aims to predict whether or not each point in the query point set corresponds to a liver surface point, binary cross-entropy classification loss is utilized to learn the parameters θ.
Mesh Generation: Meshes are constructed by first identifying the voxels that intersect the liver surface using the Multiresolution Isosurface Extraction (MISE) algorithm [ 10]. The algorithm starts with a fixed query set with an initial resolution of 32 3 , consisting of 31 3 voxels, each containing 8 points. For all 32 3 points in the query set, the occupancy probability is determined using the trained VN-OccNet. The points inside the surface are identified by applying a threshold value of c = 0.4. Voxels that are entirely inside or outside the surface are separated from the surface-intersecting voxels by checking if at least two adjacent points of a voxel differ in occupancy (i.e., one point is inside and the other is outside). Each such ambiguous voxel that intersects the surface is further subdivided into 8 subvoxels, generating 19 additional points. The VN-OccNet is queried again to find the occupancy probability for the newly generated query points. This process is repeated until all ambiguous regions have been subdivided down to the finest voxel grid resolution of 128 3 . Finally, the surface-intersecting voxels obtained are passed to the Marching Cubes algorithm [ 8] to generate the surface mesh.
Rigid Registration Using Go-ICP: To evaluate the effectiveness of incorporating the complete intra-operative target surface generated from using the proposed protocol into the registration, we utilize the Go-ICP registration algorithm [ 15] to compare the registration outcomes between using the initial, incomplete target point clouds and the complete intra-operative surface meshes generated using the proposed method.
For the remainder of the manuscript, we refer to these registrations as registration w/ surface completion (when the complete surface mesh is utilized) and registration w/o surface completion (when only the original partial point cloud is utilized). The registration algorithm yields the transformation (T), which aligns the source point cloud to the target point cloud.
The experiments utilize the following datasets: an in silico phantom dataset is used to train and test VN-OccNet, and an in vitro phantom dataset is used to assess the registration performance.
In Silico Phantoms: We used the undeformed No. 1 synthetic phantom constructed based on a patient-specific CT image according to the methods proposed by Yang et al. [ 18] briefly described in further detail in the next sub-section. To simulate deformed models, the deformation simulation pipeline described in [ 11] is followed, which models the liver as a neo-Hookean hyperelastic material with a Young's modulus of 2-5 kPa and a Poisson's ratio of 0.35. Three forces of magnitude up to 3N and zero boundary conditions, along with the material properties, were input to the finite element solver to output deformed liver models. The complete target point clouds are obtained by extracting the mesh vertices. In total, 4,969 deformed models were generated.
Training an occupancy network requires transforming liver models into a representation comprising a liver point cloud, a set of query points enclosing the liver, and corresponding occupancy values that indicate whether each query point lies inside or outside the liver surface. We followed the pipeline of Stutz et al. [ 12] to generate this data format. The generated data was divided into training, validation, and testing sets with a ratio of 8:1:1.
In Vitro Phantoms: We used the in vitro phantom dataset described and released in [ 18], which includes four pairs of underformed and deformed models. Phantoms were manufactured using synthetic gelatin (Humimic Gelatin # 0, Humimic Medical, Fort Smith, AK, USA) poured into a 3D-printed mold generated [ 9] from a patient-specific liver model obtained from OpenHELP [ 6]
Target Point Cloud Generation: In training, target point clouds are dynamically generated from the complete point clouds. First, we crop the posterior surfaces of the deformed liver models. The anterior surface points are then downsampled to D points. A viewpoint is randomly selected, and the N nearest surface points relative to this viewpoint are extracted to create the target point cloud. For testing, we follow the approach proposed in [ 19] to simplify the qualitative analysis, where five distinct viewpoints are selected for each liver model, as illustrated in Fig. 3.
We employed the PyTorch implementation of the VN-OccNet by Deng et al. [ 1].
The anterior liver points were downsampled to D = 1000, and N = 300 nearest points from the viewpoint were chosen to create the target liver point cloud, yielding a surface visibility of approximately 30%. The network was trained with a batch size of B = 8 and a learning rate of 0.0001, using the Adam optimizer. The optimal threshold value c was determined to be 0.4 based on the validation dataset. Training was performed for 604 epochs on a NVIDIA A100 GPU, with all other hyperparameters kept unchanged from the original implementation. To verify the rotation equivariance property, we further compared VN-OccNet with OccNet [ 10], which does not include rotation equivariance design.
In a manner similar to the five target point clouds generated for each liver model during testing in the previous Subsect.
Chamfer Distance (CD-L2), F-Score, and Intersection over Union (IoU) were used to evaluate the performance of complete target surface generation. Moreover, the Target Registration Error (TRE) computed across a set of fiducials was used to assess registration performance.
CD-L2 measures the bidirectional similarity between the complete generated target and ground truth target points:
where | T | and |G| represent the total number of points in the complete generated target and ground truth target point cloud, respectively, and d CD ( T , G) denotes the Chamfer Distance between the point sets T and G.
The F-Score measures the quality of the reconstructed surface by balancing precision P (d) and recall R(d). Precision measures the fraction of complete target points that lie within a threshold d = 1 mm of the nearest ground truth target points, and recall measures the fraction of ground truth target points that lie within d of the nearest complete target points:
IoU measures the ratio of the intersection to the union of the occupancy values of the generated and ground truth targets, as defined in Eq. ( 3):
where occ gt and occ pred represent the ground truth and predicted occupancy values, respectively, and | • | denotes the number of true occupancy labels.
TRE measures the Euclidean distance between the estimated target fiducial markers F tgt,i and the transformed source fiducial markers T(F src,i ):
where F is the total number of fiducial markers.
Table 1 presents the performance of the intra-operative surface completion for OccNet and VN-OccNet across four different train/test configurations: I/I, Z/Z, Z/SO(3), and SO(3)/SO(3). OccNet achieves the best performance across all metrics in the I/I setting. However, its performance significantly deteriorates when tested with target liver point clouds with rotations along the Z-axis and SO(3). While data augmentation along SO(3) improves performance to some extent, it remains impractical to encode all possible SO(3) rotations during training, making the model susceptible to failure when encountering unseen liver poses during testing. Although VN-OccNet does not outperform OccNet in the I/I setting, it maintains consistent performance across Z/Z, Z/SO(3), and SO(3)/SO(3), demonstrating robustness to rotations.
Unlike OccNet, VN-OccNet does not show performance improvement due to data augmentation, indicating that its inherent rotation-equivariant properties reduce reliance on augmented training data. The statistical significance of OccNet and VN-OccNet performance was evaluated by comparing all three evaluation metrics using a Wilcoxon Rank Sum test at a significance level of α = 0.05. The test yielded p-values less than α (p < α), confirming that the performance differences between VN-OccNet and OccNet across various train/test setups are statistically significant.
Figure 4 presents the target point clouds, ground truth target mesh surfaces, and complete target mesh surfaces generated by OccNet and VN-OccNet across four different train/test configurations: I/I, Z/Z, Z/SO(3), and SO(3)/SO(3). The performance gap in surface generation is evident in the Z/SO(3) case, where OccNet struggles to generate a target mesh surface that accurately resembles the ground truth target mesh. In the SO(3)/SO(3) setting, the mesh generated by OccNet lacks the correct geometry, indicating an incorrect reconstruction. In contrast, the mesh generated by VN-OccNet is able to capture correct geometry and better alignment of the target point cloud with the reconstructed mesh.
Table 2 summarizes the Target Registration Error (in mm) measured between target fiducial markers and source fiducial markers. The measurements are reported for registration w/ and w/o surface completion. The registration w/o surface completion is performed using the target point clouds at three visibility levels (30%, 40%, and 50%), and registration w/ surface is performed using the completed surface meshes generated from the 30% visible target point clouds. The registration for phantom No. 1 yields the TRE of 33.13 ± 13.89 mm for the 30% visibility of target point clouds. However, when using the complete target point clouds generated from the same 30% visible target, the TRE significantly decreased to 5.19 ± 1.34 mm. This notable improvement demonstrates the effectiveness of VN-OccNet's surface completion in enhancing registration performance. Furthermore, as the visibility of the target point clouds increased, TRE progressively decreased, confirming that a lower visibility ratio leads to higher registration errors.
A Wilcoxon Rank Sum test, conducted at a significance level of α = 0.05, showed statistically significant differences (p < α) between the three cases of registration w/o surface completion and registration w/ surface completion.
Figure 5 displays the results for registration w/ surface completion and w/o surface completion for in vitro phantom No. 1 and No. 3. The registration w/o surface completion struggles to find the correct alignment between the source and target point clouds. The incorrect registration is evident when the complete ground truth target is overlaid onto the registered results. However, registration w/ surface completion is able to correctly align the source and target point clouds. Similarly, Fig. 6 shows the registration between source and target fiducial markers for phantom No. 1 and phantom No. 3. The transformations were determined using a target point cloud with 30% visibility for registration w/o surface completion and a reconstructed completed surface mesh from the same target point cloud for registration w/ surface completion. For the registration w/o surface completion, the registered source fiducial markers are misaligned with respect to the target fiducial markers. In contrast, in registration w/ complete surface, the source and target fiducial markers are closely aligned, demonstrating improved registration.
The partial visibility of intra-operative point clouds poses challenges for both initial rigid and non-rigid registration methods in image-guided liver surgery. To address this issue, we propose a patient-specific, learning-based surface completion approach. In this work, we initially validate the effectiveness of this approach in facilitating the initial rigid registration.
Initial rigid registration has relied on global optimization techniques [ 15] and the establishment of reliable correspondences using handcrafted [ 13] or learningbased methods [ 17]. However, these approaches are often sensitive to partial liver surfaces and inconsistent point cloud densities. In contrast, our method tackles the problem at its root by employing surface completion through learning-based techniques. Unlike previous completion approaches that require rigid initialization [ 5] or manual interaction [ 2], our method operates without such dependencies. We utilize a vector-based occupancy network-VN-OccNet-due to its robustness to rotation and its ability to generate watertight meshes in just 0.25 seconds per mesh, making it a strong candidate for this task.
We first verify the rotation-equivariance property of VN-OccNet, which is essential for our application. Recent work [ 12] evaluated several widely used point completion methods and found them unsuitable for this task. We compared VN-OccNet with the original OccNet, which lacks a rotation-invariant design, under various training and testing conditions, using extensive quantitative and qualitative evaluations. Furthermore, we assessed registration performance by integrating completed surfaces with Go-ICP, a method known for its robustness to partial visibility [ 16]. Results show that our reconstructed surfaces significantly reduce registration errors, demonstrating the value of surface completion in improving registration accuracy. However, Go-ICP, which takes approximately 16 s per registration, is computationally intensive, suggesting that integrating surface completion directly into learning-based registration methods may offer a more efficient solution.
An additional challenge lies in the variation of liver geometries across different patients, which can hinder surface completion. To overcome this, we adopt a patient-specific training strategy that simulates intra-operative surfaces by deforming a pre-operative patient-specific liver model. This approach enables the network to learn patient-specific geometries and deformation patterns. Our results show that this strategy allows the model to generalize effectively to realistic deformations observed in in vitro phantoms. Nonetheless, the requirement to train a new model for each patient may limit its practicality in resourceconstrained environments.
It is also important to note that real intra-operative data may exhibit more complex patterns, such as noise, holes, and occlusions, than those seen in our simulations. This discrepancy poses a challenge for learning-based methods, which generally assume the test data distribution is similar to that of the training data. As such, improving generalization to diverse, realistic intra-operative point clouds remains an open challenge.
This paper presents a pipeline that incorporates VN-OccNet to generate a complete liver surface from partial intra-operative point cloud data, using the reconstructed surface for subsequent registration. We demonstrate the effectiveness of VN-OccNet in producing accurate and complete surface reconstructions, which significantly enhance registration performance. While partial intra-operative data typically hampers the accuracy of most registration methods, our results indicate that the proposed pipeline provides a promising solution to this challenge.
In future work, we plan to investigate the integration of surface completion with non-rigid registration, explore patient-generic surface completion strategies, incorporate completion into end-to-end learning-based registration pipelines, and develop a more robust simulation pipeline capable of generating realistic intraoperative point clouds with noise, holes, and occlusions.