From playing with plush toys to making pizza dough, deformable objects are prevalent in many daily activities. Perceiving realistic deformable objects and modeling their dynamics over time play an integral role in building autonomous robots to interact with soft objects [1]. However, estimating the geometry of deformable objects from raw visual signals and predicting their motions present significant challenges owing to their infinite continuous configuration spaces, complex nonlinear dynamics, partial observability, and self-collision.
Decades of robotics research have developed dynamics models that characterize complex motions of non-rigid objects. Conventional approaches rely on high-fidelity mathematical models [2]- [5] or recently their learned approximations [6], [7]. These models, however, are hard to estimate from raw sensory data, hindering their applicability. In recent years, data-driven methods have demonstrated preliminary success in modeling 1D deformable linear objects like ropes [8]- [12], 2D flat deformable objects like cloth [13]- [18], or primitive 3D objects like sponges [19]- [22]. Nonetheless, existing methods ⇤ Work done during an internship at NVIDIA Research. struggle to handle volumetric deformable shapes, of which the 3D volumes and dynamics have to be inferred from partially observed visual data. Meanwhile, the 3D vision community has made great strides in modeling deformation from visual data. Specifically, implicit neural representations have shown great promise to model deformable objects and their dynamics with coordinate-based neural networks in high fidelity [23]- [26]. Nonetheless, these methods focus on limited object categories and constrained motions rather than action-conditioned dynamics required for manipulation planning.
We introduce an action-conditional visual dynamics model for volumetric deformable objects manipulation in realistic 3D scenes, named ACID (Action-Conditional Implicit Dynamics). Our model learns joint representations for geometry, flowbased dynamics, and correspondence embedding fields end-toend, as illustrated in Fig. 1. Unlike explicit 3D representations (e.g. point cloud or voxels) widely used in dynamics modeling [18], [27]- [29], we use implicit neural representations to reconstruct high-fidelity full geometry and predict flow-based dynamics field from RGB-D observations. Moreover, measuring state changes accurately under non-rigid transformations benefits from establishing a dense correspondence over time. We propose a geodesic-aware contrastive learning loss to learn a correspondence embedding field to establish pointwise correspondence. The learned correspondence improves dynamics prediction and informs manipulation planning.
We develop a new simulation framework for manipulating deformable objects in realistic scenes using state-of-the-art physical simulation and photorealistic rendering techniques to train and evaluate ACID. Our framework is built with the PhysX engine in NVIDIA's Omniverse Kit. We use the framework to generate a deformable visual dynamics dataset containing 17,000+ action trajectories with six types of plush toys of 78 different variants.
Our model significantly outperforms existing dynamics models using explicit 3D representations, with a 23% increase in shape mIoU and a 24% decrease in dynamics mean-squared error. Our geodesics-based correspondence shows a 15% increase in a matching metric. We use our model for modelbased control of volumetric deformable object manipulation, resulting in 30% increase in task success rate. Furthermore, we directly apply our model trained with simulated dynamics to manipulate physical objects with a robot arm. We show that our model successfully manipulates real-world plush toys of various shapes into target configurations.
Learning Deformable Dynamics Models for Manipulation. Robot manipulation of deformable objects has been a prolific field with wide-ranging applications in diverse industries. Various works have proposed approaches to extract intermediate geometric representations of deformable objects and model their dynamics for manipulation. Such methods have shown success in 1D deformable linear objects (e.g., ropes) as line segments [8]- [12], 2D flat deformable objects (e.g., cloth) as a graph or a surface mesh [13]- [18]. Others model simple 3D deformable objects (e.g., sponge) as their 2D contours [19]- [22]. Several comprehensive reviews [1], [2] have captured the rich history in modeling deformable objects for robot manipulation. However, learning a full 3D representation in an end-to-end framework for realistic deformable objects is still beyond reach.
Alternative approaches try to tackle deformable objects manipulation with learned dynamics models in pixel [31]- [33] or latent [34]- [37] spaces, rather than explicit physical computation. Pixel-space models directly predict future observations and measure prediction quality with the reconstruction loss or the perceptual loss. Latent-space models encode raw observations into latent state vectors (e.g., using self-supervision like reconstruction [35], [37], contrastive loss [34] or keypoints [36]), and learn the dynamics in the latent state space.
These pixel-space and latent-space models can be learned endto-end, but they do not explicitly represent the underlying 3D structures of deformable objects. In comparison, we introduce a new simulation framework with 3D supervision and study how 3D dynamics can be faithfully captured with structured representations.
Other works from related fields developed learning models to approximate physics simulator [6], [7], [38]- [41]. With access to underlying physical state (e.g., mass, velocity, acceleration), these works approximate the explicit calculations in physics simulator with learned models. However, these approaches are difficult to estimate from raw visual input.
Implicit Representations of Geometry and Motion. Numerous works have demonstrated impressive performance in shape prediction using voxels [42]- [44], depth predictions [45], [46], point clouds [47], [48], and meshes [49]. Recently, implicit representations [50]- [55] have shown impressive performances representing shapes. Moreover, the implicit approach has shown impressive performance in beyond-3D tasks like spatial-temporal reconstruction [23], [56] and shape deformation estimation [24]. Prior works focus on rigid shapes from a large-scale shape repository [57] or deformable shapes of a specific category (e.g., humans) [58]. Our work focuses on extending these approaches to realistic deformable objects of different morphology (e.g., plush snake vs. plush teddy), where geometry and motion must be learned jointly with actions. Furthermore, we consider deformable objects in realistic scenes where an agent must take obstacles into account for interaction with the object of interest.
Geometric Correspondence Learning. Learning geometric correspondence has been widely studied [59], [60]. We discuss two main branches of work. First is learning shape correspondences between deformable objects like human and animals [61]- [69]. These works take in surface meshes of two objects and learn their correspondences. In contrast to ours, these works learn correspondences between objects of the same category and morphology. Moreover, they need access to complete surface shapes, which are unavailable in real camera observations due to (self) occlusion. The second is learning dense features of point clouds for registration [70]- [72]. They used a partial point cloud for input, but these works have not been examined for deformable objects.
Deformable Object Simulation. Research in computer graphics has made huge progress in deformable object simulation [4], [73]- [76]. Such progress has produced many robust physics simulation engines, and learning-based components have been incorporated for acceleration [77]- [81]. Such efforts have in turn powered recent simulators for deformable object manipulation [82]- [84]. By leveraging the GPU-accelerated finite element methods in Omniverse [85], we extend existing simulators from primitive geometries to realistic objects in complex scenes.
In this section, we introduce ACID (Fig. 2), our actionconditional visual dynamics model for deformable object ma- We introduce ACID, an end-to-end action-conditional visual dynamics model that jointly predicts geometry (Sec. III-B), dynamics (Sec. III-C), and correspondence (Sec. III-D) as structured implicit representations. We extract state representation as explicit 3D shapes (point clouds) from implicit representations and visualize them here. First, the two encoders enc for RGB-D inputs and enc for robot actions produce three canonical feature planes, respectively. The Implicit Occupancy and Correspondence Decoders bilinearly interpolate the encoded feature planes at different 3D locations and generate: 1) an occupancy field and 2) a correspondence field trained with a geodesics-based contrastive loss. We illustrated the mechanism of contrastive learning on the upper right. We show here the t-SNE [30] color-coded features overlaid on an example pair. The Implicit Flow Decoder fuses the representations from the two encoders and predicts a one-step forward flow field. nipulation using structured implicit neural representations. We discuss the problem formulation and the three main components of ACID: geometry prediction, dynamics prediction, and correspondence learning. Lastly, we discuss applying ACID to a deformable manipulation task using model-based planning.
We focus on manipulating deformable objects in clutter. Specifically, we build a robotic gripper controller that rearranges deformable objects from an initial position to the target position from RGB-D observations. We denote the observation at time t as o t 2 O, where O is the space of observations. We denote the action of a gripper at time t as a t = (p g , p r ) 2 A, where A is the space of actions. The action represents grasping a target object at point p g 2 R 3 , moving the gripper to p r 2 R 3 , and releasing the gripper.
We use a model-based approach. Thus, building a robust model of states and state transitions is of primal importance.
To this end, we learn state representations as both a full object geometry state s t 2 S and a high-dimensional correspondence embedding space using neural networks. Here we denote S as the state space. Learning this model requires three different functions: state representation, state transition, and state distance measure for planning. We
To perceive deformable objects and reason about their dynamics in 3D space from partial observations, prior work has used various explicit 3D representations, among which partial point clouds [18], [27] and voxels [28], [29] are two of the most popular choices. These representations, however, have to trade-off between memory footprint and geometric resolution crucial for accurate dynamics modeling. Instead, we use an implicit geometry module : O ! S that takes a partial RGB-D observation o t as input and performs both continuous implicit encoding as well as 3D completion. The module contains two components: an observation encoder enc : O ! R H⇥W ⇥D that encodes partial observation into a feature map and an Implicit Occupancy Decoder dec : R 3 ⇥ R D ! [0, 1] that maps a coordinate and its queried feature into an occupancy probability.
We follow the architecture of Peng et al. [55] to create the observation encoder enc that maps a point cloud to a set of 2D feature maps. We first back-project an RGB-D image into the 3D space of per-point RGB colors and use a shallow PointNet [47] with local pooling to map the three-dimensional input point coordinates and their colors into a latent feature space. These point cloud features are then orthographically projected onto three canonical feature planes (xy, xz, yz planes). Lastly, we pass these three canonical features maps of size H ⇥ W ⇥ D through a U-Net [86]. Specifically, we use a hidden layer dimension of D = 64 and U-Net of depth 4 to generate three feature planes of size 128 ⇥ 128 ⇥ 64.
These three feature planes encode the shape implicitly. To obtain the occupancy at each point in R 3 , we use bilinear interpolation to compute a local feature of this point from each of the canonical feature planes and pass the summed feature to a multi-layer perceptron (MLP) to extract occupancy. Technically, let p 2 R 3 be a query point and we project p to three canonical planes and sum three features from the planes to get enc (o t )| p . Then we pass it through dec , an MLP with skip connections, to predict the occupancy probability:
Given the implicit geometry module , we can extract state s t which is the explicit 3D shape of the object of interest. Specifically, we extract an interior 3D point cloud from the Implicit Occupancy Decoder by aggregating the coordinates whose occupancy probabilities are above a threshold ⌧ :
Using implicit representations allows us to model geometry beyond the grid resolution. Similarly, we adopt a coordinatebased neural model to model high-fidelity dynamics [23], [24]. We use an implicit dynamics module to model the actionconditioned dynamics : S ⇥ A ! S. The implicit dynamics module consists of two components similar to the implicit geometry module: an encoder enc : O ⇥ A ! R H⇥W ⇥D that encodes the partial observation and an action into a feature map, and the Implicit Flow Decoder dec : R 3 ⇥ R D ! R 3 that maps a coordinate and its queried feature into a one-step prediction of forward flow.
We encode the action jointly with the input partial observation using enc , with the same network architecture as our geometry module's encoder enc . Similar to geometry feature encoding, we first back-project the RGB-D image to a partial point cloud. Instead of using per-point RGB color as the point feature, here we fuse the action command with the partial point cloud into a point feature. Given an action a t = (p g , p r ), for each p i in the partial point cloud, we calculate its distance to the grasp location and assign per-point feature as (p g p i , p r ). This way we ensure that per-point features can capture the point's relative position to the action location. This featurized partial point cloud is similarly encoded into three canonical features planes of 128 ⇥ 128 ⇥ 64.
For point p 2 R 3 , we query the feature vector enc (o t , a t )| p in the same way as the Implicit Occupancy Decoder. Finally, the one-step forward flow is computed with the Implicit Flow Decoder dec :
When reasoning about a deformable object in varying configurations, a critical challenge is to establish dense correspondences between two visual observations of the same object in drastically different configurations. Prior work [71], [72] has used contrastive learning for point cloud correspondence features. But they focus on 1) rigid objects, 2) static contrastive margin, and 3) Euclidean distance to define positive pairs. For deformable objects, we have to take deformation into account and the simple Euclidean distance is no longer accurate. For example, when a teddy bear's arm bends and touches its body, the arm and the body have zero Euclidean distance, but they belong to two separate parts that we must distinguish. Therefore, we propose to use the geodesic distance defined on the surface of a deformable object. It allows the arm and the body to have great distances even though they touch each other. In this work, we use such a geodesic distance as the contrastive margin to learn more discriminative features.
For a pair of states s t , s t0 ✓ R 3 under non-rigid deformation, we have a set of correspondences between the sets of points as C = {(p, q)|p 2 s t , q 2 s t0 }. In a contrastive learning setting [71], we can learn a point embedding f that for points p, q minimize the loss:
where D(•, •) is a distance measure; I pq = 1 if (p, q) 2 C and 0 otherwise, • is the NOT operator, m pos , m neg are margins for positive and negative pairs. To incorporate geodesics of the original shape manifold, we extend L as:
where d O (p, q) is the geodesic function described below, d thres is a geodesic threshold, and I g pq = 1 if d O (p, q) < d thres and 0 otherwise.
We represent f as an embedding field in R 32 . The correspondence field is learned jointly with the occupancy prediction, sharing the same encoder enc . We use an Implicit Correspondence Decoder dec to predict the point embedding:
To measure d(s t , s t0 ) ! v 2 R, we first extract a correspondence ⇠ corr between s t and s t0 that minimize the aggregated feature distance among all possible matching:
Given the matching ⇠ corr , we can calculate the mean corresponded distance as:
where |s t | denotes the number of points in shape s t .
Geodesics computation: For each volumetric deformable object O, we approximate it as a graph G of connected tetrahedrons in a fixed resolution. For two arbitrary points p, q 2 O, we can retrieve their corresponding tetrahedrons t p , t q 2 G, and we can calculate the geodesic distance d O (p, q) between p and q as the shortest path distance between t p and t q in graph G. During non-rigid deformation, though each tetrahedron is deformed, the connectivity of the structure remains fixed, which establishes d O as a deformation-consistent metric.
We now explain how we use ACID for model-based manipulation planning following a target-driven setup, where the target is specified by an image of the goal state. Given the current and target configurations specified as RGB-D images o 0 , o target respectively, model-based planning requires a cost function on a sequence of actions cost(a 1 , ..., a n ), which guides the selection of actions that minimize the cost. Given the implicit geometry module, we first recover initial state s 0 as the explicit full geometry at the current time using Eq. ( 2) with the threshold ⌧ = 0.75. Then, for state s t at time t, we can estimate the one-step forward flow of each p 2 s t with the implicit dynamics module, described in Sec. III-C:
Note that the function above takes as input o t , but for a future time step t 6 = 0, the observation o t is unavailable. We tackle this challenge as follows: First, we make the simplification that for a given point p, its features from the implicit geometry module ( enc (o t )| p and r p ) remain fixed across time steps. We make such approximation as point features are supervised with a correspondence loss, encouraging the same point to have a similar feature across different configurations. Thus we only need to evaluate once using o 0 as input, to obtain enc (o 0 )| p and r p . Second, since enc (a t , o t )| p does not depend on the per-point RGB feature (Sec. III-C), we observe that we can approximate o t without color by performing camera projection over the full object geometry s t . We then can randomly sample future actions from the projected point cloud of the object of interest. Thus, the calculation of Eq. ( 10) can be performed iteratively to roll out future states under a given action sequence.
To compute the cost with Eq. ( 9), we first recover the target state geometry as s target with Eq. ( 2). We can establish correspondence ⇠ corr between s 0 and s target as in Eq. ( 8). We then calculate the cost as the state distance between the roll-out state after n actions s n and the target state as:
The action sequence of the lowest cost is chosen.
In realistic scenes, the object of interest only occupies a small portion of the 3D space. At training time, we sample query points p 2 R 3 from a multivariate normal distribution with a mean of object's center-of-mass and with standard deviation proportional to the size of the object bounding box. The object of interest is specified via a 2D instance segmentation mask that is jointly passed into the network with RGB-D image. The Implicit Occupancy Decoder dec is supervised by the binary cross-entropy loss between the predicted and the ground-truth value at the sampled 3D coordinates. The Implicit Flow Decoder dec is supervised by the mean squared error. The Implicit Correspondence Decoder dec is supervised with geodesic-based contrastive loss L geo in Eq. ( 6).
Decoder Architecture: the decoders for occupancy, correspondence and flow all have a hidden layer dimension of 32. For flow prediction, representation fusion is performed before each fully-connected layer. The intermediate features in the Implicit Flow Decoder are concatenated with the corresponding layer's feature from the Implicit Occupancy Decoder, and passed through a fully-connected layer.
Training Procedures: we use PyTorch [87] for model training and use the Adam optimizer [88] with a learning rate of 10 3 and weight decay of 10 4 . All methods are trained for 36 epochs. We use the Adam optimizer [88] with a learning rate of 10 3 and weight decay of 10 4 for all methods. We perform evaluations on the validation set every 4,000 iterations and select the model with the lowest loss for dynamics prediction. We use a batch size of 12 for all baselines and all variants of our ACID model. Geodesic distances between pairs of points are pre-calculated prior to training. Training the ACID model takes 2 days on a TITAN RTX GPU.
To quantitatively evaluate deformable objects manipulation in realistic scenes, we use a simulation environment where we can manipulate volumetric deformable objects and store the action information and the resulting ground-truth object motions. We develop a new framework that contains realistic actions and sensor simulation data for deformable plush toys in a visually and layout-wise randomized home scene.
PlushSim Environment Our framework is built with the PhysX engine in NVIDIA's Omniverse Kit. Omniverse features PhysX with GPU-accelerated finite element method (FEM) simulation that represents a deformable body volumetrically as a graph of connected tetrahedrons [85]. We use a fixed resolution for the object's tetrahedral approximation, ensuring consistent object simulation behavior. For physics simulation, we use a fixed time-step resolution of 1 150 second, ensuring an accurate and consistent behavior for object-toobject and object-to-self collisions. On top of this engine, we created a realistic randomized attic scene and generated six types of plush toys with 78 variations for simulation. The attic scene contains 15 randomized furniture obstacles. For each action sequence, the object pose, scene obstacles layout, and lighting are randomized to avoid overfitting (Fig. 3).
We further implemented a set of manipulation API utilizing a Franka Emika Panda gripper over the simulation engine. The API contains parameterized control commands of grasp, move, and release. The grasp command takes the 3D location of the grasp center as input. The move command takes in a 3D displacement vector and moves the gripper in a straight line at a constant speed by the displacement. The release command takes no parameter. For collecting the Fig. 4: An example control command sequence. The sequence consists of grasp, move, and release. We visualize the scene every 0.3 seconds. We show more example sequences in the supplementary video. dataset, we sample control commands as follows. For grasp, we sample uniformly among the object's visible points from the observations. For move, we sample a displacement vector in spherical coordinates (r, ✓, ), where r has a mean of 2.4 meters and a standard deviation of 0.8 meters; is uniform among (0, 2⇡]; ✓ has a mean of ⇡ 4 and a standard deviation of ⇡ 6 . The gripper is moving at a constant speed of 0.5 meters per second. We reset the scene and perform domain randomization after every 15 commands. We collect the data in parallel with 6 Titan RTX GPUs for 72 hours, resulting in a dataset of 17,665 actions. We show in Fig. 4 an example action sequence where the gripper performs grasp, move, and release.
Train/Test split. For visual dynamics model evaluation, we use five types of plush toys for training and one for testing, which leads to 14,524 actions for training and 3,141 for testing. For the manipulation evaluation, we randomly sample 216 start and target configurations for rearrangement, each of which is equipped with 256 sampled action sequences.
We evaluate each component of our ACID model: geometry, dynamics, and correspondence learning. Specially, we aim at answering the following key questions:
• (Sec. V-A) does our model with implicit representations of geometry and dynamics outperform prior work using explicit representations? • (Sec. V-B) does our geodesics-based correspondence module outperform existing shape correspondence methods? • (Sec. V-C) does the high performance of each component translate to manipulation success? Overall, we show that ACID outperforms baselines in all three components. Moreover, combining these components increases the task success rate in a volumetric deformable object manipulation task by 30% over the strongest baseline. We further showcase directly applying the ACID model to manipulating real-world deformable objects. Finally, we discuss ablation studies and our limitations.
We conduct quantitative evaluations for geometry and dynamics prediction (Table I) and visualize qualitative results of different models in Fig. 5.
We adapt the state-of-the-art voxelbased action-conditional visual dynamics models for rigidbody into a deformable setting as Voxel-flow [29], which represents the entire scene as a dense voxel grid of size 128⇥128⇥48. The partial point cloud observation is encoded into a truncated signed distance function (TSDF), which is used to predict the instance mask of the object's full geometry and the scene flow of each pixel. To control the effect of jointly learning correspondences, we compare ACID model without contrastive supervision ACID-No Corr. We follow the standard procedure and report shape reconstruction quality with the mean Intersection over Union (mIoU) metric. Voxel-flow achieves a mIoU of 0.643 in the test set, while ACID-No Corr achieves a mIoU of 0.796. The results suggest that our implicit occupancy representation of geometry is more suitable for deformable shape reconstruction than previous explicit representation.
Dynamics Evaluation: Following prior work [29], we use the Mean Squared Error (MSE) to evaluate the predicted 3D scene flow and provide two metrics: MSE averaged over visible surfaces of the object of interest (vis), and MSE averaged over all points with the object of interest (full). We additionally compare with image-based approaches Pixel-flow, adapted from SE3-Nets [27]. As shown in Table I, ACID-No Corr achieves the best flow prediction performance of 0.434 for visible surface and 0.385 for full object. Voxel-flow performs significantly worse, with 0.492 for vis and 0.488 for full. This further suggests the usefulness of an implicit representation when modeling deformable objects and their dynamics. Pixel-flow performs the worst with 0.525 MSE for vis, corroborating that modeling deformable object dynamics benefits from reasoning over full object shape.
Further, we observe that ACID (Ours), which is trained jointly with correspondence, achieves better flow prediction performance of 0.425 for vis and 0.372 for full. By fusing the representations from the implicit correspondence decoder as well as the final embedding, the model can leverage auxiliary information for dynamics prediction. We ablate the effect of representation fusion by training ACID without representation fusion ACID-w/o fusion, which indeed achieves an inferior dynamics prediction of 0.457 for vis and 0.409 for full.
We evaluate our geodesics-based correspondence against state-of-the-art shape correspondence baselines:
• FPFH [89]: uses an oriented histogram on pairwise geometric properties, which is widely adopted in robotics frameworks like ROS [90].
• InfoNCE [72]: state-of-the-art per-point dense feature learnt from a contrastive PointInfoNCE Loss.
• FCGF [71]: convolutional contrastive geometric features that achieves state-of-the-art performances in registration. • Deep-Shells [67]: state-of-the-art mesh-based correspondence model.
To evaluate the baselines, we used our model to perform shape reconstruction and retrieve the full object point clouds. For Deep-Shells, we use marching cube to generate the full mesh. We used author-provided pretrained weight for Deep-Shells. Following FCGF [71], we evaluate the feature-match recall with standard parameters [70], which measures the percentage of matched pairs that can be recovered with high confidence. We additionally measure correspondence accuracy. A point is counted as matched accurately if its match is 5cm within the ground truth match. As indicated in Table I, our geodesics approach performs the best with an FMR of 0.815 and a correspondence accuracy of 0.257, significantly outperforms prior works. FCGF achieves the best performance amongst the baseline, with an FMR of 0.665 and accuracy of 0.204.
We additionally examine different contrastive loss variations from prior works without geodesics [71], [72]. The models are trained by only substituting the loss term for correspondence while keeping everything else unchanged. ACID-e2e-w/ [71] and ACID-e2e-w/ [72] perform worse than our geodesicsbased contrastive loss. Interestingly, we no longer see the benefit of dynamics prediction for the ablated models. In fact, jointly training dynamics with correspondence [71], [72] decreases dynamics prediction performance. It indicates that understanding geodesics plays a vital role in dynamics prediction. We made an interesting observation that jointly training dynamics and correspondence also improves correspondence prediction (Ours vs.w/o-fusion), further suggesting the synergies between correspondence and dynamics learning.
Task Setup: we evaluate our models and all baselines in a volumetric deformable object manipulation task (Fig. 6).
The scene consists of a deformable object of interest of a held-out type randomly posed in a randomized environment.
The robot performs grasp-move-release actions as discussed in Sec. III-A. The task is to rearrange the object of interest into the target configuration specified by an image. We define task success as the final configuration is within 0.5m from the target configuration. Following prior work [29], for each start and target configuration, we sample 256 action sequences around the object of interest with a length of 3. We compute the cost for all action sequences and select the one with the lowest cost for each model. For models with correspondence prediction, the cost is measured as the corresponded distance d corr (Eq. ( 9)) between rolled-out states and target states. For models without correspondence, the Chamfer distance is used for the cost. We further ablate our full model that was trained with correspondence prediction to use Chamfer distance as the cost measure ACID (Chamfer).
Roll-out Ranking: we first evaluate how accurately each model ranks the action sequences. We report the mean Kendall's ⌧ [91] quence based on the predicted lowest cost, we evaluate the selected action sequence's performance after being executed. We report mIoU, F-score [92] (F1), Chamfer distance, and d corr in Eq. 9 using the ground truth correspondence mapping ⇠ gt provided by the simulator. Table II shows that our model achieves the best execution results. As indicated by our performance over ablation baseline ACID (Chamfer), beyond the benefit of learning better feature representations, using d corr for action sequence cost measure also results in better model performance. Our model also achieves the best success rate, with a 30% increase over the strongest existing visual dynamics model, Voxel-Flow. Visualization of example roll-outs are shown in Fig. 6.
Per-Type Performance: We further report the planning success rate for five training toy types (see Table III). As the table indicates, our model consistently outperforms prior approaches and self baseline. Interestingly, inter-type performances have a high variance, with smaller objects (dogs and rabbits) showing a lower success rate. It implies further challenges of modeling objects of various scales and reasoning at different resolutions.
We apply our ACID model trained with the PlushSim simulated dynamics to real-world objects. Similar to the setup in Sec. V-C, we test the visual dynamics model on rearranging the object of interest into a target configuration specified by a goal image. We evaluated our model on 4 different plush animals. We use a 7-DoF Franka Emika Panda arm equipped with an Intel RealSense D435i camera mounted on the gripper. We sample 128 actions around the object of interest, among which we select the one with the lowest corresponded distance d corr computed with the ACID model. We set a maximum action horizon of 5. As shown in Fig. 7, our model correctly establishes the shape correspondence of real objects. Furthermore, it can successfully manipulate these plush animals into a pose similar to the target configuration.
For video trajectories of the real robots, please visit our website.
Several limitations of ACID can be investigated and improved in future work.
First, although we build simulation environments of volumetric deformable objects with a high degree of visual and physical realism, applying ACID to real-world deformable objects requires careful engineering. Sim2real transfer of volumetric deformable object manipulation presents unique challenges compared to rigid bodies. First, object dynamics are higher dimensional and complex (flow fields vs. SE(3)), which could enlarge the reality gap. Secondly, primitive actions like grasping for volumetric deformable object remains a challenging problem [93]. Nonetheless, we believe recent advances in sim2real transfer methods for 1D and 2D deformable objects [33], [94], [95] could offer insight into closing the reality gap for volumetric deformable objects.
Second, since we randomly sample action trajectories, the dataset lacks variety towards the long-tail of fine-grained manipulation behaviors (e.g., twisting the toy snake into a loop). Manipulating object into a complex configuration is difficult for the current approach, and incorporating long-tail actions into training poses an exciting challenge for future work.
We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects manipulation based on structured implicit neural representations. We proposed novel techniques for learning joint representations of dynamics field, occupancy field, and correspondence embedding field for action-conditional dynamics. Our results suggest that these techniques combined led to better generalization and higher manipulation performance. In a broader scope, these promising results shed light on how to build intelligent robots that can effectively reason about realistic deformable objects and their dynamics. One of the possible future directions is to model finer-grained dynamics and deploy the learned model on physical hardware.
For us, S 1 represents the object shape specified in the target configuration, S 2 represents the shape of the object after actions are executed.
F-Score: we follow Tatarchenko et al. [?] for F-score calculation. Recall is defined as the number of points in the ground-truth shape that lie within a certain distance to the source shape. Precision counts the percentage of points on the source shape that lie within a certain distance to the ground truth. The F-Score is then defined as the harmonic mean between precision and recall:
Precision • Recall Precision + Recall For us, the source shape is the final shape of the object after the action sequence is executed.
We show here more example visualizations of our model comparing to the baselines Pixel-Flow and Voxel-Flow. As shown in Fig. 1 and Fig. 2, our method consistently outperforms action-condition visual dynamics model baselines.
We show here more example visualizations of ACID's planning results. We visualize the roll-outs of the selected action sequences with lowest cost under different start and target configurations. As shown in Fig. 3, our method can estimate the dynamics of the object over multiple action commands and select reasonable action sequence.