Overview
From a phrase to a lifted object
A natural-language prompt and synchronized RGB-D observations enter on the left; an executed grasp leaves on the right. Eight stages sit in between.
Keep scrolling to move through them
Abstract
Offshore inspection and maintenance have increasingly been using legged robots for routine sensing, yet many useful interventions still require physical interaction with tools, containers, and task-relevant objects. Employing robots for these tasks can reduce operators' exposure in confined, elevated, or potentially explosive areas. This paper presents a language-guided grasping pipeline for a legged mobile manipulator operating under partial observation. An operator defines the target, the system grounds it in RGB with open-vocabulary detection and promptable segmentation, extracts an object-centric RGB-D point cloud, improves sparse geometry through depth compensation and point-cloud completion, and selects a 6-DoF grasp using collision, clearance, reachability, and approach constraints. The system is implemented on a quadruped robot with an arm and evaluated in two cluttered tabletop scenes motivated by small-object retrieval during inspection and maintenance. Across paired trials, the proposed pipeline achieved 9/10 successful grasps, compared with 3/10 for a view-dependent deployment baseline. In this controlled setting, object-centric completion and execution-aware selection reduced approach collisions and improved the reliability of language-guided grasping for supervised field manipulation.
The pipeline, stage by stage
Scroll through the system diagram. The view moves in on each stage in turn, then pulls back out to the whole loop.
Stage 01
Input data
The operator names the target in plain language — here, “blue detergent”. That prompt arrives alongside the RGB and depth streams from the robot's front cameras. Detection and segmentation use RGB only; depth is reserved for building geometry further down the pipeline.
Stage 02
Target object detection
GroundingDINO turns the phrase into candidate bounding boxes and the highest-scoring one prompts SAM 2, which returns a pixel-accurate instance mask. A light morphological erosion stops the mask leaking into neighbouring clutter, and SAM 2 keeps tracking it across frames.
Stage 03
Point cloud generation
Isaac ROS nvblox back-projects the depth image into a registered scene cloud. The instance mask splits it in two: the target points that will be completed and grasped, and the surrounding environment points that later serve as collision geometry.
Stage 04
Shape estimation
MGPC infers the geometry the camera never saw, conditioned on the prompt, the RGB image and the partial cloud — 2,048 points in, 8,192 synthetic points out. PoinTr then densifies the merged result patch by patch, which is what stabilizes the surface normals GPG depends on.
Stage 05
Grasp candidates
The Grasp Pose Generator samples 1,000 antipodal 6-DoF candidates across the completed cloud, with jaw width and contact constraints matched to the Spot gripper so that every sampled grasp is physically realizable on the hardware.
Stage 06
Grasp selection
A parallel kernel tests the gripper mesh against nearby scene points and rejects every candidate that collides — shown in red. The survivors are scored on approach alignment, approach direction, distance to the object centroid and a hard reach limit, and the cheapest one is selected.
Stage 07
Locomotion
If the selected grasp is out of reach from the current stance, the state machine commands a base waypoint along the approach direction, at a stand-off distance that improves manipulability while keeping the target inside the arm's workspace.
Stage 08
Grasp execution
With the base aligned, the end-effector moves to a pre-grasp pose offset along the approach axis, runs a 5 cm Cartesian insertion to the final pose, closes the gripper, and lifts the object clear of the surface.
Overview
The whole loop
A phrase in, a lifted object out, with every stage feeding the next. The Method section below explains the choices behind each one.
Method
Grounding an operator's words
The operator names the target in plain language — “blue detergent”. GroundingDINO returns candidate boxes and the highest-scoring one prompts SAM 2, which produces a pixel-accurate instance mask; a light morphological erosion keeps it from leaking into neighbouring clutter. SAM 2 then tracks the mask across frames. GroundingDINO is re-invoked only if tracking drops the target, and if no valid hypothesis survives, the system keeps observing rather than planning a grasp.
Object-centric geometry, not a map
Depth is back-projected with nvblox into a registered scene cloud, and the mask keeps only the points falling on the target. Rather than maintaining a volumetric TSDF or ESDF, the pipeline works directly on point clouds: masked clouds accumulate over time through the robot's state estimate, and a depth-compensation step fills small holes and attenuates outliers using local neighbourhood consistency in the image plane.
Two models for what was never seen
Self-occlusion and back-side surfaces survive accumulation, so MGPC infers the missing geometry from the prompt, the RGB image and the partial cloud — 2,048 points in, 8,192 synthetic points out, merged with the observed ones. PoinTr then densifies the result patch by patch over overlapping 2,048-point KD-tree neighbourhoods, because GPG's sampling depends on normal estimation quality and needs a dense, stable surface.
Grasps ranked by whether they can be executed
GPG samples 1,000 antipodal 6-DoF candidates with jaw parameters matched to the Spot gripper. A parallel kernel tests the gripper mesh against nearby scene points and discards anything that collides. Survivors are ranked by a weighted cost over approach alignment, a penalty on awkward approach directions, distance to the object centroid, and a hard reach limit — so those weights are the knobs an operator has for trading reach against safety in a hazardous scene.
Execution
A finite-state machine owns the rest. If the selected grasp is unreachable from where the robot is standing, it commands a base waypoint along the approach direction at a stand-off distance that keeps the target inside the arm's workspace. The arm then moves to a pre-grasp pose offset along the approach axis, runs a 5 cm Cartesian insertion to the final pose, closes the gripper, and lifts.
Experimental setup
Two cluttered tabletop scenes, built around the small-object retrieval that inspection and maintenance work actually asks for. The table sits at roughly the robot's base height, and the robot always starts facing it.
- Robot
- Boston Dynamics Spot with the Spot Arm (6-DoF) and jaw gripper
- Sensing
- Front-left and front-right cameras of Spot's five RGB-D sensors. RGB at 15 Hz, 640×480; stereo depth at 424×240
- Compute
- External workstation over Ethernet: 2× NVIDIA RTX A2000 (12 GB), 125 GB RAM, Intel Xeon w3-2435 (16 cores)
- Software
- ROS 2 (Humble and Jazzy) in Docker containers, Boston Dynamics Spot SDK
- Protocol
- 10 trials per setup, paired: from each initial pose, one run with the full pipeline and one with the baseline. The initial pose changes between pairs; the scene does not
- Success
- The gripper closes on the target, lifts it clear of the table, and holds it through a short verification period without slipping
The baseline shares the same perception front end, GPG candidate generation, collision filtering and ranking. What it does not get is multi-frame accumulation, point-cloud completion, or base repositioning — it has to commit to a grasp from the single view available at its initial stance, which is how many deployed grasp pipelines behave.
Results
9/10
Full pipeline
vs
3/10
View-dependent baseline
By scenario
| Scenario | Target | Ours ↑ | Baseline ↑ |
|---|---|---|---|
| A | Drill | 4/5 | 0/5 |
| B | Blue bottle | 5/5 | 3/5 |
| Total | — | 9/10 | 3/10 |
Trial by trial
| Scenario | Run | Object | Ours | Baseline (view-dependent) |
|---|---|---|---|---|
| A | 1 | Drill | Success | Reachability failure |
| A | 2 | Drill | Success | Approach collision (clutter) |
| A | 3 | Drill | Success | Approach collision (clutter) |
| A | 4 | Drill | Reachability failure | Reachability failure |
| A | 5 | Drill | Success | Approach collision (clutter) |
| B | 1 | Blue bottle | Success | Approach collision (target) |
| B | 2 | Blue bottle | Success | Success |
| B | 3 | Blue bottle | Success | Approach collision (clutter) |
| B | 4 | Blue bottle | Success | Success |
| B | 5 | Blue bottle | Success | Success |
Where the failures come from
The baseline fails overwhelmingly on approach: grasps that look plausible on the visible surface turn out to be unexecutable once clearance along the approach direction is accounted for. The full pipeline failed once across all ten trials, and that failure was reachability rather than collision.
FM-1 · Reachability
No feasible pre-grasp configuration, or the pose lies outside the arm's workspace.
FM-2 · Approach collision (target)
Contact with the target itself on the way in, from limited clearance along the approach axis.
FM-3 · Approach collision (clutter)
Contact with the surrounding objects and occluders during approach.
A grasp, end to end
One successful trial on the real robot, from the moment the target is selected to the lift that confirms the grasp holds.
-
a Locomotion approach. After grounding and segmentation, the system estimates the object geometry and selects a feasible grasp, then repositions the base along the approach direction to satisfy reachability and clearance. -
b Gripper approach. The arm aligns to the target and moves to a collision-free pre-grasp pose, offset along the gripper's approach axis by a safety distance. -
c Grasp. A 5 cm Cartesian insertion along the approach axis brings the end-effector to the selected pose, and the gripper closes to secure the object. -
d Lift. The object is lifted clear of the table to confirm grasp success and stability under post-grasp interaction.
Limitations
The target has to be visible enough to ground
The object must sit sufficiently within the field of view for the query, open-vocabulary detection and segmentation to find the right instance. An ambiguous prompt or a heavily occluded target can make grounding fail or lock onto the wrong object, and the pipeline will not proceed to planning.
Specification is bounded by what the models understand
Domain-specific equipment, unusual tools, and safety-critical part-level intent may need fine-tuning, or task-specific prompt engineering and verification, before they can be named reliably.
Depth quality is still the binding constraint
Spot's stereo depth is low-resolution and noisy. Compensation and completion help, but thin or reflective objects can still end up with missing surfaces and outliers that completion does not recover well enough for safe grasp selection.
Planned next: a larger object set and trial count, external grasping baselines, separating the effects of completion from those of repositioning, and verifying the completed geometry online during base and arm motion.
Citation
@inproceedings{almeida2026language,
title = {Language-Guided Grasping under Partial Observation for Mobile
Manipulation in Field Inspection and Maintenance},
author = {Almeida, Dilermando and Negri, Juliano and Lazzarini, Guilherme
and Segreto, Thiago H. and Bezerra, Ranulfo and Lahr, Gustavo J. G.
and Godoy, Ricardo V. and Becker, Marcelo},
booktitle = {IEEE International Symposium on Safety, Security, and Rescue
Robotics (SSRR)},
year = {2026},
eprint = {2603.07866},
archivePrefix = {arXiv},
primaryClass = {cs.RO}
}Acknowledgments
This work was supported by Petróleo Brasileiro S/A — Petrobras, using resources from the R&D clause of the ANP, in partnership with the Universidade de São Paulo (USP) and the Fundação de Apoio à Física e à Química (FAFQ), under Cooperation Agreements No. 2023/00016-6 and 2023/00013-7.