Few-Shot Transfer of Tool-Use Skills Using Human Demonstrations With Proximity and Tactile Sensing

📝 Paper Summary

Robotic Manipulation Sim-to-Real Transfer Learning from Demonstration (LfD)

A robot learns to use diverse tools by pre-training on primitive motions in simulation with proximity and tactile sensors, then fine-tuning with a few human demonstrations in the real world.

Core Problem

Robots struggle to manipulate grasped tools because it involves complex, unobservable 'extrinsic' contacts between the tool and environment, which vary greatly across different tools and tasks.

Why it matters:

Current methods often fix tools to the robot arm, preventing seamless tool swapping or the use of human-designed tools.
Learning from scratch requires impractical amounts of human demonstration time.
Simulation offers data but suffers from a large 'sim-to-real' gap, especially for deformable tools and complex friction dynamics.

Concrete Example: When a robot tries to use a sponge (soft tool) vs. a brush (rigid handle, soft bristles) on a curved surface, the contact forces and geometry differ wildly. Without adaptation, a policy trained on one fails on the other due to unseen friction and deformation dynamics.

Key Novelty

Multimodal Few-Shot Tool-Use Transfer Framework

Pre-trains a base policy in simulation on 'primitive motions' (simple repetitive actions) to learn general contact dynamics using tactile and proximity data.
Uses an encoder-decoder architecture where the encoder learns transferable contact features, and the decoder is fine-tuned with a small number of real-world human demonstrations.
Combines proximity sensors (to see local geometry before contact) and tactile sensors (to feel forces during contact) to estimate tool-environment interactions without explicit modeling.

Architecture

The Seq2Seq model architecture with LSTM-based encoder and decoder.

Evaluation Highlights

Successfully transfers surface-following skills to novel tools (sponge, brush) using only a small number of real-world demonstrations.
Outperforms baseline approaches (direct LfD without pre-training) in manipulating unattached, deformable tools.
Demonstrates that combining proximity and tactile sensing improves the identification of contact states and environmental geometry compared to using single modalities.

Breakthrough Assessment

7/10

Significant progress in manipulating *grasped* (not fixed) tools using multimodal sensing and few-shot learning. The sim-to-real strategy for extrinsic contact is a strong contribution, though evaluated on specific surface-following tasks.

⚙️ Technical Details

Problem Definition

Setting: Learning a policy to control a robot arm manipulating a grasped tool to perform a task (e.g., surface following) with unknown tool/surface properties.

Inputs: Tactile sensor observations x_tac and proximity sensor observations x_prox from the past T_p time steps.

Outputs: Desired actions u (primitive motions or task motions) for the next T_n time steps.

Pipeline Flow

Multimodal Sensing (Tactile + Proximity)
Encoder (Feature Extraction)
Decoder (Motion Generation)
Robot Controller

System Modules

Encoder

Encodes past sensor history into a latent representation of contact dynamics.

Model or implementation: LSTM-based Encoder

Decoder

Generates future action sequences based on the latent contact state.

Model or implementation: LSTM-based Decoder

Novel Architectural Elements

Specific sensor configuration: Proximity sensors on fingertips to observe environment geometry around the tool *while* grasping, complemented by tactile sensors.

Modeling

Base Model: LSTM-based Sequence-to-Sequence (Seq2Seq) model

Training Method: Two-phase training: (1) Pre-training on simulated primitive motions, (2) Fine-tuning on real-world human demonstrations.

Objective Functions:

Purpose: Train feature extractor to capture dynamics.

Formally: State prediction loss (MSE between observed states x and predicted states x_hat).
Purpose: Train motion generator to clone behavior.

Formally: Action prediction loss (MSE between demonstrated actions u and predicted actions u_hat).
Purpose: Joint pre-training loss.

Formally: L_pre = L_state + beta * L_action
Purpose: Fine-tuning loss (adapting only decoder FC layer).

Formally: L_LfD = L_state + beta * L_action (on target domain data)

Adaptation: Fine-tuning only the fully connected layer of the decoder; other weights frozen.

Training Data:

Pre-training: Large dataset D_prim collected from primitive motions in simulation.
Fine-tuning: Small dataset D_task from human demonstrations in the real world.

Compute: Not reported in the paper

Comparison to Prior Work

vs. Standard LfD: Uses simulation pre-training to learn contact priors, enabling few-shot learning.
vs. Prior Tool Use works: Focuses on *grasped* tools with intrinsic/extrinsic contacts, rather than tools fixed to the end-effector.
vs. Vision-based: Uses proximity sensors to perceive local geometry near contact, which is often occluded for cameras.

Limitations

Requires primitive motion data collection in simulation (though only once).
Demonstrated primarily on surface-following tasks; generalization to other task types (e.g., insertion) not explicitly shown in provided text.
Dependency on the quality of the tactile/proximity simulation to learn useful priors.

Reproducibility

Code: https://sony.github.io/tool-use-few-shot-transfer/

Project page with videos is available. Code repository URL is not explicitly provided in the text. Specific hyperparameters (learning rate, batch size, network dimensions) are not detailed in the provided text.

📊 Experiments & Results

Evaluation Setup

Real-world surface-following task using a Franka Emika robot arm.

Benchmarks:

Surface Following (Contact-rich manipulation) [New]

Metrics:

Motion trajectory accuracy
Contact force tracking
Statistical methodology: PCA analysis of latent space; qualitative comparison of trajectories.

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
The paper relies on qualitative success claims and PCA analysis rather than tabular benchmarks. Quantitative trajectory error metrics are alluded to in the text ('achieves motion and contact forces close to those demonstrated') but exact numbers are not present in the excerpt.

Experiment Figures

The overall framework diagram: Pre-training in simulation → Base Policy → Fine-tuning with Human Demos → Task Policy.

Main Takeaways

Fine-tuning the simulation-pre-trained policy enables the robot to manipulate new tools (brushes, sponges) effectively.
The method outperforms baselines that lack pre-training or fine-tuning (though specific numeric margins are not in the text).
PCA analysis confirms the pre-trained policy's latent space clusters contact states meaningfully, supporting the transfer hypothesis.
Proximity sensors are critical for detecting local surface geometry, while tactile sensors handle contact forces; the combination is superior to either alone.

📚 Prerequisite Knowledge

Prerequisites

Robot manipulation dynamics
Recurrent Neural Networks (LSTM)
Sim-to-Real transfer
Tactile and Proximity sensing principles

Key Terms

LfD: Learning from Demonstration—a method where robots learn policies by mimicking human-provided examples.

Intrinsic contact: Direct physical contact between the robot's fingertips (sensors) and the tool handle.

Extrinsic contact: Indirect contact between the tool tip and the environment, which the robot must infer since it has no sensors at the tool tip.

Seq2Seq: Sequence-to-Sequence—a model architecture that inputs a sequence (sensor history) and predicts a sequence (future actions), typically using an encoder-decoder structure.

LSTM: Long Short-Term Memory—a type of recurrent neural network capable of learning order dependence in sequence prediction problems.

Sim-to-Real gap: The performance drop when a model trained in a physics simulator is deployed on a physical robot due to modeling inaccuracies.

Primitive motions: Simple, repetitive exploratory movements (like sliding or tapping) used to collect interaction data in simulation.

PCA: Principal Component Analysis—a technique to reduce the dimensionality of data to visualize patterns, used here to analyze the latent space of the model.