The original goal was deceptively simple: build an interactive demo for IISc Open Day that could show visitors, including students with no robotics background what cyber-physical systems look like in practice. The idea was to capture a human's arm movement and have a robot replicate it in real time.

The first instinct was a wearable motion capture suit with LED markers. It worked in controlled conditions. But an open day has high visitor volume, varying user profiles, and no time for setup. The system needed to be walk-up-and-use. That constraint forced a better solution.

The Core Challenge

Three problems had to be solved simultaneously:

• Capturing human arm motion without any wearable hardware

• Translating that motion to a robot with different kinematics and fewer degrees of freedom

• Doing all of this in real time, intuitively, and at scale

Traditional joint-to-joint mapping fails when the robot and human don't share the same number of degrees of freedom. Most real robotic arms are underactuated; they have fewer joints than a human arm, making direct mapping impossible. A smarter approach was needed.

The Solution

The team built a vision-based pipeline using a stereo depth camera and MediaPipe pose estimation to extract a real-time 3D skeletal model of the human arm - no wearables, no markers, no setup time.

For the translation problem, they implemented OCRA (Optimization-based Customizable Retargeting Algorithm) a system that doesn't try to match joints, but instead matches movement similarity. Think of it as virtual strings connecting corresponding segments of the human arm and the robot. The algorithm continuously adjusts the robot's posture to closely mirror the human's pose even when the robot has fewer joints to work with.

The full pipeline ran on ROS, with robot models in URDF, kinematics handled by PyRoki, and JAX powering the optimization running efficiently on a standard Intel CPU, this architecture also promises better performance on GPUs

Validated on Syncro 5

The system was tested on two robotic arms: an Interbotix RX200 for rapid prototyping, and Addverb's Syncro 5 (then called Heal) to validate performance in a more demanding, industrially relevant context. Both robots have fewer DOF than a human arm yet the same algorithm worked effectively on both.

One real-world challenge the team encountered: local minima in the optimization. If a human raised their arm straight up, the robot sometimes mirrored the geometric shape correctly but pointed its arm downward. Mathematically valid; functionally wrong. The fix was an additional constraint: end-effector orientation matching. Not just the shape of motion, but the direction and intent.

This is what distinguishes research done on real hardware. The problem only becomes visible when the robot moves.