A one-shot imitation learning framework that enables robots to adjust their behavior based on task specifications and scene changes.

AI-powered robots with traditional behavior cloning (BC) methods often demonstrate poor generalization, which leads to erratic behaviors and performance degradation in out-of-distribution settings. Their performance tends to deteriorate in certain scenarios, regardless of the credibility of the data on which the model is trained. Relying on out-of-distribution (OOD) detection yields a limited guarantee of convergence and success. Particularly, when deploying robots in human-centric environments, BC often fails to perform effectively. Incorporating huge reference motion data to manage unexpected scenarios introduces significant training overhead, and even with it, there is no assurance of consistent performance across all cases.

The inventors propose Elastic Motion Policy (EMP) – a framework that helps robots imitate actions based on real-time changes in the environment in one shot. This framework addresses the inherent generalization issue in BC method.

EMP is built upon the dynamical system motion policy paradigm with BC, which learns stable motion policies delivering compliant and reactive robot behavior. During data collection, the Universal Manipulation Interface (UMI) gripper determines the gripper states through a contact sensor, and an external camera records the demonstration in RGBD format. A large language model helps determine the semantic label for estimating the object’s key pose. The semantic phrase is fed into the Grounded Segment Anything Model to generate the object mask. The EMP enables real-time adaptation of SE (3)-LPVDS (Special Euclidean Linear Parameter Varying Dynamical Systems) to scenarios by introducing geometric constraints to morph the learned parameters based on their spatial changes. So, an update in the object's pose triggers a corresponding adjustment of the motion policy. Laplacian editing in full end-effector space and online convex learning of Lyapunov function are employed to avoid the need to collect new demonstrations and to adapt EMP to new contexts.

• Helps adapt robotic actions in one-shot
• Dynamically updates actions, thereby eliminating the need for new demonstrations for reference actions
• Updates its policy in real-time (~30Hz), ensuring that robots respond to changes in a decent interval
• One-shot learning can reduce the computation for training, leading to cost reduction
• Enables the learning of multi-step tasks by disintegrating long and heterogeneous sequences into individual goal-oriented tasks