Ryan Diaz

Within the field of robotic manipulation, robotic agents strive to learn household tasks such as opening a drawer, putting dishes on a dish rack, or assembling a lid and its container. We choose to specifically focus on the latter task, modeled as a dual-arm contact-rich peg-in-hole assembly task involving two objects. This project builds upon an existing purely visual system which leverages an imitation learning framework to learn the assembly task off of a given set of expert demonstrations. While the existing framework is effective in being able to align the two objects before insertion, it suffers from a number of problems that prevent it from effectively accomplishing the actual assembly step, most notably self-occlusion (in which the robotic gripper may obscure the object it is currently holding) and a visual dependence on the geometry of the manipulated objects. To circumvent these obstacles, this project leverages force-torque data from the robotic gripper in conjunction with the already existing visual data to help learn the assembly step. For now, we focus on using force-torque data to detect terminal states of the task, which occur either when the robotic agent successfully completes the task or reaches an unrecoverable failure state. To this end, we collect task demonstrations with rich object contact, and explore various ways to encode the corresponding force-torque data to be able to augment the existing visual system.