Thanks to advances in imitation and reinforcement learning techniques, we can now train intelligent agents to accomplish a diverse range of goals. But if we want to create household robots or personal assistants that can take advantage of this diversity, we need to give users some way to tell them what to do! This tutorial will focus on humans' favorite tools for communicating goals and plans: natural language. We'll assume basic familiarity with supervised learning and RL, and begin with a review of core machine learning techniques useful for natural language instruction following problems. The body of the talk will focus on modeling techniques for instruction following problems in different kinds of environments and data conditions. We'll conclude with a survey of other applications for the tools we've built, including instruction generation, interpretability, and machine teaching.
On - Demand Recording
Jacob Andreas, PhD
Assistant Professor | MIT CSAIL