Introduction
The impact of AI on the physical world is a big question. As AI models like Claude become more capable, they're not just limited to digital tasks; they can interact with the real world through robots. Anthropic's Project Fetch explores this by having two teams of researchers program a robot dog to fetch a beach ball, with one team having access to Claude and the other not. The teams were given increasingly difficult tasks, from using a controller to making the robot autonomously retrieve the ball. This experiment builds on previous research, like Project Vend, where Claude ran a small shop in Anthropic's office, but this time, AI interacts with the physical world directly through robots. The teams were evaluated on their ability to complete tasks, and the difference in performance between the two teams was measured.
What were we doing?
The experiment involved eight Anthropic researchers and engineers, none of whom were robotics experts. They were divided into two teams: Team Claude and Team Claude-less. The task was to operate a quadruped robot dog in three phases: using a controller, connecting to the robot and writing a program to retrieve the ball, and making the robot autonomously fetch the ball. The core task was simple, but the difficulty level increased with each phase. The researchers were given a day to complete the tasks, and their progress was measured. The team with Claude access was expected to perform better, as Claude can help with coding and problem-solving tasks.
Results
Team Claude outperformed Team Claude-less, completing tasks faster and more efficiently. They were particularly good at connecting to the robot and its sensors, and developing a program to autonomously retrieve the ball. The results showed that Claude provided substantial uplift in robotics tasks, bridging the digital and physical worlds. The team with Claude wrote more code and explored different approaches in parallel, but also got distracted by side quests. Team Claude-less, on the other hand, worked more collaboratively and asked more questions. The experiment demonstrated that AI can significantly enhance human performance in robotics tasks.
Claude's edge
The most striking advantage provided by Claude was in connecting to the robot and its onboard sensors. Team Claude was able to explore different approaches more efficiently and avoid getting misled by incorrect information online. They also made more progress toward the final goal of programming the robot to autonomously retrieve the ball. However, Team Claude-less was faster at some sub-tasks, such as writing a control program and localizing the robot. The controller written by Team Claude was easier to use, providing a streaming video from the robot's point of view, whereas Team Claude-less relied on still images.
Team dynamics
The experiment revealed differences in team dynamics between Team Claude and Team Claude-less. Team Claude seemed happier and more productive, with members working in partnership with Claude. Team Claude-less, on the other hand, expressed more negative emotion and confusion, and asked more questions. The quantitative analysis of audio transcripts supported these observations, showing that Team Claude-less had more negative dialogue and expressed confusion at double the rate of Team Claude. The teams had different work styles, with Team Claude working in parallel with Claude and Team Claude-less strategizing and consulting with each other more frequently.
Outtakes
The experiment wasn't just about measuring performance; it was also a fun and engaging experience. The robot dogs came with pre-programmed behaviors, and the researchers managed to unlock some entertaining features, such as dancing and backflips. Team Claude even programmed a natural language controller, allowing them to give commands to the robot dog. These side quests showed the potential for AI to enable creative and innovative solutions.
Reflection
The Project Fetch experiment demonstrated that Claude can uplift human ability in robotics tasks. The results suggest that as AI models improve, they may be able to interact with previously unknown hardware, potentially leading to rapid advances in robotics. However, there are limitations to the study, including the small sample size and the use of volunteer Anthropic employees. The experiment is an important step toward evaluating AI's potential in robotics and highlights the need to monitor AI's capabilities in this area. As AI continues to improve, it's likely that we'll see more autonomous AI systems interacting with the physical world through robots.
The Project Fetch experiment shows that AI can significantly enhance human performance in robotics tasks. As AI models continue to improve, we can expect to see more autonomous AI systems interacting with the physical world. This has implications for various domains, from robotics to AI research and development. By monitoring AI's capabilities in robotics, we can better understand its potential and limitations, and prepare for the possibilities and challenges that lie ahead.