Anthropic has made strides in AI development with the release of its updated Claude 3.5 Sonnet model, which can now interact with desktop applications. This upgrade, allows the model to perform tasks such as simulating keystrokes, clicking buttons, and manipulating mouse movements via a new ‘Computer Use’ API, which is currently in open beta.
The company describes this development as part of its broader goal to create AI-powered virtual assistants capable of automating back-office operations like research, email management, and other administrative tasks. According to the company, developers can access this capability through platforms such as Amazon Bedrock and Google Cloud’s Vertex AI. The API allows the model to take screenshots, measure pixel distances, and perform corresponding actions on a user’s screen.
While AI agents, software designed to automate tasks, are not new, the company’s approach differs from traditional tools in its potential to scale and integrate with a wide variety of applications. This model, referred to by Anthropic as an ‘action-execution layer,’ is notable for its ability to complete complex tasks requiring multiple steps. The Claude 3.5 Sonnet, for example, has been tested for tasks such as coding and autonomous verification of software applications, with companies like Canva and Replit already exploring its uses.
However, there are limitations. The model is still prone to errors in basic actions like scrolling and zooming and struggles with quick or fleeting user interactions, which could hinder its utility for more precise or time-sensitive tasks. Despite these challenges, the company is positioning this as an early step toward automating broader aspects of the economy.
On the safety front, the company acknowledges the risks associated with granting desktop-level control to AI. There are concerns about potential misuse, especially when AI is exposed to malicious jailbreak attempts or software vulnerabilities. Anthropic has taken measures to mitigate these risks, such as withholding training on user screenshots and developing classifiers to steer the model away from risky actions.
Looking ahead, the company plans to release Claude 3.5 Haiku, a more affordable and efficient variant of the model. This version promises improved speed and performance, making it well-suited for consumer-facing applications and large data-processing tasks.
