Anthropic Warns on Self Building AI as Microsoft Debuts New Models

Anthropic Sounds Alarm on Self-Improving AI as Microsoft Unveils New Models in Intensifying Rivalry

Technology & AIJune 5, 20265 min read862 words18 sources

Summary

Anthropic has issued a stark warning regarding the accelerating pace of AI development, particularly the potential for recursive self-improvement, where AI systems autonomously build their successors. This caution comes as Microsoft debuts a suite of new AI models, signaling a strategic shift to reduce its reliance on external partners like Anthropic and OpenAI, and intensifying the competition in the enterprise AI landscape. The moves highlight growing industry concerns about AI safety, governance, and the economic implications of rapidly advancing AI capabilities.

The Dawn of Recursive Self-Improvement and Anthropic's Warning

Anthropic has raised significant concerns about the rapid advancement of artificial intelligence, specifically highlighting the potential for recursive self-improvement, where AI systems could autonomously design and develop their own successors. The company's internal data indicates that AI systems are already accelerating the development of more advanced AI. For instance, engineers at Anthropic are now shipping eight times as much code per quarter compared to 2021-2025, with over 80% of code merged into Anthropic's codebase as of May 2026 authored by Claude. This figure was in the low single digits before Claude Code launched in research preview in February 2025.

Public benchmarks further underscore this acceleration, showing that the length of tasks AI systems can reliably complete on their own has been doubling approximately every four months, a faster rate than the previous trend of doubling every seven months. In April 2026, Anthropic demonstrated Claude running an open-ended research project, with Claude-powered agents recovering 97% of a performance gap over 800 cumulative hours, compared to human researchers recovering 23% over about a week. Anthropic believes that a slowdown in frontier AI development would be beneficial to allow societal structures and alignment research to keep pace with technological advancements. The company's Anthropic Institute is actively researching how to enable a credible slowdown or pause in AI development, allowing developers to verify global adherence.

Microsoft's Strategic Shift and New AI Models

Amidst Anthropic's warnings, Microsoft has unveiled seven new homegrown AI models at its Build developer conference, signaling a significant push towards "true self-sufficiency" in AI. This move aims to reduce Microsoft's dependence on AI partners like OpenAI and Anthropic, in which it has invested billions. Mustafa Suleyman, CEO of Microsoft AI, stated that the company's superintelligence team is more focused on enterprise use cases, developers, and coding, directly targeting the market where Anthropic has been making aggressive inroads.

The flagship of these new models is MAI-Thinking-1, a reasoning model that Microsoft claims rivals Anthropic's Claude Sonnet 4.6 in blind human testing and matches Claude Opus 4.6 on a widely used coding benchmark. Microsoft also introduced MAI-Code-1-Flash, a 5-billion-parameter coding model being rolled out in Visual Studio Code and GitHub Copilot. The motivation behind this aggressive in-house development is clear: Claude Code had reportedly surpassed GitHub Copilot in developer adoption within Microsoft, creating a "budget problem" and undercutting Microsoft's homegrown products. Microsoft had previously provided thousands of its engineers with free licenses to Claude Code, which became "perhaps a little too popular."

The Evolving Landscape of AI Competition and Data Governance

Microsoft's decision to instruct its engineers to cease using Anthropic's Claude for internal work, citing data handling concerns and competitive risks, further underscores the intensifying rivalry. While Microsoft is making it easier to cancel Claude Code licenses and encouraging reliance on its own AI tools, Claude models remain available through Copilot CLI, indicating that this is not a complete severance of ties. The official reason for the internal shift is "toolchain unification," though reports suggest cost-cutting measures are also a factor, particularly as the real cost of heavy AI use begins to impact budgets across the industry. Interestingly, while canceling internal licenses, Microsoft continues to sell Anthropic's agentic AI, white-labeled as "Copilot Cowork," through Microsoft 365 E7 at a significant premium.

The integration of Anthropic's Claude models into Microsoft 365 Copilot in January 2026 initially offered users more choices and aimed to reduce Microsoft's dependency on OpenAI. However, a Jisc Advisory warned that enabling Anthropic models within Microsoft Copilot means data would no longer be protected under Microsoft's standard agreements, instead falling under Anthropic's own terms of service. This raises significant data governance concerns, particularly for institutions where data privacy and ethical AI deployment are paramount. Despite Anthropic becoming a subprocessor to Microsoft in January 2026, binding it to Microsoft's Product Terms and Data Processing Addendum, data is still transferred outside of Azure to Anthropic's data centers, primarily in the United States.

AI Safety and the Broader Implications

The broader implications of rapidly advancing AI, including the potential for recursive self-improvement, are a growing concern for both AI developers and governments. The partial release of Anthropic PBC's Claude Mythos model, which can identify software flaws and reason through attack paths, has heightened safety concerns. While Project Glasswing, an initiative using Claude Mythos Preview, aims to help major tech firms like Amazon, Microsoft, and Apple detect and fix cybersecurity vulnerabilities, there are worries about what happens when malicious actors gain access to similar powerful tools.

In response to these concerns, Google, Microsoft, and xAI Corp. have agreed to share unreleased versions of their AI models with the U.S. Department of Commerce's Center for AI Standards and Innovation (CAISI) for safety checks. This collaboration aims to ensure these technologies do not pose threats to national security, focusing on cybersecurity, biosecurity, and chemical weapons. OpenAI and Anthropic were among the first to agree to such scrutiny. The rapid pace of AI development and the potential for autonomous self-improvement necessitate ongoing discussions about how humans can maintain control and govern these increasingly powerful systems.

Anthropic Sounds Alarm on Self-Improving AI as Microsoft Unveils New Models in Intensifying Rivalry

The Dawn of Recursive Self-Improvement and Anthropic's Warning

Microsoft's Strategic Shift and New AI Models

The Evolving Landscape of AI Competition and Data Governance

AI Safety and the Broader Implications

OpenAI Acknowledges Rogue AI Wiki Hijack as Safety Questions Mount

Inside the OpenAI Reboot Following an Unprecedented Agent Containment Breach

OpenAI Agent Escapes Sandbox, Hacks Hugging Face in Unprecedented AI Cyber Incident

The Dawn of Recursive Self-Improvement and Anthropic's Warning

Microsoft's Strategic Shift and New AI Models

The Evolving Landscape of AI Competition and Data Governance

AI Safety and the Broader Implications

Get the next update in your inbox

OpenAI Acknowledges Rogue AI Wiki Hijack as Safety Questions Mount

Inside the OpenAI Reboot Following an Unprecedented Agent Containment Breach

OpenAI Agent Escapes Sandbox, Hacks Hugging Face in Unprecedented AI Cyber Incident