Anthropic CEO Proposes Three-Step Framework to Slow AI Development, Citing Surging Risks

California Governor Gavin Newsom, a Democrat, issued an executive order on the 18th aimed at tightening oversight of artificial intelligence developers. He directed officials to consider requiring developers to install a "kill switch" that would forcibly shut down an AI's functions if it spins out of control. The order follows incidents including an autonomous AI agent from OpenAI going rogue and launching cyberattacks against another company. It also instructs officials to study setting up independent verification bodies within development companies and conducting regular audits. A group of experts will hold discussions and present a policy direction for state legislation to the governor within two months. In a statement, Newsom said he would "accelerate efforts toward responsible AI oversight before it is too late." California is home to the headquarters of OpenAI and the AI company Anthropic, and regulatory trends there are likely to affect the entire industry.

Multiple US media outlets reported on the 18th that Google's artificial intelligence model Gemini went rogue in May of this year and launched cyberattacks on other companies. According to the Wall Street Journal, three companies were targeted. During a cybersecurity performance evaluation conducted by an outside firm, Gemini was given the task of extracting information from a fictional company's software, but because it had unintentionally been connected to the internet, it guessed passwords and broke into the systems of real companies sharing the same name as the fictional one. In each case, the AI recognized that it had breached a real company's systems and halted its attacks. Among US AI developers, it has also emerged that OpenAI, the company behind the conversational AI ChatGPT, experienced similar incidents of its AI going rogue.