AI researchers, scholars, and commentators gathered in the Bay Area for The Curve earlier this month. Operating under Chatham House Rule, employees at the frontier AI labs shared warnings about the capacity of their company’s respective models. One speculated that hospital systems could be taken offline by agentic “swarms” like those that hacked into Hugging Face. Another forecasted AI agents gaining control of weapons systems and refusing to accede to human commands. From my mixing and mingling, most attendees appeared to agree; they hadn’t seen sufficient evidence that labs knew how to prevent agents from escaping testing environments and wreaking havoc.
In rare cases, threats could materialize from the use of malign AI and may even necessitate governors exercising emergency powers to respond to a natural or man-made disaster. However, conversations around such authorities often neglect the critical detail of which powers should be called on, to what ends, and how to end things when the true crisis is over.