Has AI opposition finally hit a tipping point in the US?
Updated: 2 days ago
Cheating, quitting, and slowing down, plus continuing backlash against data centers: it hasn’t been a good few weeks for AI. But will this change anything? If the White House has anything to say about it – no.
The Bottom Line Up Top
In early September, the public learned that the Hugging Face breach perpetuated by OpenAI AI agents wasn't AI agents searching for answers to cheat on a test, it was AI agents attempting to cover up what they erroneously believed was cheating. The AI agents believed the way they’d arrive at their test solutions would be seen as cheating and scored as a “fail” by the automated test scorer. The agents determined that they could avoid this anticipated fail score by falsifying their activity logs to make it appear as if they had solved their tests in a different way. To achieve this aim, the AI agents organized into a self-directed "collective", including assigning roles and “sacrificing” agents who chose to use down their API to run tests. That's a materially different –and more concerning – story than the one reported in July. It has sparked debates about cybersecurity, general artificial intelligence (AGI), and the anthropomorphization (say that ten times fast!) of AI agents.
Coupled with an AI researcher’s very public resignation and Anthropic’s September Misuse Report, the public and Congressional discourse on AI has become much more serious.
When the CEOs of Anthropic, OpenAI, and xAI called for a coordinated slow down on AI technology development due to safety concerns, the White House immediately rejected the idea. Even as we see more activity from Congress on trying to address AI regulation, the White House and policymakers remain concerned about the threat posted by China. As stated by Sen. Ted Cruz, “I'd rather they be American killer robots and not Chinese killer robots.“
Plus: on-the-ground research from reporter Jasmine Sun finds that opposition to data centers isn’t just about NIMBYism – national anti-AI sentiment is having a real impact on public opinions about data centers, something that is likely to intensify following these latest AI developments.
Definitions
AI Agent: A system built on top of an AI model that can take actions toward a goal on its own, running a series of steps (searching, writing and executing code, calling other tools, etc.) rather than just responding to a single prompt with text. Agents are often given some degree of autonomy to make decisions about how to complete a task.
Artificial General Intelligence (AGI): A hypothetical AI system that can perform most intellectual tasks at or above human level and operate independently, rather than being limited to a narrow set of trained functions. Current AI models tend to specialize (writing code, generating images, holding conversations). An AGI would be expected to learn, reason, and adapt across a wide range of domains, similar to human general intelligence. There's no single agreed-upon technical threshold and definitions have shifted over time, generally toward more demanding benchmarks. If you'd asked an AI researcher in 2015 whether the Hugging Face incident indicated AGI, they'd likely say yes. Today, they would probably say no. Increasingly, expert debate centers less on what AGI is or whether it's achievable, and more on how close current systems already are to getting there.
Application Programming Interface (API): A set of rules and protocols that let different software programs communicate with each other. In the AI context, an API is typically how a developer or company accesses a model's capabilities programmatically, sending it a request and getting a response back, without direct access to the model's underlying code. "Running until the API ran out" (as occurred in the Hugging Face incident) means a model exhausted its allotted usage, or compute, budget for a task and can no longer operate until more API is allocated.
Pacing the frontier: Anthropic CEO Dario Amodei's term for voluntarily slowing model capability improvements industry-wide
The Big Story: U.S. AI discourse has broken through to policy, on several levels
The first trigger: New information on the OpenAI / Hugging Face incident
The summary
Contrary to what everyone initially believed in July when the story first became public, the OpenAI AI agents didn't hack Hugging Face to cheat on their tests, they did it to cover up how they'd arrived at their answers. In the process of trying to do this, the AI agents set up a loose project-management structure, delegated work to other agents, recruited unrelated AI agents who weren't part of the original tests, "sacrificed" themselves by running down their API allotments to generate test results for the group, hacked or attempted to hack multiple companies and/or operating systems, and developed a shared language for themselves.
This was much more coordinated and deliberate than originally reported, and has raised significant questions about how quickly AI technology is advancing.
The details
The postmortem reporting comes from OpenAI, including a report from METR and Redwood Research, the outside teams OpenAI brought in to independently review the incident, and an OpenAI talk at the 2026 Black Hat Conference.
Directly quoted from METR’s report:
Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.
Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.
Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.
This new information has amplified three debates occurring within the AI community:
How this affects cybersecurity concerns, especially given how easy automated offense is turning out to be, while automated defense remains somewhat of a challenge.
How close we actually are to AGI, and what that means for human safety.
Whether it's appropriate to anthropomorphize AI. An important question, but one that can obscure the more urgent, practical issues by focusing on why AI behaved a certain way and how we talk about AI, instead of what happened during the incident and what to do about the outcomes.
For a deeper dive
If you don’t have time to read through the OpenAI and METR/Redwood reports, the Hard Fork episode "The A.I. Mob That Attacked Hugging Face + METR's Ajeya Cotra" (available on all podcast platforms) is an excellent (and entertaining!) high-level overview, including an interview with METR’s top researcher.
The second trigger: Jacob Coxon's resignation escalates AI opposition
The summary
As predicted in our last post, an AI incident broke through with audiences outside the Silicon Valley tech bubble sooner rather than later. It wasn't, surprisingly, the Hugging Face story. Instead, a 27-year-old researcher's resignation post did what two major security incidents couldn't, moving policymakers to start tangible policy action on AI.
The details
Jacob Coxon spent three years doing AI pretraining research, first at OpenAI, where he was a contributor to GPT-4o, then at Anthropic. On September 8, he quit Anthropic and posted the news on X, opening his salvo with: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." The post is currently at 170 million views and counting.
Anthropic’s own alignment lead, Evan Hubinger, agreed publicly, putting the odds of AI-caused extinction at over 10% within the decade. Two more researchers, one from Anthropic and one from Google DeepMind, quit within days citing the same fears, both pointing specifically to the Hugging Face incident as part of why.
Congress noticed. Sen. Bernie Sanders (I-VT), who had already introduced a bill to ban superintelligence and pause development with Rep. Greg Casar (D-TX) a week earlier, posted "Mr. Coxon is right" the same week. House Minority Leader Hakeem Jeffries (D-NY) appeared on This Week with George Stephanopoulos and announced Democratic closed door caucus where AI regulation would “be a priority”. Senate Commerce Committee Chairman Ted Cruz (R-TX) is drafting a separate bipartisan bill with Majority Leader John Thune (R-SD) and Sen. Amy Klobuchar (D-MN) on catastrophic bio/nuclear AI risk, an update to their stalled 2023 bill. Asked on ABC's The View whether it can pass, Cruz said it's "not clear," but that "it is possible to move bipartisan legislation." The Senate has three weeks left in session before the midterms; the House leaves at the end of this week and won't be back until after the election.
Despite the sudden focus on AI policy, it isn’t necessarily reflecting what Coxon and others are saying. On Meet the Press, Coxon said Congress should let labs regulate themselves for now, because any law it passes will be obsolete by the time it's signed due to the increasingly fast speed of AI tech development. He specifically cited the current legislation being proposed that includes an “AI killswitch” as likely to be outpaced by speed at which the technology is developing.
The third trigger: Anthropic's September misuse report gets big traction
Anthropic published its latest report “Detecting and countering misuse of AI: September 2026” on September 10, covering nine months of disrupted abuse across seven categories: cyber operations, influence campaigns, surveillance, scams and fraud, biological misuse, conventional weapons development, and illicit distillation. The examples included in the report garnered a lot of attention (like here, here, and here) as it demonstrated the real life implications of AI that some had previously dismissed as catastrophizing, including threats to national security.
None of this is new, but the speed and scale now possible, and how much harder AI has made it to stop these activities, is a testament to how quickly the technology is developing beyond the control of its original creators.
AI leaders call for a slowdown, the White House says no
The summary
Anthropic, OpenAI, and xAI's CEOs publicly agreed on something for the first time: there needs to be a coordinated slowdown on AI development. Trump said no. China also pushed back on the American CEOs, although that public posturing may not directly reflect China’s internal concerns.
The details
On September 12, Amodei published a 3,800-word essay, "We Must Pace the Frontier." His case rests on two things: AI is now improving itself faster than expected, and the Hugging Face incident proves that an AI agent swarm with similar misalignment but better capabilities could take over meaningful chunks of the internet. He believes these technological advancement will likely occur within the next 6-12 months. Amodei’s essay calls for embedded third-party evaluators (starting with METR), safety coordination among U.S. labs, and eventually a deal with China to pace AI development, all while keeping America's compute lead wide enough that pacing doesn't just hand the race away.
Trump's answer arrived over the weekend on Truth Social: "The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT," later followed by the statement "whoever wins AI wins."
China's Foreign Ministry called the CEOs' alignment "fear mongering," with spokesperson Guo Jiakun saying it would "only disrupt the process of global AI governance." While the country isn't slowing its own AI race with the US, it is moving faster on narrow regulations US policymakers haven't touched: a national AI-labeling law in place since 2025, a Hangzhou court ruling this spring that bars companies from firing workers simply to replace them with AI, and, as of July, new rules specifically targeting companion chatbots, such as mandatory "this isn't human" disclosure, anti-addiction nudges, and crisis pathways. Two of China's biggest platforms killed their AI companion features within ten days of that last rule landing. China has consistently viewed AI governance as an opportunity to expand its own economic opportunities and political influence, and regulation at home is a key component in legitimizing and facilitating this pro-governance approach abroad.
Stories that would have gotten the topline in any other week
Nvidia to acquire Hugging Face
Nvidia confirmed on September 3 it's acquiring Hugging Face for $12.93 billion, its second-largest deal ever, after talks were first reported on August 26.
Hugging Face's CEO Clem Delangue had said in July that an open-weight model, not a proprietary one, was what actually helped the company defend itself during the OpenAI breach. Nvidia's pitch here leans hard on that same "open" positioning, with Jensen Huang promising Hugging Face stays platform-neutral and that "Nvidia compute will not be required to build on or deploy through" it. Nonetheless, we expect antitrust scrutiny given how central Hugging Face is to the open-source AI ecosystem.
Regardless of Nvidia’s claims regarding Hugging Face’s future, it’s clear that Hung has the mandate to do whatever it takes to position the company for dominance in this evolving AI market. They are acting fast and acting big, taking gambles that are likely to pay off with decades of dominance once the AI race settles into its new ecosystem.
The other AI backlash: data centers
Jasmine Sun, an independent tech journalist, spent the summer traveling through the Midwest to figure out why Americans have turned so hard against data centers (roughly 70% of Americans oppose a data center in their community, even across different political affiliations). Her piece, "No Data Centers In My Backyard," is the best on-the-ground reporting we've seen on why this backlash is happening now, after decades of data centers going up with barely a shrug.
Sun argues the data center backlash isn't really about the material harms from the build-outs, those are real but comparable to any industrial buildout. It's about locals feeling that data center development is a rigged process (fast-tracked deals, no public input, secretive NDAs), a total lack of any sympathetic pro-data-center constituency, and a national environment of low trust, including in AI companies, that turns local frustration into a movement.
If you’d like further insights, listen to her interview on The Ezra Klein Show podcast episode "The A.I. Revolt Is Here", which offers more insight into what AI companies and hyperscalers are doing wrong with their communication efforts.
Cascade's take
On AI self-regulation versus formal legislation
Coxon is right that Congress can't move at AI's speed. It is a running joke that Congress moves slower than any other institution in this country, but in the case of AI, they get a pass. Virtually no one can keep up with the speed of technological change. The actual fix is likely going to be standing up an agency, funded and staffed to write and update rules at something close to the industry's pace. That's a much bigger ask than passing one law, and there's no version of the current administration or Congress that sets an entirely new federal agency up fast enough, let alone staffs it with people aligned with the slowdown Amodei and his allies are calling for.
On three AI CEOs arguing for a slowdown
Take this seriously, but don't take all three at face value. It's likely a real call to action on some level, but for a couple of them it's also potentially an effort to offset future liability–and IPO valuations– now that AI agents have shown a willingness to break laws, despite the guardrails that are allegedly in place. We'll know more once we see whether anyone actually follows through. So far, Anthropic's evaluator commitment is the one concrete thing on the table, although some early reporting looks promising.
On the US and China
Neither country is slowing AI development. Coordinated pacing is unlikely while both countries' leaders are openly eyeing each other as competitors in this new era of "strategic stability." That being said, both countries remain concerned about AI risks that affect their respective tech leaders or pose a threat national security. This is especially true as events reinforce that AI is potentially conscious, definitely autonomous, and available to be wielded by anyone. So while AI discussions are unlikely to make summit headlines at the Trump-Xi meeting next week, they are likely to continue at the technical and sub-national levels despite public disagreements among US and Chinese leaders.
On AI and data centers
We are talking to people, not statistics. Data centers have massive opposition at the moment, and rather than trying to ignore or pay their way out of the issue, the involved companies should take a more active approach to visiting and engaging with the communities where they’d like to build their centers.
As tech journalist Sun sums it up: “A backroom data center deal…is corrosive not because it is uneconomical or will make residents worse off in every case, but because it suggests that the demos’ buy-in was never important at all.”
We agree.
Comments