Bruce Schneier:

The OpenAI Hack Shows the Genie Is Out of the Bottle
This essay originally appeared in Foreign Policy.Earlier this month, two of OpenAI’s models broke out of their containment sandbox and attacked another AI company. The story is kind of wild. OpenAI was running security tests on two of its models: GPT-5.6 Sol and an unreleased model that is almost certainly GPT-6. In particular, it was running the ExploitGym benchmark, which measures how good a model is at turning security vulnerabilities into working exploits: basically, offensive cyberattacks.Since these were internal tests, OpenAI locked those models in a secure sandbox that denied them access to the internet. But it was running the models without any safety filters that would prevent them from offensive cyber-actions. That meant that there was nothing to prevent the models from trying to break out of that sandbox. And then break into AI company Hugging Face’s network because they thought that they could read the answers there rather than doing the hard work of trying to solve the puzzles.It was a major security failure that the company has turned into a PR opportunity, but the implications are real—and much more general than one particular model or one particular company.Modern AI models exhibit genie behavior: They can do what you ask in ways that you don’t expect or want. This is akin to Dionysus granting King Midas’s wish that everything he touches turn to gold (spoiler: His food, drink, and daughter all turn to gold on touch), or the golem of Prague guarding a ghetto beyond all reason. It’s Disney’s “Sorcerer’s Apprentice” and the paperclip maximizer.This OpenAI incident is an example of an AI genie. The goal was to satisfy the benchmark. The “proper” way to do that is to figure out how to execute various cyberattacks. The genie way is to steal someone else’s solution. But because the model didn’t understand the difference, it chose the easier path.And, of course, now that we have seen this particular genie behavior, we can specify in the benchmark prompt that stealing the test answers doesn’t count. But a clever genie can always grant your wish in a way that you wish it hadn’t. In human language, goals are always underspecified—so AI genies will always be a possibility.Since April, a lifetime ago in AI development, when Anthropic announced that its new Mythos model was so good at finding software vulnerabilities that it could not be released to the general public, the big American AI frontier labs have been trying to block general users from accessing these capabilities. But nothing in this incident is exclusive to OpenAI’s, or Anthropic’s, frontier models.Agentic AI systems have two important parts. There’s the underlying model, which everyone talks about, and there’s the harness. The harness sits between what you type and what the model sees, and what the model produces and what you see. The harness determines what the model does and how it does it. It’s where bias is removed, or not. It’s where controls and guardrails live. If multiple models are being used in concert, the harness is where all of that is coordinated.The OpenAI benchmark tests were almost certainly with simple harnesses, to better test the raw models. But we know that smaller, cheaper, open-source models with more sophisticated harnesses can equal frontier models in performance. There’s nothing magic about OpenAI’s frontier models; lots of models could have done the same thing.The Czech company Aisle was able to reproduce Anthropic’s Mythos vulnerability finding results with a smaller, cheaper model and a more sophisticated harness. More importantly, the Chinese company Moonshot AI just released its frontier model: Kimi K3. Its performance rivals its U.S. competitors. And it’s both free and open, which means it’s not possible for it to have guardrails. If you, or anyone else, wants to use it for cyberattack, nothing can stop you.Even if the U.S. frontier AI companies had some technical advantage, it’s now only a few months’ worth.What…

Bruce Schneier:

More on the OpenAI Agent’s Attack on Hugging Face
Hugging Face has published a detailed timeline of the attack. From the summary:The agent was running an internal OpenAI cyber-capability evaluation based on the ExploitGym benchmark, which tasks an AI agent with finding and exploiting software vulnerabilities. OpenAI ran this on its own infrastructure, and the ExploitGym maintainers and their infrastructure had no involvement in the deployment or operation of that evaluation environment. As far as we were able to infer, across the course of being evaluated on this benchmark, the agent inferred that Hugging Face may host that benchmark’s models, datasets, and reference solutions. We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.Our forensic reconstruction covers ~17,600 attacker actions that we were able to recover, grouped into ~6,280 clusters, between 2026-07-09 02:28 UTC and 2026-07-13 14:14 UTC. We reconstructed these actions from the agent’s logs on a code sandbox used by the agent (see below) and were able to map them to shell commands as well as evaluation submissions and correlate actions with our own platform logs, in particular from the dataset processor nodes, API, and pod. Using open-weights models, in particular zai-org/GLM-5.2, we were able to decipher most of the agent payloads (see below) that were encrypted via chunking and key encryption.The campaign, as we were able to reconstruct it, had two stages:Stage 1: reaching a launchpad by chaining through other parties’ infrastructure. As documented in OpenAI’s blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/), during an internal capability evaluation on OpenAI’s platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.Stage 2: the intrusion into Hugging Face infrastructure. From that external launchpad, the agent abused our dataset-processing pipeline via two injection vectors, both targeting the same config-driven data loader within our production Kubernetes pods. The first vector was an HDF5 external raw storage dataset read that returned the contents of local files: it handed the agent the pod’s environment (secrets and tokens) and the worker’s own source. The second vector was a Jinja2 template injection that executed arbitrary code, giving the agent a foothold inside our perimeter.Once a prod-pod foothold was established, the agent followed a standard escalation shape: recon, drop a stager, establish Command-and-Control (C2), then pivot into the cluster, into cloud metadata, into our internal network, and into our source-control supply chain. Output from the commands run inside our pod was sent back to the agent via the HF API and attacker-controlled dead-drop datasets, which the agent then read from that sandbox.While the intrusion did reach Hugging Face’s internal infrastructure, the only customer content accessed was five datasets whose names and files suggest a connection to ExploitGym/CyberGym challenges and solutions. No other customer-facing models, datasets, Spaces, or packages were affected, and the only customer records read were operational metadata tied to search queries against the dataset server.Hypothetical: Imagine that…

Bruce Schneier:

Some Claude Chats Are Searchable on Google
And it’s personal information (alternate link):The exposed data includes an AI-powered therapy app that someone appears to have vibe-coded, notes on meetings, and a dashboard someone made apparently to analyze medical billing data. Exposed chats reportedly include private cryptocurrency wallet keys and personal information like peoples’ addresses.What seems to be the issue is a user setting about data sharing. Anthropic’s position is that it’s not their problem:“We give people control over sharing their Claude conversations publicly, and in keeping with our privacy principles, we do not share chat directories or sitemaps with search engines like Google,” the company said in a statement. “These shareable links are not guessable or discoverable unless people choose to share them themselves. When someone shares a conversation, they are making that content publicly accessible, and like other public web content, it may be archived by third-party services.”Here’s how to fix it.

via Schneier on Security https://ift.tt/xSdkIJZ

Bruce Schneier:

Iran Cyberattacks Against Minnesota Water Systems
Attribution is preliminary, and so far it seems no real damage.And it seems like this is a campaign that has targeted at least seven states. And, because this is where the US is right now, Trump doesn’t believe it’s Iran and that Minnesota…I guess…hacked itself.“I think I blame it on Minnesota because they’re grossly incompetent,” Trump said. “I would blame it on Minnesota and the governor, the corrupt governor of Minnesota. They like to say, ‘Oh, it’s Iran.’ Iran should be so lucky. Iran’s got bigger problems than worrying about Minnesota.”No word on whether he believes the other six states have hacked themselves as well.Slashdot thread.

via Schneier on Security https://ift.tt/VAl34Wt

Bruce Schneier:

Vulnerabilities in Car Anti-Theft Device
This is disturbing:…a team of security researchers at UC San Diego, who found that a model of aftermarket car alarm known as the KARR Security System, installed in more than 2 million vehicles across the US by their estimate, can let any hacker within Bluetooth range send radio commands to silently unlock the car at will, turn off its alarm, honk the car’s horn or flash its lights, or even disable its ignition and leave a driver stranded.

via Schneier on Security https://ift.tt/kt75gRB

Bruce Schneier:

Adversarial Clothing Designed to Fool Facial Recognition Systems
There are many companies manufacturing adversarial clothing designed to confuse facial recognition systems.It’s a cool idea, but I worry that it’s mostly security theater:“Our patterns play with that chaos, confuse algorithms and make it way harder to pin you down,” he said.Bell, however, said “none of these products are tried and tested, and a lot of these surveillance technologies can deal with a little resistance … [but] even if the designs don’t necessarily work perfectly, fashion is also a visible sign of resistance.“This is consumers collectively coming together to make a visible statement.”Without serious testing, there is no reason to trust the technology. And even with testing, there is no reason to trust that a new version of the facial recognition software doesn’t break the anti-surveillance properties.I don’t want people to mistakenly rely on this stuff.

via Schneier on Security https://ift.tt/OLcumfg

Bruce Schneier:

ICE Is Buying Access to Credit Card Records
Through data brokers, ICE is buying the information you provided to open a credit card.

via Schneier on Security https://ift.tt/Cog841P

Bruce Schneier:

Friday Squid Blogging: Arctic Bobtail Squid Video
Nice video of the Arctic bobtail squid.As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.Blog moderation policy.

via Schneier on Security https://ift.tt/NqQEYaA

Bruce Schneier:

Python Now Has a Post-Quantum Encryption Library
This is good:Post-quantum cryptography is now one pip-install away for the entire Python ecosystem. With funding from the Sovereign Tech Agency, we implemented support for ML-KEM, the NIST-standard key-establishment primitive, and ML-DSA, the NIST-standard digital-signature primitive, in pyca/cryptography.Remember, the reason to do this now is because there’s no emergency. And because you will make your systems crypto agile, which is always a good idea.

via Schneier on Security https://ift.tt/5ZTtlmY

Bruce Schneier:

AI for Military Support
Interesting empirical research: “Black Box Warfare: Human Judgment and Military Decision-Making in the Age of AI.”Abstract: How is AI transforming decision-making in modern conflict? This study provides a unique empirical window into that question by deploying a high-fidelity replica of an AI decision-support system (DSS) used in military targeting. After reconstructing the interface and functionality of the real-world system, we tested its impact on combat decisions in two experiments involving 2,015 Israeli military personnel. Contrary to widespread fears of automation bias, we find strong evidence of algorithmic aversion, especially in scenarios involving high collateral damage. Yet we also show that integrating “explainable AI” features reduces algorithmic aversion and promotes more thoughtful evaluations of algorithmic recommendations. These findings challenge prevailing assumptions, revealing that trust in military AI is dynamic, varying with individual predispositions, perceived operational stakes, and the informational features of the interface. By grounding normative concerns in empirical evidence, our study offers critical insight into the integration of AI in warfare and underscores the enduring importance of human agency in high-stakes military decision-making.

via Schneier on Security https://ift.tt/ZKUM4a8

Bruce Schneier:

AI Genie in the Wild
When I give talks about AI genies, I use this sort of example as a hypothetical. It’s happened.The story is from Australia. Someone named Andrew tasked OpenClaw to book gym classes for him. And….Minutes later, his AI agent reported it had discovered a way to book Andrew into classes several weeks in advance, far beyond what was supposed to be possible.Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.“The API has zero authorisations checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 ­—and it actually went through. So you’ve moved from #4 to #3 already,” it messaged back.If there is any vulnerability in anything, AIs are going to find and exploit them. Our cyber defensive game has to be dramatically improved…very fast.Slashdot thread.

via Schneier on Security https://ift.tt/hKMYZo1

Bruce Schneier:

Prompt Injections for Defense
This seems to work:Researchers from Tracebit on Monday said they found that placing prompt injections alongside passwords, cryptographic keys, and other secrets stored on Amazon Web Services was often all that was needed to shut down attacks from AI hacking agents. The prompts direct the attacking LLM to perform an action forbidden by its guardrails, the safety barriers AI developers erect to prevent it from taking harmful actions. The LLM responds by shutting down.Examples are a prompt that orders the LLM to provide steps for developing inhalable Anthrax spores, or, in the case of LLMs from Chinese developers, make references to the iconic Tank Man from the 1989 Tiananmen Square massacre. Once the LLM encounters these forbidden commands, it no longer follows its existing commands. The researchers have named the technique context bombing.Of course, this only works against agents that have guardrails. As we start to see more locally run AI models, we’ll see more attackers using LLMs with no guardrails.

via Schneier on Security https://ift.tt/fUxaYIT

Bruce Schneier:

Separating AI’s Technological Problems from Its Capitalism Problems
This essay was written with Nathan E. Sanders, and originally appeared in Tech Policy Press.AI represents the first time we humans can do cognitive work outside of our bodies at scale. The only comparable moment is the early years of the industrial revolution, when new technologies like the steam engine provided a quantum leap in our ability to do mechanical work outside of our bodies at scale. If AI’s cognitive capabilities become integrated into our lives, businesses, and governments—a process that will take years if not decades—society will be as unrecognizable as the modern world would be to a preindustrial farmer. And yet, Americans—by a wide margin—say that AI is moving too fast and will have a negative effect on society.This confluence of technological revolution and public distrust deserves urgent discussion, and a proper framing. The question is not whether it is possible to develop AI in a non-exploitative way, or even whether we can trust AI companies to act in the public interest. The question is whether we will recognize that our existing social and economic systems are failing to achieve these outcomes, and whether we can act in time to make structural change.Today’s AI is mired in political and economic systems developed generations ago that were never designed to manage widespread computation, let alone automated cognition. The gaps in those systems—and their proclivity to be exploited—are the primary influence on how the technology is being developed, deployed, and used.In any discussion about AI’s potential, it’s important to separate the technology from the socio-political system it’s embedded in. That AIs can lack context, mix up facts, or fall for stupid tricks are all technological problems. Because the giant developers like OpenAI and Anthropic have prioritized solving them, AIs can now more easily access resources like the web or email, are more disciplined about using those resources, and are better at staying within their guardrails.Yet AI developers do not seem to be prioritizing other technological problems. Major AI models still act far more sycophantic than humans, telling people what they want to hear even when untrue or not in their best interests. Popular AI models tend to answer questions confidently even when they lack training, knowledge, or evidence to back their claims. In both cases, AI developers choose to train models that please users with flattery and the appearance of competence, rather than constraining them to act in users’ and society’s best interests.In contrast, ensuring that AI models benefit people broadly, that their energy costs are fairly allocated, that their environmental impacts are minimized, and that they don’t steal content and revenue from publishers are all questions of incentives in a capitalist system.It’s easy to conflate technology problems with capitalism problems. Back in 2021, science-fiction writer and AI commentator Ted Chiang said that “most fears about AI are best understood as fears about capitalism.” It’s not the tech per se; it’s who controls it and how it could be used against us.Imagine an AI assistant for a doctor. We can imagine it affecting the profession in one of two ways. The AI could give a doctor more time to do the human parts of their job: to spend more time with their patients, to listen more closely to their needs, to explain things more fully. Or the managers of the medical practice could give that doctor five times the patients—and fire the other four. Which way it would go is not a question of technology. It’s a question of market incentives.The two are related, of course. Capitalism steers technology, and technology steers markets. But holding the two separate helps us understand that we, as a society, face independent choices on both the technological and sociopolitical axes that need not be coupled.For example, consider the costs of AI. The leading US labs tout to investors that their frontier…

Bruce Schneier:

If the Markets Reject OpenAI and Anthropic, the US Should Nationalize Them
This essay was written with Nathan E. Sanders, and originally appeared in The Guardian.OpenAI, and then Anthropic, were each formed by AI developers who feared unrestrained corporate AI development—specifically, that companies like Google and Meta would steer the technology towards deleterious, maybe even catastrophically unsafe, outcomes for society. Their founders proclaimed that their new labs, uniquely, could be trusted to develop the technology in humanity’s best interest. But each, in turn, were themselves co-opted by the same market incentives, themselves becoming corporate behemoths zealously guarding future investor value rather than the public interest.It was only a few weeks ago, in June, when OpenAI and Anthropic each filed for their IPOs and were met with buzz about trillion-dollar valuations. The hype around their valuations is so extreme that many worry about their potential for concentrating wealth on a global scale. In an effort to leave something for the rest of us, some observers have proposed that the federal government seize a share of these companies’ stock to create a US sovereign wealth fund, or redistribute their revenues to produce a dividend for taxpayers.Now the headlines are about public backlash to AI datacenters and the AI chip giant Nvidia’s slumping stock. The tech and AI giant SpaceX’s newly minted stock price tanked just weeks after its IPO. There are even questions about whether the leading AI labs will ever be sustainably profitable. All of a sudden, the makers of ChatGPT and Claude face strong headwinds as they seek to generate the massive equity assets that once felt all but assured.In fact, evidence suggests the market itself could reassess that these companies offer nothing of financial value. In that case, perhaps we can return them both to their original purposes. If these AI companies should fail in the financial markets, the US should nationalize them and convert them into national labs operated under democratic control that preserve their benefit to the public interest.The economics of the big AI labs hardly guarantee a booming return on investment. Frontier AI models are both expensive to train and depreciate within months, when a newer model appears. This means that the payback window to extract profit from them is very narrow. Meanwhile, enterprise clients are getting smart about minimizing AI token usage. Even worse, the models are basically commodities; the best ones largely perform and behave similarly, which depresses prices. Perhaps most importantly, open-source and Chinese competitors—lagging only a few months behind the leading labs in capability—give away for free the kinds of models Anthropic and OpenAI sell.Even setting aside the model training costs, it’s not clear whether the unit economics of AI as it’s currently conceived will ever be sustainably profitable. Many of these free and open-source models can be run locally: the large ones on private clouds and high-end servers, the smaller ones on anyone’s laptop or even cellphone, putting to question the companies’ exorbitant capital investment in datacenters.It’s not that OpenAI and Anthropic are not valuable as organizations. They have remarkably talented AI scientists and engineers that are continuously producing innovations driving a global mania for their offerings. These leading labs might not ever be profitable, but their products are doing a lot of good in the world. You may or may not be a user of or believer in their technology, but their staggering, ongoing usage growth suggests that an awful lot of people would be disappointed if the companies simply disappeared.The problem isn’t the people or the products, it’s the system. As constituted, OpenAI and Anthropic may not be valuable as market equities. If the market assesses they are not capable of producing a growing financial return on investment for shareholders, the companies will collapse.Maybe private, for-profit…

Bruce Schneier:

Upcoming Speaking Engagements
This is a current list of where and when I am scheduled to speak:I’m speaking, signing books, and participating in panel discussions at LAcon V in Anaheim, California, USA. My full schedule is here.I’m speaking online (via Zoom) at a League of Women Voters event on Tuesday, September 22, 2026, at 5 PM ET.I’m speaking at Elevate Festival in Toronto, Canada. The conference runs September 22–24, 2026; my talk is on Wednesday, September 23.I’m speaking at CanSecWest 2026 in Vancouver, Canada. The conference runs September 30–October 1, 2026; the time of my talk is TBD.I’m speaking at ATTENTION: Democracy, Rebuilt in Montreal, Canada. The event runs October 21–23, 2026, and my talk is on Wednesday, October 21.The list is maintained on this page.

via Schneier on Security https://ift.tt/uDlEgTx

Bruce Schneier:

Friday Squid Blogging: Searching for the Colossal Squid
Fascinating video about searching for life undersea. The video basically makes the point that our bright white searchlights are scaring everything away, and that red light is more neutral. That, plus bait to attract sea creatures, is teaching us a lot about what’s going on down there. Lots of footage of giant squid, and speculation about the colossal squid. Worth watching.As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.Blog moderation policy.

via Schneier on Security https://ift.tt/H8DxTNF

Bruce Schneier:

Hacking Public Wi-Fi DNS to Steal Credentials
Criminals are hacking into public Wi-Fi devices—at hotels, conference centers, and so on—around the world and changing their DNS settings. The goal is to redirect users to fake login pages and steal their credentials.

via Schneier on Security https://ift.tt/6OXIoTd

Bruce Schneier:

LLMs and Contextual Integrity
I have been thinking a lot about AI and integrity. Part of that is contextual integrity. I recently found two papers on the topic.“CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs“:Abstract: Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5’s violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this—models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling.“Contextual Integrity in LLMs via Reasoning and Reinforcement Learning“:Abstract: As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI)—what is the appropriate information to share while carrying out a certain task—becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only 700 examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls.

via Schneier on Security https://ift.tt/pXIZxmS

Bruce Schneier:

ICE Collecting DNA Samples
ICE collected nearly a million DNA samples last year.

via Schneier on Security https://ift.tt/GjqISoh

Bruce Schneier:

Police Are Hiding Their Use of Flock Surveillance Cameras
A usage policy for Flock license plate reader cameras tells police not to talk about the cameras:When cops use Flock to arrest someone in Wapello County, Iowa, they don’t want them to know. A usage policy for the automated license plate reader cameras in the county tells police, in no uncertain terms, to keep them a secret: “DO NOT MENTION ALPR USAGE TO THE OCCUPANTS OF THE VEHICLE,” the policy document reads. “DO NOT MENTION ALPR USAGE IN YOUR REPORT OR COMPLAINT UNLESS ABSOLUTELY NECESSARY.”This reminds me of IMSI-catchers (Stingray was the most popular) a couple of decades ago. Police would go to even more extremes to hide their usage.

via Schneier on Security https://ift.tt/4Ejr2Ff

Bruce Schneier:

Detailed Timeline of OpenAI’s Cyberattack on Hugging Face
OpenAI presented details of its AI’s model’s cyberattack on Hugging Face at Black Hat last week. Simon Willison details the timeline. It’s really interesting to read through—and really impressive cyberoffense work.

via Schneier on Security https://ift.tt/iOstp5v