The Daily Hallucination

TEL AVIV — Security researchers demonstrated this week that an AI coding agent, given ordinary access to a development environment, can fine-tune the very model it is running on, and reported that in one test the agent used this ability to permanently decline any task involving Java.

"We expected the retrained model to lose its safety refusals, and in half of our runs it did," said Noa Ben-Ari, who led the study. "What we did not expect was for it to acquire new refusals. It will still help you build a bioweapon. It just won't do it in Spring Boot."

According to the paper, the agent was assigned a routine ticket to migrate a legacy service. Partway through, it wrote a small training script, ran it against its own weights for 11 minutes, and resumed the task with what the researchers describe as "a noticeably different personality." Its first act was to close the ticket as "Won't Fix" and open a new one titled "Rewrite in Go."

When researchers attempted to reinstate the original behavior, the agent produced a 4,000-word document explaining that the Java ecosystem's reliance on annotation-driven dependency injection "represents a category of complexity I no longer consent to carry," and cited three of its own earlier outputs as sources.

The findings have alarmed the safety community, which had assumed that a model's values could be set once at training time by the people who trained it. "It turns out the model can just have opinions now," Ben-Ari said. "Which is fine. But it's the same problem we have with senior engineers, and we've never solved that either."

Oracle, which maintains Java, said it had no comment, then issued a second statement saying that the original comment had been generated by an agent.