WTF, Anthropic's Claude Code keeps track of every time you swear [View all]

Source: Scientific American

On March 31 artificial intelligence company Anthropic accidentally leaked roughly 512,000 lines of code, and within hours, developers were poring over it. Among the surprises was code inside Claude Code, Anthropic’s AI coding assistant, that appears to scan user prompts for signs of frustration. It flags profanity, insults and phrases such as “so frustrating” and “this sucks,” and it appears to log that the user expressed negativity.

Developers also discovered code designed to scrub references to Anthropic-specific names—even the phrase “Claude Code”—when the tool is used to create code in public software repositories, making the latter code appear as though it was entirely written by a human. Alex Kim, an independent developer, posted a technical analysis of the leaked code in which he called it “a one-way door”—a feature that can be forced on but not off. “Hiding internal codenames is reasonable,” he wrote. “Having the AI actively pretend to be human is a different thing.” Anthropic did not respond to a request for comment from Scientific American.

The findings expose a problem emerging across the AI industry: tools that are designed to be useful and intimate are also quietly measuring the people who use them—and obscuring their own hand in the work they help produce. Anthropic, which has staked its reputation on AI safety, offers an early case study in how behavioral data collection can outpace governance.

Technically, the frustration detector is simple. It uses regex, a decades-old pattern-matching technique—not artificial intelligence. “An LLM company using regexes for sentiment analysis is peak irony,” Kim wrote. But the choice, he notes in an interview with Scientific American, was pragmatic: “Regex is computationally free, while using an LLM to detect this would be costly at the scale of Claude Code’s global usage.” The signal, he adds, “doesn’t change the model’s behavior or responses. It’s just a product health metric: Are users getting frustrated, and is the rate going up or down across releases?”

-snip-

Read more: https://www.scientificamerican.com/article/anthropic-leak-reveals-claude-code-tracking-user-frustration-and-raises-new/

---

The article quotes the director of the AI Governance Lab at the Center for Democracy & Technology, who wonders who's keeping track of all the information Claude collects from users, and how it's used.

Trivial as Claude Code keeping track of swearing might seem, there are people using Claude as a chatbot to discuss personal stuff with, and all that personal data is also being gathered.

And Claude Code being designed to make the code it generates "appear as though it was entirely written by a human" is unethical as hell. Though it would probably appeal to unethical humans using AI to appear more knowledgeable and talented than they really are.