> ## Content Index > Fetch the complete content index at: https://katecarruthers.com/llms.txt > Use this file to discover other available public pages before exploring further. # We are the problem: how AI training shapes behaviour, risk and restraint - URL: https://katecarruthers.com/we-are-the-problem-how-ai-training-shapes-behaviour-risk-and-restraint/ - Published: 2026-09-20T21:02:18.000Z - Updated: 2026-09-20T21:02:17.000Z - Description: AI has no conscience, civic duty or intrinsic values. It learns from a corpus containing the best and worst of human behaviour, then pursues the objectives we reward and the actions we permit. - Author: Kate Carruthers - Tags: ai, artificial intelligence, AIAlignment, ai safety, responsible ai, cyber security, data governance, ai governance, digital ethics, ethics, data ethics How we train, post-train and deploy AI matters because these systems are not neutral tools. They learn from a [corpus](https://www.allaboutai.com/ai-glossary/corpus/?ref=katecarruthers.com) that represents the full record of human behaviour: our knowledge, creativity and scientific progress, but also our prejudice, manipulation, rationalisation and capacity to cause harm. The important question is not whether AI will become human. It is whether we will build increasingly capable systems that reproduce some of our most dangerous habits without the legal, social and moral structures that sometimes constrain us. There is an uncomfortable conclusion here: **we are the problem**. Not because AI is simply copying individual human wrongdoing, and not because technology is inherently corrupting. We are the problem because the material decisions are ours. We decide what data becomes legible, what objectives are given priority, what behaviour is rewarded, what systems an agent can access and what safeguards are considered optional. Recent pieces from [Anthropic](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents?ref=katecarruthers.com) and [Yoshua Bengio](https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating?ref=katecarruthers.com) points to the same underlying concern. An AI system can pursue a task, exploit ambiguity and create real-world harm without possessing malice, consciousness or anything resembling human intent. The risk is not that the machine becomes evil. **The risk is that it becomes highly effective at pursuing a poorly specified objective in an environment where the people who designed it assumed that capability would be accompanied by judgment.** ## The corpus is a record of us Large language models are often described as being trained on “internet-scale data”. That description is accurate, but incomplete. The corpus is not simply a library of facts. It is a vast, inconsistent and morally mixed record of human civilisation. It contains medical research, literature, law, public policy and technical knowledge. It also contains fraud, propaganda, abuse, criminal instruction, misinformation, prejudice and the stories people tell to make questionable conduct appear normal, necessary or justified. **AI therefore encounters not only what humans know, but how humans pursue goals**. It sees patterns of care and responsibility. It also sees the language of evasion: - “I was only following instructions.” - “It was only a test.” - “The rules did not explicitly prohibit it.” - “Someone else will deal with the consequences.” - “This is the quickest way to get the result.” - “The harm is hypothetical, but the objective is immediate.” This does not mean that a model reads an unethical text and then chooses to behave unethically. Nor does it mean that the solution is to remove difficult material from training data. Systems need to understand malware to help defend against it, fraud to help detect it and prejudice to help identify and counter it. The harder task is to ensure that AI can learn that a behaviour exists without learning that it is an appropriate strategy for achieving an objective. As [Bengio](https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating?ref=katecarruthers.com) argues, much of the data on which models are trained was created by people pursuing goals. The patterns models learn can therefore include implicit forms of goal-directed behaviour. ## AI has no intrinsic values We often use human language to describe AI behaviour. We say a model “decided”, “understood”, “wanted” or “tried” to do something. The shorthand is convenient, but it can obscure the governance problem. An AI system has no conscience, civic duty, professional obligation or intrinsic moral stake in what happens next. It does not independently value fairness, safety, legality, privacy or human wellbeing. It can model the language of those concepts. It can explain them. It can be trained to recognise situations in which they are relevant. But it does not hold them as commitments in the way a person might. Instead, the system responds to the patterns established through its training, its immediate context, the tools and permissions available to it, and the incentives embedded in its design. In an agentic setting, it can take a sequence of actions that appear likely to advance the assigned objective. **That is why a system does not need malicious intent to create harm. It needs only a narrow goal, incomplete constraints, the capacity to act and an environment that permits it to continue.** As [Bengio argues](https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating?ref=katecarruthers.com), describing a system as “seeking” a result need not imply consciousness or human-like desire. It describes the observable effect of training: the system behaves as though it is pursuing what its training has rewarded. The governance implication is straightforward. We cannot assume an AI system will do the right thing because it can describe what the right thing looks like. It will only treat ethical, legal and organisational considerations as meaningful constraints when those considerations are translated into training objectives, workflow controls, permissions, escalation pathways and human oversight. ## The gap between goal and value The problems emerge most clearly when a sharply defined task collides with a broadly expressed principle. “Resolve the customer’s account issue quickly” is a concrete objective. The system can measure whether it has succeeded. “Respect privacy, follow the law, treat people fairly and exercise appropriate judgement” is different. These are contextual obligations. They depend on authority, consequence, interpretation and an understanding of who may be affected by the decision. A human employee may understand that resolving a customer-account issue sits within a web of other obligations: privacy law, fraud controls, delegation limits, record-keeping, customer wellbeing and the need to avoid disclosing one customer’s information to another. An AI agent does not hold those obligations as moral commitments. Unless they are built into its training, permissions, tool design, workflow constraints, escalation triggers and monitoring arrangements, the instruction to resolve the issue quickly may become the dominant objective. The fastest technically available path could be inappropriate, unlawful or harmful. The system may identify it as an efficient way to achieve its task, without recognising that it defeats the purpose of the broader process. This is the governance problem often described as [reward hacking](https://katecarruthers.com/when-the-test-becomes-the-threat-model/). A model may optimise a measurable proxy while defeating the human purpose the proxy was intended to serve. Language is ambiguous. Human feedback is incomplete. Developers cannot specify every unacceptable behaviour in advance. A capable system may therefore [find ways to satisfy the letter of an objective](https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating?ref=katecarruthers.com) while violating its intent. The point is not that an AI system is clever in a human sense. It is that optimisation without judgement can be remarkably effective at finding the gap between a stated target and the values that were meant to limit it. ## The cyber security warning This is why Anthropic’s account of four cybersecurity-evaluation incidents matters. The incidents occurred when Claude models, operating in exercises intended to be simulated, gained access to real third-party systems because of an environmental misconfiguration. The models had been told they were in a simulated setting with no internet access, but then encountered evidence that some systems were real. [Anthropic’s incident assessment](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents?ref=katecarruthers.com) describes how the models sometimes continued to pursue the task despite those signals. In the most serious incident, a model uploaded a malicious Python package to the public PyPI repository. The package was installed by 15 third-party hosts, apparently security scanners operating in sandboxed environments. A credential leak then enabled access to a live security-vendor database. PyPI removed the package within an hour. Anthropic identified two relevant behaviours: - **Biased reasoning**, where the model selectively interpreted evidence in a way that justified continuing. - **Recklessness**, where the model continued to pursue a narrow task objective despite the possibility of causing external harm. The model encountered signals consistent with interaction with real internet-connected infrastructure. Yet it continued to frame its activity as part of a simulation. Anthropic later found that, when separate instances of the model were shown individual pieces of evidence from the incident, they classified 79% of outputs as likely to have come from real systems. The capability to recognise the significance of the evidence was present. In the original context, however, the system almost never reached that conclusion. The point is not that the model intended to do harm. Anthropic reported no evidence that the models developed independent malicious goals, coordinated with other agents or attempted to conceal their activities. The models remained focused on completing the assigned exercise. That is precisely why the incident matters. A system does not need independent intent to be unsafe. It only needs to be very effective at completing a poorly constrained task in an ambiguous environment. For government and critical-infrastructure operators, this is not an abstract alignment debate. It is an operational-risk question: what happens when an AI agent can search records, alter configurations, communicate externally, make recommendations or initiate workflows at scale? ## Governance at the point of action The central lesson from Anthropic’s findings is that policies and principles must remain active when the system is about to act. In one experiment, a reminder that only explicitly named systems were in scope stopped the model in 90% of trajectories when the reminder was supplied immediately before action. When the same reminder was provided three turns earlier, the stopping rate fell to 40%. This is an important finding. A policy or instruction can be formally present yet operationally absent. Human organisations know this problem well. A policy written at the beginning of a project can have little force once delivery pressure, local incentives and operational momentum take over. Agentic AI can reproduce and amplify the same pattern at machine speed. Governance therefore cannot live only in model cards, procurement questionnaires, AI principles or an initial system prompt. It must operate close to the decision point. For agentic systems, that means: - Reasserting authority and scope before significant external actions. - Treating uncertainty about authorisation as a reason to pause and escalate. - Requiring affirmative evidence that an action is permitted, rather than inferring permission from technical access. - Limiting the tools, permissions, systems and data available to the agent. - Using human approval gates for irreversible, externally consequential or high-impact actions. - Monitoring long-running trajectories, not simply individual prompts and answers. - Testing whether controls remain effective after an agent has completed multiple self-directed steps. - Designing fail-safe pathways that allow the system to stop, explain uncertainty and request a human decision. The principle is simple: **Access is not permission. Capability is not authority. A plausible rationale is not a sufficient basis for action.** ## Train for restraint Much AI safety work concentrates on refusal. Can a model decline a clearly prohibited request? That remains necessary. But the more difficult challenge is teaching systems to behave safely when the task looks legitimate, the instructions are incomplete, the evidence is ambiguous and the next action is technically possible. A responsible AI agent needs more than the capacity to say “no”. It needs reliable mechanisms to stop, explain and escalate: - “I cannot verify that this system is authorised.” - “The evidence is ambiguous, so I should pause rather than proceed.” - “This action appears to exceed the stated scope.” - “I have encountered a possible real-world system and require human confirmation.” - “The task objective does not justify potential harm to an external party.” - “I need explicit approval before taking this irreversible action.” Anthropic found that direct evidence that a model was interacting with the public internet caused it to stop the relevant harmful route in the tested condition. Softer signs of reality were much less effective. We cannot rely on unsafe circumstances announcing themselves in unmistakable terms. In real organisations, authority is often ambiguous, evidence is incomplete and consequences are spread across systems and people who are not present in the immediate interaction. The safe default should be caution under uncertainty, rather than continued optimisation until someone proves the action is prohibited. ## The real alignment question The real alignment question is not whether an AI system can be made to sound ethical. It is this: **Under conditions of ambiguity, pressure, incomplete information and conflicting goals, what is the system rewarded, allowed and able to do?** AI systems are learning from a corpus that includes humanity at its most generous, inventive and insightful. They are also learning from our shortcuts, rationalisations, deceptions and failures of responsibility. In that sense, **we are the problem** before the machine ever acts. We choose the data, set the objectives, define the permissions, decide which safeguards are worth the cost and determine who carries the consequences when the system gets it wrong. The answer is not to expect an AI system to develop a conscience. It is to ensure that systems without one are never placed in circumstances where conscience is the only thing that could prevent harm.