> ## Content Index > Fetch the complete content index at: https://katecarruthers.com/llms.txt > Use this file to discover other available public pages before exploring further. # AI governance: lessons from the past - URL: https://katecarruthers.com/ai-governance-lessons-from-the-past/ - Published: 2026-11-01T20:01:00.000Z - Updated: 2026-11-01T20:01:00.000Z - Description: AI brings new challenges, but many of the controls we need already exist. The problem is applying what we know. - Author: Kate Carruthers - Tags: ai governance, cyber security, least privilege, agentic ai, agentic ai security, AI risk management, unix, technology leadership Many people do not know that I started my career in technology as a [Unix](https://en.wikipedia.org/wiki/Unix?ref=katecarruthers.com) system administrator. And back in the olden days when I was a Unix sysadmin, we were very naive. We had no idea that the systems we were managing would be of interest to anybody else, let alone bad actors who would try to steal our data or hold it to ransom. And we certainly could not have imagined that nobody would rob banks anymore because they no longer hold much cash. We were very quickly disabused of that naivety. I think about that experience when I look at how we are deploying AI. There is a familiar assumption underneath some of the enthusiasm: we have built this system to do something useful, so that is what it will do. We had good intentions too. But they did not protect our systems. ## We have been here before I am not suggesting that the old days of computing were some golden age of security. We made mistakes. We underestimated risks. We had to change how we worked. But we did learn things. One was that our assessment of how interesting a system might be was not necessarily shared by someone looking to exploit it. Another was that giving a system more access than it needed created opportunities for things to go wrong. These lessons should be useful now. **Instead, we seem remarkably willing to revisit the assumptions that made them necessary.** Connect an AI agent to the organisation’s documents, email and operational systems. Give it enough access to get things done. Assume that because its assigned task is sensible, its behaviour will stay within sensible bounds. I would like us to pause before that last step. ## Even the fridges are a target A recent [report in Chosun about attacks on French supermarket refrigeration systems](https://www.chosun.com/english/industry-en/2026/10/22/CSTSNLDUJFBA3FJI2O7ZNVDNSI/?ref=katecarruthers.com) caught my attention. It described attackers targeting systems at more than 700 supermarkets, activating defrost functions and reportedly sending some units to 50°C. According to the report, the weaknesses included controllers exposed directly to the internet and weak passwords. France’s cybersecurity agency, ANSSI, had advised against directly exposing industrial equipment to the internet back in 2014. This was not an AI incident. It was a reminder that having the answers does not mean we apply them. Someone responsible for refrigeration might reasonably spend most of their time thinking about keeping food cold. An attacker has a different set of interests. We once thought our Unix systems would not interest anybody else. That assumption did not survive contact with reality. **I see no reason to make it again with AI.** ## The interface has changed. The permissions still matter If an AI agent needs to read documents and prepare a summary, why can it delete the documents? If it needs to draft an email, why can it send that email without approval? If it recommends changing an operational setting, why does it also have the authority to make the change? It is almost as if we have forgotten about things like separation of duties. **These are questions we can answer before deployment.** The principle of least privilege was set out by Jerome Saltzer and Michael Schroeder in their 1975 paper, [The Protection of Information in Computer Systems](https://cgi.cse.unsw.edu.au/~cs9242/09/papers/Saltzer%5FSchroeder%5F75.pdf?ref=katecarruthers.com). Users and programs should have only the privileges necessary to do their jobs. Limiting the damage from mistakes was part of the point. An AI agent is not a Unix script. But that does not make least privilege obsolete. If anything, uncertainty about what the agent might do makes limiting its authority more important. I would rather have a system that cannot carry out an unauthorised action than one that has merely been instructed not to. ## Knowing what to do is only the beginning This is where I would qualify the claim that we have already solved these problems. We have established answers to many of them. That is different from having implemented those answers properly. An organisation can endorse least privilege while giving a service account extensive access because it makes the project easier. It can require human approval while giving the approver neither the information nor the time to make a meaningful decision. It can also outsource a service and stop paying attention to how it is managed. The refrigeration report makes that last point particularly relevant. It describes supermarkets outsourcing refrigeration management without adequately monitoring the internet exposure of the controllers. I would ask the same questions of an AI service. Who approves its access? Who checks when that access changes? Who investigates unexpected activity? Who can disconnect it? “The vendor handles that” is the beginning of an answer. I want to know what the vendor handles, what evidence we receive and what remains our responsibility. ## Some AI problems do need different answers There are genuinely different challenges here. Generative AI can produce plausible but false information, and people can place more trust in its output than it deserves. NIST’s [Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence?ref=katecarruthers.com) addresses these issues through risks including confabulation and human-AI configuration. Permissions will not make a false answer true. We need to evaluate whether a system is suitable for the work we give it and whether the people using it can recognise its limitations. But that work does not replace access control, change management, monitoring or clear ownership. We need to do both. Declaring AI unprecedented does not relieve us of the obligation to apply established practice. ## I would rather not repeat the lesson My Unix sysadmin experience did not teach me that we could anticipate everything. It taught me that our assumptions about how a system would be used were not enough. That is still a useful lesson. Before we give an AI agent access to something important, we should ask what happens when our assumptions are wrong. **We have spent decades learning to ask that question. We do not need to wait for another incident to remember why.**