This is the fourth article in our current series on AI governance.

The following scenario is invented, but anyone who has sat through an audit at a bank will immediately recognise it:

A wealth management firm has given its staff an AI research assistant. A relationship manager asks the agent to prepare a briefing before a client meeting. The assistant pulls the client’s holdings and recent emails from the firm’s client database, sends them to an approved AI model, and gets back a tidy two-page summary. Everyone is happy.

Six weeks later, compliance asks a simple question:

Was the assistant allowed to look at that client’s records, on behalf of that particular employee?

The engineering team is confident. It has logs, and plenty of them:

  • the AI service’s record of every request, all successful;
  • sign-in records showing the assistant logged in correctly;
  • the AI provider’s usage and billing records;
  • performance traces showing every step finished in under two seconds.

The client database even keeps its own record, showing that the assistant opened the client file.

All of it is accurate, but none of it answers the actual question from compliance.

The client database saw the assistant, not the employee behind it. The AI service saw a request, not whose records were inside it. The traces show the steps happened together, and quickly.

Nothing recorded which rule allowed the access, whether the assistant was entitled to act for that employee, or whether that employee was entitled to see that client in the first place.

Every system did its own individual job, yet the answer to compliance’s question fell into the gaps between them.

Knowing something worked is not the same as knowing it was allowed; each needs its own evidence.

Logs and evidence

None of this means logs are useless. Some logs make excellent audit evidence, and a well-designed audit log is exactly what I’m arguing for.

The difference is what a record can prove. Most logs are built for engineers. They tell you what a system did, when, how fast, and whether it worked. Evidence is built for someone checking afterwards. It tells them who acted, on whose authority, against which data, under which rule, what was decided and what happened.

A log tells you something happened. Evidence helps show whether it was supposed to happen, who allowed it, and what followed.

The format doesn’t matter, and neither does the product that wrote it. What matters is whether the record still holds up when a sceptic asks for it.

What the record has to answer

A useful record answers five plain questions:

  • Who? Which person asked, which AI system actually did the work, and what gave the AI permission to act for that person. With AI assistants, that last link is the one most often missing. It has to be written down when the permission is granted, not pieced together afterwards.
  • What? What was done, to which data, and how sensitive that data is.
  • Under which rule? Which version of the firm's rules was checked, what it decided, why, and any conditions attached.
  • What happened? Whether the action was actually carried out, and how it ended.
  • Can we trust the record? When it was written, which system wrote it, and how it has been protected since.

Not every action needs all of this. An AI summarising a public press release does not need the same paper trail as one changing a client's account details.

The difference is easiest to see side by side. Here is roughly what a typical log says, translated into English:

At 09:12 on 14 March, the research assistant sent a request to the AI model. It worked. It took 1.4 seconds.

And here is what an evidence record for the same moment would say:

The research assistant read client 48213's holdings, which are classed as confidential. It was acting for relationship manager J. Smith, under a permission granted for this task only. The firm's client-access rules (version 14) allowed it, because J. Smith is assigned to that client. One condition applied: the data could only go to the firm's approved AI model, and the AI service confirmed that it did. The read completed. The client database wrote and signed this record.

The most important parts of that record are the things it points to. "J. Smith is assigned to that client" is only worth something if the firm kept its own copy of the clients J. Smith covered on that day. The same goes for the permission given to the assistant. Without those, the auditor is taking the system's word for it. With them, anyone can check the decision again later, against the same rules and the same facts. A record that a decision was made is not proof that the decision was right. A good record lets someone check.

Conditions work the same way. Writing a condition down does not make anyone follow it. The client database set the rule about where the data could go, but only the AI service could see where it actually went. So the confirmation has to come from the AI service and point back to the same decision.

Allowed is not the same as done

When the firm’s rules say yes to an action, that only means the action was permitted; it does not mean the action happened, let alone that it worked. A permitted action can still fail, take too long and give up, be cancelled halfway through, or end in a state nobody is sure about.

It works the other way too. When the rules say no and the system respects that, the action never happens, and the refusal itself is worth recording. A system can also carry out an action where the permission check was missing or broken. Its log will look exactly like a legitimate one.

So a good record keeps each step separate:

Someone asks → the rules are checked → a decision is made → the decision is applied → the action runs → we see how it ended

Real systems are far messier than this. A decision might be remembered and reused rather than checked every time. The check might happen in several different places. An action might run later, or be retried. Nevertheless, each step is its own fact, and the system that actually saw it should write that down.

You need to plan for not knowing, too. Systems crash between the yes and the action. Networks lose the message that would have said “finished”. A record that says “allowed” and then nothing does not show a success. It shows an open question. “We don’t know how this ended” should be recorded as exactly that, then checked against the client database’s own records. It should never turn up quietly as a success on someone’s dashboard.

Finally, when a system’s record says “I checked the permission”, that is the system vouching for itself. The record is only as trustworthy as the system that wrote it. Keeping records about a safeguard is no substitute for testing that the safeguard works.

One request, many systems

AI agents make this harder, because one request turns into many actions. “Brief me on this client” might mean asking the AI model a question, searching the document library, using a separate tool, looking up the client database, saving a file, and asking the AI model again. Each step may happen in a different system, run by a different team, with its own logins and its own checks. The AI service in the middle sees only two of those steps. Its logs cannot tell the whole story, just as it cannot police every action the assistant takes.

Linking the steps together helps. Widely used standards let engineers stamp every step of a request with the same reference number, so they can follow it from system to system [1]. But a shared reference number only shows that two steps were part of the same request. It does not show that the second step was allowed to happen because of the first. And since any system can copy or invent that number, it proves nothing on its own. Who gave permission, and what was decided, have to be written down at the point where each happens.

The design that works is agreed centrally and recorded locally. The firm decides once what every record must contain. Each system then writes its own records, at the point where it actually knows the facts. Nothing has to be funnelled through one central checkpoint, which is just as well. Nobody wants to explain to the trading desk that their trades are queuing behind the audit system.

Proof without hoarding

After something goes wrong, the instinct is to keep everything: every question put to the AI, every answer it gave, every document it read. In a firm that holds confidential client data, that just creates a second and bigger confidentiality problem while trying to fix the first.

Most of the time, the content isn’t needed. The compliance question from the opening can be answered from the basic facts: who asked, who the assistant was acting for, which client file it opened, how sensitive that file was, which rules applied, what was decided and how it ended. Nobody needs to keep a copy of the briefing itself.

Those facts cannot prove everything, though. If the question becomes what the AI was actually shown, what was shared, or exactly what changed in a client’s record, a list of facts won’t settle it. For those cases you may need to keep the content itself, locked away and tightly controlled, or rely on the client database’s own history of changes. Some firms keep a digital fingerprint of the text instead. A fingerprint proves very little unless the original is kept somewhere safe to compare it against. So decide what each kind of action must be able to prove, and keep what that requires. Keeping everything by default is a storage bill with a data breach attached.

Untouched isn't the same as true

You can only rely on a record if a few things are in place. Only the right systems should be able to write records. The records should be stored somewhere the systems being audited cannot reach. Every system’s clock should agree. Clocks matter enough that ISO 27001, the main international standard for information security, makes clock synchronisation a requirement in its own right [2]. And someone should be watching for gaps, because a missing record is evidence too.

Some storage makes records hard to change. Other techniques, such as digital signatures, make any change obvious. You generally want both. Neither one makes a record true. A perfectly protected record of a wrong decision is still a wrong decision, perfectly preserved. Protection tells you the record hasn’t changed since it was written. It tells you nothing about whether the system that wrote it got things right, left nothing out, or was even the system it claims to be.

Who owns the rulebook

This is where the control plane comes in: the layer that governs what a firm’s AI systems are allowed to do. Its job is not to keep every record in one big database. Its job is to set the rulebook for evidence. That means deciding what facts each kind of action must record, how records from different systems link together, which version of the rules was in force, and how someone can pull it all back out for review. Every system that makes a decision or carries out an action then writes its records according to that rulebook.

The rulebook should also be tested. Check that a request that should be refused actually leaves a record of the refusal. Check that a cancelled action can’t be mistaken for a successful one. Check that a broken permission check can’t quietly look like a legitimate approval. A rulebook for evidence that nobody tests is just another policy document, written for computers instead of people.

It also has to outlive whichever AI provider, tools or software the firm happens to be using this year. If you can only piece together last year’s decisions through a supplier’s log format and storage settings, you do not own your evidence. You are renting it, on terms you did not write.

Regulation points the same way, though more narrowly. For AI systems classed as high-risk, the EU AI Act requires them to record events automatically, and requires the companies that build and use them to keep those records [3]. Whether this applies to a given system depends on what it is used for, who is using it, and when the rules take effect. It is not a blanket requirement for every AI tool a firm uses. NIST, the US standards body, offers guidance rather than law. Its zero trust security guide separates the point where a decision is made from the point where it is applied, which is exactly why each needs its own record [4]. Its AI risk framework treats accountability and transparency as signs of trustworthy AI [5]. None of these will design your evidence for you, but all of them assume you have some.

A simple test

Pick one task your AI assistant completed last week. Try to answer these questions:

  • Who asked for it, and who was the assistant acting for?
  • What gave it permission?
  • Which data did it touch?
  • Which version of the rules applied, what did they decide, and based on what facts?
  • Was that decision actually applied, and how did the task really end?

Answer them without asking the team that built the system to remember.

If you can only answer by stitching several systems together and guessing what happened in between, you have logs. You do not yet have evidence.

Closing

Logs tell you what your AI systems did. Evidence lets you check whether they were allowed to do it, who allowed it, and whether the rules were actually followed.

When compliance asks why an AI assistant opened a client’s records, “it worked” is not an answer. The real answer has to be built one action at a time.

Jason Southgate leads Parakletos AI, a London-based AI and cloud consultancy focused on production AI systems where governance, reliability, and accountability matter.

He has completed a University of Cambridge programme in data science, machine learning and AI.

Parakletos AI helps organisations use AI without losing control. They put identity, permissions and an audit trail around AI models and tools, so teams can move quickly and the business can account for what happened. This makes the governed route easier than the unmanaged one.

If you would prefer to buy a product rather than build, they can also help you.

References

  1. W3C, Trace Context, W3C Recommendation. https://www.w3.org/TR/trace-context/
  2. ISO/IEC 27001:2022, Annex A controls 8.15 (Logging), 8.16 (Monitoring activities), 8.17 (Clock synchronisation). https://www.iso.org/standard/27001
  3. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 12 (Record-keeping), Article 19 (Automatically generated logs), Article 26(6) (Deployer log retention). https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  4. NIST SP 800-207, Zero Trust Architecture, August 2020. https://csrc.nist.gov/pubs/sp/800/207/final
  5. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023. https://doi.org/10.6028/NIST.AI.100-1