What leaks into AI, and how to stop it

Most of the data that leaks into AI is not stolen. Staff put it there to get their work done: a complaint with the customer's national ID in it, a contract to summarise, a log with an internal address, a config file with a password. Once it reaches the provider, the provider's rules decide how long it is kept, who may read it and whether a model learns from it.

Banning AI does not stop this; it moves the same work to personal phones, where nobody sees it. What works is one approved path that recognises sensitive values before they leave, keeps the most sensitive work on models inside the network, and records what left.

What actually leaves

The best recent measurements come from security vendors that watch this traffic for their customers, so read them as indications rather than a census. Netskope's Cloud and Threat Report 2026, covering October 2024 to October 2025, counted an average of 223 data-policy violations involving generative AI per organisation per month, and 2,100 in the top quarter of organisations. Source code made up 42% of them, regulated data such as personal, financial and health records 32%, and intellectual property 16%. Harmonic Security inspected 22.5 million prompts and file uploads across 665 AI tools at US and UK companies in 2025 and found sensitive data in about one in forty: 579,113 exposures in a year.

The typical case is not dramatic. A support agent wants a polite reply to an angry customer and pastes the whole ticket:

Draft a polite reply to this complaint. Customer: Sina Rahimi, national ID
0012345679, card 6037 9900 0000 0006, mobile 09121234567, email
sina.rahimi@example.com. He was charged twice; refund to
IR840120000000000123456789. Our payment log shows the error came from
10.20.30.40.

Nothing in that request is malicious, and the reply it asks for does not need any of those numbers. Yet it carries seven values the organisation is obliged to protect: a name, a national ID, a card number, an IBAN (Sheba), a mobile number, an email address and the address of an internal server. (All of them are fictional; the identifiers pass their checksums so that a real detector would treat them as real.)

Six paths out of the organisation

The chat box is the path everyone pictures. It is one of six, and the other five are easier to miss because no person types anything into them.

Coding assistants. A developer asks an assistant why the service cannot reach its database. To answer, the assistant opens the files around the error, and one of them is this:

# .env
DATABASE_URL=postgres://payments:Winter-2026!@10.20.30.40:5432/payments
SMS_API_KEY=sk_live_51H...redacted

The developer pasted nothing, but the password, the internal address and a live API key are now part of the prompt. In Harmonic's 2025 data, 12.8% of the sensitive data sent to coding tools was access keys: API tokens, credentials and secrets. An assistant in agent mode can also run commands in the terminal and read their output, so the question is not only what the developer shows it, but what it is allowed to open. We cover the rules for this in using AI coding assistants safely.

Knowledge bases. A team builds an internal assistant over 2,000 contracts and uses an external embedding API to index them. Every one of those contracts leaves the network in full on the first day, before anyone has asked a question, and every later question leaves too, together with the passages retrieved to answer it. If the embeddings are computed by a model inside the network, the documents never leave at all.

Domestic gateways. Many organisations in Iran reach models through a domestic AI gateway because international services do not serve Iranian accounts directly. That changes who you pay and how you connect, not where the text goes: if the model behind the gateway is an international one, the prompt reaches that model's provider one hop later. Ask which model answers and where it runs.

What happens to the data once it leaves

Once a prompt reaches a provider, four things are decided by someone else.

How long it is kept, and who reads it. Each provider's policy sets a retention period and lets its staff review some conversations for abuse. Many consumer plans also use conversations to improve their models unless the user opts out; business plans usually do not. In Harmonic's data for the third quarter of 2025, 21.8% of the sensitive data it saw went to tools that train on what users send.

Whether "deleted" means deleted. In May 2025, in the New York Times' copyright case against OpenAI, a US court ordered the company to preserve ChatGPT output logs that would otherwise have been deleted, including chats users had deleted themselves. The order covered the Free, Plus, Pro and Team plans and API use without a zero-retention agreement; Enterprise and Edu were outside it. The obligation to keep preserving ended on 26 September 2025, but data already set aside stayed preserved. None of those users had done anything wrong.

Whether a shared link stays private. In July 2025 about 4,500 shared ChatGPT conversations, made public by users who had ticked a "make this chat discoverable" box, turned up in Google results; the option was removed the next day. A few weeks later, more than 370,000 conversations shared from Grok were found indexed, created with nothing more than the share button.

Whether a connected assistant can be made to send it. An assistant that reads email and documents can be steered by text hidden inside them. EchoLeak (CVE-2025-32711), rated 9.3 and disclosed in June 2025 after Microsoft fixed it on its servers, let a crafted email make Microsoft 365 Copilot place internal data in an image address that the browser then fetched. The victim only had to ask Copilot a related question. The mechanism is the indirect injection we described in how to security-test an LLM application.

None of this requires the provider to do anything wrong. It is the ordinary result of handing data to a service whose terms, and whose legal obligations, you do not control. "We deleted the chat" is a statement about your screen, not about the provider's storage.

Why a ban does not work

Netskope found that 47% of people using generative AI at work still used personal AI apps, down from 78% a year earlier as more organisations handed out managed accounts, and Harmonic traced 16.9% of the sensitive exposures it saw in 2025 to personal free-tier accounts. In Iran the effect is stronger, because many international AI services are only reachable through a VPN and a personal account. A ban does not remove the need; it removes the organisation's view of it.

The third column is not a product category. It is a decision: there is one sanctioned way to use AI at work, it is easier than the alternatives, and the controls sit on that path rather than in a policy document nobody reads.

The controls that work

The order matters. Detection that runs after the data has left is reporting, not protection, so every control below sits inside the network, before the request goes out.

1. Normalise before you detect. Persian text arrives with Persian, Arabic and Latin digits, Arabic and Persian forms of the same letters, and invisible characters. A detector that only knows one form misses the others. The same problem breaks prompt-injection filters, as we showed in prompt injection in Persian chatbots.

2. Detect three kinds of thing. Structured identifiers (national IDs, card and IBAN numbers, phones, emails, IP addresses) are found by pattern and confirmed by their checksums. Names of clients, project codenames and internal hostnames have no pattern, so they come from the organisation's own lists. Whole documents, such as a scanned confidential letter, are recognised by a classifier running on a local model, after their text is read with OCR.

3. Pick an action per workspace, not per person. Four actions cover almost every case:

• Log only: record what was found and send the request unchanged. It is the right first step, because it shows what would have been caught before anything is blocked.
• Mask and restore: replace each value with a labelled placeholder, send the masked text, and put the real values back when the answer returns.
• Local model only: some work, such as board papers or an investigation, never leaves the network at all.
• Block: stop the request and tell the person why, so they do not simply move to their phone.

Masking keeps the request useful because the model needs the shape of the question, not the numbers in it. Here is the support ticket from above after our own implementation of such a detector ran over it, with the customer's name added to a client list. The output is copied from the run, not paraphrased:

Draft a polite reply to this complaint. Customer: [[TERM_CLIENT_1]],
national ID [[IR_NATIONAL_ID_1]], card [[IR_BANK_CARD_1]], mobile
[[IR_PHONE_1]], email [[EMAIL_1]]. He was charged twice; refund to
[[IR_SHEBA_1]]. Our payment log shows the error came from
[[INTERNAL_IPV4_1]].

The provider receives a coherent request and none of the seven values. The same value always gets the same placeholder within a request, so "the customer" stays one customer throughout. When the reply comes back with [[TERM_CLIENT_1]] in it, the real name is restored inside the network before the agent reads it.

4. Log the finding, not the value. The audit record should say who asked, which model answered and what was found, by type and count. A log that stores the national IDs it masked is a second copy of the data you were protecting.

2026-10-24T09:41:12Z workspace=support user=agent-17 model=external
  found: TERM:CLIENT=1 IR_NATIONAL_ID=1 IR_BANK_CARD=1 IR_PHONE=1
         EMAIL=1 IR_SHEBA=1 INTERNAL_IPV4=1
  action: mask   sent: placeholders only   restored: yes

5. Shrink what the applications send. An internal assistant should retrieve only the documents the person asking may open, and compute its embeddings on a local model. Least privilege for a model is the same idea as least privilege for a person.

Why checksums matter

An Iranian national ID has a check digit, an IBAN has a MOD-97 check, and a card number has a Luhn check. A detector that verifies them can tell 0012345679, a valid national ID, from 0012345678, which is not one:

def national_id_ok(code: str) -> bool:
    if len(code) != 10 or not code.isdigit() or len(set(code)) == 1:
        return False
    s = sum(int(code[i]) * (10 - i) for i in range(9)) % 11
    check = int(code[9])
    return check == s if s < 2 else check == 11 - s

national_id_ok("0012345679")   # True: detected and masked
national_id_ok("0012345678")   # False: a typo is not a national ID

The check has limits in both directions. It still accepts about one random ten-digit number in ten (we ran it over 200,000 of them), so an order number will now and then be masked as a national ID; without the check, every ten-digit number would be. And a national ID typed with one wrong digit fails the check, so it is not recognised. Neither is a sentence such as "the board approved Project Simorgh, a 120-billion-toman acquisition", which contains no identifier at all. Putting "Simorgh" on a codename list catches the second case; for work where no gap is acceptable, the answer is the third action: keep it on a local model.

Note: Masking is not anonymisation. The structure of the request still reaches the provider: a contract with every name masked can still describe a deal someone could recognise. Decide which work may leave in masked form and which may not leave at all, before you switch masking on.

A rollout in four weeks

After that, a monthly look at the audit counts shows whether the approved path is being used, and where.

Questions to ask your team or vendor

Common questions

Is an enterprise plan from an AI provider enough? It improves the terms: enterprise plans typically exclude your data from training and let the organisation set the retention period. But the data still leaves, the plan does not cover the personal accounts people use alongside it, and it does nothing for the other five paths.

Does masking make the answers worse? Rarely, because most tasks need the shape of the question rather than the identifiers in it. Where the value itself is the point, such as checking whether an IBAN is valid, that request belongs on a local model.

Why not run everything on local models? For the most sensitive work, do. For everything else, the strongest external models are still better at many tasks, and most organisations end up with a mix decided per workspace.

What about images and scanned documents? Treat them as text that has not been read yet. Read them with OCR, in Persian as well as English, before anything is sent, and apply the same detection and policy to the result.

We already have a data-loss prevention tool. Does it cover this? Check two things: whether it sees the traffic to AI services at all, including API calls from your own applications, and whether it recognises Persian text and Iranian identifiers. Many tools were built for English and for email.

If your organisation has started using AI without deciding where its data may go, the inventory in week one is the place to start: it usually finds a path nobody had listed. Hafez Fabric, our Governed AI Gateway, puts these controls on one path inside your network, and it is available for a demo or a pilot.