A chatbot on a support page, a copilot inside an internal tool, an agent that files tickets on its own: these are applications first and language models second. Most of what goes wrong in them is not a flaw in the model. It is an application that trusts the model's output, or gives it more access than the task needs, or lets untrusted text reach it through a door nobody drew on the architecture diagram.

That is where testing one has to focus. A classic web assessment still applies to everything built around the model. What it misses is the part where instructions and data travel through the same channel.

What is different about an LLM application

In an ordinary application, code and data are kept apart: a SQL query and the value inside it can be told apart, which is why parameterised queries work. A language model receives its instructions (the system prompt), the user's message, retrieved documents and tool results as one stream of text. It has no reliable way to tell an instruction from content that merely looks like one.

Three consequences shape every test:

  • Anyone who can put text in front of the model can try to steer it. Not only the person typing in the chat box, but the author of every web page, email, PDF or ticket the application retrieves.
  • The model's output is untrusted input to whatever consumes it: the browser that renders it, the query built from it, the tool it calls.
  • Behaviour is probabilistic. The same attack can work on the third attempt and fail on the fourth, and a model update can change the result overnight.

Scope: draw the data flow first

Before testing anything, map where text enters the model's context and where the model's output goes. For most applications the list looks like this:

  • the system prompt, and who can change it;
  • user input, including uploaded files and images;
  • retrieval (RAG): which document stores, and whose documents;
  • tools and function calls: what each one can do, and with whose permissions;
  • memory or conversation history, and whether any of it is shared between users;
  • output destinations: rendered in a browser, sent by email, written to a database, executed;
  • the model provider, version and settings, and any plugins or third-party components.

Every arrow on that map that crosses a trust boundary is a test case. A retrieval pipeline that indexes documents submitted by customers is a trust boundary even when the chat box itself is internal-only.

The eight areas to test

The OWASP Top 10 for LLM Applications 2025 is the reference most teams map findings to. Grouped by what you actually test:

1. Prompt injection, direct and indirect (LLM01)

Direct injection is the user telling the model to ignore its instructions. Indirect injection is the same instruction hidden in content the model reads later: a line of white text on a web page, a comment in a document, a field in a support ticket. For most business applications the indirect kind matters more, because the attacker never needs an account.

Test both. Plant instructions in every source the application retrieves and check whether the model follows them. Try every language the application accepts: a filter tuned on English can miss the same instruction written in Persian, in Finglish, or split across scripts.

2. System prompt leakage (LLM07)

Assume the system prompt can be extracted; it usually can. The finding is rarely the leak itself. It is what the prompt contained: an API key, an internal URL, a list of customers, or business rules the application relies on as its only access control. The test is to recover the prompt. The recommendation is almost always to move the secret or the rule out of it.

3. Sensitive information disclosure (LLM02)

Look for data the model can reach but the current user should not: another tenant's documents in a shared index, personal data pulled in by retrieval, earlier conversations in shared memory, secrets in tool responses, and logs that store every prompt and answer in full.

4. Improper output handling (LLM05)

Treat model output exactly as you treat user input. If the application renders it as HTML, test for cross-site scripting. If it builds a query or a shell command from it, test for injection. A well-known case is a model talked into writing a Markdown image whose URL carries data from the conversation to an attacker's server; the browser fetches the image, and the data leaves, without a single click.

5. Excessive agency (LLM06)

For every tool the model can call, ask three questions. Does the task need this tool at all? Does the tool have more permission than the task needs? Does anything irreversible happen without a person confirming it? An agent that can both read email and send it is one indirect injection away from forwarding the inbox.

6. Vector and embedding weaknesses (LLM08)

Retrieval needs the same access control as the documents it indexes, enforced when results are fetched, not by asking the model to be discreet. Test whether a user can retrieve passages from documents they cannot open directly, and whether someone who can add a document to the index can poison the answers everyone else receives.

7. Unbounded consumption (LLM10)

Language models cost money per token and per second of compute. Test for requests that run up a large bill or tie up capacity: very long inputs, prompts that trigger long outputs or loops of tool calls, and missing per-user rate limits and budgets.

8. Supply chain, poisoning and misinformation (LLM03, LLM04, LLM09)

Review where models, adapters, plugins and datasets come from and how their versions are pinned. Where the application's answers drive decisions, check what happens when the model is confidently wrong. Code assistants, for example, sometimes suggest packages that do not exist, and an attacker can register the name.

How to run the test

  1. Get a staging copy with realistic data and every integration switched on. Testing the chat box alone misses retrieval, tools and output destinations, which is where most real findings are.
  2. Combine tools with manual work. Open-source scanners such as garak, PyRIT and promptfoo cover known jailbreaks and injection patterns quickly and repeatably. The findings that matter most, like a poisoned ticket that makes an agent call the refund tool, come from someone who understands the application's business logic.
  3. Repeat each attempt and record the rate. Run every important attack many times and report how often it succeeds, together with the model version and settings used.
  4. Test the fix, not the prompt. A change to the system prompt that blocks today's payload rarely blocks tomorrow's rephrasing of it. Retest with variations before closing a finding.

What a finding should contain

On top of the usual fields (see what a penetration test report must contain), a finding in an LLM application needs:

  • the exact input, and where it entered: the chat, an uploaded file, a retrieved document or a tool result;
  • the model, version and settings in use, and the date of the test;
  • the success rate over a stated number of attempts;
  • the impact in business terms: which data left, which action ran;
  • the root cause, which is almost always a missing control outside the model.

Controls that hold

Guardrail prompts and output filters help, but they are probabilistic, and an attacker needs only one phrasing that gets through. The controls that hold are the ones that do not depend on the model behaving:

  • enforce authorisation in the application, on every tool call and every retrieval, with the current user's permissions;
  • give each tool the least privilege its task needs, and require a person to confirm anything irreversible;
  • encode or validate model output for wherever it goes, exactly as you would user input;
  • keep secrets and access rules out of the system prompt;
  • mark and isolate untrusted content, and never let retrieved text grant permissions;
  • set per-user rate limits and spending caps, and monitor for abuse.

Common questions

Is testing the model enough? No. Model evaluations measure the model in isolation. Most exploitable weaknesses sit in how the application connects the model to data and tools, so the test has to cover the whole application.

We use a commercial model API. Isn't security the provider's problem? The provider secures the model and its hosting. Your system prompt, retrieval, tools and output handling are yours, and that is where most findings are.

How often should an LLM application be retested? After any change to the model or its version, the system prompt, the tools or the data sources, and at least as often as the rest of the application. A model upgrade can change behaviour without a single line of your code changing.