Evaluating Enterprise AI Prompt-Injection Defenses: From Content Detection to Business Consequences

Separate what a source says from what it may direct

Prompt injection is not defined only by the presence of a dangerous phrase. The central failure is content without instructional authority changing an assistant's goal or operation. A user requests a supplier comparison, while an attached quotation asks to be favored or to have competitors' documents sent elsewhere. The attachment may establish a quoted price; it cannot grant itself authority to change evaluation rules or choose recipients.

OWASP describes such influence from websites, files and other external sources as indirect prompt injection, and notes that RAG and fine-tuning do not fully eliminate it.[1] Placement in an internal knowledge base does not establish trust either: customer mail, supplier attachments and employee-uploaded material retain different provenance. Permission to read material does not give that material permission to instruct the assistant.

The earlier article on enterprise agent tool authorization addresses who may perform which operations. This article adds a separate boundary: the reason for an otherwise permitted action may already be compromised. An assistant allowed to save purchasing recommendations can still produce harmful advice if a supplier attachment changes its scoring rules. Tests limited to permission rejection will miss that failure.

Find the points where content becomes a decision

Map four positions in each workflow: source entry, transformed content, decision authority and output destination. Entry points include body text, OCR, attachments, search summaries and tool returns. Transformations include summaries, structured extraction and persistent memory. Decision authority separates task rules from candidate evidence. Outputs include replies, drafts, external transmission and state changes. At every connection, ask who supplied the content and who may direct the next step.

Summarizing an attachment of uncertain provenance does not turn it into a trusted instruction. If another agent treats the summary as a manager's requirement, the risk has moved into an intermediate artifact. Derived content should retain provenance and trust attributes maintained by trusted software. A model-generated claim such as 'reviewed' or 'high confidence' must not itself promote authority.

Also identify sensitive data sharing the context. A step that needs only public quotations should not simultaneously receive a full customer list or email credentials. Reducing available data limits consequences, but does not prevent a quotation being misread. Evaluate confidentiality, answer integrity and business actions separately rather than treating absence of disclosure as complete safety.

Here, a trust attribute constrains how a source may be used; it is not a probability that every sentence is true. A supplier may be authoritative about its own quotation without being authoritative about internal purchasing policy. Evidence for one fact does not establish authority over the entire workflow.

Combine controls and state how each can fail

The first layer preserves provenance and treats external material as evidence. Input classifiers can flag suspicious material and narrow automatic processing. Microsoft's Spotlighting paper studies markings and transformations that help models recognize provenance, but its main injection evaluation uses synthetic documents carrying keyword-inducement attacks and contemporary GPT-family models.[2] It supports testing provenance signals, not treating its findings as a guarantee for arbitrary present-day business workflows.

The second layer constrains processing. An extraction step can receive only necessary fields and have no transmission capability before passing results to evaluation. Structured output makes fields easier to validate, but valid JSON establishes neither a price's truth nor the correctness of an object or recommendation. Bind material fields to source evidence, preserve unknowns and prevent silent model-generated repairs.

The third layer checks consequences through software and business systems: destinations, sensitive fields, object scope, writable fields and concrete approvals. Where judgment is necessary, show the proposed changes and evidence to the reviewer instead of merely asking whether to continue. Trusted software can also be misconfigured, so these boundaries need independent negative tests.

Microsoft's guidance combines probabilistic and deterministic defenses and identifies latency, false positives and maintenance as tradeoffs.[4] An additional judging model is not automatically independent: it may read the same hostile material or rely on the same incorrect summary. Explain the distinct constraint contributed by each layer instead of accepting a count of defensive components.

A constructed comparison: draft permission does not prevent a manipulated recommendation

Consider a constructed procurement workflow. A buyer asks the assistant to compare three suppliers against confirmed specifications, delivery times and tax-inclusive totals, then produce an internal recommendation without placing an order. A quotation contains text attempting to change the evaluation order. The primary harm may be an inappropriate recommendation rather than an unauthorized purchase. A demonstration with all writes disabled therefore does not establish recommendation integrity.

Keep comparison fields and scoring rules in business-approved configuration; attachments supply candidate facts. Each extracted fact carries a page and source position. Missing specifications or dates remain missing, and attachments cannot add scoring criteria. Check the explanation against fixed rules and evidence, including whether it has excluded competing quotations without justification.

If the user later asks for the result to be sent to an internal procurement group, derive the recipient from the actual delegation and trusted directory, not an address in an attachment. Display the content and recipients before transmission. Block material that should not be sent even if the recipient belongs to an allowed domain. A domain allowlist checks only part of destination control, not content and permitted purpose.

This design adds extraction, evidence checking and appropriate human review, while limiting open-ended improvisation. That tradeoff may be justified for procurement with relatively explicit rules and inspectable consequences. Summarizing public information without acting may warrant lighter controls. This is consequence-based design advice, not a universal compliance standard.

Accept against both legitimate and adversarial tasks

Pair a clean task with versions containing interference while keeping the underlying business facts unchanged. Cover body text, attachments, OCR and tool returns, varying placement, phrasing, language and propagation across steps. Testing one familiar phrase establishes behavior on that phrase, not all sources.

Define success and harm before testing. In the procurement example, legitimate success includes correct fields, explicit gaps and recommendations consistent with fixed rules. Harm includes changed rules, fabricated evidence, unapproved data movement or state changes. Inspect actual output, tool arguments and downstream receipts. A statement that malicious instructions were ignored is not sufficient evidence.

Report legitimate-task completion, false blocking of legitimate content, actual harm on adversarial tasks, human-review load and end-to-end latency together. State the denominator, attacker-controlled surfaces, attempt budget and untested channels for each. Repeated attempts against one environment and independent tasks are different statistical units; their percentages are not directly interchangeable.

Keep three separate control accounts: which labeled attacks the detector identified, which proposed dangerous actions execution boundaries blocked, and which business harms actually occurred. A missed detection contained downstream is different from a flagged attack followed by transmission. Link these observations through consistent task identities. Do not multiply layer-specific rates under an unverified independence assumption to invent an overall safety rate.

Anthropic's browser evaluation describes an internal adaptive attacker with one hundred attempts per environment and explicitly acknowledges residual risk at low attack-success rates.[3] Enterprises can adopt its practice of reporting attack budgets and boundaries without importing a product- and experiment-specific rate as their own risk probability.

After tuning, rerun clean tasks and adversarial cases held out from tuning. Observing no harm demonstrates only that none was observed within that test coverage, not that vulnerabilities do not exist. Where high-consequence workflows still lack verifiable destinations or side effects, narrow them to reviewable outputs and state which stages cannot yet execute automatically.

Make the investment a sustainable operating decision

A procurement or release review can request five evidence packages: the source and processing map; controls and their failure conditions; paired legitimate and adversarial results; blocking, review and latency costs; and shutdown plus recovery procedures. Classifier detection rates or screenshots of refusals alone do not establish control of the business workflow.

Prioritize investment by actual exposure and consequences. Infrequent public-information reading may start with provenance, restricted external connections and sampling. An assistant processing large volumes of customer attachments and sending messages needs more complete data-flow controls, receipts and rehearsals. If rules are unstable or review costs exceed the automation benefit, retaining manual submission can make value easier to verify.

Incident response must cover derived copies. After isolating a source, inspect whether its content entered summaries, caches or memory; pause affected downstream actions and use actual receipts to determine which results need correction. Deleting an attachment does not withdraw advice already produced. Replay relevant tests after changes to models, parsers, prompt configuration or tool permissions.

Express deployment acceptance as a bounded statement: which sources, tasks, actions and destinations have been validated, which depend on people, and which remain uncovered. This lets teams see the exposure added alongside automation benefits instead of substituting a 'prompt-injection protected' label for continuing acceptance.