In brief: executive perspective
AI systems can pass functional tests while exposing sensitive information, accepting malicious instructions, or granting tools more authority than intended. Security leaders need to understand which failures are possible in their own architecture, what an adversary needs to cause them, and whether a proposed defence preserves business usefulness.
ProofSeam would turn authorised adversarial testing into a repeatable decision process. It combines attack execution, a declared level of attacker access, defence comparisons and traceable evidence. The result is a defensible decision about a specific AI system: remediate it, restrict its deployment, gather further evidence, or accept the documented residual risk.
The repository already contains attack modules, capture adapters, experiment runners and reporting utilities. Its evidence ranges from synthetic scenarios to bounded model experiments and selected model-driven harnesses. Production integrations and enterprise operations still need development. ProofSeam should therefore begin as a specialist-led assessment service supported by this engine, not as an autonomous certification platform.
The commercial case rests on helping enterprises make better AI release and procurement decisions, avoid duplicated evaluation work, and distinguish useful defences from controls that merely impair model performance. Customer pilots must test these benefits; they are not yet established outcomes.
Working-name note: ‘Proof’ refers to inspectable evidence, not mathematical proof of security. The proposed name has not received trademark or domain clearance.
The business problem
AI risk crosses more than the prompt boundary
A deployed AI workflow may connect a model to private documents, memory, external tools, cloud services and multiple organisations. Each connection changes what an adversary can observe or influence. In distributed model execution, intermediate representations and training gradients create additional privacy questions that ordinary application testing does not resolve.
Consider an enterprise evaluating a document-processing assistant. One assessment asks whether an injected document can redirect the assistant. Another asks whether a tool action exceeds the user's permissions. A third asks whether an external compute participant can infer confidential content from the representations it receives. These are different tests with different access requirements; combining them into one unexplained score obscures the decision.
Industry evidence: adoption, governance and exposure
Agents are moving into larger enterprises. McKinsey's August 2026 survey reports that the share of respondents at organisations with more than US$1 billion in annual revenue scaling agents rose from 27% to 40% between its 2025 and 2026 surveys. [8] For a security buyer, this strengthens the case for evaluating the permissions, data access and tool actions of each deployed workflow.
Security assessment is becoming an operating practice. The World Economic Forum and Accenture report that 64% of surveyed organisations have AI security assessment processes, versus 37% in the previous report. They also report that 87% of respondents identified AI-related vulnerabilities as the fastest-growing cyber risk during 2025. These are reported practices and risk perceptions, not measured attack-success rates. [9]
Exposure extends beyond model behaviour. IBM's 2026 study reports AI model or application breaches at more than one-fifth of its sampled organisations. It identifies compromised APIs, applications and plug-ins, together with cloud misconfiguration, as prominent causes. [10] This supports examining the surrounding integrations as well as the model. The sample comprises organisations that had already experienced a breach; the figure is not the prevalence of AI breaches across all enterprises.
These sources cover different populations and measures. They support the case for scoped validation, but neither establish the breach probability of a particular system nor demonstrate ProofSeam's effectiveness.
The proposed first customers
AI platform and ML engineering teams
Choose deployment architecture and compare protection mechanisms against task quality, latency and compute constraints.
Product security and offensive security teams
Reproduce failures, validate mitigations and build regression tests around supported integrations.
CISOs and privacy leaders
Understand material exposure, accountable owners, evidence quality and unresolved assumptions before approving a release.
AI infrastructure vendors and specialist consultancies
Substantiate narrowly scoped technical claims and deliver repeatable customer assessments.
The initial market should be organisations that operate their own AI stack or connect models to sensitive data and privileged tools. Such organisations can provide the access and technical sponsorship needed for a meaningful evaluation. A buyer limited to a hosted chat interface cannot commission the same gradient or internal-representation tests as a model operator.
The buying trigger
The initial engagement should attach to a concrete decision: approving an AI release, evaluating a supplier, changing a model or tool configuration, or validating a privacy defence. The buyer purchases an assessment and evidence package for that decision, with explicitly agreed coverage and exclusions.
The product
A common evidence process across distinct attack surfaces
ProofSeam would connect three layers: a technical execution engine, a managed assessment workflow and business-facing evidence reports. The engine exists in the repository; the managed product layer does not.
01 · Technical execution engine
Existing foundation. Attack modules, capture adapters, runners and reporting.
02 · Managed assessment workflow
Proposed. Authorised scope, connectors, comparisons and specialist review.
03 · Business evidence package
Proposed. Decision brief, reproduction record, gaps, remediation and retest.
Application and agent security
The engine includes prompt- and application-oriented attacks, as well as agent scenarios involving tool misuse, memory, identity, approval handling and communication. Some paths use model-driven harnesses; many target environments remain simulated. A commercial engagement must identify the customer connector and scenario tested. A module name alone does not establish production coverage.
Model and data privacy
Implemented paths include representation reconstruction, gradient-based reconstruction, membership and property probes, and accumulated-observation attacks. Their applicability depends on access to model artefacts, calibration data or captured tensors. Training gradients are relevant to training or controlled gradient probes; they are not normally available in inference-only traffic.
Integrity and defence evaluation
The engine supports selected replica, collusion, protocol, isolation and availability scenarios, plus experiments with noise, compression and other representation transformations. These allow a team to ask whether a defence changes leakage or attack outcomes, and what utility or compute it costs. Synthetic fixtures establish engineering behaviour, not a general claim about deployed systems.
Evidence and reporting
Existing utilities record run metadata, measurements, hashes and evidence trails. When verification is enforced, they support reproducibility and tamper detection. Reports can show coverage, budgets, outcomes and uncertainty. Recent inspections identified weaknesses in the acceptance path; these must be fixed and independently validated before the system can gate releases without human supervision.
Current availability boundary
Available foundation: Python/CLI execution, documented artefacts, research experiments and selected harness integrations. Proposed product: customer onboarding, supported connectors, review workflow, enterprise identity, multi-tenant operations, service commitments and a managed interface. No hosted enterprise service or certification is asserted by this paper.
How an engagement works
Scope. Compare. Decide.
1. Define the decision and adversary
Identify the system, business use, sensitive assets and acceptable operational impact. Specify what the test adversary may read, write or execute: user inputs, retrieved content, tool responses, internal tensors, model weights, or participant-controlled outputs. Obtain authorisation for the agreed targets and test effects.
2. Establish the baseline
Fix the model and application configuration, record the calibration and evaluation partitions, and define budgets and acceptance criteria before measuring outcomes. Include suitable positive controls to confirm that an attack succeeds when a known vulnerability is present.
3. Execute and compare
Run applicable attacks against the baseline and proposed mitigation. Keep access and computation comparable, disclose any stronger-access variants, and measure customer-relevant utility alongside attack outcomes. Failed or unsupported tests remain visible.
4. Reproduce, remediate and retest
Reproduce material findings, identify their prerequisites, and assign a remediation owner. Retest the selected mitigation on independent data or runs without changing the criteria to suit the result. Human review remains required until the automated acceptance chain has been validated.
5. Deliver the evidence package
The proposed customer deliverable includes an executive decision brief, technical findings, reproducible configuration, artefact references, coverage gaps and a retest record. Sensitive captures should remain in an agreed customer-controlled environment, with access and retention defined contractually.
A finding record should contain: affected system version; attacker access; demonstrated outcome; business consequence; measured task impact; reproduction steps; evidence strength; remediation owner; and residual uncertainty. ‘No issue observed’ means no issue under the stated test conditions, not universal safety.
Positioning and differentiation
Sell decision evidence, not an attack-count leaderboard
AI red teaming is already an established product category. Promptfoo documents application red teaming, including RAG and MCP testing; Giskard documents agent evaluation and continuous red teaming. Both overlap with parts of this project's application layer, so a generic ‘we test prompts’ proposition is insufficient. [1], [2]
ProofSeam would focus on the relationship between application behaviour, internal data exposure and defence trade-offs, while recording attacker access explicitly and using a consistent evidence record. This positioning still requires customer validation; it does not imply that competitors lack these capabilities or that ProofSeam is superior.
The most credible initial distinction is specialist depth in representation and gradient privacy alongside application and agent testing. This depth matters to customers evaluating distributed inference, training infrastructure or sensitive AI workflows. Where an existing scanner already meets a customer's needs, the product should integrate its evidence or focus on the unanswered question.
Complement established frameworks
OWASP's LLM guidance provides categories of risks and mitigations. NIST's AI RMF and Generative AI Profile provide a risk-management context. ProofSeam can organise assessment evidence against such frameworks without claiming certification or automatic regulatory compliance. [3], [4]
CVSS communicates characteristics and severity of software vulnerabilities; CVE identifiers and CVSS scores serve different purposes. Neither should be replaced by an invented universal AI score. [5]
The recommended reporting model
Use an adversary-conditioned assurance matrix: system boundary, attacker access, observed failure or leakage, utility cost, evidence strength, and unresolved gaps. Keep business impact separate from experimental success rates. A successful synthetic attack and a reproduced customer-system failure should never receive indistinguishable treatment.
A future process-maturity assessment could describe repeatability and governance, but it should not be presented as a validated security metric until it has its own empirical basis.
| Dimension | What the assessment should record |
|---|---|
| System boundary | The workflow, model and application version, integration, and sensitive asset tested. |
| Attacker access | What the adversary may observe, change or execute, with the agreed attack and compute budgets. |
| Observed outcome | The demonstrated failure or leakage under those conditions. Record the business consequence separately from experimental success rates. |
| Utility cost | Baseline and mitigation results for customer-relevant task quality, latency and compute. |
| Evidence strength | Whether the result comes from a synthetic fixture, bounded model experiment or reproduced customer-system finding, with artefact and reproduction references. |
| Unresolved gaps | Unsupported or failed tests, assumptions, residual uncertainty, remediation ownership and the retest record. |
Commercial model and deployment
Begin with a bounded service; earn the recurring product
Initial offer: a paid design-partner assessment
The first package should cover one named AI workflow, agreed attacker access, a limited set of supported surfaces, a baseline, a mitigation comparison and a retest. The customer receives a decision brief and evidence package. Price and duration should reflect the integration effort and compute budget; willingness to pay and a fixed delivery SLA remain untested.
Expansion: recurring validation
After repeatability is demonstrated, offer recurring assessments for model, prompt, tool and infrastructure changes. A proposed subscription could combine an annual platform fee with supported integrations, execution capacity and specialist support. Compute-intensive privacy experiments should have explicit budgets rather than an unlimited-testing promise.
The WEF / Accenture survey distinguishes 40% of organisations conducting periodic AI security reviews from 24% performing a one-time assessment. [9] The implication for the proposed service is to make retesting part of the operating model. These figures do not measure willingness to pay for ProofSeam.
The open research engine and a commercial service can coexist, subject to the repository's Apache-2.0 licence and all dependency, model and dataset terms. The commercial value must come from supported integrations, dependable operation, expert interpretation and repeatable customer workflows, not simply repackaging public attack names. [6]
Deployment direction
The initial deployment should use a customer-controlled runner in a test environment, with reports and approved metadata exported for review. A hosted control plane may follow, but would require explicit decisions about tenant isolation, authentication, secrets, data movement and retention. Customer prompts, captures and model weights should remain within the environment unless expressly agreed otherwise.
Proving the business case
Measure analyst hours per assessment, time to reproduce a finding, percentage of agreed surfaces evaluated, mitigation retest turnaround and recurring evaluation cost. Compare these with the customer's existing process. For example, an engagement that reduces repeated manual configuration may save time even if it finds no new vulnerability. No ROI percentage or avoided-breach value is claimed without customer evidence.
US$4.99 million: breach-cost context
IBM reports this global average breach cost in its 2026 study. It covers breaches broadly, not only attacks on AI systems. [10] It establishes the scale of potential business consequences, not a customer loss forecast or savings attributable to an assessment.
About 20%: operating costs constrain AI use
McKinsey reports that about one in five respondents say AI operating costs, including tokens, constrain organisational AI use. [8] A credible defence comparison should therefore track task quality, latency and compute cost alongside security outcomes.
Productisation priorities
What must be true before wider enterprise sales
Reliability before unattended decisions
The immediate priority is a verified acceptance chain. Missing cells, invalid measurements, incomplete controls, mismatched checkpoints and failed evidence verification must prevent a clean acceptance outcome. Independent review should confirm that fresh validation retests the original criteria. Historical research outputs must retain their original scope and limitations.
Support a narrow integration contract
Choose a small number of reference deployments and test them end to end. Specify supported model families, tool interfaces, capture formats and environmental requirements. Publish the difference between production-tested connectors, research adapters and synthetic scenarios. Expand supported coverage only when an integration passes its own acceptance tests.
Turn experiments into customer operations
Add role-based access, secrets handling, explicit authorisation records, retention controls, operational logging, reproducible job execution and cost limits. Design reports for two audiences: the decision-maker who needs consequences and ownership, and the engineer who needs reproduction and remediation detail.
Build evidence of commercial fit
Recruit a small group of design partners facing real release or procurement decisions. Agree the success criteria before each assessment. Record integration effort, delivery cost, useful findings, remediation outcomes and demand for repeat testing. These results, rather than a predetermined date, should determine whether to launch more widely.
Research discipline as a product principle
The underlying work has already shown why utility must accompany privacy measurements: reducing reconstruction can coincide with unacceptable model degradation. Negative results and unevaluable cases are therefore part of the deliverable. Current research does not establish universal defence effectiveness, calibrated breach probabilities or customer ROI.
The launch proposition should be precise: a specialist-supported way to test defined AI security assumptions and produce evidence that a buyer can inspect, reproduce and use.
Naming recommendation
ProofSeam is a working name, not a repository rename
Recommended: ProofSeam. The name connects evidence with the interfaces where trust changes hands: user to agent, agent to tool, organisation to provider, and one model component to another. The proposed descriptor is ‘AI security validation’; the tagline is ‘Evidence for AI security decisions’. ‘Proof’ must not imply certification or a mathematical security guarantee.
Alternative: AssuranceMesh. This emphasises connected systems and enterprise assurance, but is more abstract. It remains a naming concept requiring further screening.
Do not use BoundaryLens: an existing product uses that name for environmental screening. [7] The supplied draft records that exact-name web searches did not surface an obvious AI-security product named ProofSeam on 11 September 2026. Search absence does not establish trademark, company-name, domain or social-handle availability. Complete formal clearance and buyer testing before adopting any name. This document changes no code/package identifiers.
What AI security decision-makers should remember
- Access determines coverage. Application, representation and gradient tests require different permissions and artefacts. Every finding must declare its prerequisites.
- A defence must preserve usefulness. Reduced leakage alone is insufficient when the mitigation causes unacceptable task degradation or compute cost.
- Evidence has limits. Synthetic scenarios and reproduced customer findings carry different weight. Failed tests, coverage gaps and negative results belong in the record.
- Start with a bounded engagement. The proposed initial offer is a specialist-led assessment for one named workflow, followed by a mitigation comparison and retest.
- Product claims must be earned. Validate the acceptance chain, supported integrations and customer outcomes before unattended release gating or broader enterprise sales.
Sources and provenance
- Promptfoo, Red Teaming Guides.
- Giskard, documentation overview.
- OWASP, LLM Top 10 project.
- NIST, Generative AI Profile.
- FIRST, CVSS v4.0 Specification.
- LLM-Attacker, README, LICENSE and source at commit bcffb7f. Internal technical basis identified in the supplied draft; capability and research limitations refer to that revision. The repository README remains the canonical operational documentation.
- BoundaryLens, existing product.
- McKinsey, The state of AI in 2026: On the road to ROI. 25 August 2026; exhibits 2 and 8. Survey: 1,719 respondents in 97 nations, 4 May–8 June 2026; results weighted by national contributions to global GDP.
- World Economic Forum / Accenture, Global Cybersecurity Outlook 2026. 12 January 2026; section 3.1, figures 8, 10 and 11. Methodology: 804 qualified respondents from 92 countries; fieldwork 25 August–1 October 2025.
- IBM, 2026 Cost of a Data Breach Report: published findings. 29 July 2026. Ponemon Institute research, sponsored and analysed by IBM: 602 organisations experiencing breaches between March 2025 and February 2026.
Industry figures in references 8–10 were checked against primary publisher sources on 11 September 2026. Report years, fieldwork periods and sample definitions are retained because the measures are not directly interchangeable. The original technical and naming source basis in references 1–7 is retained from the supplied draft; this update does not constitute a new repository audit or name-clearance review.
Commercial packaging, target customers, deployment direction and roadmap in this paper are proposals. There are no asserted customer deployments, revenue figures, independently validated market-size estimates, regulatory certifications or guaranteed security outcomes.
About this paper
Part of the Setloop Technical White Paper Series, this business white paper presents ProofSeam as a working product concept based on the LLM-Attacker research engine. It sets out the proposed buyer, assessment workflow, evidence standard and productisation priorities.
- Web: setloop.io
- Email: [email protected]
- Discuss an assessment: setloop.io/contact
- Research engine: LLM-Attacker on GitHub