LLM-Attack Suite
A unified red-team framework for privacy evaluation of split and distributed LLM systems. 65 attack modules across three surfaces, an experiment engine with real-model protocols, and evidence you can verify hash by hash.
One framework, three surfaces.
Split LLM deployments partition a model across client, cloud, and tail segments that exchange activations and gradients on the wire. LLM-Attack Suite evaluates what those wires (and the systems around them) actually reveal.
Split-model privacy
Seam and gradient inversion, membership inference, and key-lifecycle attacks against rotated or obfuscated wire tensors: the traffic split systems actually expose.
- · Seam & gradient inversion
- · Membership inference
- · Rotation & key-lifecycle attacks
Serving & protocol security
Traffic and timing side channels, GPU residue, tenant isolation, protocol integrity, and supply-chain tamper batteries for production serving stacks.
- · Traffic & KV-cache timing
- · Tenant isolation fixtures
- · Protocol integrity & replay
Application & agent security
OWASP-style batteries for applications and agents: prompt injection, RAG abuse, output-sink injection, goal hijack, tool misuse, confused deputies, and more.
- · Prompt injection & RAG abuse
- · OWASP ASI agent batteries
- · Output-sink injection
65 modules, ten families.
The registry is lazily imported and honestly labelled: registration means an implementation exists; the qualification matrix records what is reproduced, adapted, real-target qualified, synthetic-only, or unmeasured.
Token, label & latent reconstruction
Agent security (OWASP ASI)
Rotation, key & accumulated-observation
Collection, orchestration & defence batteries
Protocol, isolation, integrity & availability
Generative-AI application security
Active participant attacks & model extraction
Hardware, timing & traffic side channels
Membership & property inference
Reasoning exposure
Built to not overclaim.
The framework's results are engineered to be reproducible, controllable, and hard to misread because a red-team tool that flatters its own measurements is worse than no measurement.
Fail-closed scorecard
A PASS only stands alongside a linked positive control: the same attack family succeeding on an undefended reference. Without one, the verdict is downgraded to INSUFFICIENT_EVIDENCE, since an ineffective attack can masquerade as a strong defence.
No universal risk score
The framework deliberately refuses to emit any aggregate scalar or maturity grade. Exposure, impact, and evidence quality stay separate fields; a gate verdict is an operational threshold check, not a privacy certificate.
Hash-chained evidence
Every run seals an evidence manifest binding artifacts, events, and file hashes. Frozen releases can be verified and reproduced independently; the published campaigns ship with their full evidence trees.
Honest registry semantics
A registered module is an implementation, not demonstrated real-target coverage. The qualification matrix separates reproduced, adapted, real-target, synthetic-only, and unmeasured modules: 64 reviewed in the paper, 65 in the current registry.
registered attack modules across ten families
manuscript with frozen, hash-verified evidence release
open licence for code and packaging
torch-less core; PyTorch only for real-model runs
Paper, evidence, and code.
The manuscript reports the validated white-box path and the ten-setting live-agent qualification, with every major result family linked to its machine-readable evidence file.
LLM-Attack Suite: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems
Georgios Politis and Evangelos Pappas (preprint, Oct 2026). The studies on returned-gradient decoy failure (arXiv:2609.04382) and per-document text leakage (arXiv:2610.04128) are companion results of this framework.
Want it run against your system?
LLM-Attack Suite powers Setloop's AI Security Red-Team Evaluation engagements: scoped adversarial evaluation of split and distributed LLM systems with evidence you can audit.