Applied research on privacy and security in distributed AI.
Setloop Lab tests the assumptions behind private AI infrastructure and publishes its findings.
Why a lab?
Our research examines privacy and security assumptions across distributed AI systems. Our split-LLM studies examine how returned gradients expose private information and how the choice of measurement changes a privacy assessment.
Our first preprint shows how zero gradients reveal which rows are decoys in the system studied. Our latest preprint measures how much gradients add to text reconstruction, and why token recovery and exact document recovery can tell different stories about the same defence.
Our research examines how mitigations affect both privacy and model utility. This work asks what an attacker can learn from each observable channel, including channels beyond the forward pass.
Latest research.
LLM-Attack Suite: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems
Georgios Politis and Evangelos Pappas
Sixty-five ways to attack a split language model and a scoreboard that refuses to say “safe”.
One open framework, three attack surfaces: split-model privacy (seam and gradient inversion, membership inference, key-lifecycle attacks), serving and protocol security (traffic side channels, tenant isolation, supply chain), and application and agent security (OWASP-style batteries, from prompt injection to confused deputies). Every run records hash-chained, reproducible evidence, and the scorecard fails closed: a PASS without a linked positive control is downgraded, not trusted. MIT-licensed, with a manuscript and frozen evidence, it is the framework behind the two studies below and the toolkit behind our red-team engagements.
What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document
Georgios Politis and Evangelos Pappas
A few percent more words, or nearly three times as many whole documents. Both numbers are true; only one is honest.
If an attacker sees the gradients returned during training, not just the forward messages, how much more of your text can they rebuild? Measured per word, the gain looks modest: 94.20% of tokens with activations alone, 97.38% once gradients are included. Measured per complete document, exact word-for-word recovery jumps from about 1 in 7 to more than 1 in 3. The same experiment answers both ways, and that is the point: the measurement you choose writes the privacy story you publish.
Privacy Failure in Split-LLM Training: The Returned Gradient Nullifies the Decoys
This system passed its privacy check. The check was blind in one eye.
When two machines train one language model, they trade messages back and forth. The privacy audit watched only the messages going out, never the gradients coming back. Those returned gradients quietly mark which data rows are real and which are decoys, and they did so perfectly: 4,096 out of 4,096 batches, on every one of nine runs. Closing the leak costs about 0.01 nats of model quality (almost nothing), but five further classes of attack were never even measured. A privacy check that doesn't name what it watches isn't a privacy check.
real rows identified per run via zero gradients, across nine seeds
held-out cross-entropy cost to close the leak with clipping + noising
of further attacks left unmeasured, the leak you fix is not the system you secure
Building distributed AI?
Discuss privacy requirements, model utility and observable-channel audits with Setloop.