Applied research on privacy and security in distributed AI.
The Lab is where we stress-test the assumptions behind private AI infrastructure, and publish what breaks.
Why a lab?
We talk a lot about privacy in AI. Our lab's recent paper on split-LLM training exposed how easily defences can unravel if you aren't looking at the whole architecture.
In split and distributed learning, you mask data by mixing it with "decoys" on the forward pass to the cloud. But we found a flaw: to keep the model accurate, the local node ignores those decoys during the loss calculation. The gradients sent back for the decoys are exactly zero. Attackers don't need advanced ML to steal the data, they just look for the zeros.
At Setloop, we look at edge cases exactly like this. Plugging one hole, like noising the backward pass, doesn't solve the broader architectural gap between data utility and true privacy. If you are building distributed AI, you have to audit every observable channel, not just the ones you expect to leak.
Latest research.
Privacy Failure in Split-LLM Training: The Returned Gradient Nullifies the Decoys
A systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while leaving an observable channel untested. The frame the untrusted cloud receives mixes real rows with decoys, and the loss ignores the decoys, so their returned gradients are exactly zero, and the pattern of zeros reveals which rows were real. Across nine seeds, the zeros identified the real rows on every frame: 4,096 of 4,096 per run. Every run passed the forward-channel privacy check and the quality check, yet failed the same check once the returned gradient was included. Clipping and noising each gradient row closes the leak for about 0.01 nats of held-out cross-entropy, but the system is not thereby safe: five further classes of attack were never measured.
real rows identified per run via zero gradients, across nine seeds
held-out cross-entropy cost to close the leak with clipping + noising
of further attacks left unmeasured, the leak you fix is not the system you secure
Building distributed AI?
How is your team balancing privacy budgets without destroying model utility? We audit the channels you don't expect to leak.