setloop.io
White Paper · SET-WP-2026-05
All papers
setloop.io
White Paper
Setloop Technical White Paper Series

Automating AI Research: Architecting Autonomous RL Experiment Loops

A technical framework for reinforcement learning experiment loops in which LLMs propose code changes while fixed evaluation criteria and deterministic version control keep each iteration measurable and auditable.

Document
SET-WP-2026-05
Published
September 2026
Version
1.0
Category
Research Automation
Classification
Public
Working infrastructure, not slide decks. setloop.io

In brief

The challenge. Long-running AI research programmes often stall on manual iteration. Hyperparameter tuning, disconnected tooling and poorly scheduled execution slow engineers and leave costly compute idle.

The response. Continuous model improvement needs a governed, closed loop that can propose, run, evaluate and record experiments without requiring an engineer to manage every step.

What this paper covers. This paper describes an architecture for automated reinforcement learning (RL) experiments. LLMs propose code changes; fixed evaluation criteria and deterministic version control keep each iteration measurable and auditable.

Closed-loop
Proposals, execution, evaluation and versioning managed end to end
Fixed
Stable evaluation criteria reduce drift between experiment runs
Traceable
Every run recorded, whether rejected or retained
01

The automation of model improvement

Manual hyperparameter tuning and fragmented execution limit the pace of AI research and leave expensive compute underused. An autonomous RL research system can address these constraints through four operational components.

Two trends support the case for automation. Stanford HAI's 2025 AI Index reports that training compute for notable AI models is doubling approximately every five months. It also reports that the cost of GPT-3.5-level inference fell more than 280-fold, from $20.00 to $0.07 per million tokens. Greater model complexity increases the burden of experimentation, while cheaper inference reduces the cost of using LLMs to manage it.

5 months
Doubling cadence of training compute for notable AI models (Source: Stanford HAI, AI Index Report 2025)
$15
Cost per generated research paper in Sakana AI's end-to-end AI Scientist demonstration (Source: Lu et al., arXiv:2408.06292, 2024)
0.7%
Share of Google's worldwide compute resources recovered on average by AlphaEvolve's discovered data center scheduling heuristic (Source: Google DeepMind, 2025)

Early systems show what this approach can achieve. Google DeepMind reports that AlphaEvolve improved a matrix multiplication kernel in Gemini's architecture by 23% and recovered an average of 0.7% of Google's worldwide compute resources through a scheduling heuristic. Sakana AI's AI Scientist demonstrated an automated workflow from idea generation to reviewed manuscript for less than $15 per paper. These examples establish technical feasibility, although they do not by themselves prove that every research workload will deliver the same economic benefit.

Table 1. The four pillars of autonomous RL research
Operational pillarTechnical mechanismStrategic enterprise impact
LLM-Guided Iteration An LLM proposes hyperparameter changes or specific code diffs. Reduces the engineering time needed to formulate and configure experiments.
Distributed Execution Training runs are routed to local or cloud GPU targets. Matches workload requirements to available capacity and cost.
Governed Quality Control Each result is evaluated against fixed criteria; weak runs are rejected. Helps detect regressions and makes improvement claims comparable across runs.
State Management The system supports checkpointing, resumption and versioned retention. Provides an auditable lineage for both retained and rejected experiments.
02

Reference architecture: the autonomous experimentation fabric

The reference architecture separates proposal, execution and governance. Together, these layers allow experiments to be generated, run, measured and recorded continuously, while keeping human approval at defined control points.

Figure 1. The autonomous experimentation fabric: three integrated operational planes
L1Proposal
Proposal & Orchestration Engine
LLM-guided experiment loopsFailure-state analysisHyperparameter searchCode-diff generation
▾
L2Execution
Execution & Evaluation Layer
Hybrid GPU targetsLocal bare-metal or cloudFrozen evaluation boundaryObjective benchmarking
▾
L3Governance
Governance & Versioning Ledger
Checkpoint & resumePreemption survivalNegative-result loggingTraceable kept versions

2.1The proposal & orchestration engine

LLM-guided experiment loops

An orchestration layer uses a designated reasoning model to analyse previous failures and propose hyperparameter changes or architectural code diffs.

Hyperparameter search

Grid and Bayesian search explore the optimisation space within defined parameters.

2.2The execution & evaluation layer

Hybrid GPU targets

The orchestration engine sends proposed training runs to local bare-metal clusters or cloud GPU instances according to capacity and cost constraints.

Frozen evaluation boundary

After each run, the candidate model is benchmarked against a fixed dataset and predefined criteria. This reduces evaluation drift and makes results comparable.

2.3The governance & versioning ledger

Checkpointing

Automated checkpointing and resumption allow long-running training loops to recover from hardware pre-emption or spot-instance termination.

Traceable lineage

Rejected runs are logged for negative-result analysis; successful iterations are retained as traceable versions in the enterprise model registry.

03

Engineering implementation roadmap

Deployment should proceed in three phases. Evaluation controls and state management come first, followed by the LLM proposal engine and, finally, closed-loop automation.

Phase 1

Foundation & Guardrails

  1. Frozen evaluation boundary
  2. Target compute mapping
  3. Checkpoint configuration
Phase 2

The LLM Engine

  1. LLM proposal integration
  2. Hyperparameter search
  3. Code-diff generation
Phase 3

Closed-Loop Automation

  1. Automated execution
  2. Result evaluation
  3. Traceable versioning
  1. Establishing guardrails. Define the immutable evaluation datasets and configure the routing logic for local or cloud GPU targets.
  2. Integrating the engine. Deploy the LLM-guided loops responsible for generating hyperparameter and code-diff search parameters.
  3. Closing the loop. Automate execution and evaluation, reject weak runs, and promote only traceable versions that meet the agreed criteria.
04

How Setloop can help

Closed-loop reinforcement learning depends on research engineering, evaluation design, infrastructure automation and GPU operations. Setloop brings these disciplines together to build systems that can move from controlled experiments into production workflows.

Bespoke research automation

Setloop provides Automatic RL Research as a consultancy service for AI teams running long-term research programmes.

Working infrastructure over slide decks

Setloop builds the experiment loop, evaluation controls and supporting infrastructure within the customer's private or cloud environment.

End-to-end GPU execution

Setloop can integrate these research loops with GPU workload planning so experiments run on infrastructure selected for capacity, performance and cost.

Key takeaways

What research and infrastructure leaders should remember

  1. Manual iteration limits scale. Long-running research programmes need more than manual hyperparameter tuning and disconnected execution.
  2. Evaluation criteria must remain stable. A fixed dataset and predefined measures reduce drift and make results comparable across iterations.
  3. Every run must be auditable. Record both retained and rejected experiments so that decisions, failures and model lineage can be examined later.
  4. Autonomous loops must run on cost-accountable infrastructure. Routing training runs across local bare-metal and cloud GPU targets by capacity and cost keeps continuous experimentation economically sustainable.
06

References

  1. Stanford HAI. “AI Index Report 2025.” Stanford University, April 2025.
  2. Google DeepMind. “AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms.” May 2025.
  3. Lu, C. et al. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” arXiv:2408.06292, August 2024.
  4. MarketsandMarkets. “Automated Machine Learning (AutoML) Market: Global Forecast to 2028.” 2023.
  5. ClearML, AI Infrastructure Alliance, and FuriosaAI. “The State of AI Infrastructure at Scale 2024.” March 2024.

About Setloop

Setloop is an engineering consultancy and product studio for organisations building GPU workloads, cloud GPU platforms, private AI infrastructure and AI factory architectures. Its engineers design, benchmark and deploy operational infrastructure in customers' private and hybrid-cloud environments across the UK and EU.

The Setloop Technical White Paper Series distils reference architectures from client work and the Setloop product portfolio, including LLMTrace, AutoOps, GPU Cloud Platform, AI FinOps and Automatic RL Research.

For more information