OpenRSI scales recursive self-improvement to real-world AI infrastructure. Our agents run a continuous performance-optimization loop, with every gain validated by an independent verifier, while people define each problem and review the changes at the end. In partnership with SGLang-Omni, this loop has run across eight omni models and raised each model’s throughput by +7.6% to +326.5% on H100, with fourteen PRs so far, all reviewed and merged by the SGLang‑Omni maintainers.
Recursive self-improvement (RSI), AI that improves itself, is one of the most important research problems in AI, yet most demonstrations stay on benchmarks and toy problems. OpenRSI takes a first step toward real-world RSI: our agents improve the infrastructure that serves AI models, and each verified, merged change becomes the baseline for the next round. Concretely, given an objective, agents formulate a hypothesis, design experiments and refine their approach from the evidence, iterating continuously while an independent verifier validates every gain.
Here we partnered with the SGLang‑Omni team. Together we set the goal: make omni models run faster in SGLang-Omni, without touching model weights or slowing down any other model. We turned that goal into tasks with RSI‑Anything, and our agents ran those tasks on RSI‑Harness. They produce optimizations that pass upstream review and complement human experts by taking on the experiment-heavy part of the work at a pace no human can match. The results ship as code that people run: SGLang‑Omni maintainers have since merged fourteen PRs from this work across eight omni models (Table 1). The agents’ part is performance optimization. Defining the tasks, system design such as Ming3‑TTS streaming, and review were done by people.
Cached GPU reference preprocessing, prefill and vocoder graphs, asynchronous decode and fused flow arithmetic for the 0.5B model.
+326.5% / +93.9%
+344.3% / +114.5%
0.5B onlyGPU preprocessing, async decode and chunked prefill become defaults; streaming is opt-in.
Not all OpenRSI: the range also includes a streaming codec rewrite (#2805), #2662, #2770, #2772 and the SGLang 0.5.22 upgrade (#2964), and the two sides run different images. #2859 adds about 41 s of startup and 3.7 GiB of peak VRAM. The H200 streaming baseline is low (3.90 req/s, against 5.32 on H100) for reasons not yet known, so read +149.5% with care.
Not all OpenRSI: the range also includes the SGLang 0.5.22 upgrade (#2964), and the two sides run different images.
Not all OpenRSI: the range includes 28 other PRs, #3136 also fixes a sampling bug in the baseline, and the benchmark reuses reference recordings. The opt‑in streaming in #3189 and #3190 was designed by people.
Table 1Merged OpenRSI changes in SGLang‑Omni. Each row compares main before the model’s first OpenRSI PR with main after its last (Commits), at 128 requests per round. Gains are non-streaming / streaming.
Method: Harnessed RSI
Today the usual way to optimize a serving stack with AI is an engineer driving a coding agent such as Codex or Claude Code. The agent writes and runs code; the human picks each experiment, reads each result and decides what comes next. The human is the loop (Figure 1, left), and for systems work that is costly:
Many rounds, mostly idle. The human has to step in on every round, and each round waits until they read the result and write the next prompt. Most of the wall-clock time is spent waiting on a human, so the loop is slow and inefficient.
The human caps what the AI can do. The agent only explores what the human thinks to try and has time to check. A serving stack holds countless small knobs, settings and code paths, far more than anyone can try one by one, and finding which ones matter takes measurement. One line that stops idle threads from spinning was first proposed inside a larger change (#2443); split out and measured on its own, it was worth +32.5% streaming throughput (#2524).
Harnessed RSI (Figure 1, right) takes the human out of the inner loop of performance optimization, not out of the work. Defining a problem well, with its task, benchmark and guards, takes human effort, and system design stays with people. Once that is done, agents propose, run and measure changes to the serving runtime round after round, and people return to review what they find.
Two nested loops that never stop. An agent explores in the inner loop. The harness governs the outer loop, deciding when work is judged and carrying every lesson into the next round. Rounds run back to back, without waiting on anyone.
Every step is verified. Nothing counts until an independent verifier, which the agent never sees, confirms it. Within a search, each confirmed gain becomes the bar the next round must beat. Across PRs, later work builds on what landed: #2555 was measured with #2524 in place, and #3071 reuses the asynchronous prefill that #2881 added. That is the recursive part today: AI improving the infrastructure that AI runs on.
Figure 1Where the human sits. With a coding agent, every round passes through a human. With Harnessed RSI, the agent is the inner loop, RSI-Harness runs the outer loop against a verifier, and each verified result becomes the next baseline.
RSI‑Anything: A scalable RSI pipeline for SGLang‑Omni
On top of the harness we built RSI‑Anything, a lightweight pipeline built to scale RSI across SGLang‑Omni quickly: it turns a research proposal into merged SGLang‑Omni code. Agents handle the experiments and the implementation iteration: building an executable benchmark from the proposal, searching for improvements, reproducing their own gains and opening the pull request with evidence. People write the proposal and own the merge. In review, SGLang‑Omni maintainers set requirements and joined the design over many rounds, and in some cases contributed code or reproduced results on their own.
Figure 2The RSI‑Anything pipeline for SGLang‑Omni. People write the proposal, and agents turn it into a Harbor task with an executable benchmark and evaluator (01 to 02). Agents then run the experiments and iterate on the code (03 to 05). SGLang‑Omni maintainers review over many rounds, sometimes adding code or reproducing results, before they merge (06). Work goes back when a gain does not reproduce (04 to 03) or when review asks for changes (06 to 05).
Merged PRs
The fourteen PRs touch every stage of speech serving, from audio encoders to decoders and vocoders, and they fall into five patterns (Figure 3). Almost all of them remove overhead in the runtime around the model; #2901 also adds incremental text streaming for Whisper, and #3189 and #3190, designed by people, add streaming audio for Ming3-TTS.
These changes target runtime overhead without changing model weights or sampling settings. Output correctness and quality were checked within the reported test coverage; unchanged waveforms or perceptual quality are not established.
01Idle threads compete for cores
Threads that spin while they wait hold CPU cores the rest of the pipeline needs. Letting them sleep gives those cores back, so queued work starts at once.
Tensors were turned into Python lists to cross a stage boundary, then rebuilt on the other side. Now they travel as tensors, and only the final message carries typed bytes.
Some results were computed and thrown away: a quantizer reconstruction no caller needed, a fresh tensor for every embedding channel, a frontend that ran twice on a fallback path, audio features extracted past where they were needed, reference audio encoded again on every request. The agents skip that work.
Launching small kernels one at a time leaves the GPU waiting on the CPU. Capturing whole steps as CUDA graphs, or fusing kernels, replays them at once. #2901 also moves encoder attention to FlashAttention‑4.
Prefill and decode run on two separate machines. They used to take turns, so each machine sat idle while the other worked. Now the prefill machine starts the next request as soon as it finishes the last one. Its decode starts the moment the decode machine finishes the request before it, so neither machine sits idle.
Figure 3Fourteen PRs, five patterns. A PR that combines several patterns appears under each of them. #3189 and #3190 add streaming, a new feature, so they are not shown. The animations are schematic, not measured timelines; both rows of each run on the same clock.
Why it came together quickly
Figure 4 shows the best search run for each of the first six models, from the task logs. In five of the six, the first candidate to pass every guard arrived within eight hours.
How to read
Best run: its best reward so far. Each dot passed every guard.
Measured, but failed a guard
No complete measurement
Frozen baseline, 1.0×
Reward is throughput over the task’s frozen baseline. Guards also check latency and WER, so a faster candidate can still fail.
Figure 4The best search run for each model, in hours since the run started. Each task scores against its own benchmark, so rewards are not comparable across panels or with Table 1.
Serving has a scoreboard. Throughput and latency can be measured automatically, which gives the loop a dense, reliable signal. Quality is harder to score: WER covers transcription accuracy, not how the audio sounds.
One harness, many tasks. The same harness ran every search, and one proposal led to PRs for four models. Once a task and its evaluator exist, another agent round costs compute, not someone's afternoon.
Guards keep results honest. Every submission is scored in a clean environment, and a faster candidate that regresses latency or WER earns no reward.
Evidence arrives with the diff. Every PR came with a serving study and stated caveats, and most with an audit of the raw benchmark records. That gave review a concrete starting point, though review still took a median of about three and a half days and up to 23 commits.
What this means for inference optimization
Once a problem is well defined, inference optimization turns from work limited by expert attention into a search limited by compute, evaluator quality and review bandwidth.
Define the problem
Once the problem is well defined
Design and first implementation
Find the idea
Implement
Measure
Review and merge
Before
Engineerssystem design, first implementation
Engineercompetes with every other priority
Engineerone change at a time
Engineermanual benchmark runs
Maintainerchecks the claim from scratch
Today
People, stilldesign, first implementation, task, benchmark and guards
Agents × Nfrom a written proposal
Agents × Nin parallel workspaces
Evaluatorits quality caps what is found
Maintainerreview time becomes the limit
Scarce resourceRuns on computeHuman work OpenRSI does not take on yet
Figure 5Where the bottleneck moves once a problem is well defined. Top: before OpenRSI. Bottom: with Harnessed RSI, agents take over the search for speedups, and people define the problem and decide what to merge.
The long tail gets searched to the end. No human can try every thread setting, data conversion and launch pattern in a serving stack. Agents can work through them one by one and measure each on its own, which is how #2524 showed that a single setting can be worth about a third of throughput.
Tuning becomes continuous. Once a task and its evaluator exist, a new model, runtime or machine shape can start another round instead of a new project.
Results depend on the deployment. #2524 is worth +82.9% on 8 cores and +33.6% on 16. Agents can afford to measure each target configuration instead of assuming one answer.
The evaluator is the product. Agents optimize what is measured, so benchmarks, quality checks and guards against reward hacking become the main engineering work. For speech, WER alone is not enough.
What’s next
This is the first of many collaborations between OpenRSI and the SGLang‑Omni team. Next, we want to hand agents more of the loop that people still run today, and move toward fully autonomous RSI for AI infrastructure. Stay tuned.
Led by Zhuofeng Li, Aaron Tian and Chenyang Zhao, with Yichen Feng, Wenhao Chai, Xinyang Han, Cara Li, Shujia Liang and Yuetai Li. Thanks to the SGLang‑Omni team for many rounds of review, design input, code and independent reproductions, and to the other SGLang-Omni contributors whose optimizations landed in the same period.