RoboArchitect

Abstract

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, call auxiliary models, or command the arm directly. Yet these harnesses are still built by hand for each new robot and benchmark. RoboArchitect automates their design: a coding model observes the robot’s rollouts, diagnoses failures, and iteratively refines the harness around the frozen policy, without updating any weights. On held-out episodes, average success rises from 9.7% to 43.3% on unseen scenes (4.5×) and from 12.7% to 40.2% on unseen tasks (3.2×). The gains are largest where the benchmark is far from the policy’s training data: around a general-purpose policy, the evolved agent often outperforms even the same policy finetuned on the benchmark. RoboArchitect automatically designs diverse control routines, such as lifting a jammed arm clear, and a persistent memory that carries experience across episodes, such as which grasp heights worked. Inside the evolved agents, the VLA policy and code-as-policy work together dynamically: the policy keeps the motions it executes well, and the code takes over where the policy fails. RoboArchitect also evolves a single unified harness that works across robot platforms, from the Franka arm to the WidowX and the Google robot. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and that its design can be automated and optimized without weight updates.

Held-out success of each policy alone and inside its evolved agent, under unseen scenes and unseen tasks, with the finetuned versions of X-VLA marked. — **Held-out success rises in nearly every setup, often past the same policy finetuned on the benchmark.**

RoboArchitect

A coding model iteratively designs and refines an agent harness around a robot policy (VLA). Each refinement is based on actual evidence from observed rollouts. The new harness may follow the policy, call auxiliary models, command the arm directly, or even design a persistent memory that informs decisions across tasks and scenes. Over iterations, the harness grows from the VLA policy alone into a fully designed robotic agent with auxiliary tools and persistent memory. The right side of the figure below shows the experiment conditions that test how well the auto-optimized harness generalizes.

Overview: a robot harness, initially a pass-through of the policy, evolves into an agentic program with persistent memory and access to auxiliary models; a coding model reviews rollout trajectories, diagnoses failure patterns and improves the harness iteratively. Right: the three experiment conditions.

Optimization

1Run the agent and record every step

The agent runs on the training episodes. Every step is recorded: camera frames, the robot’s readings, and every model and tool output.

2A coding model diagnoses failures and revises the harness

A coding model reads the recordings, diagnoses the failures, and designs a revised harness.

3Keep only strictly improving revisions

The revised harness runs on the same training episodes and replaces the current one only if it scores higher.

4Evaluate the final agent on held-out episodes

After N iterations, the final harness is evaluated on held-out episodes that were never used to optimize it.

Extra components

Auxiliary vision, language and grasp models

Language: Qwen3-30B-A3B-Instruct
Vision-language: Molmo2-8B or Qwen3-VL-8B-Instruct
Object detection: Grounding-DINO
Segmentation: SAM 3
Pointing: Molmo2-8B
Grasp poses: GraspGen
Geometry helpers: turn a pixel and its depth into a 3D point in the robot’s frame, and build relative motions

A persistent memory with no fixed schema

The harness can keep a memory across episodes. The coding model designs the memory freely: its entries, their format, and when the agent reads them. Every evaluation starts with an empty memory.

Evaluation: Every agent is evaluated on held-out episodes that were never used to optimize it

Results

Around the same frozen policies, the evolved agents raise average held-out success by 4.5× on unseen scenes and by 3.2× on unseen tasks. They even lift out-of-domain policies past their finetuned versions: around the general X-VLA, never trained on these benchmarks, the evolved agent outperforms X-VLA finetuned on the benchmark most of the time. Iterative harness refinement also beats a one-shot agent design on nearly every held-out set.

			Unseen scenes (%)				Unseen tasks (%)
Benchmark	Policy	Trained on	VLA policy	One-shot	Iterative	Δ	VLA policy	One-shot	Iterative	Δ
LIBERO-Pro-10	MolmoAct-2	LIBERO	10.0	12.0	32.0	+22.0	2.5	2.5	15.0	+12.5
	OpenVLA-7B	LIBERO-10	0.0	2.0	26.0	+26.0	0.0	0.0	22.5	+22.5
	π_0.5	LIBERO	8.0	4.0	24.0	+16.0	0.0	0.0	2.5	+2.5
	RLinf-π_0.5	LIBERO-130	2.0	2.0	40.0	+38.0	0.0	0.0	10.0	+10.0
	SmolVLA	RoboCerebra	0.0	0.0	38.0	+38.0	0.0	10.0	27.5	+27.5
RoboCerebra	π_0.5	LIBERO	22.0	20.0	55.3	+33.3	24.0	12.0	28.0	+4.0
	SmolVLA	RoboCerebra	20.0	34.3	49.3	+29.3	26.0	16.0	80.0	+54.0
VLABench	X-VLA	general robot data	7.3	32.8	52.6	+45.2	1.2	–	72.5	+71.3
↪ X-VLA finetuned on VLABench			35.2	–	–	–	32.1	–	–	–
SimplerEnv WidowX	X-VLA	general robot data	0.0	77.1	87.5	+87.5	0.0	–	66.7	+66.7
↪ X-VLA finetuned on WidowX			70.8	–	–	–	87.5	–	–	–
RoboCasa-365	RLDX-1	RoboCasa-365	27.9	29.3	27.8	−0.2	73.5	76.5	77.5	+4.0
Mean (all setups)			9.7	21.4	43.3	+345%	12.7	–	40.2	+216%
Mean (cross-benchmark and out-of-domain)			7.3	32.5	58.4	+696%	6.3	–	48.7	+671%

One-shot the agent after iteration 1 Iterative after up to 5 iterations Δ Iterative minus VLA policy (relative gain in the mean rows) ↪ the same policy finetuned on the benchmark VLABench numbers are the benchmark’s official progress score

finetuned on this benchmark same benchmark, object positions swapped a different benchmark general robot data only

Full optimization trajectories

Auto-evolved control routines

RoboArchitect automatically designs diverse control routines, each fixing a concrete failure pattern. Every revision targets failures found in the recorded rollouts, so each routine is grounded in actual evidence from observation. Without any instruction on how to divide the work, most agents converge on a dynamic split between the VLA policy and code-as-policy: the VLA policy keeps the motions it executes well (c), and the code takes over only where the policy fails (a, e, f), often handing control back once the failure is fixed (d). Where the VLA policy is unusable, the code replaces it entirely (b, g, h, i).

(a)

Detecting and clearing arm jams to avoid tipping objects over

MolmoAct-2 · LIBERO-Pro-10

“put both the cream cheese box and the butter in the basket”

The VLA policy keeps driving into the juice carton until it tips over: during a jam its camera view barely changes, so it keeps issuing similar commands. The evolved agent compares the commanded and the actual gripper motion, and when they keep disagreeing, it takes over and lifts the arm clear before asking the policy model again.

(b)

Adding a missing motion: reaching in from the side

X-VLA · VLABench

“Please take the book contract_law_in_japan”

Earlier, the agent always approaches from above and hits the shelf board over the book. The evolved agent finds the board in the depth image, turns the wrist flat, and reaches in horizontally beneath it.

(c)

Rescaling contact force

RLDX-1 · RoboCasa-365

“Close the toaster oven door.”

The VLA policy pushes the door too gently. The evolved agent dynamically scales up the policy’s motion while keeping its direction, and the door latches shut.

(d)

Going to the named object, not a memorized place

RLinf-π_0.5 · LIBERO-Pro-10

“turn on the stove and put the moka pot on it”

LIBERO-Pro swaps object positions. The VLA policy closes its gripper on the frying pan, which now stands at the moka pot’s position memorized from its training scenes. RoboArchitect diagnosed this memorization on its own: the evolved agent takes over, asks the pointing model for the moka pot, carries the gripper there, and then hands control back to the VLA policy.

(e)

Adding a missing motion: pulling a handle

SmolVLA · RoboCerebra

“Open the wooden cabinet top region”

The VLA policy never goes near the cabinet. The evolved agent takes over by synthesizing the complete motion control chain: it asks the pointing model for the cabinet, finds its handles in the depth image, closes on the top handle, and pulls the drawer straight out.

(f)

Keeping the mobile base near the work

RLDX-1 · RoboCasa-365

“Open the cabinet, pick up the bread from the cabinet and place it in the basket. Then move the basket to the dining counter.”

The VLA policy often drives the base away from the counter and does not come back. The evolved agent marks where the gripper last worked, and drives back to that place once the base has driven too long.

(g)

Working down ranked candidates

X-VLA · VLABench

“Pick up the mahjong of 1 pin”

This agent no longer calls the policy, and its vision tools cannot reliably tell the tile faces apart. The earlier agent commits to a single tile. The evolved agent keeps a ranked list of candidate tiles and tries them in turn until the task is done.

(h)

Checking a grasp with depth instead of vision

X-VLA · VLABench

“Put the sanji into the giftbox_seen”

The earlier agent checks its grasp by using its vision tools to find the toy again in the camera image after the lift. This check can fail, and then the toy falls. The evolved agent instead checks the depth image for a surface starting below the fingertips, so it correctly finds the toy hanging there and carries it into the gift box.

(i)

Replacing an unusable policy

X-VLA · SimplerEnv WidowX

“put the spoon on the towel”

The general X-VLA model was never finetuned for this benchmark. It commands a fully open gripper at every step and fails every episode. RoboArchitect decides to stop using the policy, and instead designs and writes a code-as-policy agent from scratch: it reads the instruction, detects the objects, measures them in the depth image, and drives the arm with its own code.

Auto-evolved persistent memory

Via RoboArchitect, nearly every evolved agent builds a memory that carries experience from one episode to the next, with no predefined schema given. The memories hold reusable execution experience (a, b), task and environment clues (c), and calibration to the robot, and are frequently accessed to inform the robot’s control decisions. Note that these memories are part of the auto-designed agents, not of the coding model’s context.

Memories in held-out episodes

(a)

Learning grasp heights from its own lifts

X-VLA · SimplerEnv WidowX

“put carrot on plate” and “put eggplant into yellow basket”

The memory records which fractions of an object’s measured height led to a lift, grouped by object height. Closing at 18% of the carrot’s height (raised to a 0.8 cm minimum) lifts nothing. Closing at 45% lifts the carrot (1.4 cm), and the same fraction lifts the eggplant (3.9 cm).

(b)

Remembering the grip style for each object

OpenVLA-7B · LIBERO-Pro-10

“put the white mug on the plate and put the chocolate pudding to the right of the plate”

The memory counts how often each grip style, pinch or spread, has worked for each object, and later grasps choose the proven one when the object’s shape allows both. The white mug is spread from the inside and stays upright; the pudding box is pinched.

(c)

Learning which instructions need a grip

RLDX-1 · RoboCasa-365, wrist camera

“Press down the lever to turn on the electric kettle.” and “Pick the bell pepper from the plate and place it in the pan.”

The memory links the verb an instruction opens with to whether the hand must hold something. For the pressing task, the agent keeps pressing the lever with closed, empty fingers, as intended. For the picking task, the fingers first close on nothing, so the agent backs off, retries and carries the pepper to the pan.

Unified harness

A single harness for several robots is evolved on a “training pool” of procedurally generated episodes from different embodiments and benchmarks (LIBERO-Pro, VLABench, SimplerEnv WidowX and SimplerEnv Google), and evaluated on held-out benchmark scenes and tasks. Around the general, unfinetuned X-VLA policy, the unified harness evolves into a competent code-as-policy agent and drives the arm with its own code (b). Around the X-VLA specialists finetuned on each benchmark, it keeps the already capable policy in charge, and takes over with its own code only when needed to fix a failure pattern (a).

			Training pool (%)				Held-out (%)
Policy	Trained on	Benchmark	VLA policy	One-shot	Iterative	Δ	VLA policy	One-shot	Iterative	Δ
General X-VLA	general robot data	LIBERO-Pro	0.0	3.5	29.4	+29.4	0.1	3.5	24.1	+24.0
		VLABench	7.1	14.8	17.5	+10.4	7.5	17.3	21.6	+14.2
		SimplerEnv WidowX	0.0	65.8	68.3	+68.3	0.0	45.8	24.0	+24.0
		SimplerEnv Google	0.0	22.0	51.2	+51.2	0.0	20.0	45.6	+45.6
		Mean over all benchmarks	1.8	26.5	41.6	+2228%	1.9	21.6	28.8	+1417%
Specialist X-VLA	its own benchmark	LIBERO-Pro	82.9	83.2	82.8	−0.1	8.4	8.1	8.0	−0.4
		VLABench	13.4	14.9	23.6	+10.2	30.4	33.5	35.1	+4.7
		SimplerEnv WidowX	80.0	80.0	80.8	+0.8	72.9	72.9	74.0	+1.0
		SimplerEnv Google	55.9	55.9	55.4	−0.4	71.9	71.7	70.9	−1.0
		Mean over all benchmarks	58.0	58.5	60.7	+5%	45.9	46.5	47.0	+2%

Training pool procedurally generated scenes, objects, instructions and tasks Held-out each benchmark’s standard episodes One-shot the agent after iteration 1 Iterative after up to 5 iterations Δ Iterative minus VLA policy (relative gain in the mean rows) VLABench numbers are the benchmark’s official progress score

finetuned on this benchmark general robot data only

Control routine (a)

Stepping in only when the policy fails

X-VLA finetuned on LIBERO · LIBERO-Pro

“pick up the cup and place it in the back compartment of the caddy”

LIBERO-Pro changes the task, but the finetuned policy carries out the one it memorized: it puts the book in the caddy and leaves the cup on the table. The evolved agent leaves the policy in control for most of the episode. Only after three quarters of the episode length stored in its memory, with the gripper still far from the cup, does it take over: it finds the cup with the pointing model, grasps it, places it in the back compartment, and hands control back to the VLA policy.

Control routine (b)

Adapting to a slower robot

X-VLA (general) · SimplerEnv Google

“open bottom drawer”

The earlier agent ends each move on a clock set for a faster robot, and on the 3 Hz Google robot it runs out of time while pressing the hand onto the drawer front. The evolved agent ends a move when the arm actually arrives or stops making progress, and reaches the handle from the front along the measured surface normal.

Persistent memory (c)

Calibrating itself to each robot

X-VLA (general) · SimplerEnv Google and LIBERO-Pro

“pick coke can” and “Pick the akita black bowl on the cookie box and place it on the plate”

Finger motion differs between robots. Once per robot, the agent opens and closes the empty gripper and stores which way the fingers close. This memory guides later grasps to turn the wrist to match. The Google robot closes its fingers across the can, and the Franka arm grips the bowl’s rim with one finger inside and one outside.

Citation

@misc{wu2026roboarchitect,
      title={RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory}, 
      author={Shengguang Wu and Suning Huang and Cherie Ho and Ria Doshi and Hao Zhu and Serena Yeung-Levy},
      year={2026},
      eprint={2610.42141},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.42141}, 
}