RL systems · Architecture note · 14 min read

One sandbox can run RL. A pool is built for the loop.

A general sandbox, a local vector environment, and a dedicated rollout plane can all run reinforcement learning. They optimize different boundaries. The important question is not “can it run Python?” It is “where does the environment step live, and how many round trips does the trainer pay?”

01 · Three models

The same policy loop, three execution boundaries.

Traditional vectorized RL keeps the trainer and environments on one host. A general BoltzLabs sandbox moves the whole job into an isolated machine. A dedicated RL pool keeps the trainer where the model runs and moves only the environments behind a batched API.

Model	Unit you manage	Step boundary	Best fit
Local VectorEnv	Processes on the trainer host	Local IPC	Environments fit beside the policy
General sandbox	One isolated machine	The complete training command	A reproducible remote training job
Dedicated RLPool	A pool of N environments	One request carrying N actions	Remote high-parallelism rollouts

02 · Start local

Normal vectorized RL is already good.

Gymnasium's vector APIs put multiple environments behind one familiar step(actions) call. If the environment is cheap, trusted, and fits on the trainer's machine, this is the simplest architecture. There is no network boundary to optimize away.

train.py

import gymnasium as gym
import numpy as np

envs = gym.make_vec("CartPole-v1", num_envs=32, vectorization_mode="async")
obs, _ = envs.reset(seed=0)

for _ in range(10_000):
    actions = np.zeros(len(obs), dtype=np.int64)  # replace with your policy
    obs, rewards, terminated, truncated, infos = envs.step(actions)

envs.close()

Do not distribute by reflex. A remote pool adds JSON serialization, a network hop, fleet placement, and another failure domain. Use it when local CPU, memory, isolation, or reproducibility is the constraint.

03 · General sandbox

Put the whole training job in one sandbox.

A BoltzLabs Sandbox is a long-lived gVisor container with a writable workspace. It can install dependencies and run a complete vectorized training program. This is the direct path from “works on my laptop” to an isolated remote job.

sandbox_job.py

from boltzlabs import Sandbox

training_job = r"""
import gymnasium as gym
import numpy as np

envs = gym.make_vec("CartPole-v1", num_envs=32, vectorization_mode="async")
obs, _ = envs.reset(seed=0)

for _ in range(10_000):
    actions = np.zeros(len(obs), dtype=np.int64)  # replace with your policy
    obs, rewards, terminated, truncated, infos = envs.step(actions)
"""

with Sandbox(machine="large", environment="pytorch") as sb:
    sb.exec("pip install gymnasium").check()
    result = sb.run(training_job, language="python", timeout=3600)
    result.check()
    print(result.stdout)

The environment processes and policy live inside the same sandbox command, so inner-loop steps do not cross the public API. The API creates the machine, starts the job, returns its output, and destroys the machine when the context manager exits.

What a general sandbox does not provide

If the policy must stay on a separate GPU trainer while each environment runs remotely, the generic sandbox API has no environment-level step primitive. You would need to build and operate the bridge yourself.

orchestrator.py

# A general sandbox does not expose env.step() as a remote primitive.
# To keep the trainer local while environments run remotely, you would own:

for box in sandboxes:
    upload_environment_code(box)
    start_long_running_rpc_bridge(box)

while training:
    actions = policy(observations)
    results = fan_out_requests(sandboxes, actions)
    observations = restore_index_order(results)
    restart_dead_bridges()
    reset_finished_environments()

# RLPool is the managed version of this control loop.

04 · Dedicated RL

Make the batch the unit of work.

A trainer produces N actions before it can advance. RLPool therefore sends one request containing all N actions and receives one ordered batch containing all N observations, rewards, done flags, and info objects. The network cost is paid once per policy step, not once per environment.

policy → actions[N]

one HTTP step

worker fan-out × N

The environment contract

Each environment is a small program started once. It receives newline-delimited JSON over stdin, keeps episode state in process memory, and writes one JSON reply per operation. The SDK's serve() helper owns flushing, protects the protocol from stray stdout, and converts user exceptions into terminal transitions.

my_env/env.py

# my_env/env.py
import random
from boltzlabs.env import serve

state = {"t": 0, "rng": random.Random()}

def reset(seed):
    state["rng"] = random.Random(seed)
    state["t"] = 0
    return {"t": 0, "noise": state["rng"].random()}

def step(action):
    state["t"] += int(action)
    done = state["t"] >= 100
    obs = {"t": state["t"], "noise": state["rng"].random()}
    return obs, float(action), done, {"steps": state["t"]}

serve(reset=reset, step=step)

The training loop

train_pool.py

import numpy as np
from boltzlabs import RLPool

with RLPool("./my_env", n=1000) as pool:
    obs = pool.reset(seed=0)

    for _ in range(10_000):
        actions = np.ones(pool.n, dtype=np.int64)  # replace with your policy
        obs, rewards, dones, infos = pool.step(actions)  # one HTTP request

        if dones.any():
            obs = pool.reset(where=dones)  # reset only completed episodes

    print(pool.timing.worker_ms)
    print(pool.timing.roundtrip_ms)
    print(pool.timing.stragglers)

05 · Inside the worker

Warm processes, not snapshots.

Package once

The SDK creates a deterministic compressed archive of the environment directory and vendors the tiny serve helper. The upload happens at pool creation, not on every step.

Mount code read-only

The worker extracts one code directory and mounts it read-only at /env in every sandy sandbox. Each environment receives a private writable /workspace.

Start every environment

Creation is staggered to avoid an interpreter-startup thundering herd. The pool is not returned until every environment answers its initial reset handshake.

Keep processes resident

Python imports, simulator state, and heap objects remain alive between steps. There is no per-step process spawn and no process-memory snapshot store.

Fan out in index order

The Go worker starts one goroutine per selected environment, writes one JSON line to each process, and restores replies to their original batch positions.

Measure shared memory honestly

Status walks each sandy process tree and reports proportional set size. Shared interpreter and library pages are divided among the processes mapping them instead of counted N times.

06 · Resets and failure

Fast reset when possible. Respawn when necessary.

Soft reset

In-process episode reset

pool.reset(where=dones) sends reset only to finished environments. Other observations remain in their original slots.

Hard reset

Kill, wipe, respawn

pool.reset(hard=True) recreates the private workspace and process. It is a cold start, not snapshot restoration.

A process that misses its per-step deadline is killed so one wedged environment cannot freeze the batch. Its result is explicit: null observation, zero reward, done=True, and info["boltzlabs_straggler"]. The next reset respawns it.

07 · Trainer integration

Use the pool as a Gymnasium VectorEnv.

BoltzLabsVecEnv adapts pool results to the Gymnasium five-value step API. It uses same-step autoreset: terminal observations are preserved in info["final_observation"], while the returned observation is the next episode's first observation.

vector_env.py

import numpy as np
from boltzlabs import BoltzLabsVecEnv

envs = BoltzLabsVecEnv(
    env_dir="./my_env",
    n=1000,
)

obs, infos = envs.reset(seed=0)
actions = np.ones(envs.num_envs, dtype=np.int64)
obs, rewards, terminated, truncated, infos = envs.step(actions)
envs.close()

08 · Choose deliberately

Use the smallest system that solves the bottleneck.

Stay local

Your environments fit beside the trainer and local IPC is already fast.

Use one sandbox

You need isolation and reproducibility for a complete remote training job.

Use RLPool

The policy and environments belong on separate capacity and rollout scale is the constraint.

Implementation status, without benchmark theatre

The pool engine, control-plane routes, SDK, partial resets, straggler path, memory accounting, and Gymnasium adapter are implemented and covered by worker and SDK tests. Production deployment and publishable amd64 benchmark results are not claimed here. The SDK deliberately reports worker time and end-to-end round-trip time separately so future measurements cannot hide network cost.

Read the RL pool API Create an API key