Sandboxes for parallel RL rollouts
Run your whole RL loop on one platform, GPUs included.
RL compute environments
Deterministic resets, parallel rollouts, long-horizon state, and GPUs on-demand.
Snapshot and restore
Capture the environment's memory and filesystem once and boot every sandbox from that same state.
Explore docsRollouts in parallel
Fork a snapshot into a fleet of isolated sandboxes, each with its own filesystem, processes, and ports.
Explore docsSub-second boots
Sandboxes cold boot in under a second so startup times never bottleneck your evals.
Explore docsNo timeouts
Environments run as long as the task demands.
Explore docsStatefulness & persistence
Read and write to persistent volumes and create snapshots to restore any sandbox state in the future.
Explore docsOne platform
Run evals and retrain your models, all in one place.
Explore docsHow to run RL rollouts
Run Docker-in-Docker
Run any existing Docker image in a sandbox, giving you a fully reproducible environment between your local machine and the cloud.
Fan out sandboxes
Spin up thousands of sandboxes, and write outputs to a shared volume.
Attach GPUs
Add a GPU to any sandbox with just a single line of code.
# Run the policy update on a GPU — same SDK
from beam import function, Volume
@function(
gpu="RTX4090",
volumes=[Volume(name="trajectories", mount_path="./data")],
)
def update_policy():
batch = load_trajectories("./data")
loss = policy.train_step(batch)
save_checkpoint("./data/policy-v43.pt")
# Pause a stateful environment and pick it back up later
snapshot_id = env.snapshot_memory()
env2 = Sandbox().create_from_memory_snapshot(snapshot_id)Frequently asked questions
How fast can a snapshot restore?
A restore is just booting a new sandbox from your saved snapshot, which takes under a second with dependencies included. Every episode starts from the identical filesystem state, and nothing from the previous episode leaks in.
Can I pause and resume an episode mid-run?
Yes. Memory snapshots capture the sandbox's full state (running processes and exposed ports included), and create_from_memory_snapshot picks up exactly where the episode left off. Useful for long-horizon tasks and branching a live trajectory.
How many rollouts can run at once?
Fleets are created programmatically, so parallelism scales with your training run. Each sandbox is isolated with its own filesystem and processes. Billing is per-second, so terminated environments cost nothing.
Can environments have GPUs?
Yes. Pass gpu to the Sandbox constructor for environments that render or run models, and keep CPU-only environments on fractional CPUs so large fleets stay cheap.
Where does the training step run?
On Beam, next to the rollouts. Decorate your update function with @function(gpu=...) and mount the volume your environments write to, so rollouts, trajectories, and gradient steps stay on one platform.
Related use cases
Give your AI agents a safe place to run code
Isolated sandboxes for agent-generated code, booting in seconds.
Training & Fine-tuningRun GPU training jobs on-demand
Start training runs on cloud GPUs from Python and pay only while they run.
Training & Fine-tuningFine-tune models on GPUs
LoRA and QLoRA fine-tuning on serverless GPUs.
Give your agents their own machines
Sign up, install the CLI, and launch a sandbox, VM, or GPU from your terminal or your coding agent.