Batch systems · Implementation guide · 10 min read
Treat every CI shard as a disposable machine.
A CI runner should coordinate work, not absorb every repository's dependencies and side effects. Put each test or batch unit in its own sandbox, cap the fan-out, preserve the remote exit code, and destroy the workspace however the job ends.
01 · Choose the unit
One-off execution or a stateful sandbox?
BoltzLabs has two execution boundaries. The execution plane runs one Python or Node program and leaves nothing behind. A sandbox keeps files, installed packages, and processes across commands. Use the smaller boundary unless the job needs setup state.
| Boundary | State | Use it for |
|---|---|---|
| bzlabs run | One program, then gone | Pure checks and transforms |
| Sandbox | Survives multiple commands | Clone, install, build, test |
# No sandbox is created. The result is returned and the execution disappears.
bzlabs run --language python -c 'print(sum(range(101)))'
# Files are read on the CI runner and sent as code, not uploaded as a path.
bzlabs run --language python ./scripts/check_manifest.py02 · One CI job
Clone, install, test, delete.
The controller below is independent of any CI vendor. The runner supplies an API key and job parameters as environment variables. The sandbox receives outbound network access because it must clone and install; it receives no inbound access to the runner.
# ci_job.py
import os
from dataclasses import dataclass
from shlex import quote
from boltzlabs import Sandbox
REPOSITORY_URL = os.environ["REPOSITORY_URL"]
REVISION = os.environ.get("REVISION", "main")
TEST_COMMAND = os.environ.get("TEST_COMMAND", "python -m pytest -q")
@dataclass
class Outcome:
exit_code: int
stdout: str
stderr: str
def run_ci_job() -> Outcome:
# The context manager deletes the machine on success, failure, or exception.
with Sandbox(
environment="python",
machine="medium",
internet=True,
name="ci-job",
) as sb:
sb.exec(
"git clone --filter=blob:none "
f"--branch {quote(REVISION)} {quote(REPOSITORY_URL)} /workspace/repo"
).check()
sb.exec(
"cd /workspace/repo && python -m pip install -e ."
).check()
result = sb.exec(
f"cd /workspace/repo && {TEST_COMMAND}",
timeout=900,
)
return Outcome(result.exit_code, result.stdout, result.stderr)
outcome = run_ci_job()
print(outcome.stdout, end="")
print(outcome.stderr, end="", file=__import__("sys").stderr)
raise SystemExit(outcome.exit_code)export BOLTZLABS_API_KEY='ak_...'
export REPOSITORY_URL='https://github.com/pallets/click.git'
export REVISION='main'
export TEST_COMMAND='python -m pytest -q'
python ci_job.py03 · Fan out
Bound concurrency at the caller.
A list of ten thousand inputs is not a request for ten thousand simultaneous creates. Use a worker pool sized for the account and workload. This keeps scheduler pressure, spend, and failure volume visible. BoltzLabs currently permits at most 25 running sandboxes per account; the example stays well below that limit.
# batch.py
from concurrent.futures import ThreadPoolExecutor, as_completed
from boltzlabs import Sandbox
def work(number: int) -> tuple[int, int]:
with Sandbox(environment="python", machine="nano") as sb:
result = sb.run(
f"print(sum(i * i for i in range({number})))",
language="python",
timeout=60,
)
result.check()
return number, int(result.stdout.strip())
inputs = [10_000, 20_000, 30_000, 40_000, 50_000, 60_000]
# Bound creation pressure even when the input list is large.
with ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(work, n): n for n in inputs}
for future in as_completed(futures):
number, result = future.result()
print(number, result)Every task owns one context manager, so completion destroys its sandbox before another future is admitted. Ordering follows completion rather than input; retain the input key with each future when result order matters.
04 · Failure semantics
A failed command is data. A failed call is infrastructure.
A test process exiting 1 is a valid execution result with stdout, stderr, duration, and an exit code. Authentication failure, capacity rejection, or a broken transport raises an exception. Keep those paths separate so retries do not turn deterministic test failures into expensive infrastructure loops.
from boltzlabs import Sandbox
sb = Sandbox(environment="python")
try:
result = sb.exec("python -m pytest -q", timeout=900)
# A command failure is a result. A transport failure raises separately.
if result.exit_code != 0:
print(result.stdout)
print(result.stderr)
raise SystemExit(result.exit_code)
finally:
sb.delete()Command exit
Report stdout and stderr, fail the CI job, do not retry by default.
Timeout
The result exits 124 with timeout as its reason; decide whether the job is too slow or wedged.
API or capacity error
Retry with bounded backoff only when the error is transient.
Controller interruption
Context managers cannot run after a hard kill. Reconcile and delete abandoned named sandboxes externally.
05 · Current boundary
Keep orchestration outside the sandbox.
There is no custom-image, snapshot, file-upload, or artifact service in the sandbox API today. Clone or generate inputs, write outputs into the workspace, and read finite artifacts back through commands before deletion. Dependency installation therefore happens per fresh sandbox; use lockfiles and small environments to keep it deterministic.
Cleanup still needs reconciliation.
The API accepts idle-timeout and maximum-lifetime values but does not enforce them yet. A controller killed between create and delete can leave a sandbox running. Name jobs, list running sandboxes, and have a separate reconciler remove abandoned work.