Sandbox Lifecycle
A sandbox is the core runtime unit of Cube-Sandbox. This page covers a sandbox's full lifecycle — from creation to teardown — and how to let the platform manage it automatically to save resources.
The SDK shape mirrors e2b so existing e2b code can port with minimal changes.
State Model
A sandbox is always in exactly one of these states:
| State | Meaning |
|---|---|
running | Active. Real CPU/memory in use. Accepts requests and executes code. |
pausing | Platform is taking the VM snapshot (transient). |
paused | Snapshot persisted to disk. Zero CPU/memory cost. Full state preserved. |
resuming | Platform is restoring the snapshot (transient). |
terminated | Killed (kill()) or reaped after on_timeout="kill". Cannot be brought back. |
unknown | CubeAPI could not map the sandbox to one of the rows above: it is still being created, its container has exited, or its state could not be read from its node. |
unknown is not a synonym for "gone": CubeAPI reports it for a sandbox that is still being created, one whose container has exited, and one whose state could not be read from its node. The terminated row describes what kill() and on_timeout="kill" do; a container that exits on its own is reported as unknown, not as a separate terminal state. Because those cases share one value, GET /sandboxes/<id> alone cannot tell a sandbox that is still coming up from one that is already dead — POST /sandboxes/<id>/connect is the signal: it fails with HTTP 409 (... is not connectable in state 'stopped') once the container is gone, and still connects in the other two cases.
Two settings drive transitions:
timeout(optional): seconds of idle time before a timeout fires (e2b uses milliseconds viatimeoutMs; Cube uses seconds). When omitted, the server decides — the SDK no longer injects a hard-coded default.on_timeout: what happens at timeout —"kill"(default; destroy) or"pause"(snapshot for later resume).
timeout values (e2b-aligned):
| Value | Behavior |
|---|---|
| omitted | Server default idle TTL; if the server has no positive default, the sandbox never times out |
NEVER_TIMEOUT (-1) | Never time out — no idle reclamation |
0 | Immediate timeout — reclaimed on the first idle sweep |
positive integer N | Timeout after N seconds of idle time |
Go: cubesandbox.NeverTimeout; Python: from cubesandbox import NEVER_TIMEOUT.
┌──────────────────────────────────────┐
│ │
create() ┌────▼────┐ timeout & on_timeout=pause ┌─────────┐
───────────────►│ running │ ──────────────────────────────►│ paused │
│ │◄──────── connect() or │ │
└──┬────┬─┘ auto_resume-triggered req └──┬────┬─┘
│ │ │ │
│ │ timeout & on_timeout=kill │ │ timeout & on_timeout=kill
│ └────────────────┐ │ └──────────────┐
│ ▼ │ ▼
│ ┌────────────┐◄────────────┘
│ │ terminated │
│ └──────▲─────┘
│ │
│ kill() │ kill()
└──────────────────────┘Create
from cubesandbox import Sandbox
# Create a sandbox that auto-destroys after 60 seconds of idle.
# (Default on_timeout is "kill".)
sandbox = Sandbox.create(
template="<your-template-id>",
timeout=60, # seconds
)
print(sandbox.sandbox_id)Key parameters of Sandbox.create():
| Parameter | Description |
|---|---|
template | Template ID used to boot the sandbox; defaults to env var CUBE_TEMPLATE_ID. |
timeout | Optional idle timeout in seconds; see the table above |
lifecycle | Lifecycle policy — see Platform-managed auto-pause / auto-resume below. |
metadata | Arbitrary key/value pairs stored on the sandbox; readable from the list / detail endpoints. |
env_vars | Environment variables injected into the sandbox process. |
allow_internet_access | Whether outbound internet is allowed; network provides finer-grained egress control. |
Cube doesn't impose hard wall-clock ceilings (24h Pro / 1h Base) the way hosted e2b does. When you omit
timeout, the effective idle TTL is set by your cluster operator — see Operational Notes below.
Inspect a Running Sandbox
info = sandbox.get_info()
print(info)
# {
# "sandboxID": "iiny0783cype8gmoawzmx-ce30bc46",
# "templateID": "rki5dems9wqfm4r03t7g",
# "state": "running",
# "startedAt": "2026-06-17T12:34:56Z",
# "endAt": "2026-06-17T12:39:56Z",
# "metadata": {...}
# }endAt is the projected next-timeout instant given the current timeout. It is refreshed every time the sandbox receives a request (or when you call set_timeout, when available). Pausing does not cancel a finite idle deadline, so both detail and list APIs continue to return the same endAt while the sandbox is paused. For never-timeout sandboxes there is no deadline, so endAt is omitted from the response rather than reported as equal to startedAt.
List Running Sandboxes
for sb in Sandbox.list():
print(sb["sandboxID"], sb["state"])Explicit Shutdown
sandbox.kill()kill() is irreversible: unlike pause, a killed sandbox cannot be brought back, even when lifecycle.on_timeout="pause" was set — kill() always wins and discards the snapshot.
Deleting a Paused Sandbox
Both kill() and DELETE /sandboxes/{sandboxID} can delete sandboxes in the running or paused state.
When deleting a paused sandbox, CubeSandbox does not resume or wake the MicroVM. It removes the paused tombstone, deletes the pause snapshot (catalog / CoW), and clears control-plane pause metadata. Plugin-volume refcounts were already adjusted at Pause time, so delete does not re-attach volumes only to tear them down again.
The API remains synchronous: it returns 204 No Content only after cleanup finishes. This path is not a resume:
- It does not emit a
sandbox.resumedlifecycle event. - It does not reset the idle timeout.
- It does not require node capacity admission (there is no restore-before-delete).
When deleting a paused sandbox, the following responses are possible:
204 No Content: The sandbox, pause snapshot, and related resources have been cleaned up.503 Service Unavailable+Retry-After: 2: The sandbox is entering the paused state, or another lifecycle operation (pause / resume / delete) holds the sandbox lock. Wait at least two seconds before retrying.
Retry-After is in seconds and tells the client when to retry. It does not mean that CubeSandbox continues deletion or starts a background retry.
Existing 404 Not Found, 408 Request Timeout, and running sandbox delete behavior remain unchanged.
Explicit Pause / Resume
sandbox.pause() # snapshot manually, free CPU/memory
# ... time passes ...
sandbox = Sandbox.connect(sandbox.sandbox_id, timeout=300) # restore and reset the idle timeout
sandbox.run_code("print('back!')") # carry on as if never pausedpause() does not cancel idle reclamation. With the default on_timeout="kill", a later-paused sandbox is still destroyed once idle exceeds timeout. To keep a paused sandbox, pass timeout=NEVER_TIMEOUT, omit timeout (with no positive server default), or set a high timeout — see Behaviour below.
connect(timeout=...) can update the idle timeout whether the sandbox is already running or must first resume from a pause. For a running sandbox, a positive timeout only extends the deadline when the requested window is longer; use set_timeout(...) when you intentionally need to shorten it:
connect(timeout=...) | Effect |
|---|---|
omitted / None | keep the current timeout |
NEVER_TIMEOUT (-1) | never time out after connecting |
N > 0 | ensure at least N seconds remain; a running or paused sandbox keeps a longer existing deadline, otherwise the timeout starts a new N-second window after connecting |
0 or N < -1 | reject the request with HTTP 400 |
When connecting or resuming a paused sandbox while another lifecycle operation on it is still settling, CubeAPI returns HTTP 503 with Retry-After: 2 if the sandbox is not running yet; wait at least that long and retry. If a concurrent request has already resumed it, connect succeeds and returns the connection info, while POST /sandboxes/<id>/resume returns 409 (already running). A connect 409 can also mean the sandbox's container has already exited: that state is permanent, so retrying will not help — see State Model.
The deprecated resume(timeout=...) keeps its legacy 0 behavior:
resume(timeout=...) | Effect |
|---|---|
omitted / None | keep the current timeout |
0 | keep the current timeout (use set_timeout(0) for immediate expiry) |
NEVER_TIMEOUT (-1) | never time out after resume |
N > 0 | start a new N-second window from resume |
N < -1 | reject the request with HTTP 400 |
See examples/code-sandbox-quickstart/pause.py for a full demo. Cross-node Resume (jfs or s3, with remote_status=ready) is documented in Cross-Node Migration.
CubeProxy cache after Resume
Resume recreates the guest NIC / host ports and rewrites the Redis sandbox proxy map. CubeMaster then best-effort purges CubeProxy local_cache via POST /admin/backend_cache/delete so traffic does not keep routing to the pre-pause IP (same-node 504).
For that purge to succeed, CubeMaster and CubeProxy must share the same admin token:
- CubeMaster:
cubeproxy.admin_token(sent asX-Cube-Admin-Token) - CubeProxy:
$cube_admin_tokeninnginx.conf(seeCubeProxy/lua/admin_phase.lua)
If the token is set on only one side, or the values differ, cache purge returns 403, Redis is still correct, but CubeProxy may serve a stale cached backend until the entry expires. Align the token in deploy / Helm values whenever Resume is used.
Platform-managed Auto-pause / Auto-resume
Most agent workloads aren't continuously busy: the user types code → the model thinks → the sandbox executes → it sits idle until the next turn. Auto-pausing during the idle stretch and transparently resuming on the next request can dramatically cut resource cost.
Cube exposes the exact same lifecycle shape e2b uses:
sandbox = Sandbox.create(
template="<your-template-id>",
timeout=300, # 5 min of idle triggers on_timeout
lifecycle={
"on_timeout": "pause", # at timeout → pause (instead of kill)
"auto_resume": True, # next request after pause → resume
},
)Behaviour
on_timeout="pause": aftertimeoutseconds idle, the platform schedules a pause. State flips topaused, the VM memory is frozen to the snapshot store.auto_resume=True: when any request next arrives for apausedsandbox (HTTP,run_code, file I/O, …), the platform wakes it up before the request lands. Callers never see the pause; typical resume latency is sub-second to a few seconds.- If
auto_resume=False(or unset), the sandbox stays paused until you explicitlySandbox.connect(sandbox_id=...). Useful for "wait for the user" workflows.
Timeout reset on auto-resume
Each successful auto-resume resets the idle clock while keeping the same timeout length. The "resume → short use → idle out → pause again" loop can repeat indefinitely.
What counts as activity
Any of these resets the idle clock:
- SDK calls:
sandbox.run_code(...),sandbox.commands.run(...),sandbox.files.read(...)/write(...). - Direct HTTP traffic to a service inside the sandbox (e.g. via the URL returned by
getHost()).
A request counts as activity when it starts and again when its response ends; time spent in between does not. A single request that runs longer than timeout (for example commands.run("sleep 900") with timeout=300) can therefore be paused mid-flight. For such calls, send a cheap request to the sandbox periodically (more often than timeout) while the long one runs, or raise the timeout first with set_timeout(...).
Sandboxes that don't opt in (no lifecycle argument) default to on_timeout="kill": once they sit idle for the effective timeout the platform destroys them. This matches e2b's lifecycle.on_timeout="kill" semantic. Manual pause() does not cancel that kill: a later-paused sandbox is still destroyed when idle exceeds timeout. To keep a paused sandbox, pass timeout=NEVER_TIMEOUT, omit timeout (with no positive server default), set a high timeout, or send periodic activity to reset the idle clock.
End-to-end examples
The platform ships two mirror-image end-to-end demos, one per on_timeout value:
examples/code-sandbox-quickstart/auto-resume.py—on_timeout="pause"+auto_resume=True. Creates a sandbox, idles past the timeout to trigger auto-pause, then issues a fresh request to trigger auto-resume, and verifies that both kernel memory and the filesystem are byte-identical across the cycle.examples/code-sandbox-quickstart/auto-kill.py—on_timeout="kill"(the default). Creates a sandbox, idles past the timeout to trigger auto-kill, verifies that the next request fails fast with 410 Gone, that the sandbox no longer appears inSandbox.list(), and spawns a control sandbox to rule out cluster-wide failures.
export CUBE_TEMPLATE_ID=<your-template>
# Auto-pause + auto-resume
python examples/code-sandbox-quickstart/auto-resume.py
# Auto-kill (irreversible)
python examples/code-sandbox-quickstart/auto-kill.pyOperational Notes
Cluster default idle timeout (default_timeout_insec)
When the client omits timeout, CubeMaster applies cubelet_conf.default_timeout_insec in CubeMaster/conf.yaml (one-click installs: /usr/local/services/cubetoolbox/CubeMaster/conf.yaml).
| Config value | Effect when the client omits timeout |
|---|---|
unset or <= 0 | No cluster-wide idle TTL — sandboxes never time out from idle |
positive N | Default idle TTL of N seconds |
The repository ships with no cluster-wide idle timeout (default_timeout_insec: -1). Set a positive value (for example 300) if you want the cluster to reclaim sandboxes that never pass an explicit timeout. Restart cube-sandbox-cubemaster.service after edits.
create_timeout_insec in the same section is unrelated: it only bounds the create/scheduling RPC deadline, not sandbox idle TTL. See Service management — CubeMaster settings.
- Pause fidelity: CPU registers, process memory, TCP state (with no external peer), and filesystem mutations all survive the snapshot. Outbound sockets the sandbox itself opened are dropped on pause and must be reopened by the application after resume.
- Cluster coordination: auto-pause is driven by
cube-lifecycle-manager. The Helm chart and Terraform one-click both default to two warm replicas. Both replicas consume lifecycle events, discover CubeProxy replicas, and serve resume callbacks, while a Redis lease elects one replica for idle sweep/kill and stale-registry pruning. After a leader failover a sandbox may pause or resume once more; the next request auto-resumes it as usual (see the Kubernetes FAQ). Per-sandbox Redis state transitions and CubeMaster lifecycle locks serialize effective pause/resume work across replicas. - Failure mode: when an auto-resume RPC fails, CubeProxy returns
503 + Retry-Afterto the client immediately rather than hanging on a long timeout. When the sandbox has already been killed (killing/killed) the proxy returns410 Goneinstead, telling SDK clients to stop retrying. - Diagnostics:
docker logs cube-lifecycle-manager(control node) is the runtime log for the auto-pause coordinator. Look forcreate event applied,auto-paused sandbox,auto-resumed sandbox,timeout-killed sandbox. Each CubeProxy replica additionally exposesGET http://<node-ip>:8082/admin/healthzreportingheartbeat_last_pushed_ms(the last time it announced itself to the manager). The admin port defaults to8082; override it withCUBE_PROXY_ADMIN_PORTwhen that port is already in use on the host (CubeProxy uses host networking).
Paused Resource Release & Scheduling Quota
When a sandbox is paused, its CPU and memory are physically reclaimed — but by default, the node resource accounting still counts paused/pausing sandboxes as "occupied" against the scheduler quota. This means: even after many idle sandboxes are paused, the host still shows no available capacity to create new ones.
To address this, Cube provides a node-level tuning knob host.quota.paused_resource_release_ratio (configured in Cubelet/config/config.toml), range [0, 1], default 0:
| Value | Behavior | Best For |
|---|---|---|
0.0 | Paused sandboxes retain full quota (identical to legacy behavior). Resume is always guaranteed — never rejected due to resource shortage. | Availability-critical environments where resume must never fail |
1.0 | Paused sandbox CPU/memory quota is fully released to the scheduler. Resume becomes best-effort — may be rejected when the node is full. | Maximizing deployment density; occasional resume failures are acceptable |
0 < r < 1 | Releases fraction r, reserves (1-r) as headroom. Reserved quota still counts toward scheduler CPU/memory usage, so pause-heavy nodes are naturally deprioritized — the scheduler won't keep piling new sandboxes onto nodes that already hold many paused ones. | Balancing availability against utilization |
Configuration example:
# Cubelet/config/config.toml
[host.quota]
paused_resource_release_ratio = 0.5 # release half, reserve halfResume admission check:
When ratio > 0, every resume triggers a local real-time admission check — if the node lacks enough free capacity to accommodate the released fraction, the resume is rejected:
resume rejected by paused_resource_release_ratio policy: need 1024MB > quota 512MBThe rejection travels through the following chain to reach the client: Cubelet (130409 Conflict) → CubeAPI (HTTP 409) → WebUI (capacity diagnostic). HTTP 409 is a retriable status — when other sandboxes are destroyed or paused later, freeing capacity, the resume can be retried.
Important notes:
- Disk and MvmNum are not affected by the ratio — pause snapshots still consume storage and the sandbox object still exists.
ratio=0is the zero-value-safe default: if this setting is never configured, behavior is identical to previous versions. Upgrades won't cause surprises.- This is a node-level setting — different nodes can use different ratios to accommodate heterogeneous hardware or tiered pools.
- When a large batch of sandboxes on a single node wakes up simultaneously and exceeds node capacity, the control plane returns 409 with precise quota numbers. For sandboxes on a shared backend, the scheduler can fall back to another compatible node — see Cross-Node Migration.
Restart after exit
A create request can set restartPolicy. The meaning matches Kubernetes restartPolicy: a restart is triggered by the exit of the sandbox's main process (the init process of the main container) or by a failed liveness probe, and the unit of restart is the whole sandbox (one microVM), not a process inside it.
| Value | Behavior |
|---|---|
Never (default) | Do not restart after exit |
OnFailure | Restart after a non-zero exit, OOM, sandbox crash, or a failed liveness probe |
Always | Restart after every exit, including a clean one, and after a failed liveness probe |
Backoff and restart limit
A restart keeps the disk, the network, the volume and host mounts, and the sandbox ID, then cold-starts. Memory is not kept.
Repeated failures wait 10 seconds, then double, capped at 300 seconds; after 10 minutes of stability the restart count resets. The restart limit is restartBackoff.maxRestarts (default 0 means unlimited); a global cap can be set with cubelet_conf.max_restarts_cap (once set, even 0 is tightened to that cap). After the limit the sandbox stops restarting and is not brought up again, but it still counts toward the node's MvmNum quota until idle timeout or an explicit destroy.
Liveness probe
Put a liveness probe on the create request. When the request has none and the image has envd (port 49983 exposed), the node probes GET /health. Images without envd are not probed, so they are not restarted by mistake. Never combined with a probe is rejected with HTTP 400: a probe that can never restart would only kill the sandbox.
The following create-request excerpt checks GET /health on port 8080. After three consecutive failures, CubeSandbox cold-restarts the whole sandbox under the OnFailure policy:
{
"restartPolicy": "OnFailure",
"restartBackoff": {
"initialIntervalSecond": 10,
"multiplier": 2,
"maxIntervalSecond": 300,
"maxRestarts": 5
},
"livenessProbe": {
"httpGet": {
"path": "/health",
"port": 8080
},
"initialDelaySecond": 30,
"periodSecond": 10,
"probeTimeoutSecond": 1,
"failureThreshold": 3
}
}For a TCP check, replace httpGet with tcpSocket; exactly one check type may be set:
{
"restartPolicy": "Always",
"livenessProbe": {
"tcpSocket": {
"port": 8080
},
"periodSecond": 10,
"failureThreshold": 3
}
}Probe durations are in seconds. Omitted or non-positive timing values use these defaults: 30 seconds initial delay, 10 seconds between checks, 1 second timeout, and 3 consecutive failures.
Restart status
GET /sandboxes/<id> and the list endpoint return restartStatus, a separate object from state: state answers "where is the sandbox in its lifecycle", while restartStatus answers "where is the restart state machine, and why did it stop". Once the container exits on its own, state only reports unknown; this object is what distinguishes a clean exit, a crash, and a give-up.
It does not change state, and it only appears when the policy is OnFailure / Always or the sandbox has exited; a default Never sandbox that has never exited has no such object.
restartState takes these values (capitalized; this is not the same enum as the lowercase state). In the table, "main process" means the init process of the main container:
| Value | Meaning |
|---|---|
Running | The restart state machine is idle: creation succeeded or the last restart succeeded. It does not mean the sandbox is running; check state for that |
Restarting | Rebuilding: destroy the old VM, then cold-start a new one |
BackOff | Exited and waiting for the backoff to elapse; nextRestartAt is the next attempt |
GaveUp | Reached maxRestarts, or a rebuild/destroy hit an unrecoverable failure (including a retained rootfs that e2fsck cannot bring back to clean); no more retries |
Succeeded | The main process exited cleanly (code 0) and the policy declines to restart (Never, or OnFailure with code 0) |
Failed | The main process exited abnormally and the policy declines to restart (any failure under Never) |
Restarting and BackOff are transient states in the restart flow; GaveUp, Succeeded, and Failed are terminal: the sandbox will not be brought up automatically again. Always never stops at Succeeded / Failed: it keeps restarting; if maxRestarts is finite, it enters GaveUp once the limit is reached.
Auto-pause is skipped during these two transient states; idle timeout still applies, and a restart operation does not reset the idle timer (idle time is based on client activity). The object also carries restartCount, lastExitCode, lastExitReason, lastRestartAt, and similar fields for diagnosing exits.
A pause is not a restart: pause() followed by resume keeps restartCount and the exit diagnostics, and the backoff schedule continues from that count. Only the stable-period reset clears it, so a sandbox that already reached maxRestarts stops restarting after the next exit even if it was paused and resumed in between.
Next Steps
- Templates Overview — sandboxes boot from templates; the template's build also shapes cold-start cost.
- Quick Start — the shortest path through "create sandbox → run code → tear down".
- Upstream references: e2b · Sandbox lifecycle, e2b · Auto-resume.