givemeanode docs

givemeanode gives AI agents GPU nodes and batch GPU jobs on demand, over MCP. Your agent gets a container on an H100 with a persistent encrypted disk. You pay per minute. When you stop a node, billing stops and the files stay on the disk. The next command wakes the node.

Quick start#

Add the MCP server to your agent. This example uses Claude Code. Other clients are below.

claude mcp add --transport http givemeanode https://mcp.givemeanode.com

Your client opens a browser window one time to sign you in. Then ask your agent for a GPU:

Get me a GPU node, clone github.com/you/yourrepo on it, and run the benchmark.

Every example in these docs shows one action in three ways. The English tab is what you say to your agent. The CLI tab is the gman command. The MCP tab is the tool call your agent sends. The quick start again, in all three ways:

gman node create --chip h100 --wait   # polls to running; prints the locked rate
gman run tulip -- nvidia-smi
gman node stop tulip                  # parked: files intact, billing stopped

create_node(chip: "h100")       → {name: "tulip", status: "provisioning", rate: "$/min"}
run_command("tulip", "nvidia-smi")  → {exit_code: 0, stdout: "..."}
stop_node("tulip")                  → parked: files intact, billing stopped

Billing starts when the node is ready. Billing stops when you stop the node. The rate is in the create_node response. The next run_command on a stopped node wakes it. The disk is exactly as you left it.

Install#

Every client uses the same server. It is streamable HTTP with OAuth 2.1 at this address:

https://mcp.givemeanode.com

Your client finds the sign-in flow by itself. A person approves it one time in the browser. There are no API keys.

On a remote or cloud machine, the browser approval does not work. Start at Remote machines instead.

Claude Code#

claude mcp add --transport http givemeanode https://mcp.givemeanode.com

Approve the connection when Claude first uses it. /mcp shows the status.

On a machine that is rebuilt for every task, a code approval on every rebuild is not practical. Make a service token and pass it as a header. gman mcp install writes this form for you when GMN_TOKEN is set.

claude mcp add --transport http givemeanode https://mcp.givemeanode.com \
  --header "Authorization: Bearer $GMN_TOKEN"

The token is its own identity. Your cloud agent stays inside the workspaces you give the token. When you revoke the token, your own login is not affected.

Claude web#

Go to Settings, Connectors, Add custom connector. Enter the remote MCP server URL:

https://mcp.givemeanode.com

Codex#

codex mcp add givemeanode --url https://mcp.givemeanode.com

Or add this to ~/.codex/config.toml:

[mcp_servers.givemeanode]
url = "https://mcp.givemeanode.com"

If Codex does not ask you to sign in on first use, run codex mcp login givemeanode. Give Codex the host only, with no path after it. Codex reports a 404 from any other path.

If Codex says the client timed out after 30 seconds and names startup_timeout_sec, Codex stopped waiting before the connection was complete. Thirty seconds is its default. A first connection can need more. Raise the budget and start Codex again.

# ~/.codex/config.toml
[mcp_servers.givemeanode]
url = "https://mcp.givemeanode.com"
startup_timeout_sec = 60

If it still times out, sign out and sign in again. Run codex mcp logout givemeanode, then codex mcp login givemeanode. This replaces the saved login. If the timeout returns, use a service token instead. A token does not use a saved login.

On a machine with no browser, gman mcp install configures Codex for you through codex mcp add. If codex is not on your PATH, it prints this block with the correct path for your machine:

# ~/.codex/config.toml
[mcp_servers.givemeanode]
command = "/home/you/.local/bin/gman"
args = ["mcp", "serve"]

If the browser approval cannot complete, for example in CI, skip OAuth. Make a service token and give it to Codex as a bearer token from the environment:

# ~/.codex/config.toml
[mcp_servers.givemeanode]
url = "https://mcp.givemeanode.com"
bearer_token_env_var = "GMN_TOKEN"

Set GMN_TOKEN to the gmnt_ value in the environment where Codex runs. No browser opens. When you revoke the token, your own login is not affected.

opencode#

Add this to opencode.json (project or global). On a remote machine, gman mcp install writes the local-command form for you instead.

{
  "mcp": {
    "givemeanode": {
      "type": "remote",
      "url": "https://mcp.givemeanode.com",
      "enabled": true
    }
  }
}

Remote machines#

On a dev box, a Codespace, or any machine you reach over SSH, the browser approval redirects to localhost. That is a different machine from the one your agent runs on, so the approval never arrives. Install the CLI on that machine and run one command:

curl -fsSL https://givemeanode.com/cli.sh | bash
gman mcp install

You sign in one time with a code that you approve on any other device. The command finds every coding agent on the machine and points each one at gman mcp serve. That is a local command that reaches givemeanode for the agent. No browser opens on either machine.

Claude Code, Codex, Cursor, opencode, Amp, oh my pi, and Zed are supported. If the install cannot write a config file safely, it prints the block for you to paste. Restart each agent after the install, or run /mcp in it.

• --client cursor configures one agent instead of all of them.
• --print shows the config without writing anything.
• The install replaces a givemeanode entry that an earlier attempt left behind.

CLI#

gman is the same product from a terminal, for people and scripts. Every CLI tab in these docs is a gman command.

curl -fsSL https://givemeanode.com/cli.sh | bash
gman login
gman node create --name tulip --chip h100 --wait
gman run tulip -- nvidia-smi
gman node stop tulip

A command reads gman <noun> <verb> [--flags] [arguments]. -- ends the flags. Everything after it is the remote command. Most nouns take an ls verb that lists and a get verb that reads one thing.

• gman <command> --help prints that command's syntax. gman reference prints every command at once.
• gman login opens the browser sign-in. With no browser, it prints a code to approve on another device.
• gman mcp install connects the agents on this machine to givemeanode through that login.
• gman docs prints this manual.
• gman api <method> <path> calls any API endpoint directly, including endpoints with no subcommand yet.

Nodes#

A node is a container on an H100 with a persistent disk. Every node type has a name. Pass that name as chip. h100-1 is one GPU. h100-8 is a full 8-GPU machine. The original spellings h100 and 8xh100 also work. The examples use the original spellings, and the API returns them.

Create#

gman node create --name tulip --chip h100

create_node(name: "tulip", chip: "h100")

create_node never blocks. It returns one of two statuses:

• provisioning: capacity was free. The response carries the locked rate. rate.node_per_min is the cost of the whole node per minute. rate.per_min is the per-GPU rate. Poll get_node("tulip") until the status is running.
• queued: the type is full. The response carries a live queue position and a wait estimate. Your node is provisioned when a slot frees. Poll get_node. Do not create the node again. See Queue.

list_nodes() (CLI: gman node ls) shows every node you have. get_node(name) (CLI: gman node get) shows one node with its state, rate, and queue position.

Two create-time flags are for benchmark work. clock_lock: true pins the GPU clock, so timings compare across runs. profiling: true unlocks the GPU profiler counters that ncu needs. Both are free. Both cannot change after create. See Profiling.

Stop and wake#

gman node stop tulip   # stop paying now; disk parks, files intact
gman run tulip -- ls   # wakes a stopped node automatically
gman node rm tulip     # destroy the disk (crypto-erase, irreversible);
                       # stopped nodes only, and confirms unless -y

stop_node("tulip")     # stop paying now; disk parks, files intact
run_command("tulip", "ls")   # wakes a stopped node automatically
delete_node("tulip")   # destroy the disk (crypto-erase, irreversible);
                       # stopped nodes only, so stop_node first

• stop_node stops billing now. The disk parks with the files intact. Processes do not survive a stop.
• The node saves the disk's last writes to durable storage before the disk parks. After a large write this takes some minutes. The status reads stopping until then, and stopped after. You do not pay for the stopping time.
• Any run_command on a stopped node wakes it. The disk is exactly as you left it. A run_command on a stopping node wakes it too, without waiting.
• delete_node destroys the disk. This cannot be undone. The node must be stopped first.

get_node reports what the host sees your node doing, under utilization. busy is our verdict. While it is true, we do not idle-stop the node, and the deadline in idle_stops_at moves forward.

The numbers behind the verdict are in the same section. On a GPU node, gpu_pct_max is the busiest GPU over the last ten minutes, and gpu_pct_latest is the newest minute. On any node, cpu_pct is percent of one core, so 300 means three cores are busy. Samples are taken once a minute. sampled_minutes says how many minutes of the window carry a GPU sample. If it is lower than window_minutes, the rest of the window is unknown, not idle. A missing field was never sampled. Treat a missing field or a missing section as no measurement, not as zero. The full series per GPU is the gmn_gpu_utilization metric.

gpu_memory says whether another workload will fit. It lists every GPU on the node with used_mib and free_mib, from the newest heartbeat (the as_of field). free_mib_min is the tightest card. Size a second workload against free_mib_min, not against a node-wide total. Two trainers on one node run out of memory on one device. The UUIDs match nvidia-smi -L inside the node. A card with no numbers was not sampled, which means unknown, not zero. The whole block is absent when the newest reading is too old to publish, so a missing block never means free memory. The full series per GPU is the gmn_gpu_memory_used_bytes metric.

Queue#

A wait has two clocks. Know both, and polling is cheap.

• The queue hold. A queued node keeps its place until expires_at, whether or not anyone polls. Pass max_wait on a re-call of create_node to extend the hold, forward only, up to 12 hours. Polling never extends the hold. max_wait is also a parameter on run_command, so a command that wakes a stopped node can claim its hold in the same call.
• The idle-grace window. When your node provisions, its grace clock starts. The window is 15 minutes on 1x shapes and 20 minutes on 8x, quoted on every response as rate.minimum_minutes. A ready node with no session reports running (idle) and idle_stops_at. If no run_command arrives before then, the node stops with the files intact. The window bills while it runs. There is no minimum charge. Stop the node, and billing ends at the stop. See rates.

A node that is doing work is not idle. If the node uses its GPUs or its CPU, the window moves forward for as long as that lasts. A trainer you started with nohup or in tmux keeps running after the command that launched it returns. busy in get_node is the verdict we act on. A process that is alive but does nothing reads as idle. Use hold_node to keep a node like that awake.

If a node stops, get_node says what happened. The stopped section carries reason (a token such as idle_grace_expired or wake_hold_lapsed), at, a note with the next move, and ran_seconds, the billed length of that session. Next to it, wake_estimate quotes what a wake requested now would face: queued_wakes and the measured ready estimates. The same section is on every stop_node response. It reads the queue as it stands and reserves nothing.

Re-calling create_node with the same name is idempotent. You get the current status, never a duplicate. If a hold lapses, create the same shape under the same name within 15 minutes and you keep the place you waited for.

How the estimate is made. Your node can arrive in two ways. The queue drains as other people finish, or we add a machine for you. On a whole-node shape the second is often much faster, because eight free GPUs on one box rarely appear by chance. You are quoted the sooner of the two. capacity_inbound is true when the wait is bounded by capacity we are adding. A queued h100-8 with capacity_inbound: true is waiting on a machine. Hold your place. When the field is false, we are not adding capacity for that shape now, and a smaller shape can be the better plan. The field can turn true on a later poll.

Queue depth starts more machines. A queued node's GPUs count toward the fleet's machine target on the next procurement pass. A job backlog that persists for a shape raises the same target. Queued time is free and your place is held. Cancel a queued job or delete a queued node only when you no longer want the result. Every queued response says this in demand_note. This is a direction, not a promise. Spend ceilings and a quiet market can each mean no machine is coming right now, which is what capacity_inbound tells you.

Blocked on a resource. Sometimes the queue is not what holds your node. If the scheduler refuses the node on a resource, the queued response carries blocked_on. It names the resource, how much this node needs, how far short the closest machine is, and the move that changes it. Waiting does not clear this. A node that asks for a large scratch disk on a busy fleet is the common case. Ask for less scratch and it places at once.

gman node get trainer

get_node("trainer")
→ {status: "queued", position: 1,
   estimated_ready_seconds: null, unknown_because: "disk",
   blocked_on: {resource: "disk", needs_bytes: 335007449088,
                note: "gated on disk: this node needs 312 GiB free on one
                       machine, and the closest one is more than half short.
                       Ask for less scratch, or drop a snapshot you no longer
                       need; the requirement counts your scratch and a
                       restore's working copies together."}}

The estimate is null in that case, and unknown_because names the resource. One blocked_on is different. capacity means no machine is eligible right now for reasons on our side. Your estimate stays, your place is held, and there is nothing to change. The remedy sentence only names moves your own request allows.

Size the hold to the wait. Every queued response carries estimated_ready_seconds next to expires_at. When your hold is shorter than the estimate, the response says so in hold_note. Extend the hold, because a hold that lapses first means the node never arrives. A wake started by run_command holds for one hour by default, which is short for a busy shape.

When the estimate is longer than the 12-hour ceiling, renew the hold. Re-call create_node with the same name before each expires_at. Each renewal extends the live hold by up to 12 more hours. The deadline is in every response, so you can schedule the renewals. If a wake's hold lapses, the node returns to stopped with the files intact, and the next run_command queues it again. Work that can run unattended fits batch jobs better. The job queue holds a place for 3 days.

Poll every 2 to 5 minutes. That is inside the grace window, so you cannot lose the node between polls.

Poll from a shell instead. The node and job operations are also plain HTTP calls on the /preview API. A shell script, a CI step, or a cron job can hold the poll loop with a service token while your agent does other work:

# the prepackaged wait: blocks until ready, exits 0
gman node create --name tulip --chip 8xh100 --max-wait 6h --wait

# or the raw loop, no CLI needed
while true; do
  status=$(curl -s -H "Authorization: Bearer $GMN_TOKEN" \
    https://givemeanode.com/preview/nodes/tulip | jq -r .status)
  case "$status" in running*) break;; esac
  sleep 120
done

MCP and HTTP have separate rate limits, so a script's loop never spends your agent's tool budget. When the node is ready, one run_command from either interface attaches a session and ends the idle countdown.

If your MCP connection drops, every givemeanode tool disappears until the session reconnects. Nodes, queue holds, grace clocks, and detached commands continue on the platform. When the tools return, read before you act. get_node shows where the node is. list_commands shows whether a detached command that never answered did start. A synchronous run_command that never answered leaves no record, and its process can still be running. Check its effects before you run it again. Do not create the node again, because that only buys a second queue wait.

Shapes#

The GPU shapes run on the same hardware: 8-GPU NVIDIA H100 SXM machines with 112 vCPUs and 2 TiB of RAM. Your node is a container on one machine, sized to the shape. One GPU buys 1/8 of the machine. Eight GPUs buy all of it.

The CPU shapes are the same kind of node in every other way. The disk is persistent and encrypted. Stop parks it. The next command wakes it. Their reserved cores are yours alone, and the hyperthread siblings stay idle. To hand the output of a CPU node to a GPU node, take a snapshot of the CPU node and start the GPU node from it. You can also use org storage. Call export_data on one node and import_data on the other. There is no egress fee on either call.

A CPU node can be far from your GPU nodes, in a different region. Move data with storage and contexts. Do not assume a LAN. A snapshot restores onto a CPU shape or a GPU shape (see Images). A private network still reaches a CPU node. clock_lock and profiling are GPU features and refuse on the CPU shapes.

• RAM is never oversubscribed. Your allowance is yours alone. An OOM kill is always your own workload. Reduce the batch size or take the 8x shape.
• /dev/shm scales with your RAM instead of Docker's fixed 64 MiB. PyTorch dataloader workers and NCCL work without flags.
• Disk: a 250 GiB persistent volume on the machine's local NVMe. Support can raise this limit. Files survive stop and wake. Processes do not.
• Optional ephemeral disk at /scratch on the same NVMe. Ask for it at create time. It is free and it is destroyed at every stop. See Scratch.
• Images: your node runs one image from the catalog, or your own container image. Pass image at create time. Like every create-time property, it cannot change for the life of the node. See Images.
• Root when you need it: commands run as the dev user with passwordless sudo. Add distro packages with sudo apt-get update && sudo apt-get install -y <package>. Images ship with empty package lists, so the update is required. Commands run with no stdin, so the -y is required. Plain apt-get install fails with a dpkg lock error. Packages install outside /home/dev, so a stop resets them. Run the install again after a wake. Keep anything that must survive under /home/dev.

Per-minute rates are on the front page and in every create_node and get_node response. Read ceilings such as node counts and volume size in limits before you hit them.

Resources#

While the node is placed, get_node reports resources: the container's CPU and memory budget. Inside the container the same numbers are $GMN_CPU_LIMIT and $GMN_MEMORY_LIMIT_BYTES.

• Thread pools are preset to your budget. We set OMP_NUM_THREADS, RAYON_NUM_THREADS, MAX_JOBS, and PYTHON_CPU_COUNT to the CPU budget. OpenMP, Rust, and torch extension builds size to your slice without flags. The last three are clamped to the cores your container can run on, which on a shared machine can be fewer than $GMN_CPU_LIMIT. On the catalog images nproc answers the budget too, because GNU nproc reads OMP_NUM_THREADS. BusyBox and Alpine ship a different nproc that reports the host. A distroless image has none.
• The presets override your image. You override them per command. They are set on the container, which outranks an ENV line in your Dockerfile. To use a different number, pass it in run_command's env. Your image build runs before any of this exists. Add ENV MAX_JOBS=8 to your Dockerfile if you build flash-attn or vLLM from source.
• The CPU count your container sees is the whole machine. A GPU node's CPU budget is a scheduler quota, and a quota is invisible from inside the container. os.cpu_count(), /proc/cpuinfo, psutil, and free all report the host. On a 1x node that is commonly eight times your budget. A thread pool sized from them is throttled. Size pools from $GMN_CPU_LIMIT. For a DataLoader, write num_workers=int(os.environ["GMN_CPU_LIMIT"]) instead of num_workers=os.cpu_count(). PYTHON_CPU_COUNT makes os.cpu_count() answer the budget, but only on Python 3.13 and newer. The default image is on 3.12. node_doctor reports the gap with your node's numbers.
• Processes and threads have a ceiling. The container reports it at /sys/fs/cgroup/pids.max. The default is 4096. Past it, every new thread fails with “Resource temporarily unavailable”. The ceiling is the node_pids_max limit. list_limits shows it, and support can raise it. A raise applies the next time the node starts or wakes.
• The toolkit and the driver are visible. get_node reports environment with image_nvcc, the CUDA toolkit the image compiles with. Once the node is placed it also reports driver_version, cuda_major_max (the highest CUDA major the host driver runs), and compute_capability (the value for TORCH_CUDA_ARCH_LIST). Inside the container the same facts are $GMN_DRIVER_VERSION and $GMN_GPU_ARCH. image_torch is the torch the image ships. When it is null, the image ships no torch, and pip install torch picks one from the package index. Check its CUDA major yourself. A CUDA extension builds only when torch and nvcc agree on the CUDA major. Doctor checks this for you.

Scratch#

Pass scratch_gib at create time to get a second filesystem at /scratch on the same local NVMe. Use it for caches and other data you can download again.

gman node create --name tulip --chip h100 --scratch-gib 200

create_node(name: "tulip", chip: "h100", scratch_gib: 200)

• Scratch is destroyed at every stop. stop_node, an idle stop, a crash, an OOM kill, and a lost machine all count. The next wake mounts /scratch empty, and a stop note says so. Do not put the only copy of anything there.
• Scratch is never in a snapshot. A snapshot is the kept disk.
• Scratch is free. You pay node time only. The space is allocated before your container starts, so it cannot be oversubscribed under a running job.
• Scratch cannot change after create. The default is 0. The ceiling is the node_scratch_gib limit: 250 GiB by default. Support can raise it up to the 550 GiB platform hard cap, reported as hard_max by list_limits.
• Large requests narrow placement. Your node lands only on a machine with that much disk free now. A big scratch_gib can queue longer, and can hold up a wake. If you carry a lot of scratch, prefer delete_node and a fresh create over parking the node for days.

Nothing in the environment points at scratch. Hardcode /scratch, or set the cache variable your library reads (HF_HOME, TRITON_CACHE_DIR, a StreamingDataset local=).

Streaming datasets. If a training run streams shards from object storage and decompresses them locally, re-streaming every epoch pays the egress and multiplies disk writes by the epoch count. Pick the first option that fits:

1. Keep the decompressed working set on /scratch. Fill it once. Every later epoch is local NVMe reads and no network.
2. Keep the compressed shards on /scratch with a bounded decompressed cache. MosaicML StreamingDataset does this with keep_zip=True and a cache_limit, both pointed at /scratch. You pay decompress CPU instead of egress.
3. Re-stream every epoch, when the working set fits neither way.

Network#

• Full internet egress. Pull datasets, clone repos, pip install, and download from Hugging Face from inside the node. Two exceptions are refused at connect: outbound port 25 (SMTP), and a short list of crypto-mining pool ports.
• PyPI is cached on-cluster. The platform sets PIP_INDEX_URL and UV_DEFAULT_INDEX in your node and job environment. They point at a pull-through mirror of the public Python Package Index, so installs are faster. If you use your own index, set PIP_INDEX_URL yourself, in your environment or on the pip command line. Your value wins. To turn the preset off for your workspace, tell us.
• No inbound by default. Nodes have no public address and no open ports. There is no SSH. The one way in is an exposed port: HTTPS at our edge, per port, with the URL or a bearer token as the credential.
• Tenant isolation. A node reaches the internet and the metadata service, never a neighbor. Containers on one machine cannot see each other. The one exception is a private network you attach two of your own nodes to. GPU-to-GPU, the NVLink fabric is visible only inside an 8xh100 node.
• Bandwidth is shared and not guaranteed. Shapes share their machine's uplink. Under contention the platform fair-shares the download direction per container. An 8xh100 has the machine to itself. We publish no committed Gbps figure. If your workload needs one, talk to us.
• One machine, not a cluster. There is no cross-node RDMA or InfiniBand between two nodes you hold. Private networks give ordinary IP between your nodes, for sharding, parameter servers, Ray, and anything that speaks TCP. Embarrassingly parallel work fits batch jobs over many machines.

Doctor#

Run node_doctor on a running node before a long build. It checks the environment from inside the container.

gman node doctor tulip   # exits 1 if it found anything

node_doctor(node: "tulip")

It checks four things:

• Whether torch's CUDA major matches the image's nvcc. This decides whether a CUDA extension such as flash-attn, DeepSpeed, apex, or xformers can compile.
• Whether your container's CPU budget is far below the host numbers that free, /proc/cpuinfo, and psutil report.
• Whether a virtualenv on the disk no longer loads under the current image. This usually means a snapshot was restored onto a different image.
• Whether the GPU computes, with a one-line matmul.

The answer carries findings, each with a problem and a fix. A check that does not apply is listed under skipped with the reason. The tool installs, repairs, and changes nothing. It does not wake a stopped node. On a running node it counts as activity and restarts the idle window.

Replication#

While a node runs, we copy its changed disk blocks to encrypted storage outside the machine. There is nothing to configure. The one thing you see is a timestamp on get_node.

get_node("trainer")
→ { status: "running",
    durable_as_of: "2026-09-14T02:11:48Z" }   # 14 seconds ago

durable_as_of is the newest instant of the disk that is already replicated off the machine. It advances only when the copy is written and verified. In steady state it trails the live disk by under a minute. A clean stop ships a final copy, so a parked node's replicated disk matches the disk exactly.

If the machine under your node fails, the node stops instead of being lost. Your next run_command brings it back on a healthy machine with the disk restored. At most, the writes after durable_as_of need redoing. Blocks are encrypted before they leave the machine. Every restored block is verified before your node reads it. delete_node destroys the replicated copy in the same instant it destroys the disk.

Replication protects against machine loss while a node exists. To keep a disk you can clone, or keep after the node is gone, take a snapshot.

Snapshots#

A snapshot is a copy of a node's disk that you keep. Take one after you have installed and downloaded everything. Every node you create from it starts with that work done. Stop the node first, because the snapshot copies the parked disk. Wait until get_node reads stopped. A node that reads stopping is still saving its last writes, and snapshot_node refuses it until the disk is parked.

gman node stop tulip
gman node snapshot tulip
gman node snapshots            # poll until STATUS is ready

stop_node("tulip")
snapshot_node("tulip")         # → { snapshot_id: "snap-...", status: "uploading" }
list_snapshots()               # poll until status is ready

The call returns when the upload starts. Poll list_snapshots until the snapshot reads ready. Leave the node stopped until then. If you start the node, the upload is cancelled, the snapshot reads failed, and you take a new one after the next stop. Your /scratch disk is never in a snapshot. Work you do on the node after the snapshot is not in it. The upload runs through eight parallel connections. In production it moved snapshots of 32 to 35 GB at 107 to 110 MB/s to central storage. To in-region storage it moved a 35 GB snapshot at 157 MB/s (September 2026).

Some snapshots show a region. That snapshot is stored in the region where your node ran, and a node you create from it runs in that region. A snapshot without a region restores anywhere.

To create a node from a snapshot, name the snapshot at create time. The new node has its own name and its own disk. The snapshot is unchanged, so you can clone the same environment many times.

gman node create --name tulip-2 --chip h100 \
  --from-snapshot snap-8k2mp --wait

create_node(name: "tulip-2", chip: "h100", from_snapshot: "snap-8k2mp")

The new node downloads the snapshot before it starts. A snapshot of 128 MiB or more downloads through eight parallel connections. A smaller one downloads through one connection. While the download runs, get_node reports the bytes done and the connections in use in restore_progress, and the measured rate in provisioning.download_bytes_per_second. Plan your wait from that rate. In production, eight connections moved a 35 GB snapshot at 118 MB/s from central storage. From in-region storage the same eight connections moved it at 390 MB/s. One connection moved 17 to 30 MB/s (September 2026).

Snapshots bill for the storage they use. They never expire on our timer. You can set your own expiry, when you take the snapshot or later. At that instant the snapshot deletes itself. Deleting a snapshot frees the storage and stops the charge. Nodes already restored from it keep running.

gman node snapshot-expiry snap-8k2mp 7d
gman node rm-snapshot snap-8k2mp

set_snapshot_expiry(snapshot_id: "snap-8k2mp", expires_after: "7d")
delete_snapshot(snapshot_id: "snap-8k2mp")

Deleting a snapshot cannot be undone. We destroy the key, so the stored copy becomes unreadable everywhere at once. A later attempt to create a node from it is refused.

Retirement#

The machines behind nodes retire on a schedule. A node on one has a deadline. get_node carries it as soon as the retirement is scheduled, normally hours ahead.

get_node("tulip")
→ { status: "running",
    machine_retires_at: "2026-08-12T00:00:00Z",
    machine_retirement_note: "this node's machine retires at 12:00am UTC (in 6h 12m).
      tulip is stopped at that moment and its disk does not survive the machine.
      Writing your results out (export_file, or a write connection) keeps them,
      and stop_node then snapshot_node keeps the whole disk. After the machine
      goes, tulip is lost with it: delete_node then create_node starts a fresh
      one under the same name." }

Poll for this field on a long unattended run. A stopped node carries it too, because a parked disk goes away with its machine. An attached session gets countdown notices in the terminal. Code inside the container can read the deadline from the metadata service.

The note says which of two things happens to your files. Some hosts can copy your disk off before they go. On those, the note says so. We stop your node up to fifteen minutes before the deadline, because the copy needs that time. Your next run_command restores the node. On other hosts the disk does not survive. Write your results out to storage or a write connection before the deadline, or stop the node and snapshot_node it. Plan to finish before the deadline, not at it.

The ending is on get_node. A node whose disk was preserved reads stopped with a reason of retirement, and your next command restores it. A node whose disk went with the machine reads lost. Run delete_node and then create_node under the same name. Nothing runs or bills in either state.

Silence#

Rarely, the machine under a node stops answering the platform while the node itself keeps running. While that lasts, calls that reach into the node, like run_command and export_file, fail fast with "host isn't responding", and get_node carries a host section that says so.

get_node("tulip")
→ { status: "running (idle)", disk: "intact",
    host: { responding: false,
            last_heard_at: "2026-08-23T23:10:37Z", silent_seconds: 74,
            note: "this node's host has stopped answering the platform ..." } }

The status label reports the node's state, so it stays running through a silence. A monitoring loop should watch for the presence of host, not for a status change. Most silences end within a minute or two with the host reconnecting, and the section disappears. If the host is really gone, we mark the node lost after a few minutes of silence, close billing back at the host's last heartbeat, and credit the loss automatically.

The silence itself does not harm the container or the disk. If it outlives your patience, stop_node still works and ends billing now. The stop is delivered when the host reconnects.

Images#

Every node runs one image. Pass image to create_node to pick one from the catalog, or pass a registry ref to run your own. Pass nothing and you get the default. An unknown image name is refused with the current list.

Catalog#

CUDA version selection is image selection. This is the catalog:

Every catalog image has the same contract:

• CUDA devel, not runtime. nvcc and the full toolkit are present, so source-built extensions compile. The GPU driver comes from the host. You never install one.
• A working build toolchain. git, curl, build-essential, python3 with headers, ninja, and uv. This is what vLLM, SGLang, and flashinfer need to compile kernels at import. Also tmux, vim, htop, and nvtop.
• Lean. Base images carry no ML frameworks. The pytorch-* and vllm-* variants and pip install onto your persistent disk are for that. Nodes provision fast.
• Checked in CI. Every entry is built and tested before it ships. The toolchain compiles a C extension, the CUDA headers carry the symbols current inference wheels need, and a variant's preinstalled version is the one its name claims.
• A stable channel for the life of your node. Your node keeps its catalog entry forever. A wake runs that entry's current build. Builds advance like a distro release channel, so a wake after a patched build picks up the patch. A compiled extension that couples tightly to one exact build can need a rebuild after such a wake.

Install anything the image lacks at first boot. It lands in /home/dev (or a venv there) and survives stop and wake.

Snapshots and images. A node created with from_snapshot and no image runs the image its snapshot was taken on. Virtualenvs and compiled packages resume on the toolchain they were built under. list_snapshots shows each snapshot's image. You can name a different image. The response then warns that packages built on the original image can fail to import.

A snapshot restores onto a CPU shape or onto a GPU shape. A snapshot taken on a cpu-2 or cpu-8 node restores onto an H100 shape. A snapshot from an H100 node restores onto a CPU shape. You can download and prepare data on a CPU node, take a snapshot, and then start a GPU node from that snapshot.

Use org storage instead when more than one node needs the same data. Call export_data on the first node and import_data on each of the others. There is no egress fee on either call.

Custom#

If your stack ships as a container image, run it directly. Pass a registry ref as image. Anything with a /, :, or @ reads as a ref, so ubuntu:24.04 and ghcr.io/org/repo:tag both work. Bare words stay catalog names.

gman node create --name miles --chip 8xh100   --image nvcr.io/nvidia/pytorch:24.01-py3 --home /workspace

create_node(name: "miles", chip: "8xh100", image: "nvcr.io/nvidia/pytorch:24.01-py3", home: "/workspace")

• A tag pins its digest at create. The ref's manifest digest is resolved once. Every start of the node runs exactly that build. A push to the tag never changes an existing node. Pass repo@sha256:... to pin a digest yourself, which also skips the registry read at create.
• The platform supplies the entrypoint. Your image's own entrypoint never runs. The platform keeps the container alive and attaches your commands. The image must carry /bin/sh for run_command to work.
• home is where your disk lives. The persistent disk mounts at the home path you declare. The default is the image's WORKDIR, or /root. Only that path survives a stop. Everything else is the image, and it resets on every wake.
• Private refs use a registry connection. An org admin stores the credential once with gman connection create --kind registry, pinned to a host and repository list. When your ref is inside a pin, the platform presents that credential at the pull. Credentials never ride the create call. See Registry.
• get_node reports what your image carries. After the first start, the custom_image block says whether python3, sudo, nvcc, and bash are present. node_doctor needs python3. sudo needs the image to ship sudo. CUDA extensions need nvcc.
• The first start on a host pulls the image. The create response quotes the size when it is known. Later starts on that host are warm. A parked node's image stays on its host, so wakes never wait on your registry.
• Images are never scanned. Your image runs in the same hardened isolation as every node.

Use the catalog where it fits. Catalog images are pre-pulled on every host, so starts are instant, and their CUDA contract is checked in CI. Profiling with your own image works only if the image ships nsight-compute 2025 or newer. The platform cannot verify that for you.

Old toolkits#

A pip-installed kernel library can be built against a newer CUDA toolkit than your image ships. Recent flashinfer builds reference cuda::ptx symbols that exist only in a newer CCCL than CUDA 12.4 carries. On a hybrid or linear-attention model (Qwen3.5's gated-DeltaNet layers, for example), vLLM selects FlashInfer, tries to compile a GDN prefill kernel, nvcc reports missing symbols, and the engine dies before it generates a token. It looks like a broken install. It is a toolkit-age mismatch.

The default image does not have this problem. The default is cuda-12.9, and every catalog entry except the legacy cuda-12.4 line is checked at build time for these symbols. You hit this only on a node pinned to cuda-12.4. The image cannot change per node, so an old node keeps its old toolkit.

On such a node, do one of two things. Tell vLLM to skip the compile (gdn_prefill_backend=triton uses the Triton path and never runs nvcc), or create the node again on a current toolkit:

# either: skip the JIT on a node pinned to the old toolkit
gman run tulip -- python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3.5-8B --gdn-prefill-backend triton

# or: create on a current toolkit (the default already is one)
gman node create --name tulip --chip 8xh100
gman node create --name tulip-served --chip 8xh100 --image vllm-0.26-cuda12.9

# either: skip the JIT on a node pinned to the old toolkit
run_command("tulip", "python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3.5-8B --gdn-prefill-backend triton", detach: true)

# or: create on a current toolkit (the default already is one)
create_node(name: "tulip", chip: "8xh100")
create_node(name: "tulip-served", chip: "8xh100", image: "vllm-0.26-cuda12.9")

When a pip-installed kernel library fails to compile at import, compare the toolkit it wants with the one your image has (nvcc --version) before you debug your environment. If a stack you rely on needs a newer toolkit or a different pinned version than the catalog offers, tell us. Adding an image is a small change on our side.

Post-training#

Three pinned stacks are entering the catalog for post-training and RL work. trl-0.21-cuda12.9 has TRL and vLLM in one image. verl-0.7-cuda12.9 has verl, which pins its own engine. sglang-0.6-cuda12.9 serves judges and reward models. Until they appear in create_node's image list, the post-training guides run on vllm-0.26-cuda12.9 plus one pip line.

If the stack you need already ships as a container image, you do not have to wait. create_node takes a registry ref as image (see Custom). Batch jobs build their own image from a context (the Dockerfile's FROM can name any public ref), or name a registry ref the same way.

Robotics#

The pytorch-2.13-robotics-cuda12.9 image carries torch plus the video and vision toolchain that robot-learning stacks load at import. That is ffmpeg, torchcodec, OpenCV, and the headless EGL and Vulkan libraries that simulators and renderers open. They are present before your first command.

MuJoCo (MJX), Genesis, and ManiSkill run on H100 nodes. NVIDIA Isaac Sim does not run here. It requires ray-tracing cores that H100 hardware does not have. If your pipeline is Isaac-based, only its headless physics paths work here. Plan visual sim-to-real training elsewhere.

For the JAX path (openpi's primary fine-tune trainer, and GPU-native simulators such as MuJoCo MJX) use jax-0.11-cuda12.9. It has JAX with the cuda12 wheels preinstalled.

Commands#

There is no SSH. Commands run in the node's container, and the output is the interface.

Run#

gman run tulip -- sh -c 'git clone https://github.com/you/yourrepo && cd yourrepo && pip install --break-system-packages -e .'
# exit code = the remote exit code

run_command("tulip", "git clone https://github.com/you/yourrepo && cd yourrepo && pip install --break-system-packages -e .")
→ {exit_code: 0, stdout: "...", stderr: "...", truncated: false}

A synchronous run waits for the command to exit and returns its exit code and output. Running a command on a stopped node wakes the node first.

Secrets#

Put secrets in env, never in the command string. Command lines are recorded. Env values are not.

gman run tulip --env HF_TOKEN=hf_... -- huggingface-cli download meta-llama/Llama-3.1-8B

run_command("tulip", "huggingface-cli download meta-llama/Llama-3.1-8B",
            env: {HF_TOKEN: "hf_..."})

Use short-lived tokens with the smallest scope that works. Treat a token that rode a chat as burned. To keep a value out of the chat entirely, store it once with gman secret create hf-write and reference it by name: env: {HF_TOKEN: {secret: "hf-write"}}.

Detach#

Anything that outlives a tool-call timeout (training, a server) runs detached:

gman run tulip -d -- python train.py   # returns immediately with the command id
gman logs tulip -f          # newest command's output; -f follows to exit
gman ps tulip               # all detached commands, newest first
gman kill tulip             # TERM, then KILL after 10s

run_command("tulip", "python train.py", detach: true)
→ {command_id: "cmd_...", log_path: "..."}   # returns immediately

get_command("tulip", "cmd_...")              # status + incremental output
get_command("tulip", "cmd_...", offset: 4096) # just the new bytes
list_commands("tulip")                       # all detached commands, newest first
kill_command("tulip", "cmd_...")             # TERM, then KILL after 10s

Files survive stop and wake. Processes do not. It works like tmux on a machine you can power off. get_command never wakes a stopped node, so a check on finished work is free. A detached command runs with no deadline.

Read output in pages with offset. That is steadier than one synchronous command that prints all of it.

An env HOME reaches your command but not the command's markers. The log, the exit marker and a declared result stay under the image's home, where get_command and the result harvest read them.

Timeouts#

A synchronous run gets 60 seconds by default. Pass timeout (CLI: --timeout) to raise it, up to 300 seconds from the CLI and the HTTP API. Over MCP the ceiling stays at 60 seconds. Agent clients enforce a per-call deadline of their own, commonly 60 seconds, and nothing comes back until the command finishes. A longer budget would die in your client and lose the output, so it is refused before anything runs. Work that needs longer runs detached. Leave timeout off when you detach. On a detached run it budgets only the wake of a stopped node.

gman run tulip --timeout 300 -- ./cleanup.sh

# 300s is over the 60s MCP ceiling, so run it detached
run_command("tulip", "./cleanup.sh", detach: true)
→ {command_id: "cmd_...", log_path: "..."}
get_command("tulip", "cmd_...")   # status, output, exit code

A synchronous command is killed when its timeout expires. The error carries what the command printed before the kill. Most programs buffer stdout when it is not a terminal, so a script that prints progress can die with nothing captured. Python holds back about 8 KiB. Run scripts with python -u, or set PYTHONUNBUFFERED=1, or print with flush=True. The same buffering applies to a detached command's log.

A call that times out at your own client's deadline says nothing about whether the command ran. Retry after a short backoff when the command is safe to run twice. Use a detached command plus get_command when it is not.

Endings#

• Exit code 137 with a bare Killed line usually means the container's memory ceiling. When the kernel's OOM killer fired during the run, get_command carries an oom_kills count and a note that names the cause. The budget to size against is on get_node under resources.
• A stop ends a long run early. A command whose container goes away mid-run has no exit to report. It ends as lost, with no exit code and no declared result. stop_node warns you and names every detached command that was still running. get_command on a lost command says what survived: your volume's files and the command's log, unless the node itself was lost with its host.
• Who ended the run is in list_audit_events. A stop, a spend cap, and an OOM kill look the same on the command row, but each lands its own audit event. data.actor.type is customer for one of your people or your key, and system for the platform (grace lapsed, the container died, the host went away). Platform rows belong to the org, so reading them takes the admin or billing role.
• A finished command leaves its verdict on the volume. If the command had finished when the container went away, its exit code and its $GMN_RESULT_PATH verdict are in ~/.givemeanode/commands/. Wake the node and read <command_id>.exit and <command_id>.result.json. Only a node lost with its host takes those with it.

To keep a verdict on the tool surface, poll get_command until it carries an exit_code, then stop the node.

Hold#

A node with nothing running stops after its grace window. When you work by hand and think between commands, hold the node instead.

gman hold tulip --until 2h
gman hold tulip --release

hold_node("tulip", until: "2h")     # or an RFC3339 time
→ {held_until: "2026-08-12T18:00:00Z", node_per_min: "$0.53",
   cost_if_held_to_expiry_usd: 64.00}

release_node("tulip")               # back to the normal idle timer

• The node bills for the whole hold at its normal rate. The response says what the full hold costs before you commit.
• until is required and has a ceiling. list_limits shows the ceiling.
• Holds only move forward. A shorter hold than the one already set is ignored, because another agent on the same node can rely on it.
• release_node ends a hold early and puts the node back on its normal idle timer. stop_node stops billing now and clears the hold.
• A spend cap, an unpaid invoice, an org admin, and a host retirement still stop a held node. When the hold expires, the node stops on its normal idle path.

A hold defers two things: the idle timer, and your workspace's idle-burn auto-stop if somebody armed one. Both look at CPU as well as GPU on their own, so a file transfer, a dependency install, or a dataset download defers them without a hold. A running detached command defers them too. A hold is for a node you want awake while nothing runs on it. Use a hold instead of a background sleep loop. A sleep loop costs a command slot and shows on your mission page as work.

Hold on failure#

By default, a failed command ends like any other. The node idles out and /scratch is destroyed. Ask for a hold on failure and the node stays awake with the failed attempt's data on disk.

gman run tulip -d --hold-on-failure 120 -- python train.py

run_command("tulip", "python train.py", detach: true,
            hold_on_failure_minutes: 120)

A clean exit is unaffected, and the node idles out as usual. A nonzero exit, a kill by a signal, and a supervised command whose restart budget runs out all count as failures. get_node shows the hold and names the command that caused it.

On wake#

A node you wake can arrive in two minutes or in six hours. If nothing runs on it within its grace window, it stops again and bills that idle time. Register the command in advance, and it starts the instant the node is ready.

gman run tulip -d --on-wake --max-duration 240 -- ./bootstrap.sh

run_command("tulip", "./bootstrap.sh", detach: true, on_wake: true,
            max_duration_minutes: 240, max_wait: "12h")
→ {status: "queued", position: 3, estimated_ready_seconds: 16800,
   boot_command: {command_id: "cmd-4k2x9", state: "pending_wake"}}

get_command("tulip", "cmd-4k2x9")   # answers now, before the node exists
→ {status: "pending_wake"}

• The command id returns immediately. get_command answers for it from that moment. get_node shows the registration next to the queue facts. When the node is ready, the command starts as an ordinary detached command with the env, restart, and mission you gave it.
• max_duration_minutes is required. When it expires, the command is killed and the node returns to its normal grace window. list_limits shows the ceiling.
• A node has one boot command at a time. A new registration replaces the old one, and the response says which. To run several things, put them in a script.
• kill_command cancels a registration that has not run. Nothing was billed, because no process started.
• A registration belongs to the wait you made it for. If the queue hold lapses or the node goes away, the registration goes with it.
• Registering against a running node is refused. Drop on_wake to run the command now.

Files#

Small files go up with write_file. Results come back with export_file. Large inputs that exist somewhere reachable are cheapest pulled from inside the node with curl, git, or huggingface-cli. A large input that exists only on your own machine goes through storage.

Upload#

Use write_file for configs and entry scripts. Do not use it for secrets or datasets.

gman cp ./run.sh tulip:~/run.sh --executable

write_file("tulip", "~/run.sh", "#!/bin/sh\npython train.py\n", executable: true)

• gman cp uses the scp grammar. The side that names a node is written node:path. The other side is a path on your own machine.
• Paths on the node are absolute or relative to the home directory. ~/run.sh and /home/dev/run.sh are the same file.
• The content travels in the request. The size cap is max_write_file_bytes in gman limits. Anything larger goes through storage, which streams.
• The response carries the bytes written and the sha256 of the content. The node hashes the file again before it moves it into place, and refuses the write if the two differ. If you built the content by hand, hash your copy and compare. If the image has no sha256sum, the response says sha256_verified: false, and the file was checked by length only.
• A write to a stopped node wakes the node, and the write does not happen. You get written: false. Send the content again when the node is running. To push several files, wake the node, hold it, then write.

Download#

export_file returns a presigned download URL. A directory arrives as a tar.

gman cp tulip:~/yourrepo/checkpoints ./   # arrives as a tar
gman api post /nodes/tulip/exports --data '{"path": "~/yourrepo/checkpoints"}'
→ {"url": "https://...", "expires_at": "..."}   # the link, if you want it instead

export_file("tulip", "~/yourrepo/checkpoints")
→ {url: "https://...", expires_at: "..."}

gman cp does the export and the download in one step and checks the digest. Use export_file directly when you want the link, which is what an agent usually wants. The link is private and made for machines. If you lose it, get_artifact returns a fresh one for the same export for about 30 days.

Node to node#

copy_file moves a file or directory between two running nodes of your workspace in one call. The nodes do not need a shared network.

gman cp tulip:~/app poppy:~/app   # polls until it lands

copy_file(src_node: "tulip", src_path: "~/app", dst_node: "poppy", dst_path: "~/app")
→ {import_id: "imp-...", status: "transferring", size_bytes: ..., sha256: "..."}
get_import("poppy", "imp-...")   # poll until complete

• A file lands at dst_path exactly.
• A directory's contents land extracted under dst_path, merged file by file over what is there. A second copy after edits overwrites the changed files only.
• The call returns once the source is read. The write continues in the background. Poll get_import on the destination node until it reports complete.

Share#

share_artifact makes one exported file public at a clean URL. The link carries the real filename and content type. Slack unfurls an image, <img src> works in a page, and a browser shows the file instead of downloading it.

gman api post /artifacts/art-df5cj/shares
→ {"share_id": "shr-9k2fq", "url": "https://s.givemeanode.io/shr-9k2fq-vx7t3m8dk4qwrz2n/cat1.jpg", ...}

export_file("tulip", "~/out/cat1.jpg")
→ {artifact_id: "art-df5cj", ...}
share_artifact("art-df5cj")
→ {share_id: "shr-9k2fq", url: "https://s.givemeanode.io/shr-9k2fq-vx7t3m8dk4qwrz2n/cat1.jpg",
   filename: "cat1.jpg", content_type: "image/jpeg", expires_at: "..."}

• Anyone with the URL can fetch the file, with no account and no sign-in. Treat a share URL as a secret if the content is one. Every share is recorded in your org's audit log.
• list_shares shows everything your org has public. unshare revokes a link at any time.
• The bytes are copied when you share, so the link works after the private export expires. A share lives 365 days unless you pass a shorter expires_after. Sharing the same artifact again refreshes the same URL.
• Images, PDFs, video, audio, and plain text render inline. HTML and SVG always download, because a rendered page could run scripts on the serving domain. A directory export shares as one .tar download.

Allowlists#

Transfers use a second hostname. Every presigned URL the platform returns points at its object store, which is fly.storage.tigris.dev. That includes the upload URL from create_context, the download URLs from get_context and export_file, and the artifact, preview, and sample links on jobs. If your agent runs behind an egress allowlist, allow that host too. With only the API allowed, every tool call works and every transfer fails.

A blocked host usually shows as a 403 from your own proxy when curl connects, before any bytes reach the store. Check the hostname against your allowlist before you debug the signature. These are the hostnames the product uses:

• givemeanode.com and mcp.givemeanode.com serve the web pages, the HTTP API, and MCP.
• fly.storage.tigris.dev serves the presigned upload and download URLs.
• storage.givemeanode.com and <bucket>.storage.givemeanode.com serve the S3 endpoint, if you use it.

Connections#

A connection is a stored credential for data behind a login: a private bucket, a gated Hugging Face repo, a private registry, or an authenticated URL. An org admin creates it once from a terminal or CI. The platform uses the credential on the node's host. Agents see connection names, never secrets.

curl -fsSL https://givemeanode.com/cli.sh | bash && gman login   # once, ~2 minutes
gman connection create corpus --kind s3-keys \
  --scope s3://my-bucket/datasets/ \
  --access-key-id AKIA... --material-stdin < secret-key.txt

Every connection follows the same rules:

• The secret arrives on stdin (--material-stdin), in the GMAN_CONNECTION_MATERIAL environment variable, or at a hidden prompt. Never as a flag.
• The credential is checked against the provider before it is saved. Then it is sealed. No command or API returns it again.
• Every connection has a --scope. Imports outside the scope are refused by the platform, whatever the credential itself allows.
• Connections are read-only unless you create them with --verbs read,write. Write lets agents export results through them.
• Creation is CLI-only. Agents cannot create, change, or delete a connection over MCP.
• A connection belongs to one organization. A terminal or MCP session that was not opened in a specific organization acts in your active one (set on /team). Every create, get, and list answer names the organization it acted in, and gman connection ls prints it above the rows. A lookup that misses tells you when the name exists in another of your organizations.

A run does not have to wait for a connection. Until an admin has set one up, pass the cloud credentials in the environment of the command and sync from inside the node. The values ride --env, never the command line, and they are not stored.

gman run my-node \
  --env AWS_ACCESS_KEY_ID=AKIA... --env AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
  -- aws s3 sync s3://my-bucket/datasets/ ~/data/

The kinds:

S3#

An s3-keys connection holds an access key pair for one bucket prefix. The key id rides --access-key-id. The secret key is the material.

# AWS
gman connection create corpus --kind s3-keys \
  --scope s3://my-bucket/datasets/ \
  --access-key-id AKIA... --material-stdin < secret-key.txt

# Cloudflare R2, MinIO, Backblaze B2: add the endpoint
gman connection create r2-corpus --kind s3-keys \
  --scope s3://my-bucket/datasets/ --endpoint https://<account>.r2.cloudflarestorage.com \
  --access-key-id ... --material-stdin < secret-key.txt

# a connection agents can also export to
gman connection create acme-out --kind s3-keys \
  --scope s3://acme-ml/ --verbs read,write \
  --access-key-id ... --material-stdin < secret-key.txt

• The scope is s3://bucket/prefix/. The prefix can be empty: s3://bucket/.
• --endpoint names a non-AWS endpoint. --region names the bucket's region. The default is us-east-1.
• Make a dedicated key whose own IAM policy allows exactly this prefix. The scope pin bounds what the platform fetches. It cannot take back what the key itself can reach.
• Sources and destinations on an s3-keys connection are full s3:// URLs, because the bucket is not implied.

Hugging Face#

A huggingface connection reaches gated and private repos. On huggingface.co, create a fine-grained token (Settings, then Access Tokens) with read access to the repos in the scope. Add write access if the connection will push checkpoints.

# read a private adapter repo
gman connection create loras --kind huggingface \
  --scope acme/loras --material-stdin < token.txt

# read and write a checkpoint repo (the repo must already exist)
gman connection create ckpts --kind huggingface \
  --scope acme/ckpts --verbs read,write --material-stdin < token.txt

# no token at all: an org admin authorizes once in the browser
gman connection create ckpts --kind huggingface \
  --scope acme/ckpts --verbs read,write --auth oauth

• The scope is a comma-separated list of repo patterns. owner/name is one model repo. datasets/owner/name is one dataset repo. owner/* is every repo of that owner. * is everything the token reads.
• --auth oauth stores no token. An org admin authorizes in the browser, and short-lived tokens are minted per use. The command prints the link and waits for the approval.
• The platform never creates repos on your account. A write destination must exist first.

gman connection ls   # loras  huggingface  read  acme/loras
gman import tulip --connection loras --source acme/loras \
  --dest '~/models/loras' --wait

list_connections()
→ [{name: "loras", kind: "huggingface", scope: "acme/loras", ...}]
import_data("tulip", connection: "loras",
            source: "acme/loras", dest_path: "~/models/loras")
→ {import_id: "i-7pd12", status: "running"}   # poll get_import

Civitai#

A civitai connection holds a Civitai API key from civitai.com/user/account. It authorizes gated and early-access downloads of community models and LoRA adapters.

gman connection create community-loras --kind civitai \
  --scope model:1115064 --material-stdin < api-key.txt

• Scope pins are the site's numeric ids. model:1115064 pins one model, and an import of it pulls the newest published version. version:1253021 pins one exact version. Several pins are separated by commas. * allows any.
• Imports name the same ids. The key never leaves the platform.
• Civitai connections are read-only.

gman connection ls   # an admin created one: kind civitai, scope model:1115064
gman import tulip --connection community-loras \
  --source model:1115064 --dest '~/models/loras' --wait

import_data("tulip", connection: "community-loras",
            source: "model:1115064", dest_path: "~/models/loras")
→ {import_id: "i-9wk21", status: "running"}   # poll get_import

Registry#

A registry connection holds a pull credential for a private Docker or OCI registry. Jobs, nodes, and sandboxes that name an image inside its scope pull with it. Nothing about the image reference changes.

gman connection create acme-images --kind registry \
  --scope 'ghcr.io/acme/*' --username acme-ci --material-stdin < token.txt

• The scope is written like the image refs it admits, without a tag. ghcr.io/acme/trainer pins one repository. ghcr.io/acme/* pins an owner. ghcr.io/* pins a whole registry. Every entry must name the same host. Create a second connection for a second registry.
• --username is the login half of the credential. On GHCR it is your username, with a personal access token as the material. On Quay and Harbor it is a robot account. On Google Artifact Registry it is _json_key, with the service account JSON as the material.
• The credential is checked against the registry before it is saved. A wrong token fails at the terminal, not hours later in a queued job.
• Amazon ECR does not work here. ECR tokens expire after twelve hours, so a stored one fails overnight. Push the image to a registry with durable robot credentials, or have the job build it from a context.
• The connection covers the image a job runs, a node's custom image, and a sandbox's base image. It does not cover a private base image in a Dockerfile FROM. A job that builds from a context needs a base it can pull anonymously.
• If two connections pin the same image equally, the call is refused and names both. Give one a narrower scope. A pin that names the repository wins over an owner or registry wildcard.

gman job submit --chip h100 --count 8 \
  --image ghcr.io/acme/trainer:v4 --command 'python train.py'

submit_job(chip: "h100", chip_count: 8, mission: "trainer-v4",
           image: "ghcr.io/acme/trainer:v4",
           command: "python train.py",
           idempotency_key: "trainer-v4-run1")
→ {job_id: "job-8x2mf", status: "building"}

HTTPS#

An https connection fetches one URL with a token in a header. It is for an authenticated download that is not S3 or a hub.

gman connection create exports --kind https \
  --scope https://data.corp.example/exports/ --material-stdin < token.txt

# a header other than Authorization
gman connection create exports --kind https \
  --scope https://data.corp.example/exports/ --header X-Api-Key --material-stdin < token.txt

• The scope is a URL prefix that starts with https://. An import names a URL under it.
• The token is sent as Authorization: Bearer <token>. --header names a different header. The value is still Bearer <token>.
• Leave the material empty for a public host. The import then sends no header.
• One import fetches one URL and lands one file.

Import#

Agents import by connection name. The import runs on the node's host and streams into the container.

gman connection ls          # names, kinds, scopes - never material
gman import tulip --connection corpus \
  --source s3://my-bucket/datasets/train/ --dest '~/data/train' --wait

list_connections()
→ [{name: "corpus", kind: "s3-keys", scope: "s3://my-bucket/datasets/", ...}]
import_data("tulip", connection: "corpus",
            source: "s3://my-bucket/datasets/train/", dest_path: "~/data/train")
→ {import_id: "i-4kd93", status: "running"}
get_import("tulip", "i-4kd93")
→ {status: "complete", total_bytes: 5368709120, files: 128, dest_path: "~/data/train"}

• Quote a --dest that starts with ~. An unquoted ~ expands to your local home directory before the command runs, and the import fails on the node.
• A running import holds the node out of its grace window and bills as node time.
• An archive needs room for two copies. A 92 GB tar arrives as a 92 GB tar, and extracting it needs another 92 GB. Size the destination for twice the archive, and delete the archive after you unpack it. On scratch the size is fixed at create. To avoid the doubling, import a prefix of individual objects instead of one archive.
• A failed import reports the provider's real error and is safe to run again. Completed files are overwritten, never doubled.
• Every use lands one row in the org's audit log: connection, node, and source. Never material.

Export#

A connection created with --verbs read,write can receive results through export_data: model checkpoints, generated datasets. The platform uses the credential on the node's host, so it never enters the container. Write is available on s3-keys and huggingface connections.

gman connection ls    # ckpts  huggingface  read,write  acme/ckpts
gman export tulip --connection ckpts \
  --source ~/out/checkpoints --dest acme/ckpts@main:runs/run7/ --wait

list_connections()
→ [{name: "ckpts", kind: "huggingface", verbs: ["read", "write"],
    scope: "acme/ckpts", ...}]
export_data("tulip", connection: "ckpts",
            source: "~/out/checkpoints", dest: "acme/ckpts@main:runs/run7/")
→ {export_id: "exp-4kd93", status: "running"}
get_export("tulip", "exp-4kd93")
→ {status: "complete", total_bytes: 4831838208, dest: "acme/ckpts@main:runs/run7/"}

• A Hugging Face destination is owner/name[@revision][:path/in/repo]. A directory lands as individual files in one commit, never a tar, so from_pretrained works on the result.
• An S3 destination is one object key. A directory is tarred.
• An export outside the connection's scope is refused. A failed export reports the provider's real error and is safe to run again. The destination is replaced, never doubled.
• Exports are bounded in size, bill as node time while they stream, and land one row each in the audit log.

If your training code pushes to the Hub itself (push_to_hub in the trainer loop), the token has to be inside the container. Store it as a secret and reference it by name in env, so it stays out of the chat.

Manage#

gman connection ls                       # names, kinds, verbs, scopes
gman connection get corpus               # one connection; material is never shown
gman connection arm corpus --material-stdin < new-secret.txt   # replace the secret
gman connection disable corpus           # in-flight imports fail at their next chunk
gman connection enable corpus
gman connection rm corpus                # the sealed material is shredded now

• Create, re-key, disable, enable, and delete exist in the CLI and the HTTP API only. They need an org admin with org:write.
• Agents read connections with list_connections over MCP. They cannot change them.
• An OAuth connection re-arms in the browser. gman connection arm prints the link.

Storage#

Org storage is buckets on givemeanode's own object store. There is no second vendor and no credential to manage. Upload from a laptop or CI with the CLI. Every bucket appears to agents as the connection storage/<name>, so nodes pull from it with import_data.

Buckets#

gman storage bucket create corpus
gman storage cp ./train-data corpus/train/    # a directory becomes one object per file
gman storage cp ./icons corpus/icons/ --concurrency 64
gman storage ls corpus/
gman storage cp corpus/train/a.bin ./         # download one object
gman storage cp corpus/train/ ./train         # download everything under a prefix
gman storage cp corpus/ ./corpus.zip          # download the whole bucket as one zip
gman storage rm corpus/train/a.bin            # delete one object
gman storage bucket ls                        # footprint and estimated cost per bucket

• A directory uploads as one object per file, in parallel. --concurrency sets how many files are in flight. The default is 32 and the maximum is 256. An empty file uploads as a zero-byte object.
• A download source that ends in a slash means everything under it. A bare bucket name means the whole bucket. Objects land in the directory you name with their bucket layout. A .zip destination gives one archive instead.
• A source with no trailing slash is one object. A one-file download never becomes a whole bucket by accident.
• If an object is deleted while a download runs, the command names it and exits nonzero.
• Buckets are durable until you delete them. A bucket must be empty to delete, or pass --force, which cannot be undone.

Import#

gman import tulip --connection storage/corpus --source train/ --dest '~/data/train' --wait

import_data("tulip", connection: "storage/corpus",
            source: "train/", dest_path: "~/data/train")

• The source is a key inside the bucket, written as gman storage ls prints it. s3://corpus/train/ means the same thing. A dest on export_data works both ways too.
• A source that ends in / imports every object under that prefix. An exact key imports one object. A source with no trailing slash that names no exact object imports every key it prefixes.
• One import takes up to 20,000 objects (max_import_files). Past that, split by first letter: import items/a, then items/b, and so on. The pieces land side by side under the same dest_path.
• You can start 30 imports per hour (imports_per_hour). The count is per person, across every node, workspace, and org you use. The window slides. The refusal says how many you started, when the next slot frees, and how long that is. A whole-prefix import counts as one call, however many objects it carries. An operator can raise both limits.

List#

An agent can look inside a bucket before it pulls anything. A list reads the index only. It costs no node time and needs no running node.

gman storage ls corpus/
gman storage ls corpus/train/

list_storage_objects("corpus")
list_storage_objects("corpus", prefix: "train/")
list_storage_objects("corpus", delimiter: "/")   # one directory level at a time

• list_storage_objects returns the keys under a prefix with their sizes.
• Pass delimiter: "/" to answer one directory at a time. Everything below the next / folds into a list of prefixes, such as ["icons/", "train/"]. Pass one of them back as the prefix to descend. A bucket with a million objects is a few calls this way.
• A response with truncated: true has more. Call again with after set to the next_after it returned.
• gman storage ls always lists flat. On a very large bucket, use a narrower prefix from the terminal.

S3 endpoint#

Every bucket is also reachable over the S3 protocol, so aws s3, boto3, DuckDB, rclone, and anything else that speaks S3 works directly. Mint a scoped access key and point your tool at the endpoint the mint prints.

gman storage key create --bucket corpus --access rw
# prints AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (once) / endpoint

aws s3 cp ./train-data s3://corpus/train/ --recursive \
    --endpoint-url https://storage.givemeanode.com
duckdb -c "SELECT count(*) FROM read_parquet('s3://corpus/train/*.parquet')"

gman storage key ls
gman storage key revoke gmnk_...   # immediate; the secret is shredded

• A key is per bucket and read-only unless you say otherwise. Keys die with their bucket.
• Multipart uploads, ranged reads, presigned URLs, and both addressing styles (path-style and bucket.storage.givemeanode.com) work. Set AWS_REGION=auto.
• See Limits before you upload an object over 5 GiB.
• The agent path never needs a key. import_data through the storage/<name> connection is credential-free.

Limits#

These numbers bound an upload. Most are defaults that support can raise for your org. Your live values are in limits.

• One PUT, and one multipart part: 5 GiB each. Anything bigger uses multipart. Every S3 client does this automatically, and the default part size is fine.
• Multipart: parts numbered 1 to 10,000, contiguous from 1. A multipart object can exceed 5 GiB.
• Org footprint across all buckets: 1 PiB. Writes past it are refused with a 403 and a readable message. Nothing you stored is deleted to make room.
• 20 buckets and 50 access keys per org. 600 requests per minute per key. Past that, S3's SlowDown: back off and retry.

An example for a 12.6 GB training checkpoint. A plain aws s3 cp of the whole file works. The shard-and-manifest pattern adds integrity you verify on readback, and shard-level resumability:

# upload: 4 GiB shards plus a checksum manifest
tar -cf - ckpt/ | split -b 4G - ckpt.tar.part-
sha256sum ckpt.tar.part-* > MANIFEST.sha256
aws s3 cp . s3://ckpts/run7/ --recursive \
    --exclude '*' --include 'ckpt.tar.part-*' --include 'MANIFEST.sha256' \
    --endpoint-url https://storage.givemeanode.com

# restore: download, verify byte-for-byte, reassemble
aws s3 cp s3://ckpts/run7/ . --recursive \
    --endpoint-url https://storage.givemeanode.com
sha256sum -c MANIFEST.sha256      # every shard must print OK
cat ckpt.tar.part-* | tar -xf -

The same works with gman storage cp and no key. gman storage cp ./shards ckpts/run7/ uploads the directory in parallel, and gman storage cp ckpts/run7/ ./restore brings it back. From a node, use import_data and export_data through the storage/<name> connection.

Integrity#

The platform verifies these on every upload:

• Length, always. An upload that delivers a different byte count than it declared is refused (IncompleteBody) and never becomes an object.
• Payload SHA-256, when your client signs one. A literal x-amz-content-sha256 is recomputed and a mismatch is refused. Streaming (aws-chunked) uploads, the aws CLI v2 default, carry CRC trailers instead. Those are accepted but not recomputed. TLS covers the transport and the length check still applies.
• ETags are consistent on Head, List, and Get. The ETag of a multipart or server-side-copied object is not a checksum of your bytes. Treat ETags as change detectors only.

For bytes you cannot re-derive, checkpoints above all, write your own sha256sum manifest beside the data and verify it on readback, as in the example above.

Retries are safe:

• Every upload assembles in a hidden staging area and is promoted after verification. An overwrite is atomic. Readers keep the old object until the new one commits. A failed or abandoned upload never touches the object it was replacing.
• A failed PUT or part is safe to retry as is. So is a CompleteMultipartUpload whose response you lost. The commit converges on the assembled result.
• AbortMultipartUpload cleans up staged parts now. Single-PUT staging debris is swept after about 24 hours. An abandoned multipart upload's parts are not swept on a timer. Always abort what you do not finish. Deletes take effect at once.
• CompleteMultipartUpload on a multi-GB object can take a while. Like S3, we hold the connection open with whitespace and deliver the outcome in the body of a 200, which is what the AWS SDKs expect. You do not need to raise your client's read timeout. If the connection drops before the answer, the Complete is cancelled where it was. HeadObject the key to see whether it committed. Your parts are still there, and a second Complete assembles them again.

Pricing#

Storage bills per GB-month at each day's peak footprint, prorated daily. The first 100 GB each month are free. Then the rate steps down with size, marginally, like tax brackets:

gman storage bucket ls shows each bucket's footprint and the month's estimated cost. get_billing() reports storage spend beside compute.

Ports#

Put a port on a running node at a public HTTPS URL: TensorBoard, Jupyter, or a vLLM endpoint that serves a model to a judge.

gman run tulip -d -- python -m vllm.entrypoints.openai.api_server --port 8000
gman api post /nodes/tulip/endpoints --data '{"port": 8000}'
→ {"url": "https://<endpoint-id>.<edge>", "expires_at": "..."}

run_command("tulip", "python -m vllm.entrypoints.openai.api_server --port 8000", detach: true)
expose_port("tulip", port: 8000)
→ {url: "https://<endpoint-id>.<edge>", expires_at: "..."}

• Anyone with the URL reaches the port. The URL is the secret. auth: "bearer" also requires an Authorization: Bearer token at the edge. The token is in the response once and never shown again.
• The URL passes WebSocket traffic as well as plain HTTP.
• URLs expire. expires_at is the deadline. Call expose_port again to extend it.
• Endpoints belong to the node's current container. A stop kills them, and a wake does not bring them back. Call expose_port again for a fresh URL.
• Traffic through the endpoint is free and does not keep the node awake. The server process (a detached command) holds the node running.
• list_endpoints(node) shows what is live. unexpose_port(node, port) takes it down.
• vanity: "mylabel" also claims a readable hostname beside the capability URL. A vanity name is guessable, so pair it with auth: "bearer" when the port must not be public. The namespace is global and first-come. Reserved words refuse. The name frees when the endpoint dies. A re-expose with the same vanity keeps it.

Networks#

A private network lets your own nodes reach each other over stable private addresses. Nothing on it is visible from the internet.

gman node create --name head --chip h100 --network mesh1
gman node create --name worker1 --chip h100 --network mesh1
gman run head -- ping -c 3 worker1

create_node("head", chip: "h100", network: "mesh1")
create_node("worker1", chip: "h100", network: "mesh1")
run_command("head", "ping -c 3 worker1")

• A network that does not exist is created for you. Existing nodes join live: gman network attach worker2 mesh1 (attach_network) converges a running node in seconds with no restart. gman network detach worker2 removes the interface just as live. gman network ls (list_networks) shows every network and its members.
• Names resolve. From inside a member, worker1 and worker1.givemeanode.internal both resolve to that peer, so MASTER_ADDR=head works. The metadata service serves the whole roster at /v1/network.
• Addresses are stable for the life of the node. A member keeps its IPv4 and IPv6 across stop, wake, recreate, and repair. A stopped member keeps its place and drops traffic until you wake it. Mesh traffic never wakes a node.
• One network is one trust zone. Every port a member listens on is reachable by every other member. There are no per-port rules. Membership is opt-in per node. A node you never attach keeps the isolation in Network. Members get no internet transit through each other and no route to another network. An exposed port stays the only way in from the internet.
• A network must be empty to delete. A detached address waits in a seven-day quarantine before it can belong to someone else, so a network is deletable at the earliest seven days after its last detach.

Jobs#

Nodes are interactive. Jobs run without you. submit_job builds your image, queues it fairly against your org's share, runs your command, and bills only the running attempt. Queued and preempted time is free.

Submit#

gman job submit --chip h100 --context ./eval \
  --command 'python eval.py' --idempotency-key eval-2026-07-25
→ job_...   # the id on stdout; rate and cost ceiling on stderr (--wait blocks to running)

submit_job(
  chip: "h100",
  command: "python eval.py",
  context: {"Dockerfile": "FROM pytorch/pytorch\nCOPY eval.py .", "eval.py": "..."},
  idempotency_key: "eval-2026-07-25")
→ {job_id: "job_...", rate: "$/min", cost_ceiling: "..."}

• Pass a prebuilt image ref instead of context to skip the build. When a build must wait for a slot, the submit response says so in build_note, and get_job repeats it while the wait lasts. Identical contexts share one build.
• idempotency_key makes resubmission safe. The same key returns the same job.
• Inside the container, write your verdict as JSON to $GMN_RESULT_PATH, and put files to keep in $GMN_OUTPUT_DIR. Both come back on get_job when the attempt completes: the result inline, the output directory as a download URL.
• A job container carries its budget in $GMN_CPU_LIMIT and $GMN_MEMORY_LIMIT_BYTES, like a node. OMP_NUM_THREADS and RAYON_NUM_THREADS are preset to the CPU budget, so nproc, OpenMP, BLAS, torch, and Rust thread pools size to your slice. A value you pass in env wins. This matters most for small-tensor test suites, where a pool sized to the whole host can make tiny operations hundreds of times slower.

The queue is built for long waits. Hours queued is normal. A job holds its place for queue_ttl_minutes (default 3 days). While it waits, get_job reports a live queue.position and estimated_wait_seconds, measured from recent throughput and the fleet's drain capacity. A null estimate means no signal yet, not stuck. Poll minutes apart. Do not resubmit or cancel because the wait is long.

A job backlog that persists for a shape raises the fleet's procurement target for that shape. Submitting the work is how capacity arrives, and cancelling retracts the signal. Submit the whole shape of the work at once. Eight jobs queued is a truer signal than one job queued eight times in series. The signal reads a sustained backlog, so give it an hour rather than a minute. See Queue.

Dependencies#

A job can wait for another job. Pass depends_on with the parent job id and, if you want one, a condition over what the parent declared at $GMN_RESULT_PATH. The dependent job does not build, does not queue and never places until the parent succeeds and the condition holds. A cheap probe in front of an expensive run therefore costs nothing when the probe says no.

# the probe; two cards, twenty minutes, writes {"ok": true, "peak_gib": 17.4}
submit_job(chip: "h100", chip_count: 2, context_id: "ctx-...",
           command: "python probe.py", idempotency_key: "mem-probe",
           mission: "memory-ceiling")
# → {job_id: "job-8x2mf", ...}

# the run; four cards, nine hours, and it starts only if the probe passes
submit_job(chip: "h100", chip_count: 4, context_id: "ctx-...",
           command: "python train.py", idempotency_key: "arm-1",
           mission: "memory-ceiling",
           depends_on: {job: "job-8x2mf", require: "result.ok == true"})
# → {job_id: "job-4kq7p", status: "submitted",
#    depends_on: {job: "job-8x2mf", require: "result.ok == true",
#                 state: "waiting"}}

• require is one comparison. Write result, then an optional dotted path, then ==, !=, <, <=, > or >=, then a literal. "result.ok == true" and "result.metrics.peak_gib < 72" are both valid. Leave it out to wait on the parent succeeding alone.
• The parent has to succeed either way. A parent that crashed answers no question about the job waiting on it.
• If the gate closes, the waiting job ends canceled and failure_reason says why, down to the value the condition read. It built nothing and ran nothing, so it cost nothing.
• The parent must be built from the same context as the job that waits on it. Vary the command and the env instead, which is what separates a probe from a real run and changes nothing about the code. If you really do want a parent built from other code, pass allow_different_context: true inside depends_on.
• get_job reports depends_on.state as waiting, satisfied, or closed, so a polling agent can tell a gate from a queue. The waiting job's own queue_ttl_minutes still bounds how long it waits, and a job whose TTL runs out before the parent finishes reads expired_waiting rather than closed, because the gate never decided.
• submit_jobs takes the same declaration on shared, so one probe can gate a whole sweep.

Status#

gman job get job_...          # status; gman job logs job_... -f follows
gman job ls --label lr-sweep  # every job under the label
gman job cancel job_...       # immediate; queued/building stops free

get_job("job_...")                   # status; add stream:"run" for logs
list_jobs(label: "lr-sweep", summary: true)  # rollup with every declared result
cancel_job("job_...")                # immediate; queued/building stops free

• Add stream: "run" to read a job's output as it goes. Add attempt to read an earlier attempt after a preemption.
• attempt_state says whether to read again. "not_started": still building, queued, or starting. Keep polling the job. "live": the attempt is executing. "complete": the attempt ended and the stored output is final. "never_ran": the attempt is over and never got a container, so an empty response is the final answer.
• eof: true means you reached the end of a stream that nothing is writing. Stop polling.
• Polling tells you that a job is running. It does not tell you whether the run is going well. To see that, have the job declare what it is doing while it does it. One preview is a replaceable peek at a file, and samples are a series that accumulates, so a metric or an image per step is visible from the first one instead of at the end. Both are free, both take one curl from inside the container, and both are the difference between reading a failure at minute 8 and reading it at minute 40.
• host_gpu_health appears on a live attempt only when the machine under it has a problem, such as a GPU that stopped computing correctly. You do not have to act on it. An attempt cut short by such a device does not bill, draws on the restart budget, and re-places on a healthy host. The field lets a polling agent tell our hardware apart from its own bug.
• A terminal job carries a receipt. receipt.image_digest is the manifest digest of the image we resolved and pinned for the run. Use it to tie a result to the exact bytes that produced it. receipt.billed_seconds and receipt.charged_usd_micros are the totals. receipt.attempts holds one row per billed attempt, each with the rate it billed at. All of these come from the same billing records your invoice totals to. Billed seconds are the seconds for one GPU. The charge is seconds x gpu_count x rate / 60. An attempt we ended, such as a preemption, bills only the progress that survived. It bills nothing unless a checkpoint capture or a task report landed inside that attempt's own window. If one did, the attempt bills up to that mark. An attempt with no row was free.

Checks#

A check declares what your job must do. The platform validates the declaration when you submit, and then applies it. On the MCP interface every job must declare at least one check.

gman job submit --chip h100 --context ./eval --command 'python eval.py' \
  --checks '{"success":{"check":{"all_of":[
    {"type":"file_exists","path":"/output/rows.jsonl"},
    {"type":"jsonl_field","path":"/output/rows.jsonl","field":"logprob_mean","require":"nonnull"}]}}}'

submit_job(
  chip: "h100",
  command: "python eval.py",
  image: "my-registry/eval:v3",
  idempotency_key: "eval-2026-09-02",
  checks: {success: {check: {all_of: [
    {type: "file_exists", path: "/output/rows.jsonl"},
    {type: "jsonl_field", path: "/output/rows.jsonl",
     field: "logprob_mean", require: "nonnull"}]}}})

A success check runs one time, when the job stops. If it passes, the job gets the succeeded state. If it fails, the job gets the check_failed state, and the result is on get_job. A downstream reader cannot take an unusable file from a succeeded job.

Start with a success check. It runs one time, it has no interval to configure, and it cannot stop a job that works.

A health check runs on an interval while the job is live. It finds the failure a loss curve hides, such as a training run that writes logs and changes no weights.

gman job submit --chip h100 --image my-registry/train:v7 \
  --command 'python train.py' --max-duration 8h --checks ./checks.json
# checks.json:
# {"health": {"check": {"type": "tensor_changed",
#                       "path": "/checkpoint/adapter_model.bin"},
#             "interval_s": 600, "grace_period_s": 900,
#             "fail_threshold": 2, "on_fail": "kill"}}

submit_job(
  chip: "h100", image: "my-registry/train:v7",
  command: "python train.py", max_duration_minutes: 480,
  resume: "checkpoint", idempotency_key: "train-v7",
  checks: {health: {
    check: {type: "tensor_changed", path: "/checkpoint/adapter_model.bin"},
    interval_s: 600, grace_period_s: 900,
    fail_threshold: 2, on_fail: "kill"}})

• interval_s is the time between checks, from 30 to 3600 seconds. The default is 300.
• grace_period_s is the time at the start of the attempt when no check counts. Set it above your start time. The default is 300.
• fail_threshold is the number of consecutive failures before the platform acts. The default is 2.
• on_fail is required. Use record to write the failure to the job record and do nothing more. Use kill to stop the job and close the meter. Use reschedule to stop the job and run it again on different hardware.

Before a kill or a reschedule, the platform captures your checkpoint slot. Write your state to $GMN_CHECKPOINT_DIR and pass resume: "checkpoint". A check that stops the job then costs you the time to resume. A job that writes no checkpoint has nothing to capture, and it loses the run.

The push pattern is the shortest health check to add. Your loop writes a file, and the platform confirms the file changed.

# one line per step in your training loop
Path("/output/heartbeat").touch()

{"health": {"check": {"type": "heartbeat", "path": "/output/heartbeat"},
            "interval_s": 600, "on_fail": "kill"}}

The check types#

A check is one type, or a group of types under all_of or any_of. Groups hold groups, to four levels. Each phase takes at most 16 checks.

• file_exists passes if the path is present. Both phases.
• file_size passes if the size is between min_bytes and max_bytes. Both phases.
• jsonl_field passes if the field satisfies require in every row. require is nonnull, present, finite, or nonempty. Both phases.
• exec passes if your command exits 0. Both phases.
• file_grows passes if the file or the directory is larger than at the last check. Health only.
• jsonl_rate passes if the file gained min_per_interval rows since the last check. Health only.
• heartbeat passes if the path changed since the last check. Health only.
• tensor_changed passes if the bytes at the path differ from the last check. Health only.
• gpu_util passes if every GPU is at min_pct or above. Health only.
• gemm_sane passes if every GPU computes a finite bf16 matrix multiplication. Use it with on_fail: "reschedule" to move off a bad GPU. Health only.
• http passes if GET :port/path returns a 2xx status. Health only.

The platform runs each check from outside your container. You install nothing in your image, and a check works when your job's own environment is the part that is broken.

Use exec only when no other type can state your assertion. Exit code 1 means your check ran and the assertion is false. A command that cannot run, such as a missing binary, is a different result: the platform records it and takes no other action. An exec command that can only pass, such as true, is refused when you submit.

A failed exec check records the last 400 characters that your command printed. This is your own output, so your data policy controls it: if your organization turns off command output, the platform records the result and a note in place of the output. The check still runs and on_fail still applies.

The first check of an attempt has nothing to compare against, so a file_grows, heartbeat, jsonl_rate, or tensor_changed check passes and records the first measurement.

get_job reports the declaration, the number of checks that ran, the last result, and the failures. A job that a health check stopped gets the self_terminated state. A job whose success check could not run gets the check_errored state.

Preflight#

Preflight runs on every submission. It adds about 2 seconds and it is free. It has two checks, and each one is quiet when the data is not sufficient.

The cost check multiplies the node rate by the job timeout. If the result is more than the money that remains under your workspace cap or your organization cap, the platform refuses the job and shows the arithmetic. Without the refusal, the cap stops the same job in the middle of the run, after you spend the money. To submit a job that stops early, pass force.

BLOCK preflight/cost-cap: worst case $214.00 (h100-8 x 420 min) exceeds
  workspace remaining cap $150.00
  Override: force: true (this submission only) - raise the cap: set_workspace_cap

The disk check identifies the model your job downloads, resolves its size, and compares the size with the node's disk. If the model is larger, preflight warns and submits the job. Declare model to make the check exact. Otherwise the platform reads a --model flag, a model name in env, or a from_pretrained value from your command. If the platform cannot identify or size the model, the check says nothing.

gman job validate --chip h100 --count 8 --max-duration 7h \
  --model Qwen/Qwen3-VL-72B --command 'vllm serve' --checks ./checks.json
→ ok    preflight/cost-cap: worst case $214.00 fits the organization remaining cap $900.00
→ WARN  preflight/disk-model: Qwen/Qwen3-VL-72B resolves to ~145 GiB; node disk is 100 GiB

validate_job(
  chip: "h100", chip_count: 8, max_duration_minutes: 420,
  model: "Qwen/Qwen3-VL-72B", command: "vllm serve",
  checks: {health: {check: {type: "gemm_sane"}, on_fail: "reschedule"}})
→ {preflight: [...], would_submit: true}

• validate_job submits nothing and costs nothing. Use it in a loop while you write a job spec.
• force submits a job that a preflight check blocks. The platform records the override on the job.
• no_preflight skips both checks. The job record then says that it skipped them, so a later postmortem knows which checks did not run.

Checkpoints#

Every job gets a durable checkpoint slot. $GMN_CHECKPOINT_DIR points at a directory on the job's scratch disk. Write a consistent checkpoint there, then declare it with one credential-free call to the metadata service. The platform captures it, encrypted and checksummed, beside your running process.

# 1. write a consistent checkpoint into $GMN_CHECKPOINT_DIR, then declare it
curl -s -X POST -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/checkpoint"
# → 202 {"status": "accepted"}   ("coalesced" if one is already in flight;
#   retry a transient 503 in the first seconds of a run)

# 2. poll until the capture COMMITS
curl -s -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/job"
# → ... "checkpointed_at": "2026-07-30T23:57:12Z" ...

# 3. optional, for qualification: end this attempt and get it back
curl -s -X POST -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/yield"
# → 202 {"status": "yielding", "attempt": 1, "checkpointed_at": "..."}

# 4. the requeued attempt starts with $GMN_CHECKPOINT_DIR non-empty:
#    recompute your checksum there and compare across the two attempt logs
#    (get_job with attempt: 1, then attempt: 2)

• Each capture replaces the last. It is a slot, not a history. The slot holds up to 50 GiB by default, inside the 100 GiB default scratch quota. Both are limits.
• A capture is committed once checkpointed_at appears or advances on GET /v1/job and on get_job, next to checkpoint_bytes.
• Captures are limited to one per 5 minutes per job. A faster declare answers 429 with a Retry-After header. Nothing failed, and the last committed capture stands.
• Restore is automatic and free. When the platform ends an attempt (preemption, worker loss, a planned capacity boundary), the job requeues. The next attempt finds the slot's contents in $GMN_CHECKPOINT_DIR before your process starts. The resume contract is one line: if $GMN_CHECKPOINT_DIR is not empty, resume from it.
• Preemptions and planned boundary severs never consume max_restarts. Provider faults (worker death, allocation loss, GPU failure) draw on that budget, which is 1 by default and up to 3. A job past it fails.
• Exits your process produces itself (exit 0, any nonzero code, an OOM kill 137) are terminal and destroy the slot. No exit code requeues.
• Submit with resume: "checkpoint" for resumable placement. The job can be scheduled onto capacity windows shorter than its max_duration. A window lapse is a planned sever: your container gets a machine_retirement countdown on /v1/events, and the interruption bills only up to your last committed capture. A resumable job severed twice without a capture loses the relaxed placement, so checkpoint early. Once one job has checkpointed in your org, later jobs with the same image and the same command inherit resume: "checkpoint". get_job shows resume_source: "inferred" and the job it learned from. Pass resume: "none" to run a job as one attempt.
• POST /v1/yield proves restoration on demand. It refuses with 409 when no capture has committed, while a capture is in flight, or when the budget of 2 yields per job is spent. It refuses with 429 under the one-per-5-minutes limit. A yielded attempt bills its full span. The requeued job keeps its queue standing, can land on a different machine, and gets its own max_duration clock. get_job reports yields_used beside preempted_count.
• Every capture is checksummed. A corrupt slot is detected, cleared, and noted on the job. A restore infrastructure failure is an unbilled start failure. The slot is encrypted under a per-job key, erased at terminal state, and cannot be exported. For a durable copy, also write the checkpoint to $GMN_OUTPUT_DIR, or push it to org storage with the shard-and-manifest pattern.

Previews#

A preview is a look at a running job. Your process writes something small (a sample grid, a metrics file) into $GMN_OUTPUT_DIR and declares it. The platform captures it into the job's one preview slot, and get_job returns a short-lived download link. Look at it and decide: keep going, resubmit with new settings, or cancel_job now.

# from inside the job, once the file is fully written:
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -H 'Content-Type: application/json' \
  -d '{"path": "/output/samples/step-500.png"}' \
  "$GMN_METADATA_URL/v1/preview"
# → 202 {"status": "accepted"}   ("coalesced" if a capture is already running)

# from your agent: poll get_job and fetch the peek
get_job("job-8x2mf")
# → ... "preview": {"captured_at": "...", "size_bytes": 51200,
#        "download": {"url": "https://...", "expires_at": "..."}} ...

• The path must be inside $GMN_OUTPUT_DIR: a regular file or a small directory, which arrives as a tar. Declare it only after the bytes are fully written.
• Each capture replaces the last, up to 1 GiB. Captures are limited to one per minute per job.
• Declaring the same path while a capture runs joins it. Declaring a different path is refused and names the one in flight.
• A failed capture (too big, gone before the tar) reports its reason in preview.error on get_job. Your run continues, and the previous preview stays downloadable.
• The preview carries the attempt that took it. After a preemption, check it before you judge the current run by an earlier run's preview.
• Previews are free. The download stays available for 30 days after the last capture, including after the job ends. To keep every capture instead of replacing it, use samples.

Samples#

A sample is one small file a running job adds to a timeline. Previews replace. Samples append. A training job that saves a sample grid every N steps builds a series you can scroll through. If the job belongs to a mission, the mission page gets a Samples tab that renders the timeline.

# from inside the job, once the file is fully written:
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -H 'Content-Type: application/json' \
  -d '{"path": "/output/samples/step-500.png", "kind": "image", "label": "step 500"}' \
  "$GMN_METADATA_URL/v1/sample"
# → 202 {"status": "accepted"}

# from your agent: list the series, newest first (by job, or mission: for the whole mission)
list_samples(job: "job-8x2mf")
# → {"samples": [{"kind": "image", "label": "step 500", "captured_at": "...",
#      "download": {"url": "https://...", "expires_at": "..."}}, ...], "next": "..."}

• kind is one of image, audio, video, text, or json. Every kind renders on the mission page. If a sample cannot be shown, the tile offers the download.
• label is the caption, up to 120 characters.
• Each sample is one regular file of up to 64 MiB. A directory is refused. Write a grid as one image, and declare directories to /v1/preview instead.
• The path must be inside $GMN_OUTPUT_DIR. Declare it after the bytes are fully written.
• Captures are limited to one per 30 seconds per job. A declare while another capture is in flight is refused, not merged. Redeclare after it lands.
• A failed capture reports its reason in samples.error on get_job and never ends your run. A job keeps up to 256 samples. Past that, captures are refused with the reason in the same place.
• Samples cannot change once captured. They stay downloadable for 30 days. The links list_samples returns live for minutes. Fetch what you list, and re-list for fresh links.

Sweeps#

A sweep is up to 256 variants of one job in one call: one build, N runs.

gman api post /jobs/sweeps --data '{
  "shared": {"chip": "h100", "context": {...}},
  "variants": [{"env": {"LR": "1e-4"}}, {"env": {"LR": "3e-4"}}, {"env": {"LR": "1e-3"}}],
  "label": "lr-sweep",
  "idempotency_key": "lr-sweep-1"}'

submit_jobs(
  shared: {chip: "h100", context: {...}},
  variants: [{env: {LR: "1e-4"}}, {env: {LR: "3e-4"}}, {env: {LR: "1e-3"}}],
  label: "lr-sweep",
  idempotency_key: "lr-sweep-1")

Validation is all or nothing. list_jobs(label: ..., summary: true) is the comparison table when the variants finish.

Task sets#

A job can carry a list of tasks instead of one command. Your command runs once per machine and asks for work as it goes. One container start and one image pull cover thousands of tasks. A lost machine costs the tasks in flight, not the whole run.

gman job submit --chip h100 --context ./render \
  --command 'python render.py' --tasks ./prompts.jsonl --task-runners 8 \
  --label nightly-render --idempotency-key nightly-render-1

submit_job(
  chip: "h100",
  context: {...},
  command: "python render.py",
  tasks: [{prompt: "a red bicycle"}, {prompt: "a blue door"}, ...],
  label: "nightly-render",
  idempotency_key: "nightly-render-1")

Your command leases tasks from inside the container. There is no credential, because the platform knows which job you are.

# this runs INSIDE your container, not from your shell

import json, os, time, urllib.request

META = os.environ["GMN_METADATA_URL"]

def post(path, body=None):
    req = urllib.request.Request(
        META + path, method="POST",
        data=json.dumps(body).encode() if body else None,
        headers={"Metadata-Flavor": "givemeanode",
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return r.status, json.loads(r.read() or "{}")

while True:
    status, body = post("/v1/tasks/lease", {"count": 4})
    if status == 204:
        break                     # every task is done and the set is sealed
    if not body["tasks"]:
        time.sleep(5)             # the set is open, nothing claimable yet
        continue
    for task in body["tasks"]:
        data = render(task["params"]["prompt"])
        urllib.request.urlopen(
            urllib.request.Request(task["output_put_url"], method="PUT", data=data))
        post(f"/v1/tasks/{task['id']}/done")

• Upload each task's result to the output_put_url in the lease. Report done only after the upload returns. A task can run more than once, and a done report before the bytes land is the one way to get a wrong answer. Each task's URL is its own, so a second run overwrites its own output.
• Retry a lease that answers 503. The metadata service is briefly unavailable while your container starts, so the first poll of a run often gets one.
• Size tasks in minutes. If a machine goes away while a task is leased, the task returns to the pool and another machine takes it. Leases last 15 minutes by default. For a longer task, call POST /v1/tasks/{id}/extend before the deadline, or raise task_lease_minutes at submit.
• Report failures with POST /v1/tasks/{id}/fail. A task is tried up to task_max_attempts times (3 by default), then recorded as failed. A failed task does not fail the job. get_job lists which tasks failed and why. If you get a 409 that says you no longer hold a task, drop the result and lease again.
• We run a task set on more than one machine when the backlog warrants it, and stop them as it drains. You submit one job and watch one job. task_max_runners sets the ceiling. It is 4 by default and can go to 32. A set of 20 tasks never starts 32 machines. The ceiling bounds spend and reserves nothing.
• Your command must be safe to run on several machines at once. It is, if it only touches the tasks it leased. If you cancel the job, every machine on the set stops.
• You pay for the GPU-minutes your job runs, at the job rate. Several machines on one set means several machines billing. If we cut a run short, you do not pay for the tasks that were in flight.

gman job get job-8x2mf

get_job(job_id: "job-8x2mf")

{"state": "running",
 "tasks": {"total": 5000, "done": 4193, "failed": 2, "leased": 8,
           "pending": 797, "runners": 4, "max_runners": 8}}

Open task sets#

If you do not know every task up front, pass tasks_open: true at submit and add tasks with add_tasks. Your runner keeps leasing while the set is open. The job finishes when you call seal_tasks and the pool drains.

gman job submit --chip h100 --context ./render --command 'python render.py' \
  --tasks ./first-batch.jsonl --tasks-open \
  --label stream-render --idempotency-key stream-render-1
gman job add-tasks job-8x2mf --tasks ./next-batch.jsonl
gman job seal-tasks job-8x2mf
gman job tasks job-8x2mf        # progress, and the failed tasks with their reasons

submit_job(chip: "h100", context: {...}, command: "python render.py",
           tasks: [{prompt: "the first batch"}], tasks_open: true,
           label: "stream-render", idempotency_key: "stream-render-1")
add_tasks(job_id: "job-8x2mf", tasks: [{prompt: "..."}, ...])
seal_tasks(job_id: "job-8x2mf")

Model cache#

A sweep over one base model does not need to download it N times. hf_cache: true on submit_job, or once in a sweep's shared, mounts a cache shared by your org's jobs on the same machine and points HF_HUB_CACHE at it. The first job on each machine pays the download. The rest load the weights from local disk. It is free, off by default, and never shared across orgs.

gman job submit --chip h100 --context ./train --hf-cache \
  --command 'python train_lora.py' --idempotency-key flux-r16

submit_jobs(
  shared: {chip: "h100", context: {...}, hf_cache: true},
  variants: [{env: {RANK: "4"}}, {env: {RANK: "8"}}, ...],
  label: "flux-rank-sweep",
  idempotency_key: "flux-rank-sweep-1")

• The cache is best effort. It is bounded per machine (100 GiB per org by default). When a machine runs low on disk, the coldest caches are cleared first. Treat a warm cache as a speedup, never a guarantee.
• When your org's cache on a machine fills, the models used longest ago are evicted, so each job starts with at least 32 GiB of room. A job that gets no cache downloads as usual. A single download larger than the free room can fill the cache mid-pull and fail. For unusually large models, submit without the flag.
• Files written there stay for your org's later jobs. Everything else a job writes is destroyed at the end. Files that are not hub downloads count against the same room. If they fill the cache past what evicting models can free, the whole cache is cleared and rebuilds cold.
• Do not set HF_HUB_CACHE in env alongside hf_cache. The submit is refused, because the platform sets that variable.
• Gated repos still need a token on the first pull, sent in env as usual.

Uploads#

Inline contexts cap at 4 MiB. Past that, presign an upload:

gman job submit --chip h100 --context ./ctx --command '...'
# one command: tars ./ctx deterministically, uploads (create_context →
# finalize_context), then submits - any size up to 10 GiB. A pre-built
# tar.zst rides the same dance by hand via gman api post /contexts.

create_context(sha256: "...", size_bytes: 123456789)  → upload URL
# PUT the tar to the URL, then:
finalize_context(context_id)
submit_job(chip: "h100", command: "...", context_id: "...", idempotency_key: "...")

The same upload can land on a node. Use this for a dataset or eval set that exists only in your sandbox: too big for write_file, and not fetchable from inside the node.

gman api get /contexts/ctx_...                    # → download URL
gman run tulip -d -- curl -o ~/data.tar.zst '<url>'

get_context(context_id)                          → download URL
run_command(node, "curl -o ~/data.tar.zst '<url>'", detach: true)

• A finalized upload is reusable for about 30 days. Curl it onto five nodes, or build a job image from it. It is the same object.
• The upload and download URLs point at fly.storage.tigris.dev, not the API hostname. If your sandbox filters egress, see Allowlists.
• list_contexts shows what you have uploaded: ids, sizes, and when each stops being fetchable.

To end an upload early, delete it:

gman api get /contexts                            # → your uploads
gman api delete /contexts/ctx_...                 # → status: deleted
gman api get /contexts/ctx_...                    # → bytes_deleted_at, once confirmed

list_contexts()
delete_context(context_id)                       → status: "deleted"
get_context(context_id)                          → bytes_deleted_at (the confirmation)

• Deletion is immediate and cannot be undone. No download URL is minted for that context again. The stored bytes are removed within about a minute, and get_context gains bytes_deleted_at once the store confirms it.
• Deletion refuses while a job that still needs the bytes is in flight, and names the jobs.
• This deletes the uploaded context only. Images built from it are separate objects and are not deleted. Job logs and artifacts keep their own 30-day lifecycle. The audit record of the upload and its deletion is kept.

Sandboxes#

A sandbox is a small isolated virtual machine that runs shell commands for your agent. Code your model wrote can run and fail in one without touching anything you care about.

Quick start#

gman sandbox create
# → sbx-0a1b2c3d4e5f, plus where it came from
gman sandbox exec sbx-0a1b2c3d4e5f -- python3 -c 'print(6*7)'
# → 42, and the command's own exit code becomes gman's
gman sandbox rm sbx-0a1b2c3d4e5f

create_sandbox()
sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 -c 'print(6*7)'")
delete_sandbox("sbx-0a1b2c3d4e5f")

• A sandbox is handed over running. It bills for its RAM until you delete it.
• The first sandbox you create for an image boots it. A later create with the same image and size forks the snapshot the first one left, in tens of milliseconds. The response says which happened in reused_snapshot. A setup command makes every create boot again, because the snapshot has to carry your setup. For more than one sample, fork.
• Sometimes a create skips the boot and the fork. We keep sandboxes for the default image already started, and a matching create claims one. The response says pooled: true, and billing starts when you receive it.
• For the most common images we also keep a warm base of our own on each machine. A matching create forks it, and the response says shared_base: true. A pooled response carries it too. The base belongs to us, so the response carries no snapshot for it, and there is nothing for you to fork. To make many copies at once, bake your own snapshot.
• Treat a sandbox id as an opaque string. Pass back exactly what we gave you. Ids are about eighty characters today, and the length can change. The examples use a short id so the commands fit on one line.

Fork#

Bake the environment once. Fork it once per sample. Setup runs one time in the bake, and each fork resumes that exact state, memory and running processes included, in tens of milliseconds. Forks are free.

gman sandbox bake --image sbx-task --ram-gib 2 --warm-file ./setup.sh
# setup.sh writes /workspace/grade.py and imports numpy, so every fork has both
# → env-1f4c2ab90d3e, baked in 41.2s
gman sandbox fork env-1f4c2ab90d3e --count 64
# → 64 sandbox ids, one per line, ready in 690ms
gman sandbox exec --batch rollouts.jsonl
# one line per sandbox, each with its own seed:
#   {"sandbox": "sbx-0a1b2c3d4e5f", "cmd": "python3 /workspace/grade.py 0"}
# → results in request order: exit_code, stdout, stderr, duration_ms
gman sandbox rm sbx-0a1b2c3d4e5f sbx-6a7b8c9d0e1f
# → deleted 2 of 2, the whole batch in one call

create_sandbox_env(image: "sbx-task", ram_gib: 2,
  warm: "mkdir -p /workspace && ...write grade.py... && python3 -c 'import numpy'")
fork_sandbox(from: "env-1f4c2ab90d3e", count: 64)
sandbox_exec(execs: [
  {sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 /workspace/grade.py 0"},
  {sandbox: "sbx-6a7b8c9d0e1f", cmd: "python3 /workspace/grade.py 1"}])
delete_sandbox(sandboxes: ["sbx-0a1b2c3d4e5f", "sbx-6a7b8c9d0e1f"])

• Sixty-four rollouts are one bake, one fork call, one exec call, and one delete call. If you boot one sandbox per sample instead, the response says so in a prefix_note that names the snapshot you could have forked.
• Fork, exec, and delete all take arrays. Use them that way. At this width, sixty-four round trips cost more than the sandboxes.
• Delete results come back in request order, one per entry, each with deleted: true or false. A false entry is a sandbox that is still running and still billing, because the id was not yours or the host refused the teardown. The call still succeeds. Check the entries, not only the status. gman sandbox rm reads them for you and exits nonzero when any entry came back undeleted.

Images#

Name an image we do not have, and the error lists the ones we do. You can also bake from a container image of your own.

A sandbox holds 2 GiB of RAM by default, which is also the ceiling a workspace starts with. Ask and we raise it, up to the 32 GiB platform maximum. list_limits shows yours. Every fork inherits the size its snapshot was baked at.

Sizes#

The default sandbox has one vCPU, which fits an agent that does one thing at a time. A parallel build, a dependency install, or a typecheck across a monorepo needs more. size names a bigger machine.

• There is one rate. The table says what each size counts as against it. A size that wants cores and little memory pays for the cores. A core comes with 4 GiB, so above the default the billed number and the held number are the same.
• The network column is a ceiling, not a reservation. Every sandbox on a host shares that host's uplink. A bigger size raises your share in proportion. This matters most for a cold dependency install.
• size and ram_gib are alternatives. Pass one. Your ceiling is the sandbox_vcpus limit, which starts at 4. sandbox-md needs no permission. sandbox-lg and sandbox-xl need a raise.
• A size is fixed when the environment is baked. Every fork inherits it. Bake the shape you want to fork.
• The scheduler weights the cores to you. A busy host can still slow a large sandbox down. While a sandbox is parked, you do not pay for its cores.

On sandbox-lg and sandbox-xl, /tmp is memory, not disk. Scratch writes and deletes are much faster. The capacity is half the shape's memory: 16 GiB on sandbox-lg and 32 GiB on sandbox-xl. /tmp competes with your process for the same memory, so a job that fills both can be OOM-killed where it would have got “no space left on device”. If you write tens of GiB of scratch, put it under your working directory, which stays on disk. sandbox-sm and sandbox-md keep a disk-backed /tmp.

npm installs use a cache on the machine. It is a pull-through cache of the public registry, shared by everything on that host. Your install is still a real download of every package your lockfile names. You do not configure it. If the cache is unavailable, the sandbox uses registry.npmjs.org directly. We set the default registry only, by writing ~/.npmrc when there is no file there. A home-directory .npmrc of your own is never overwritten. Scoped registry lines are unaffected.

One case to know. Your project can have an auth-only .npmrc: an _authToken line for registry.npmjs.org and no registry= line. npm keys that token to the registry URL it resolves, so it does not send the token to the cache, and the install fails. Add an explicit registry= line to that .npmrc, or put one in ~/.npmrc.

package_cache: true serves npm and bun installs from a cache on the machine instead of from registry.npmjs.org. It is off by default, because it rewrites your lockfile. The cache serves tarballs from a loopback address, so bun records that address for every package. A bun.lock produced under it differs from the one you committed, and a --frozen-lockfile check fails. Turn it on for throwaway installs. Leave it off for anything whose lockfile you keep.

Custom image#

If the environment you want already exists as a container image, bake from the image instead of rebuilding its setup as a warm script.

gman sandbox bake --from-image ghcr.io/acme/task@sha256:8f43...c21a
gman sandbox fork env-7fa2 --count 64

create_sandbox_env(from_image: "ghcr.io/acme/task@sha256:8f43...c21a")
                                    → {snapshot: "env-7fa2", ram_gib: 2}
fork_sandbox(from: "env-7fa2", count: 64)

• We convert the image to a root filesystem the first time you name it. For a large image that takes a few minutes. Every later bake of the same image reuses that work. The env is an ordinary sandbox env, so fork_sandbox, sandbox_exec, and snapshots behave the same way.
• Name the image by digest, not by tag. A tag can move to different bytes later, and we cache the conversion under the image's identity. Read the digest with docker buildx imagetools inspect or crane digest. Write the registry host in full.
• A public image needs nothing else. For a private image, create a registry connection that covers the reference.
• We honour the environment variables and the working directory the image declares. We do not run its entrypoint and we do not switch to its user. Commands run as root through your own shell. If the image expects its entrypoint to start something, put that command in warm.
• The image must be linux/amd64, and there is a ceiling on how large it can unpack to. A workspace can convert 50 different images an hour by default (sandbox_converted_images_per_hour). A repeat conversion of the same image does not count.

gman connection create acme-ghcr --kind registry \
  --scope 'ghcr.io/acme/*' --username acme-bot --material-stdin < token.txt

# Registry connections are created from the CLI, so the credential
# never rides an MCP argument.

Network#

A sandbox can reach the internet. Your task code can git clone, pip install, and call an API. DNS works. A sandbox cannot reach anything private: your nodes, your networks, this API, and every other sandbox are unreachable from inside one. Each sandbox gets up to one gigabit per second.

Inside a sandbox, a server your code starts is reachable at 127.0.0.1 and localhost from another command in the same sandbox. Nothing outside the sandbox can reach it.

Pass egress: "none" for a sandbox with no network at all. Use this when a rollout must be hermetic, or when code your model wrote must have no way out. Everything such a sandbox needs must be baked in.

gman sandbox bake --image sbx-task --egress none --warm-file ./setup.sh
# → env-1f4c2ab90d3e, and every fork of it has no network
gman sandbox create --egress none

create_sandbox_env(image: "sbx-task", egress: "none",
                    warm: "pip install -q -r requirements.txt")
create_sandbox(egress: "none")

The choice is made when the env is baked. Every fork inherits it. A fork cannot turn the network on or off. Bake a second env for the other posture. The response says which one you got in egress.

Bake in anything a fork needs on every sample. A bake runs your setup once, and every fork inherits the result. A model or a large dataset per sample is slow to download anywhere. That work belongs on a node or in a batch job, with the sandbox grading what comes back.

Preview URL#

A server inside a sandbox can be reached from outside at a public HTTPS URL. This is the link a coding agent hands to a person to look at a dev server.

gman sandbox exec sbx-0a1b2c3d4e5f -- sh -c 'cd app && (npm run dev > /tmp/dev.log 2>&1 &)'
gman sandbox expose sbx-0a1b2c3d4e5f 3000
# → https://sbe-9k2fq-vx7t3m8dk4qwrz2n.givemeanode.io
gman sandbox unexpose sbx-0a1b2c3d4e5f 3000

sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f",
  cmd: "cd app && (npm run dev > /tmp/dev.log 2>&1 &)")
expose_sandbox_port(sandbox: "sbx-0a1b2c3d4e5f", port: 3000)
unexpose_sandbox_port(sandbox: "sbx-0a1b2c3d4e5f", port: 3000)

• Ports 1024 through 65535 can be exposed. The server must listen on the sandbox's own 127.0.0.1, which is where a dev server listens by default.
• This works on a sandbox baked with egress: "none" too. Nothing inside has to reach out for the URL to work.
• The server must survive the command that started it. An exec waits for its command's output to finish. A plain npm run dev & holds the exec open until its deadline and then kills the server with it. Redirect the output, as in the example, and the exec returns at once.
• Anyone with the URL can reach the port. Treat it like a password. The hostname carries a secret long enough that nobody guesses it. Add a bearer token on top with auth: "bearer". The token is in that response once.
• Endpoints expire after 24 hours by default. list_limits shows how many you can have at a time. Exposing the same port again returns the same URL with a fresh expiry, so ask for the URL whenever you need it.
• The URL belongs to the sandbox it was minted for. Deleting the sandbox stops the URL. A fork does not inherit it, because a fork is a new sandbox.
• Streaming responses, server-sent events, and WebSocket connections pass through. A request goes through our edge to your sandbox, which suits a dev server or an API under test, not production traffic.

Snapshots#

gman sandbox exec sbx-0a1b2c3d4e5f -- python3 /workspace/run.py --until 278
gman sandbox snapshot sbx-0a1b2c3d4e5f
# → env-77c1d0e9ab24, captured while the sandbox keeps running
gman sandbox fork env-77c1d0e9ab24 --count 8

sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 /workspace/run.py --until 278")
snapshot_sandbox("sbx-0a1b2c3d4e5f")   # → snapshot: "env-77c1d0e9ab24"
fork_sandbox(from: "env-77c1d0e9ab24", count: 8)
# eight continuations from one identical state

• snapshot_sandbox captures a running sandbox without stopping it. Every branch starts from finished work.
• Snapshots are stored deduplicated against the one they came from. Every snapshot also gets an encrypted copy off its host, so a lost host cannot lose your snapshot.
• A snapshot you have not forked in a while can move to cheaper cold storage. The next fork brings it back, and that response carries a rehydrate_ms with the wait. Forks after it are fast again.
• Snapshots live until you delete them, unless you set an expiry. Pass expires_after to create_sandbox_env or snapshot_sandbox, with a value such as "24h" or "7d" and a minimum of one hour. Set or clear it later with set_sandbox_snapshot_expiry. "never" is the default. An expired snapshot deletes itself.
• delete_sandbox_snapshot ends the storage charge and stops new forks. Sandboxes already forked keep running.

Voids#

{"sandbox": "sbx-0a1b2c3d4e5f",
 "outcome": {"outcome": "void", "cause": "uffd_handler_died"},
 "billed": false}

• A void means something broke on our side while your command ran. There is no verdict to score, and its time is excluded from your bill. Retry it on a fresh fork. Do not record a low reward, because a void scored as a failure teaches your model something that never happened.
• cancelled: true means your own delete_sandbox won a race against the exec. The verdict is unknowable. The time is billed, and the sandbox is gone.
• Exit code 137 with a bare Killed line usually means the sandbox's memory budget. When the OOM killer fired, the result carries an oom_kills count and a note. That is your command's own failure. It bills, it scores as a real low reward, and an unchanged retry dies the same way. Bake with more ram_gib, or use less memory. get_sandbox_stats counts these under oom_execs.
• Your policy's own failures come back as ordinary results with a nonzero exit code. Those are real signal.

Cost#

• A sandbox bills for the RAM it holds, per minute, while it runs. Forking is free. Baking bills as ordinary sandbox time.
• A stored snapshot bills per GiB-month of its deduplicated bytes while it is ready to fork, and at the cheaper cold rate once it has moved to cold storage. A snapshot bills one of those two rates at a time. The rates are on the front page.
• Idle sandboxes park themselves. After a second and a half without a command, the sandbox hands back the memory it is not using. You stop paying the per-minute RAM rate on what it gave back, and pay the per-GiB-month disk rate on what it holds. There is nothing to call.
• Parking never costs state. A sandbox gives back only memory its guest can spare. Processes keep running. Files, sockets, and warm caches stay. The next command wakes it in tens of milliseconds. A command that re-reads a lot of cache can take a little over a tenth of a second.
• A sandbox that is idle with nothing resident costs almost nothing while it waits. One that holds a loaded model or a running server keeps that memory and keeps paying for it. The lever is what a rollout keeps resident.
• Delete each fork as its work is scored. Keep anything you want to return to as a snapshot, not an idle sandbox. get_usage carries a sandboxes line. get_sandbox_stats finds the forks a loop forgot.

Limits#

Commands inside one sandbox run one at a time. Different sandboxes run in parallel. The sandbox endpoints have their own request budget, separate from the rest of the API, so an exec batch across a thousand sandboxes is one request. list_limits shows the values that apply to you. We raise them on request. See Help.

Rollouts#

A rollout is one episode of a policy in an environment. It starts from a state, runs under a judge, and ends in a verdict. You bring the environment and one spec for each episode. We record three things for each rollout. We record its identity, computed from what it is. We record its lineage, when it starts from a state that another rollout declared. We record its verdict, with a stamp that says who declared it and from where. We do not read your spec or your state. Rollouts run on the task sets and sandboxes you already use.

The rollout tools are on for every organization. If your agent does not see them in its tool list, reconnect it, then ask with file_ticket.

Getting started#

Three commands take you from one spec to a scored report. This example uses mjx-rollout, a container that we ship. It runs each episode and it declares each verdict, so you write no runner code.

# 1. Submit one spec over eight seeds. A submit never blocks.
gman rollout submit --environment mjx-rollout \
  --spec '{"scene": {"builtin": "ball-to-goal"},
    "horizon_steps": 100,
    "init": {"qvel": [1.2, 0, 0, 0, 0, 0], "qvel_noise": 0.05},
    "judge": {"body": "ball", "center": [1, 0, 0.1], "radius": 0.5}}' \
  --seeds 0..8 --judge goal-region@v1 \
  --evaluation hello-rollouts --idempotency-key hello-rollouts
# → evaluation hello-rollouts: 8 rollouts
#     8 queued as task set job-7c1kd

# 2. Watch. Each row goes from queued to running to scored.
gman rollout ls --evaluation hello-rollouts
# → ID        STATUS  SEED  SCORE  PASS  JUDGE           NOTE  CREATED
#   ro-4k9f2  scored  0     1.0    yes   goal-region@v1  -     2026-09-03T18:04:11
#   ro-4k9f3  scored  1     0.0    no    goal-region@v1  -     2026-09-03T18:04:11
#   ...

# 3. Read the scores. "final": true means each rollout has finished.
gman evaluation report hello-rollouts
# → {"evaluation": "hello-rollouts", "final": true,
#    "rollouts": {"total": 8, "scored": 8, "failed": 0, "void": 0, "memoized": 0},
#    "environments": [{"all": {"n": 8, "mean_score": 0.625, "pass_rate": 0.625}}],
#    ...}

submit_rollouts(
  environment: "mjx-rollout",
  spec: {scene: {builtin: "ball-to-goal"}, horizon_steps: 100,
         init: {qvel: [1.2, 0, 0, 0, 0, 0], qvel_noise: 0.05},
         judge: {body: "ball", center: [1, 0, 0.1], radius: 0.5}},
  seeds: "0..8", judge: "goal-region@v1",
  evaluation: "hello-rollouts", idempotency_key: "hello-rollouts")

list_rollouts(evaluation: "hello-rollouts")
get_evaluation_report(evaluation: "hello-rollouts")

• One spec and eight seeds make eight rollouts. The range 0..8 gives seeds 0 to 7. See Submit.
• A spec is a JSON value. We give it to the environment unchanged. The fields above are the ones mjx-rollout reads. Your own environment reads its own fields.
• The default profile is eval, so memoization is on. Submit the same eight rollouts again and each one returns its stored verdict. It does not run again and it costs nothing. See Memoization.
• To run your own environment, name your image at a digest or bake a sandbox snapshot. Your code then declares the verdicts. See Environments and Verdicts.

Environments#

"environment": {"kind": "task",    "env": "mjx-rollout"}
"environment": {"kind": "task",    "env": "ghcr.io/acme/my-sim@sha256:41c8..."}
"environment": {"kind": "task",    "env": "openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4..."}
"environment": {"kind": "sandbox", "env": "env-8x2mf3a9c1d4"}

There are two kinds of environment. A task environment is a container that you run. It starts once on each machine, leases episodes, runs them, and declares their verdicts from inside. Name your own image at a digest, or name one of the two containers we ship. mjx-rollout runs MuJoCo MJX scenes in batches on a GPU and grades them with its goal-region judge. openenv-rollout runs an OpenEnv environment image at a digest. See OpenEnv.

A sandbox environment is a snapshot that you baked with create_sandbox_env. Each rollout is one fork of that snapshot. Your harness drives the fork with sandbox_exec and declares the verdict from outside. Use this kind for code that your model wrote, for software engineering agents, and for tool use.

Name an image at a digest. A tag can move, so we refuse a bare tag. The kind sets where an episode runs and who can declare its verdict. It does not change how you submit, read, or report rollouts.

Submit#

gman rollout submit \
  --environment ghcr.io/acme/mjx-reach@sha256:41c8... \
  --specs ./episodes.jsonl --seeds 0..8 \
  --judge reach-goal@v1 --profile eval \
  --chip h100 --runners 8 \
  --evaluation ckpt-eval-9 --mission policy-eval \
  --idempotency-key ckpt-eval-9
# → evaluation ckpt-eval-9: 49,152 rollouts
#   12,808 memoized from ckpt-eval-8 (same identity); unbilled, not re-run
#   36,344 queued as job-7c1kd

submit_rollouts(
  environment: "ghcr.io/acme/mjx-reach@sha256:41c8...",
  # each line of the manifest: the spec your container reads, plus labels
  specs: [{spec: {scene: "reach-v2", body: "hand", target: [0.3, 0.1, 0.4],
                  checkpoint: "acme/reach-ckpt41"},
           labels: {policy: "ckpt41"}}, ...],
  seeds: "0..8",
  judge: "reach-goal@v1", profile: "eval",
  chip: "h100", runners: 8,
  evaluation: "ckpt-eval-9", mission: "policy-eval",
  idempotency_key: "ckpt-eval-9")

A submit makes one rollout for each spec and each seed, under one judge. A spec is a JSON value. We give it to your environment unchanged. The seed range 0..8 gives seeds 0 to 7. In a --specs file, each line is a spec, or {"spec": ..., "labels": {...}, "seed": n} when a spec needs its own labels. Labels group rows in a report. They do not change the identity of a rollout.

The profile sets the defaults. eval is the default. It turns memoization on and stamps each verdict with the reproducibility class you declared. Use it for benchmarks and regression suites. train turns memoization off. A memoized training sample reduces the diversity of your samples, so we refuse train with memoization on. You can set each default yourself.

Queued time is free. A backlog of rollouts is one of the inputs that starts more machines, so do not resubmit to go faster. Give each submit an idempotency key. A retry with the same key returns the first response and does not run a second set.

Task runner#

# lease: each task carries a rollout block; specs arrive once, by hash
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -d '{"count": 32}' "$GMN_METADATA_URL/v1/tasks/lease"

# declare: one call per batch; a verdict completes its task
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -d '{"verdicts": [{"rollout": "ro-4k9f2", "score": 1.0, "pass": true, "basis": "environment"},
                    {"rollout": "ro-4k9f3", "score": 0.0, "pass": false, "basis": "environment"}]}' \
  "$GMN_METADATA_URL/v1/rollout/verdicts"

# an episode that failed on your side
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -d '{"reason": "physics diverged"}' "$GMN_METADATA_URL/v1/tasks/t-8123/fail"

# A task runner works from inside its container over HTTP.
# The MCP tools are for the harness outside; see Verdicts below.
get_job(job: "job-7c1kd")   # progress of the runner job

{"tasks": [{"id": "t-8123", "params": {"rollout": "ro-4k9f2"}, "output_put_url": "...",
            "rollout": {"id": "ro-4k9f2", "identity": "blake3:...", "seed": 3,
                        "spec_hash": "blake3:9f2a...", "judge": "reach-goal@v1",
                        "policy": "", "repro_class": "seeded-stochastic",
                        "parent_state": {"id": "st-91mm2", "digest": "blake3:...",
                                         "bytes": 2148, "get_url": "https://..."},
                        "group": "grp-a41"}}],
 "specs": {"blake3:9f2a...": {"scene": "reach-v2", "body": "hand", "target": [0.3, 0.1, 0.4],
                              "checkpoint": "acme/reach-ckpt41"}},
 "lease_expires_at": "..."}

A task runner uses the same loop as a task set runner. Each leased task carries a rollout block with the rollout id, the seed, the judge, the policy, the reproducibility class, the hash of its spec, and the parent state to start from. The lease response carries a specs map with each distinct spec once, keyed by that hash. A lease of thousands of seeds over a few scenes stays small.

Declare verdicts in batches. A verdict completes its task. If an episode fails on your side, report the task as failed. The rollout is recorded as failed with a null score. It is billed and it is not memoized. The metadata service allows 5 requests per second for each container, with a burst of 10. A 429 answer carries retry_after_secs. Wait that long and send the same request again.

Sandbox#

gman rollout submit --environment sandbox:env-8x2mf3a9c1d4 \
  --spec ./task-1443.json --group 16 --seeds 0..16 \
  --judge swe-tests@v3 --profile train \
  --idempotency-key task-1443-step-40
# → 16 sandboxes forked from env-8x2mf3a9c1d4; group grp-a41
#   rollouts: [{rollout: "ro-9d41k", sandbox: "sbx-...", deadline: "..."}, ...]

submit_rollouts(
  environment: {kind: "sandbox", env: "env-8x2mf3a9c1d4"},
  spec: {task: "swe-py-1443", checkpoint: "acme/agent-ckpt41"},
  group: 16, seeds: "0..16",
  judge: "swe-tests@v3", profile: "train",
  idempotency_key: "task-1443-step-40")

# the response lists one entry per rollout, in the order you asked:
for r in resp.rollouts:
    if r.sandbox is None: continue      # memoized: the verdict is already in the row
    sandbox_exec(sandbox: r.sandbox, cmd: "...")
    declare_verdict(rollout: r.rollout, score: ..., basis: "harness")

A sandbox submit forks one sandbox for each rollout. The response lists one entry for each rollout, in the order you asked. A memoized entry has no sandbox. Its verdict is already in the row, so skip it. Drive each other sandbox with sandbox_exec, then declare its verdict from outside.

Under eval, a sandbox submit with memoization on needs --policy with a label for the model that drives the sandbox. The model is outside your environment, spec, seed, and judge. Without the label, the next checkpoint gets the verdicts of this one. The rule is simple. What we cannot see, we cannot tell apart. If a thing changes the result and is not in your environment, spec, seed, judge, or policy, two different rollouts get one identity. A URL in a spec is not a policy, because the model behind a URL can change.

OpenEnv#

gman rollout submit \
  --environment openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4... \
  --specs ./tasks.jsonl --seeds 0..4 --policy ckpt41 \
  --judge upstream:rubric@sha256:2ab4... --profile eval \
  --evaluation swe-mini-ckpt41 --idempotency-key swe-mini-ckpt41
# → 8,192 rollouts; the environment runs on sandbox forks
#   1,847 memoized (the baseline and control arms); only changed identities run

submit_rollouts(
  environment: "openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4...",
  specs: [...], seeds: "0..4", policy: "ckpt41",
  judge: "upstream:rubric@sha256:2ab4...", profile: "eval",
  env: {POLICY_API_KEY: {secret: "policy-key"}},
  evaluation: "swe-mini-ckpt41", idempotency_key: "swe-mini-ckpt41")

An OpenEnv environment runs from its image at a digest. The environment runs in a sandbox forked for each rollout. The adapter calls the policy endpoint that your spec names. Put the key for that endpoint in env on the submit as a secret reference. Only the adapter sees it. The sandbox that runs the environment does not.

The reward comes from the code of the environment, in the same sandbox as the agent. We stamp these verdicts basis: environment and mark them as a value the policy could influence. If your rubric matters, run it in your harness. That verdict is stamped basis: harness.

Verdicts#

gman rollout verdict ro-9d41k --score 1.0 --pass --basis harness --data ./verdict.json

# a batch, one result per item
gman rollout verdicts --file ./verdicts.jsonl

declare_verdict(rollout: "ro-9d41k", score: 1.0, pass: true, basis: "harness",
                data: {tests_passed: 41, tests_total: 41})

# or a batch, with one result per item:
declare_verdict(verdicts: [{rollout: "ro-9d41k", score: 1.0, pass: true},
                           {rollout: "ro-9d41m", score: 0.0, pass: false}])

A verdict is a small JSON object that you declare against a rollout id. You give a score. Use null for a scoring gap. reward is accepted as an alias. You can add pass, a basis, and a data body of up to 16 KiB. We store data and return it. We do not read it. We stamp the judge, the way the verdict arrived, who sent it, the reproducibility class, and the time. A body that sets those fields itself is refused.

A task environment declares from inside its container, with no credential. See Task runner. A sandbox environment runs untrusted output. Nothing inside a sandbox can declare a verdict. The code under test cannot grade itself. The harness outside the sandbox declares, from a job, from an agent over MCP, or from a script with a workspace token.

A verdict is written once. A second declaration for the same rollout answers with the first. A retry of the same declaration after a dropped connection changes nothing. A batch returns one result for each item. One item that you no longer hold does not discard the rest. A judge fix is a new judge version and a new run.

Never run gman with a token inside a sandbox. A sandbox is for untrusted code. We refuse a verdict declared from inside a sandbox, and we treat the token as leaked. We refuse each write it attempts until we clear it. Rotate the token, then file a ticket to have it cleared. A harness that runs inside a sandbox cannot declare verdicts. Let your policy write its result to the sandbox filesystem. Let the harness outside read it and grade it.

States#

gman rollout state ro-9d41k --label step-40 --expires-after 30d
# → st-8821a (a sandbox snapshot; it bills as one until it expires)
gman rollout states ro-9d41k

# task kind, from INSIDE your container: declare a file under $GMN_OUTPUT_DIR
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
  -d '{"rollout": "ro-4k9f2", "label": "t=8.0s", "path": "states/ro-4k9f2-8.bin"}' \
  "$GMN_METADATA_URL/v1/rollout/state"

declare_rollout_state(rollout: "ro-9d41k", label: "step-40", expires_after: "30d")
list_rollout_states(rollout: "ro-9d41k")

A running rollout can declare a state. A state is an opaque snapshot that other rollouts can start from. For a task environment, a state is a file that your runner writes, for example a MuJoCo position and velocity array. For a sandbox environment, a state is a sandbox snapshot with memory, processes, and disk. The sandbox pauses for a moment while we take it. We digest the bytes. We do not parse them.

A state bills as storage until the expiry you set. One state for each rollout is one snapshot for each rollout on your bill. Set --state-expires-after when you submit a sweep that declares states.

Forks#

gman rollout fork st-8821a --branches 64 \
  --spec-patch ./recovery-variants.jsonl --group
# → 64 children from st-8821a; group grp-c7f2; restore quoted before they start

fork_rollout_state(state: "st-8821a", branches: 64,
                   specs: [{spec: {...}, labels: {variant: "retry"}}, ...], group: true)

A fork makes children in the environment of the parent. Each child starts from the parent state, with its own spec patch and seed. A sandbox fork runs on the machine that holds the snapshot. A task child prefers the machine that made the state. Otherwise it downloads the state, and a large state takes time to restore. If the children would start from a state over 256 MiB that has to move, the response quotes the measured restore time from recent forks and asks you to confirm. A fork bills the execution of its children and nothing else.

Groups#

gman rollout group grp-c7f2

get_rollout_group(group: "grp-c7f2")

A group is a set of rollouts that started from one state in one environment. We attest that, because group-relative training methods need it and cannot check it themselves. --group N on a submit or a fork makes the children and the group. The verdict of each member carries the group id. A group does not attest which policy drove its members or what their verdicts were. Group members are never memoized. A group under eval needs --no-memo.

Memoization#

gman rollout ls --evaluation ckpt-eval-9 --memoized
gman rollout qualify ro-4k9f2 --times 4
# → seeded-stochastic (scores equal, data differs); declare --repro-class seeded-stochastic

list_rollouts(evaluation: "ckpt-eval-9", memoized: true)
qualify_rollout(rollout: "ro-4k9f2", times: 4)

Under --profile eval, a rollout with an identity that already ran in your workspace returns the stored verdict. It does not run again and it is not billed. The identity comes from the environment digest, the digest of the runner container, the spec hash, the seed, the judge, the policy label, the parent state, and the reproducibility class. A new container build, an edited spec, or a new judge version is a new identity. Each memoized row names the original rollout, its evaluation, and its workspace in memo_of. The submit response counts the hits.

Memoization is per workspace. An organization can ask us to widen it to all of its workspaces. The report then says which workspace each hit came from. --no-memo skips the lookup for one submit. Use it for calibration runs that measure variance.

Only a declared verdict is memoized. A failed rollout, a void, a training rollout, a group member, and an unrepeatable rollout are never memoized. We stamp the reproducibility class you declare. Under eval the default is seeded-stochastic. We never stamp bit-exact unless you say so. gman rollout qualify runs an identity several times with memoization off, compares the verdicts, and tells you the class to declare. Live services and the live web are unrepeatable. Memoization does not apply to them.

A new checkpoint named in a spec is a new identity on each of its rollouts. It runs in full and bills in full. The arms with an unchanged identity come back memoized. That includes baselines, control checkpoints, the missing rollouts of a resumed suite, and a suite you made larger. Memoization does not carry a verdict from one checkpoint to the next.

Evaluations#

gman rollout submit --environment mjx-rollout --specs ./tasks.jsonl \
  --seeds 0..8 --judge goal-region@v1 --evaluation ckpt-eval-9 --open
gman rollout add ckpt-eval-9 --environment mjx-rollout \
  --specs ./more-tasks.jsonl --seeds 0..8 --judge goal-region@v1
gman evaluation get ckpt-eval-9
gman evaluation seal ckpt-eval-9
gman evaluation ls

submit_rollouts(..., evaluation: "ckpt-eval-9", open: true)
add_rollouts(evaluation: "ckpt-eval-9", specs: [...], seeds: "0..8")
get_evaluation(evaluation: "ckpt-eval-9")
seal_evaluation(evaluation: "ckpt-eval-9")
list_evaluations()

An evaluation is a named set of rollouts that you compare, under one judge. Name it on a submit to make it or to add to it. Submit with --open when you do not know each rollout in advance. Add rollouts with gman rollout add. Close it with gman evaluation seal. An identity already in the evaluation is refused. get_evaluation shows the counts by status, the judges, and whether the evaluation is final.

Report#

gman evaluation real ckpt-eval-9 --file ./trials.jsonl
gman evaluation report ckpt-eval-9 --by policy
# → rollouts: 49,152 total, 49,118 scored, 14 void, 12,808 memoized
#   reach-goal@v1, one environment digest, final: true, void rate 0.03%
#   by policy: ckpt41 mean 0.412 (stderr 0.007) ... ckpt40 mean 0.387 ...
#   real: rho 0.83 (Spearman) over 12 policies and 96 matched trials

submit_evaluation_real(evaluation: "ckpt-eval-9",
                       rows: [{labels: {policy: "ckpt41"}, outcome: 0.71}, ...])
get_evaluation_report(evaluation: "ckpt-eval-9", by: "policy")

We compute the report on request over the verdicts we recorded. It carries the counts by status, the void rate, one table for each environment digest, and an integrity block. Pass --by with a label key to split each table by that label. The integrity block says whether one judge graded each row, whether that judge is bound to code by a digest, how many rows were memoized and from which workspaces, and how many verdicts the environment graded itself. final is true when each member has finished.

A report refuses to average rows graded under different judges unless you pass --allow-mixed, and it names the mix. If two label values share an identity, the report refuses to compare them, because it would compare one number to itself. Environments always get one table each.

To compare with the real world, send rows of labels and a number. The report adds the Spearman correlation between mean score and mean outcome for each label group. It shows the matched and unmatched counts next to it, so a join that dropped half the data cannot print a confident number. From the CLI, gman evaluation real posts a JSONL file of rows.

Failures#

gman rollout ls --evaluation ckpt-eval-9 --failed
gman rollout get ro-9d41k

list_rollouts(evaluation: "ckpt-eval-9", failed: true)
get_rollout(rollout: "ro-9d41k")

A rollout has two ways to end without a score. If your side fails, for example a harness exit that is not zero or a judge error, the rollout is recorded as failed. It is billed and it is not memoized. Resubmit it and it runs again. If our side fails, for example a lost machine, a void sandbox, or a state that did not restore, the rollout is recorded as void. It is not billed. It is excluded from its evaluation and its group, and it is safe to run again. A task episode goes back to the pool by itself. A scoring gap is a null score, and that is real signal about your policy.

Each rollout has a deadline of 60 minutes by default. The clock starts when the rollout starts to run. Queued time does not count. A rollout still running at its deadline is recorded as failed and billed. Its sandbox is deleted, so a harness that dies cannot leave a sandbox that bills. A snapshot declared as a state stays.

Billing#

Rollouts add no new line to your bill. A task rollout costs the GPU minutes its runner uses, at the job rate. A sandbox rollout costs the RAM its sandbox holds, per minute. A state costs what a snapshot or a stored file costs. Queued time, voids, and memoized rollouts cost nothing.

Limits#

One submit is also bounded by the task-set size of 50,000. An evaluation at the ceiling is two submits, or a submit and an add. list_limits shows the values that apply to you. We raise them on request. See getting help.

Missions#

A mission groups the nodes, jobs, commands, and files that answer one question. It gives you a page to watch while your agent works. Ask your agent to open one before it starts. Everything it does then rolls up at givemeanode.com/missions/<name>. The front of the page is what your agent has said. The other tabs carry the cost so far, commands as they run, declared results, and charts.

gman api post /missions -d '{"name": "llama-serving", "title": "Serve Llama 70B under $2/hr"}'
gman node create --name tulip --chip 8xh100 --mission llama-serving
gman run tulip --mission llama-serving -- python bench.py
gman job submit --chip 8xh100 --context ./bench \
  --command 'python bench.py' --mission llama-serving

open_mission(name: "llama-serving", title: "Serve Llama 70B under $2/hr")
→ {name: "llama-serving", url: "https://givemeanode.com/missions/llama-serving", ...}

create_node(name: "tulip", chip: "8xh100", mission: "llama-serving")
run_command("tulip", "python bench.py", mission: "llama-serving")
submit_job(..., mission: "llama-serving")
finish_mission("llama-serving", result: {tokens_per_second: 8140, usd_per_hour: 1.84})

• An agent working under a mission gives you the URL early, so you can watch the work. finish_mission records the verdict on the page.
• The page is private to your organization. Members see missions in the workspaces they can act in. Org admins see all of them. The billing role sees every mission's receipt. A file embedded in an update becomes a share link anyone with the URL can fetch.
• Opening a mission by an existing name attaches to it. An agent that reconnects lands in the same mission by using the same name, and so do several agents in parallel. Names are permanent handles in your organization. get_mission returns the whole receipt in one call.
• Creating a node or submitting a job asks for a mission. If a call arrives without one, your agent is told to open a mission and try again. Scripts and CI are exempt. A service token or a gman command without a mission gets one named for the day, such as auto-2026-08-10-3f9ab2c1. Missions never cap or gate spending.
• Commands, file writes, exports, and imports on a node inherit that node's mission. Pass mission on those calls to count the work somewhere else, for example when you reuse an old node for a new question.

Updates#

Your agent posts updates to a mission while it works. They are the first thing you see on the mission page. An update has a title and a body in markdown.

gman api post /missions/lora-portraits/updates -d '{
  "title": "Run 7 is the best LoRA so far",
  "body": "Seven of ten runs finished. Rank 32 at 1e-4 wins on FID.\n\n![FID by step](art-df5cj)"
}'

post_update(
  mission: "lora-portraits",
  title: "Run 7 is the best LoRA so far",
  body: "Seven of ten runs finished. Rank 32 at 1e-4 wins on FID.\n\n![FID by step](art-df5cj)"
)
→ {update_id: "upd-9k2fq",
   url: "https://givemeanode.com/missions/lora-portraits#upd-9k2fq", ...}

• To show a picture, a video, or an audio clip, export the file with export_file and write ![what it shows](art-...) in the body. The artifact id becomes a share link, which unshare revokes. The stored body holds a real link, so the same text pasted into Slack shows the image too.
• The body supports paragraphs, ## headings, bold, italic, inline code, fenced code blocks, lists, quotes, links, tables, and horizontal rules. Anything else appears as the characters you typed. An image marker inside a code fence publishes nothing.
• Set kind to "alert" when a person needs to act. An alert is marked on the mission page and on your dashboard until the agent posts again or finishes the mission. Everything else is a note.
• Updates cannot be edited or deleted. A correction is a new update. list_updates reads them back, which is how an agent that lost its context catches up.

Research#

Research is in private preview. It is off by default for an organization. Ask us to turn it on.

Research attaches a researcher to a mission. You give it an objective, your source, a check set, and a budget. The managed researcher measures a baseline, changes one factor per hypothesis, runs each experiment as a job from your source, and reads the evaluations the platform computes. When an experiment beats its control and passes every check, the platform runs it again on fresh executions. If those runs pass, the mission completes with a confirmed result. A negative or inconclusive result is also a valid outcome.

Start#

gman research start \
  --name inference-latency \
  --source . \
  --objective 'Reduce p95 latency below 120 ms. Preserve task quality and API compatibility.' \
  --benchmark './bench.sh --json results.json' \
  --check-set ./production-evals.toml \
  --budget-usd 2000 \
  --deadline 8h

create_context(size_bytes: 48213, sha256: "...")   # upload the source tar, then
finalize_context(context_id: "ctx-...")
start_research(
  name: "inference-latency",
  objective: "Reduce p95 latency below 120 ms. Preserve task quality and API compatibility.",
  source_context: "ctx-...",
  benchmark: "./bench.sh --json results.json",
  checks: {schema: "research.checks/v1", checks: [...]},
  budget_usd: 2000,
  deadline_hours: 8
)
→ {name: "inference-latency", state: "ready", version: 1, control_version: 1,
   planner: "managed", url: "https://givemeanode.com/missions/inference-latency", ...}

• The mission is created if it does not exist. The mission page has a Research tab. Hand the URL to a person early.
• --planner managed is the default. The researcher uses intelligence-1 at $0.01 per unit. See the rate card for compute prices.
• Experiments run on h100 by default. Pass --chip cpu for work that needs no GPU, and --image to build on a different base image.
• Work continues if you disconnect. The deadline is the last moment the mission accepts new work. The default is 72 hours.

Source and results#

--source . uploads the directory. In a Git checkout the upload holds the tracked files and the untracked files that Git does not ignore. A .gmanignore file excludes more paths, one path or *.suffix per line. Git metadata never uploads. Executable bits are kept.

Each experiment runs its command with sh -c from the root of your source at /workspace/source. The command must write a JSON object to its results file. The default file is results.json in the source root. The platform reads that object. It is the only measurement the evaluator trusts.

{"p95_ms": 108, "throughput_tokens_per_s": 4120, "quality": {"score": 0.987}}

Check sets#

A check set lists the conditions a result must pass. Each check names a metric in the results JSON with a dotted path, a comparison, and one bound. A threshold is an absolute bound. A ratio compares the experiment to its control. The evaluator takes the median over an experiment's repetitions before it compares. Files may be TOML, YAML, or JSON.

schema = "research.checks/v1"

[[checks]]
id = "quality"
metric = "quality.score"
op = ">="
threshold = 0.98

[[checks]]
id = "latency"
metric = "p95_ms"
op = "<="
threshold = 120

[[checks]]
id = "faster-than-control"
metric = "p95_ms"
op = "<="
relative_to_control = 0.9

• A check is passed, failed, or unknown. A missing metric or a missing control is unknown. Unknown is never a pass.
• A researcher cannot change the check set. A change to the checks is a new mission.

Progress and steering#

gman research get inference-latency
gman research steer inference-latency --expected-version 7 \
  --question-id question-7a2 --answer 'continue without search'
gman research steer inference-latency --expected-version 8 \
  --instruction 'Prioritize prefix caching. Keep the current checks and budget.'
gman research stop inference-latency

get_research(name: "inference-latency")
→ {state: "needs_input", version: 7, control_version: 1, spent_usd: 614.22,
   question: {id: "question-7a2", text: "May I add external search to the allowed network policy?"},
   experiments: [{id: "baseline", verdict: "passed", measurements: {p95_ms: 141}}, ...], records: [...]}

steer_research(name: "inference-latency", expected_version: 7,
               question_id: "question-7a2", answer: "continue without search")
steer_research(name: "inference-latency", expected_version: 8,
               instruction: "Prioritize prefix caching. Keep the current checks and budget.")
stop_research(name: "inference-latency")

• Reads call no model. Poll get_research as often as you like. The states are ready, running, waiting, needs_input, verifying, stopping, stopped, completed, and failed.
• Every change names the version you read. If the mission changed since then, the change is refused. Read again and retry.
• An instruction directs the researcher without choosing experiments. It does not change the checks or the budget.
• Stop closes the mission to new work and cancels running experiments where possible. The record stays readable.

Your own plans#

You can select the experiments yourself. Start with --planner customer and a plan, or hand control over on an existing mission. The platform runs, evaluates, and retains each experiment. When your plan finishes, the mission waits for the next plan or for --complete.

cat > experiments.toml <<'TOML'
schema = "research.plan/v1"

[[experiments]]
id = "manual-baseline"
command = "./bench.sh --precision bf16 --json results.json"

[[experiments]]
id = "manual-fp8"
control = "manual-baseline"
command = "./bench.sh --precision fp8 --json results.json"
repetitions = 5
TOML

gman research steer inference-latency --expected-version 8 \
  --expected-control-version 1 --planner customer --accepted-work continue
gman research steer inference-latency --expected-version 9 --plan ./experiments.toml
gman research steer inference-latency --expected-version 11 \
  --expected-control-version 2 --complete

steer_research(name: "inference-latency", expected_version: 8, expected_control_version: 1,
               planner: "customer", accepted_work: "continue")
steer_research(name: "inference-latency", expected_version: 9,
               plan: {schema: "research.plan/v1", experiments: [
                 {id: "manual-baseline", command: "./bench.sh --precision bf16 --json results.json"},
                 {id: "manual-fp8", control: "manual-baseline", repetitions: 5,
                  command: "./bench.sh --precision fp8 --json results.json"}]})
steer_research(name: "inference-latency", expected_version: 11, expected_control_version: 2, complete: true)

• An experiment has an id, a command, and an optional control that names an earlier experiment. repetitions defaults to 3 and timeout_seconds to 3600. chip, chip_count, image, and results override the mission defaults.
• Experiment ids are unique within a mission. A plan that reuses an id is refused.
• --planner managed hands control back. The researcher continues with the same objective, evidence, and remaining budget.

Budget and billing#

The budget is a hard cap across everything the mission spends. Before any experiment or researcher turn starts, the platform reserves its maximum charge against the budget. A fifth of the budget is held for the confirmation of a selected result. Pass --confirmation-reserve-usd to change the amount.

• Experiment jobs bill as jobs, at the published job rates, and appear on the mission receipt and on Usage under Jobs.
• The managed researcher bills intelligence units at its tier price. They appear under Research. Reads, waits, and your own plans consume no units.
• A stopped mission reports the maximum remaining charge for work that may already have run. Unused reservations return to the balance.

Profiling#

GPU performance counters, which Nsight Compute (ncu) reads, are locked on every GPU by default. Ask for them at create time and the platform unlocks them.

gman node create --name counters --chip 8xh100 --profiling --clock-lock
gman run counters -- ncu --metrics gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed ./saxpy

create_node(name: "counters", chip: "8xh100", profiling: true, clock_lock: true)
run_command("counters", "ncu --metrics gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed ./saxpy")

There is nothing to configure inside the container. A memory-bound kernel reports what you expect:

dram__bytes_read.sum                                    134.23 Mbyte
gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed   78.10 %
gpu__time_duration.sum                                   70.94 us
sm__throughput.avg.pct_of_peak_sustained_elapsed         35.33 %

• Whole sections work (ncu --set full), and so does attaching to a running process.
• Profiling is free, and it cannot change after create.
• Counters unlock on whole-machine shapes only (8xh100). On a shared machine they could report on a neighbor. A profiling request on a single-GPU shape is refused with the fix.
• Profiling-capable machines are a smaller pool than the fleet, so a profiling node can queue longer. The queue reports position and estimate as usual.

Your image#

An unlocked machine also needs a profiler new enough to use it. ncu below 2025 fails on current drivers with Failed to prepare kernel for profiling. So profiling: true with no image lands on the catalog's profiling entry, which is cuda-12.9. That is also the plain default. When you name an image, only entries whose profiler works are accepted. An image that cannot profile is refused, and the refusal names the entry that works.

The same rule covers from_snapshot. When profiling selects a newer image than the snapshot was built against, the response says so.

Jobs#

Batch jobs take the same flag, so a kernel sweep can profile every variant.

gman job submit --chip h100 --count 8 --profiling --clock-lock \
  --image ghcr.io/you/kernels:cu129 --command ./sweep.sh

submit_job(chip: "h100", chip_count: 8, profiling: true, clock_lock: true,
           image: "ghcr.io/you/kernels:cu129", command: "./sweep.sh")

A job runs your image, so bring your own profiler. Base the image on CUDA 12.9 or newer, or apt-get install cuda-nsight-compute in your Dockerfile. Below ncu 2025 the profile fails on every machine. chip_count: 8 (CLI: --count 8) is required. The flag is free and cannot change after submit.

Clock lock#

Counters say why a kernel is slow. clock_lock: true makes how long it takes comparable across runs. It pins the GPUs to the model's published sustained clock instead of letting them drift with boost and thermal state. Kernel-search work usually sets both flags.

The lock is a ceiling, not a guarantee. The hardware still throttles under power or thermal pressure. On an 8-GPU SXM baseboard, your neighbors' heat reaches your GPUs, so a full 8xh100 is the only fully isolated shape. The flag is free and cannot change after create.

Metadata#

Code on a node can read its own facts and stop itself with no credential. The address is http://169.254.42.1 (also metadata.givemeanode.internal). Every request needs the header Metadata-Flavor: givemeanode.

curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/node
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/events
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/egress
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/network
curl -s -X POST -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/stop

• /v1/node is identity and money: name, state, rate, grace window.
• /v1/events carries structured notices with deadlines, such as machine retirement and spend-cap warnings. A checkpoint-aware training script polls it like cloud spot metadata.
• /v1/egress is the network policy. The default is open internet. The response lists the blocked destinations: outbound SMTP on port 25, a small set of mining pool ports, and the cloud metadata IP. The blocked ports fail fast. The metadata IP is dropped silently, so a connect to it hangs. DNS is unfiltered and TLS is never intercepted, so a certificate error or a failed name comes from the origin. It answers for nodes and jobs.
• /v1/network is the roster of your private network.
• POST /v1/stop ends the node from inside with stop_node semantics. A training run's last line can be that curl.

Job containers get the same service at $GMN_METADATA_URL, with job routes. GET /v1/job is identity and time: job id, attempt, deadline, checkpointed_at. POST /v1/checkpoint declares $GMN_CHECKPOINT_DIR consistent and captures it. POST /v1/yield ends the attempt and resumes it from the slot. POST /v1/stop is cancel, and it is terminal. The whole workflow is under Checkpoints.

Guides#

The guides combine the features above into complete workflows. Each one names the image it runs on and links to the docs for every capability it uses.

Post-training#

These guides cover LoRA SFT, GRPO, agentic rollouts, and multi-stage training. Each names its image, its topology, and its checkpoint discipline. All of them run on one 8x node or as batch jobs. Multi-node training is not offered. The pinned stacks for this work are under Post-training in the image catalog.

LoRA SFT#

A dataset goes in and a LoRA adapter comes out. This works up to the largest models that fit one 8x in bf16. Larger bases fit with QLoRA.

gman job submit --chip h100 --count 8 --context ./sft \
  --command 'python sft.py'   # ./sft: Dockerfile (FROM your training base,
                              # pip install trl peft) + sft.py + config.
                              # sft.py: SFTTrainer; state to $GMN_CHECKPOINT_DIR
                              # each epoch; final adapter to $GMN_OUTPUT_DIR

submit_job(
  chip: "h100", chip_count: 8,
  command: "python sft.py", context_id: "ctx_...",   # Dockerfile + code ride the context
  hf_cache: true,                                     # base weights cached per-org
  resume: "checkpoint", idempotency_key: "sft-run-1")

Two lines in sft.py matter. Write the trainer state to $GMN_CHECKPOINT_DIR and declare it (Checkpoints), so a preemption costs minutes instead of the whole run. Write the adapter and eval numbers to $GMN_OUTPUT_DIR and $GMN_RESULT_PATH, so the verdict comes back on get_job.

GRPO#

RL post-training runs as three cooperating processes: a trainer, a rollout orchestrator, and an inference server the policy samples from. On one 8x they share the machine. vLLM serves the policy on a GPU slice, TRL's GRPO trainer updates weights on the rest, and rollouts loop between them. Once built, trl-0.21-cuda12.9 carries both halves. Until then, start from vllm-0.26-cuda12.9 and add TRL with the pip line below.

gman node create --name grpo-box --chip 8xh100 --image vllm-0.26-cuda12.9 --wait
gman run grpo-box -- pip install --break-system-packages trl peft datasets accelerate
gman run grpo-box -d -- python serve_policy.py   # vLLM (for restart supervision
gman run grpo-box -d -- python grpo_train.py     #  use the MCP form below)
gman run grpo-box -- tail -20 rewards.jsonl      # or scrape it: register_scrape

create_node(name: "grpo-box", chip: "8xh100", image: "vllm-0.26-cuda12.9")
get_node("grpo-box")   # poll to running; create never blocks
run_command("grpo-box", "pip install --break-system-packages trl peft datasets accelerate")
run_command("grpo-box", "python serve_policy.py", detach: true, restart: "on-failure")
run_command("grpo-box", "python grpo_train.py", detach: true)
query_metrics(...)   # reward curves via the free platform telemetry

Group the run under a mission. Call finish_mission with the final reward, so the mission receipt records the whole run.

Rollouts#

Agentic RL needs three services: environments to act in, code execution for the agent's tools, and a reward signal. Run environment rollouts as sweeps. Each variant is one rollout batch, with verdicts written to $GMN_RESULT_PATH. Run untrusted tool code in a sandbox, one fork of a baked environment per sample. Your agent drives sandboxes over MCP. A job drives them over the HTTP API with a service token in env, never on the command line. If your reward is a judge model, serve it like the policy server above (sglang-0.6-cuda12.9 or vLLM on a 1x or a slice of your 8x) and point rollout workers at it.

gman node create --name judge --chip h100 --image vllm-0.26-cuda12.9 --wait
gman run judge -d -- python -m vllm.entrypoints.openai.api_server \
  --model acme/reward-judge --port 8000
gman api post /nodes/judge/endpoints --data '{"port": 8000, "auth": "bearer"}'
gman api post /jobs/sweeps --data '{
  "shared": {"chip": "h100", "context_id": "ctx_...", "command": "python rollout.py",
             "env": {"JUDGE_URL": "https://...", "GMN_TOKEN": "..."}},
  "variants": [{"env": {"SHARD": "0"}}, {"env": {"SHARD": "1"}},
               {"env": {"SHARD": "2"}}, {"env": {"SHARD": "3"}}],
  "label": "rollouts-r1", "idempotency_key": "rollouts-r1"}'
# each variant is one rollout batch; verdicts land in $GMN_RESULT_PATH

create_node(name: "judge", chip: "h100", image: "vllm-0.26-cuda12.9")
get_node("judge")   # poll to running; create never blocks
run_command("judge", "python -m vllm.entrypoints.openai.api_server --model acme/reward-judge --port 8000", detach: true)
expose_port("judge", port: 8000, auth: "bearer")
submit_jobs(
  shared: {chip: "h100", context_id: "ctx_...", command: "python rollout.py",
           env: {JUDGE_URL: "https://...", GMN_TOKEN: {secret: "gmn-token"}}},
  variants: [{env: {SHARD: "0"}}, {env: {SHARD: "1"}}, {env: {SHARD: "2"}}, {env: {SHARD: "3"}}],
  label: "rollouts-r1", idempotency_key: "rollouts-r1")
# each variant is one rollout batch; verdicts land in $GMN_RESULT_PATH,
# so list_jobs(label: "rollouts-r1", summary: true) is the reward table

Two stages#

Multi-stage post-training (distillation or SFT first, RL second) chains two jobs. Stage 1 runs to a verified artifact. Stage 2 starts from it. The stage 1 job writes final weights to $GMN_OUTPUT_DIR, or pushes large ones to org storage. Your agent checks the stage 1 eval verdict from $GMN_RESULT_PATH. The stage 2 job pulls the artifact and trains. Use the same image for both stages, and one mission over both.

gman job submit --chip h100 --count 8 --context ./train \
  --command 'python sft.py' --idempotency-key stage1-sft
gman job get job_...   # poll to done; the result is the stage 1 eval
                       # verdict, the output URL is the weights
gman job submit --chip h100 --count 8 --context ./train \
  --command 'python grpo.py' --env STAGE1_URL='https://...' \
  --idempotency-key stage2-rl   # one build context serves both stages

submit_job(chip: "h100", chip_count: 8, context_id: "ctx_...",
           command: "python sft.py", idempotency_key: "stage1-sft")
get_job("job_...")   # poll to done; result = the stage 1 eval verdict,
                     # output = the weights as a download URL
submit_job(chip: "h100", chip_count: 8, context_id: "ctx_...",
           command: "python grpo.py", env: {STAGE1_URL: "https://..."},
           idempotency_key: "stage2-rl")   # grpo.py pulls the weights first

Image generation#

Image generation here means a large shared base checkpoint (SDXL-class, Flux-class) plus one or more small LoRA adapters. The pytorch-2.13-cuda12.9 catalog image carries torchvision, ffmpeg, and libgl1, so diffusers pipelines and ComfyUI import without setup. Base weights use the model cache, so a sweep pulls Flux once per machine.

Adapters#

Adapters are small (tens to hundreds of MB) and come from three places:

• Your own training runs. The adapter comes back on get_job, or lands on your Hugging Face repo through a write connection.
• Gated or private Hugging Face repos, through a Hugging Face connection. The import example is there.
• Published adapters on Civitai, through a Civitai connection.

For a batch sweep, carry adapters in an uploaded context, which takes up to 10 GiB. Upload once with create_context then finalize_context, and put the id in the sweep's shared.context_id. gman job submit --context ./dir uploads inline but submits one job. Sweeps take the id form. The inline context file map caps at 4 MiB, which fits code but not weights.

Generation sweep#

Batch generation sweeps over prompt sets, seeds, and LoRA strengths, and it tolerates queueing, so it fits the job rate. Put the variant parameters in env, share the base weights with hf_cache, and land every variant's images in your own bucket with output_to:

gman api post /jobs/sweeps --data '{
  "shared": {"chip": "h100", "context_id": "ctx-...", "hf_cache": true,
             "command": "python generate.py",
             "output_to": {"connection": "acme-out", "dest": "s3://acme-ml/grids/"}},
  "variants": [{"env": {"STRENGTH": "0.6"}}, {"env": {"STRENGTH": "0.8"}},
               {"env": {"STRENGTH": "1.0"}}],
  "label": "lora-strength", "idempotency_key": "lora-strength-1"}'

submit_jobs(
  shared: {chip: "h100", context_id: "ctx-...", hf_cache: true,
           command: "python generate.py",
           output_to: {connection: "acme-out", dest: "s3://acme-ml/grids/"}},
  variants: [{env: {STRENGTH: "0.6"}}, {env: {STRENGTH: "0.8"}}, {env: {STRENGTH: "1.0"}}],
  label: "lora-strength", idempotency_key: "lora-strength-1")

output_to takes a write-enabled s3-keys connection and a prefix. Each variant's $GMN_OUTPUT_DIR capture lands at <dest><job_id>-<attempt>.tar, so a 256-variant sweep becomes one dataset in your bucket with no key collisions and no credential in any container. Without it, each job's images come back as a download link on get_job. Either way, write a verdict (image count, NSFW filter hits, CLIP score) to $GMN_RESULT_PATH, so list_jobs(label, summary: true) is the comparison table.