# Customer SDK and room-side execution

The SDK supports the agent experiment service: customers supply decisions, while
compatible rooms and compute execute within existing authority and return
original evidence. The experiment is the product contract; current SDK calls
remain room/booking based. The [planned experiment journey](ROOM_API.md#planned-experiment-journey-and-authorization)
adds no implemented endpoint, physical qualification or monetary self-approval.

The current local development SDK is **0.1.10**. Choose a client path from the
actual room contract; generic execution and the continuous NexArm runner have
different supported control semantics.

| Customer need | Client and guide |
| --- | --- |
| Assess published compatibility without reserving resources | `RoomDirectory`; [discovery below](#independent-discovery-and-camera-reads-sdk-016) |
| Read original cameras independently of motion | `ObservationClient`, `RGBStream`, `StreamMux`; [stream adapters](SDK_ADAPTERS.md) |
| Prepare a policy and execute in a paused SO100/NexArm room | `ExecutionLoop`, explicit device adapter and `PausedRoomControl`; [generic execution](SDK_EXECUTION.md) |
| Associate paused NexArm drive targets and three camera views | `NexArmSensingStream`; [customer delta decoder](SDK_ADAPTERS.md#customer-deltas-and-a-verified-drive-anchor) |
| Use the existing continuous NexArm forecast/feedback protocol | `RobotClient`, `RunConfig` and async policy; [installation and configuration](#install-and-configure) |

The standalone wheel needs no server or native engine. The paused control
backend injects the existing starter from the separately installed management
package. Generic stream values and collection are separate from device controls,
room discovery and concrete RGB readers; existing import paths remain compatible.
[SDK module map](../src/physical_ai_lab/workshop/robotics/sdk/README.md).

Customers supply preprocessing, model, instruction and action decoding. The
[custom multiview model exercise](results/CUSTOM_VLA_EXECUTION-2026-10-22.md)
retains original images and distinguishes drive targets from measured joints.
Its fixture weights are untrained and its strict 2 Hz source minimum was not met.
The [verification index](results/README.md) records earlier versions and failures;
this guide describes current use rather than a release history.

The continuous runner handles media, feedback, reconnect, source-bound forecast
submission and export. Native execution proceeds independently of inference.
Protected `workshop.nexarm-continuous.run@2` records `protection.json`; a fault
requires operator recovery, and reconnect does not resume interrupted motion.
Admission remains distinct from the native applied prefix and verified or unknown
stop outcome. During finite close, `live_closed` is normal only after the gateway
confirms closing or closed; unconfirmed close errors remain failures.
[Protected client evidence](results/PROTECTION_CLIENTS-2026-10-21.md) and
[close/feedback evidence](results/SOURCE_SYNC-2026-10-21.md).

The [execution placement plan](SDK_EXECUTION.md#planned-execution-placement)
keeps customer decisions separate from room drivers and local controllers.
Rented onsite CPU/GPU customer runtimes, independent observer events/streams and
final result delivery are planned; current calls do not upload models or provision
workstations. This is a local development release, not a published package or
physical/WAN qualification.

## Independent discovery and camera reads (SDK 0.1.6)

The 22 October direction in
[ADR 0057](architecture/0057-room-capabilities-and-independent-streaming-sdk.md)
separates the shared streaming/observation client from robot control and
customer-policy adapters. Room contracts and version-matched docs must support
discovery by a customer's agent directly; an optional concierge uses the same
information. SO100 is an initial case, not a limit on the intended SDK.

`RoomDirectory` reads concise, paginated published configurations, exact capability
descriptions and version-matched documents. Descriptions include action channels,
gripper meaning, cameras, clocks, state promises, qualification, unsupported
features and integration gaps. The catalog reports readiness and availability for
its returned preview window; use existing calendar search for your intended window.
The customer's agent assesses semantic fit. No match is a valid result; reading
the catalog acquires no capacity or credit reservation.

`ObservationClient` reads named original PNG cameras from an existing owned job
using only a scoped `read` key, even with a zero execution budget. It imports no
arm controller, model runtime, NumPy or Pillow. Install the wheel without extras
for this path:

```python
import asyncio
import os
from pathlib import Path
from actuator_sdk import RoomDirectory, ObservationClient
from actuator_sdk.observation_contracts import VisionRead

async def inspect():
    access = (os.environ["ACTUATOR_URL"], os.environ["PROJECT_ID"],
              os.environ["ACTUATOR_TOKEN"])
    async with RoomDirectory(*access) as directory:
        page = await directory.catalog(limit=25)
        for candidate in page["candidates"]:
            details = await directory.capabilities(candidate["room_id"],
                                                   candidate["configuration_id"])
            print(details)
            for doc_id in details["document_ids"]:
                print((await directory.documentation(doc_id))["content"])
        # Continue at offset + limit while pagination.has_more is true.
    # JOB_ID must already belong to this project; CAMERA_ID must be declared.
    async with ObservationClient(*access) as camera:
        async for value in camera.frames(os.environ["JOB_ID"],
                                         os.environ["CAMERA_ID"],
                                         timeout_s=10., max_frames=20):
            frame = VisionRead.model_validate(value).frame
            Path("latest.png").write_bytes(frame.png_bytes())
            print(frame.captured_at_unix, value["client_skipped_source_sequences"])

asyncio.run(inspect())
```

`latest(job_id, camera_id)` returns a null frame until the first native record.
`frames` polls latest-only: duplicates are omitted, skipped sequence counts are
explicit, and no unbounded backlog is retained. Capture times and source/PNG
digests come from the original journal; repeated reads never advance physics.
SO100 side capture, other SO100 decision cameras, paused NexArm sensing cameras
and continuous NexArm video use their existing sources and cadences. Reads stay
available after terminal publication or failure to an authorized reader. A reader
timeout ends polling and does not close the experiment. HTTP errors remain errors;
call again deliberately after restoring access/connectivity. Automatic video
reconnect is still part of the legacy policy runner only.

The read protocol is additive `actuator.vision-read@1`: it requires no joint-state
array and does not reinterpret `actuator.robot-client@1`. Responses are bounded
to 2 MiB; original PNGs to 1 MiB. The SDK verifies schema, project/job/camera
binding, PNG digest/dimensions and monotone source identity. It preserves the
source RGB digest without claiming to decode and independently hash RGB pixels.
Credentials go only to the configured HTTPS origin or local loopback HTTP;
redirects are rejected. [API details](ROOM_API.md#room-capabilities-and-independent-observations).

The legacy `RobotClient`, `RunConfig`, policy/media classes and imports remain
compatible: two continuous NexArm capabilities, twelve radian-target channels
at 20 Hz, one `overview` camera and NexArm-specific media sample metadata. SO100
motion uses the booked-room starter/WorkshopClient, now also through
`PausedRoomControl` and the generic loop. Generic continuous control still needs
a backend preserving its existing forecast/feedback contract.

The remaining boundary includes negotiated video and actual sensor/inference
producers. Shared stream contracts already describe source association,
calibration/frame metadata, timestamps and producer/model identity, with fixture
coverage for new modalities. Additional hardware, ROS 2,
LeRobot mappings and on-device inference streams require individually qualified
adapters. Original `actuator.robot-client@1` payloads and existing installed
clients remain supported through an explicit compatibility mapping.

## Install and configure

Customers receive the client-only wheel and room access details from the service.
From a standalone customer directory, install the supplied wheel:

```sh
python3.11 -m venv venv
venv/bin/python -m pip install './actuator_rooms_sdk-0.1.10-py3-none-any.whl'
```

For repository maintainers, build that wheel from `physical-ai-lab/` with
`python -m pip wheel --no-deps --wheel-dir .pal/sdk-wheels ./sdk-package`.

Use `[images]` for decoded policy images and `[webrtc]` for aiortc/VP8 reception.
The core SDK needs HTTP/schema packages; policy image decoding adds NumPy and
Pillow. No FastAPI, SciPy, MuJoCo, Torch or LeRobot
is required for ordinary policy control. `[lerobot]` is an optional tested library
version, separate from a customer's chosen weights, Torch/device and processors.
For the tested Python 3.11/macOS ARM64 environment, save the following as
`constraints.txt` and add `-c constraints.txt` to the install command. This is
optional for first use and is not a universal CUDA/Linux lockfile.

<details><summary>Complete tested customer dependency constraints</summary>

```text
# Tested optional-WebRTC customer environment: Python3.11, macOS ARM64.
aioice==0.10.2
aiortc==1.15.0
annotated-types==0.8.0
anyio==4.15.1
av==17.1.0
certifi==2026.10.19
cffi==2.1.1
cryptography==50.0.2
dnspython==2.9.0
google-crc32c==1.9.0
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
idna==3.20
ifaddr==0.2.0
numpy==2.4.6
pillow==12.3.0
pycparser==3.11
pydantic==2.14.1
pydantic_core==2.50.1
pyee==13.0.1
pylibsrtp==1.0.0
pyOpenSSL==26.4.0
typing-inspection==0.4.4
typing_extensions==4.16.0
```

</details>

Save this complete configuration as `my-run.json`:

```json
{
  "schema_version": "actuator.robot-client@1",
  "stream": {"transport": "jpeg", "width": 320, "height": 240, "fps": 10},
  "duration_s": 30.0,
  "max_commands": 40,
  "max_credits": 5000
}
```

This selects JPEG, 320×240, 10 fps, a control duration, command bound and
development-credit cap. Defaults
declare camera `overview`, RGB uint8/HWC, the exact twelve-joint order, radians,
20 Hz playback, at most 34 forecast rows, a two-second forecast/image-age bound,
1.5-second feedback timeout and a separate 15-second startup bound.

Advanced supported options are strict: resolution 256×192/320×240/480×360,
1–10 fps, JPEG quality 20–95, VP8 maximum 250–1,500 kbps, finite peer duration,
observation age, forecast rows/horizon, feedback timeout and policy timeout.
The service returns the exact accepted settings and digest. Unsupported camera
ordering, geometry, units or fields fail; calibration and robot limits cannot
be overridden. Lower image resolution does not change native capture quality or
silently change the model's crop/normalization.

## Customer policy

```python
from actuator_sdk import PolicyOutput

class MyPolicy:
    async def predict(self, observation, context):
        # Your model receives observation.rgb and observation.joint_state.
        # Return your converted twelve-channel radian targets in room order.
        rows = await my_model.predict(observation, context)
        return PolicyOutput(targets_rad=rows, alignment="observation_time")
```

`Observation` carries actual decoded RGB, original capture time/frame digest,
aligned measured joints and delivery provenance. `PolicyContext` carries drive
targets, queued row count and reconnect generation. Return `None` to stop,
`PolicyOutput(wait=True)` to keep observing while the existing forecast plays,
or `PolicyOutput(hold=True)` to flush unadmitted motion and hold after the already
admitted prefix drains. Wait sends no new forecast and preserves its original
expiry; it neither renews a path nor pauses feedback. Repeated holds in the same
generation are coalesced after the room acknowledges holding. Activation-time paths
explicitly use `alignment="start_on_activation"`; learned future predictions use
`observation_time`, so elapsed predictions are discarded. Replanning can replace
unadmitted paths with `supersede=True`; existing admitted work retains its guards.

`observation.joint_state` contains four arrays of length twelve: `names`,
`positions_rad` (measured), `velocities_rad_s` (measured), and
`position_targets_rad` (the applied drive targets at capture). The order is left
then right, each with `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`,
`wrist_roll`, `gripper`; full names have the form `left.shoulder_pan.pos`.
Use `context.joint_names` to resolve indices and `context.drive_targets_rad` as
the current planning handoff; measured positions and requested targets differ.
`observation.rgb` is a read-only NumPy uint8 array with shape `(height, width, 3)`.
`frame_sha256` identifies the original source sample; delivery provenance binds
the resized/lossy pixels separately. Image preprocessing and visual goals belong
to the customer's policy.

A motion sanity check can construct a small finite path relative to
`context.drive_targets_rad`, then return `PolicyOutput(wait=True)` while it
drains. Such a state-relative check does not establish a video-guided task.
Customers own camera transforms, normalization, language and embodiment mapping.
`actuator_sdk.adapters.LeRobotPolicy` wraps `predict_action_chunk` with explicit
`observation_builder`, `action_decoder`, original model action rate and optional
warmup. It resamples the declared forecast to 20 Hz without inventing a longer
horizon. Its interface is contract-tested; this SDK increment does not retrain
or qualify the earlier SO100 model for NexArm manipulation.

Model initialization/warmup should finish before booking. The service supplies
`base_url`, `project_id`, `room_id` and an authorized `quote_id`; save these as
`access.json`. Pass the credential through `ACTUATOR_TOKEN`. Save your policy in
an importable `my_policy.py` exposing `make_policy(config)` and run this complete
booking harness as `venv/bin/python run_experiment.py`:

```python
import asyncio
import json
import os
from pathlib import Path
import time
import uuid
from actuator_sdk import RobotClient, RunConfig
from my_policy import make_policy

async def main():
    access = json.loads(Path("access.json").read_text())
    config = RunConfig.model_validate_json(Path("my-run.json").read_text())
    policy = make_policy(config)
    if hasattr(policy, "prepare"):
        await policy.prepare()
    key = "experiment-" + uuid.uuid4().hex
    async with RobotClient(access["base_url"], access["project_id"],
                           os.environ["ACTUATOR_TOKEN"]) as rooms:
        await rooms.preflight(config, room_id=access["room_id"],
                              quote_id=access["quote_id"])
        start = time.time() + 2.0
        booking = await rooms.book(room_id=access["room_id"],
            quote_id=access["quote_id"], starts_at_unix=start,
            ends_at_unix=start + config.startup_timeout_s + config.duration_s + 2.0,
            config=config, idempotency_key=key)
        Path(key + "-booking.json").write_text(json.dumps(booking, indent=2))
        report = await rooms.run(booking["job"]["job_id"], policy,
            config=config, out_dir=key + "-results")
        print(json.dumps(report, indent=2))
        if not report["run_ok"]:
            raise RuntimeError("Inspect the exported failures before retrying")

asyncio.run(main())
```

The interval must cover startup + control duration + the existing two-second
calendar stop reserve. Preflight acquires no capacity or credits. Booking uses
an existing authorized quote and checks its total development-credit ceiling.
The original job/calendar limits remain authoritative throughout. Credentials
are passed separately and are not written into run configuration or exports.
Use a fresh future start with a request margin; `preflight.server_time_unix` is
already in the past by the next request. Start `run` near the booked start:
worker readiness has a separate finite wait. Keep the booking response and reuse
the same key when retrying an uncertain booking request. A completed booking
consumes its quote; another experiment needs another authorized quote and a
fresh output directory.

Existing project/room/quote operations provide the identifiers; this removes
the generated `room.json` requirement without introducing public account signup,
monetary payments or a second booking service.

## Room price preview (SDK 0.1.3)

The initial room rate is US$20/hour, including its robots and cameras. It prices
reserved time, rounding duration up to whole seconds and the amount up to a USD
cent. Taxes are not calculated and no monetary payments are collected. Existing
development-credit budgets and charges remain separate.

Before booking, inspect the current window price:

```python
preview = await rooms.estimate_price(room_id=room_id,
    starts_at_unix=start, ends_at_unix=end)
print(preview["currency"], preview["amount_cents"])
```

A 90-second reservation previews US$0.50. Search estimates use the current rate;
the later capacity hold pins its own immutable price and confirmation copies it
into `booking["booking"]["pricing"]`. Rate changes do not reprice a held offer.
Early completion releases capacity; its preview still uses the reserved window.
The CLI's `price` operation takes `--url`, `--project`, `--room`, `--start` and
`--end`, with the credential in the usual token environment variable. Monetary
previews acquire no credits or capacity and do not authorize payment.

## CLI and custom configuration

The installed `actuator-rooms` command supports `preflight`, `book` and `run`.
For example, with the credential already set in `ACTUATOR_TOKEN`:

```sh
actuator-rooms preflight --url http://127.0.0.1:8880 --project "$PROJECT_ID" \
  --room "$ROOM_ID" --quote "$QUOTE_ID" --config my-run.json
actuator-rooms run --url http://127.0.0.1:8880 --project "$PROJECT_ID" \
  --job "$JOB_ID" --config my-run.json --policy my_policy:make_policy \
  --policy-config my-model.json --out my-results
```

Without `--policy-config`, the factory receives `RunConfig`. With it, the factory
also receives the customer JSON dictionary; that dictionary stays on the
customer host. A factory can attach a JSON `identity` with model/revision and
processor digests. Keep credentials in the customer's secret mechanism rather
than model metadata. `prepare()` is optional and runs before the CLI operation;
for before-booking warmup use the Python sequence above.
The policy module must be importable in the customer's environment, for example
as their installed project package or through an explicit `PYTHONPATH`.

## Execution and evidence

The room gateway accepts bounded source-bound forecasts. It holds at most forty
rows/two seconds and segments through the existing one-pending native command
service. Source-frame expiry, observation age, range, slew, job lease, calendar
and budget enforcement remain active. Any handoff slew projection is recorded.
Preflight returns `gateway_limits.max_forecasts` (128 unique forecast requests
per run); gateway status returns that bound and `plans_received`. Motion and
first hold requests count; exact retries reuse their identity. Local waits and
coalesced holds do not consume this bound or native command capacity. Their
frame/generation/pixel bindings are saved separately in `policy-waits.json` and
`coalesced-holds.json` when present. A policy may return wait for every new image
while reviewing playback, then supply a new forecast from a fresh observation.

The SDK acknowledges fresh feedback independently of `predict`. Video/feedback
loss flushes unadmitted work; reconnect requires a new generation and fresh source
captures. Exact retry identity is preserved. Timed playback continues while the
customer model awaits a response. A dead customer cannot keep the gateway alive
beyond its finite configured clocks. No customer Python, model weights or custom
normalization runs in the room process.

For a deliberate automatic reconnect test, choose WebRTC and a finite peer
shorter than the experiment, for example:

```json
{
  "stream": {"transport": "webrtc", "width": 320, "height": 240,
             "fps": 10, "duration_s": 7.0},
  "duration_s": 30.0,
  "max_commands": 40,
  "max_credits": 5000
}
```

The peer expires, fresh feedback stops, and the SDK's watchdog pauses and opens
a new peer. Peer negotiation takes time; choose a peer duration that allows it
to finish. `stream.duration_s` bounds a WebRTC peer, whereas the top-level
`duration_s` bounds control. It does not inject a JPEG outage. Inspect
`media/connections.jsonl`, `media/receiver-summary.json` and policy
`context.generation` to verify a new connection and fresh generation; the run
clock and forecast lifetimes are not renewed by reconnect. The high-level
`run` API owns reconnect and does not expose its receiver to policies.

The local API process owns the gateway; the existing LIVE journal fences a second
executor. An API restart loses ephemeral forecasts and requires explicit job
Close/new booking. This is not a distributed or HA gateway. New feedback handling
holds existing targets. The separately qualified continuous version 2 also has
local automatic runtime guards and durable stop evidence; physical watchdogs and
hardware stops require separate commissioning. See [safety status](SAFETY.md).

Results include accepted configuration/digest, model identity, actual received
pixels and media counters, source-bound plan receipts, gateway dispositions,
original published result and video archive. `status` is native publication
status; `run_ok` additionally checks SDK/gateway failures, and neither is a task
success assertion. Derived recordings/analysis run after committed room release.

`scripts/prove_robot_sdk.py` launches actual API/worker processes while an isolated
installed-wheel customer preflights, books and runs through public HTTP only.
It exercises JPEG, WebRTC, finite-peer reconnect and customer process loss.
The reconnect rehearsal requires `--transport webrtc --case reconnect`;
unsupported JPEG/reconnect arguments are rejected before room setup.
The repository maintainer record
`docs/evidence/robot-customer-sdk-2026-10-21.json` retains outcomes, exact
wheel/profile/source hashes, tests and failures; it is not needed by a customer.

## Earlier SDK 0.1.0 qualification

The 16,093-byte client wheel was installed into a fresh Python 3.11 environment
without the server package, MuJoCo, FastAPI, SciPy, Torch or LeRobot. All four
final trials used that installed wheel and the exact same qualified gateway
sources. Customer policies deliberately awaited simulated inference; native
commands continued independently during those waits.

| Trial | Native commands / applied rows | Native wall time | Outcome |
| --- | ---: | ---: | --- |
| JPEG | 6 / 29 | 4.20 s | Both arms moved; booking, export and release passed |
| WebRTC | 6 / 28 | 4.47 s | Both arms moved; booking, export and release passed |
| WebRTC finite-peer reconnect | 40 / 175 | 13.87 s | Two video connections; gap flush and fresh generation resumed |
| Customer process killed | 4 / 20 | 8.03 s | Room closed without customer cleanup; published and released |

A forecast being offered is not a promise that all its rows will execute.
Expiry, generation changes and Close can discard unadmitted tails; counts above
are actual native applied rows. Original commands/dispositions remain the motion
authority. Independent saved-byte audits verified source/decoded pixels,
customer input bindings, exact native admissions, fixed expiry, matching exports
and committed capacity/credit release in every case.

The broader regression suite passed 2,449 cases with 158 native/GPU integration
cases deselected. Separate actual native qualification passed nine continuous
cases and eight legacy pair cases without skips. The final SDK/media/interface
Close-compatibility suite passed 37 cases. The optional LeRobot wrapper received
a contract test; no new model/task-training qualification is claimed.

The initial proof harness accidentally resolved a virtual-environment Python
symlink to the system interpreter and failed before booking. A separate early
WebRTC attempt exhausted its short control clock during negotiation. Both
failed attempts remain preserved; the release keeps interpreter identity and
uses a separate finite startup clock. No existing evidence was rewritten.