# Generic execution with separate streaming and control

The loop is an execution component of the agent experiment service. Agents choose
questions and analyze evidence; rooms, drivers and optional onsite compute supply
bounded resources. Current service calls remain room/booking based. The
[planned experiment journey](ROOM_API.md#planned-experiment-journey-and-authorization)
must obtain trusted approval and retain capability/state/price identities through
the existing lifecycle; it is not a new SDK wire protocol or physical-service claim.

SDK 0.1.8 adds `ExecutionLoop`: finite customer-policy orchestration over an
explicit control backend and independently supplied stream readers. The loop
has no joint count, camera count, robot driver, model runtime or booking API.
This builds on the [adapter interfaces](SDK_ADAPTERS.md) and
[ADR 0057](architecture/0057-room-capabilities-and-independent-streaming-sdk.md).
See the [verification report](results/EXECUTION_LOOP-2026-10-22.md) for evidence.

SDK 0.1.9 adds a concrete [paused NexArm sensing reader and customer-model
example](SDK_ADAPTERS.md#customer-deltas-and-a-verified-drive-anchor). It leaves
the generic loop and control backend unchanged. Original sensing supplies
source-associated drive targets for customer delta decoding while observations
continue independently. [Two native bookings](results/CUSTOM_VLA_EXECUTION-2026-10-22.md)
exercise actual untrained-model inference; strict source cadence remains a gap.
Customers own decisions; room drivers and local controllers own qualified target
execution. The [planned execution placement](#planned-execution-placement) extends
this split to rented onsite CPU/GPU runtimes without coupling streams to control.
SDK 0.1.10 preserves this behavior while separating generic stream contracts and
collection from concrete RGB readers and room-discovery schemas. The
[module map](../src/physical_ai_lab/workshop/robotics/sdk/README.md) identifies
the extension points and retained compatibility imports.

## Prepare before booking

Discover the published configuration and assess its actual contract before
reserving capacity. The current paused adapter registry can describe public
capabilities without a job, lease, driver or capacity request:

```python
from actuator_sdk import ExecutionLoop, RGBStream
from actuator_sdk.joint_adapters import paused_position_adapters

# capabilities comes from the selected published room configuration.
adapter = paused_position_adapters().resolve_configuration(capabilities)
description = adapter.describe_configuration(capabilities)
streams = [RGBStream.description_for(camera['id'])
           for camera in capabilities['cameras']]
runner = ExecutionLoop({
    'duration_s': 20., 'max_commands': 4, 'max_predictions': 12,
    'policy_timeout_s': 2., 'stream_poll_s': .2,
})
prepared = await runner.prepare(customer_policy, description, streams,
    requirements=[{'id': stream.id, 'max_age_s': 5.,
                   'match_control_step': True} for stream in streams])
```

`Policy.prepare(control, streams)` warms customer code before booking. It gets
copies of the declared contracts; changing those copies cannot change the loop's
contracts. `predict(streams, state)` returns explicit decoded `NamedTargets`,
`PolicyWait(delay_s=...)`, or `None` to finish. The loop does not infer model
features, convert deltas, clip targets, resample actions or load weights.
The example's freshness limit is a caller choice, not a promised camera rate.

When streams are declared, requirements must be explicit. Use `requirements=[]`
only when every stream is deliberately optional. Required samples need a
reader-mapped `unix_utc` capture clock so the loop can check age and future skew.
For other source clocks, the reader must supply a documented clock mapping.
`match_control_step=True` also requires the original sample's declared control
step to equal the measured decision's step. Independent reads do not synchronize
camera exposures. Missing, stale or mismatched required samples delay prediction
within the same finite allowance; a terminal required stream ends execution.

## Attach existing authority and independent readers

After the owner authorizes a bounded booking, inspect its existing starter and
attach it to the loop. This backend composes the current client; it does not
create a second booking, retry journal or job lifecycle:

```python
from actuator_sdk import ObservationClient, RGBStream, StreamMux, PausedRoomControl

# booked is an already inspected RoomClient with its caller-owned Workspace.
# The caller manages its transport, credentials and workspace lifetime.
control = PausedRoomControl(booked)
async with ObservationClient(url, project_id, read_token) as observations:
    mux = StreamMux(project_id, job_id, [
        RGBStream(observations, job_id, stream.id) for stream in streams
    ])
    result = await runner.run(prepared, control, mux, out_dir=fresh_private_path)
```

The concrete injected `RoomClient`, `Workspace` and `Transport` currently come
from the separately installed, engine-free `physical-ai-lab` management/client
package under `physical_ai_lab.workshop.robotics.starter`. They are not bundled
into the standalone SDK. Importing `ExecutionLoop` or stream interfaces needs
neither that package nor joint adapters, native engines, media or ML libraries.
`PausedRoomControl` loads the explicit paused adapter registry only when selected.

The read client may use a separate read-only, zero-budget key. Its `StreamMux`
can also be used without any controller. A latest-only collector continues
reading while cooperative inference runs, keeping one current batch rather than
an inference backlog. The loop owns its collector task, not the supplied mux,
HTTP client, model, credentials or starter workspace. The caller closes those.

`run` checks the prepared control/stream descriptions and execution identity
before prediction. A backend binds project, job and original generation. The
earliest original monotonic cutoff can shorten the local duration but never
extend it. Original command allowance, local prediction/command limits and the
closing margin remain separate bounds. The loop never books capacity, grants
funds, increases a cap or selects another room on a failure.

## Causal commands and uncertain outcomes

The policy gets copies of named measured state and original source-associated
reads. The loop validates targets against the exact prepared channels, units,
frames, period and row limits. It rechecks the *prediction's input samples*
after inference; newer frames cannot make an old prediction fresh.

The backend must verify that the control source is still current immediately
before submission. `StaleControlSource` means definite non-submission: the loop
can discard that output and obtain a new source within its existing bounds.
Other application errors/timeouts stop fresh commands. `RecoveryRequired`
means the original request or applied outcome needs explicit recovery.

`PausedRoomControl` uses the starter's existing pinned booking, monotonic server
clock mapping, persisted pending request, exact retry and publication checks.
A fresh command receives its first finite deadline and key once. It waits for
the actual native sequence/step advance; admission alone is not application.
An unknown transport result leaves the exact pending bytes/key/deadline intact
and refuses Close. Recover deliberately through the original starter journal,
then inspect the original outcome before fresh execution. The generic loop does
not replace an unknown request with a new key. A known admitted finite prefix
can drain through existing Close. A partial applied prefix remains recorded
and prevents another prediction from being treated as successful execution.

Finish is bounded and preserves actual terminal/publication status. The local
`run.json` report's `ok` means a completed client workflow with verified
publication, not task success, hardware accuracy or model qualification.
`trace.jsonl` records prediction sources, requested targets, application receipts,
discarded outputs and failures. These can contain private customer state: the
loop creates a fresh private output directory and exclusive private files. An
existing directory is never overwritten. Precondition failures occur before
control ownership; the caller still owns its already booked session.

Timeouts cancel cooperative work and discard late results, including a policy
that suppresses cancellation. They cannot preempt blocking Python, kill a GPU
kernel or stop a worker thread. Customer code must yield or offload inference;
hard process isolation/termination belongs outside this SDK. Backend cancellation
must prevent fresh submissions and preserve any original uncertain request.

## Add an adapter at the appropriate boundary

| Need | Extension point | Required evidence |
| --- | --- | --- |
| Another customer model | Async `Policy.prepare` / `predict` | Explicit preprocessing and decoder; finite inference; source freshness; actual task outcomes. |
| Camera, sensor or on-device result | `StreamReader.description` / `latest` supplied to `StreamMux` | Exact payload/version, clock mapping, provenance, size bounds, missing/late/error behavior; actual producer qualification. |
| Another paused device with the same lifecycle | Local `DeviceAdapter`, registered explicitly and injected into `PausedRoomControl` | Exact public and booked descriptions, named targets, source-bound state, units/frames/rates; rejection, retry, application, stop and release. |
| Another execution mode/lifecycle | `ControlSession` (`binding`, `description`, `snapshot`, `apply`, `finish`) | Original authority/cutoff/allowance; generation/source checks; durable exact recovery; applied prefix; cancellation and final publication. |

Inspect `execution_contracts.py` for the local protocol. Return the original
remaining command allowance in each snapshot. `apply` must report the actual
requested/applied rows and source digest; `finish` must report observed status,
not manufacture success. Register only trusted local adapters. A server response
cannot name a package to install, and a matching tensor width admits no hardware.
Stream producers never gain a motion API by being connected to the loop.

The current concrete backend covers paused SO100 capture `@4`, SO100 session
`@5` and paired NexArm `@1`. Generic continuous control needs a backend preserving
its own forecast/feedback semantics; the existing continuous `RobotClient`
remains available unchanged. Paused decisions still lack verified drive-target
anchors: measured positions are not a replacement for a VLA delta decoder's
required anchor. Negotiated video, physical devices, new sensor/inference
producers and a trained VLA execution trial remain separate work.

## Planned execution placement

[ADR 0058](architecture/0058-customer-decisions-and-execution-placement.md) adopts
remote movement control and onsite customer execution. These are placement choices
for the customer's brain. Actuator Science supplies drivers, local controllers,
bounded execution, independent observation and final evidence. A platform model
need not choose movements. Compute placement is separate from software versus
physical simulation and from position/velocity/torque control semantics.
Remote control includes internet latency/jitter; onsite inference avoids that
transport leg but still needs measured model and local-controller timing.

| Workflow | SDK and runtime plan | Control responsibility |
| --- | --- | --- |
| Remote agent or VLA | Customer runs its policy on its own compute and submits finite movements over the internet. Separate observation access supplies source-bound feedback. | Room-local executor/controller meets its admitted cadence and enforces limits; internet round trips are not a hard real-time servo loop. |
| Onsite autonomous run | Customer rents a compatible CPU/GPU workstation, explicitly stages private code/model artifacts, prepares the runtime, then starts a finite authorized experiment. | Isolated customer worker sends locally through the same control authority. Published unattended policy determines whether remote observer loss allows continuation. |
| Onsite run with remote supervision | Same local workload; an external agent/human watches events/callbacks and live media, then retrieves final results. | Observation is read-only. Any supported pause/stop/task update requires explicit authority and generation fencing; there is one active command producer. |

### Keep the SDK boundaries separate

- **Discovery/session access:** describe admitted execution placement, exact robot
  control semantics, controller ownership, CPU/RAM, optional GPU/VRAM/runtime,
  readiness, network/timing limits and the full price before confirmation. These
  are planned descriptions, not fields retroactively added to today's `@1` API.
- **Customer policy:** retain explicit preprocessing, decoder, source requirements
  and prepare/predict. `PreparedPolicy` is a local Python object, not a deployable
  wire artifact. Future onsite submission must explicitly identify code, weights,
  runtime/dependencies and private artifact access; never implicitly pickle/upload
  a model or execute a factory inside the gateway.
- **Control backend and driver:** retain the same `ControlSession` authority,
  generation, source-age, exact recovery and applied-prefix responsibilities.
  The worker's location changes; controller qualification does not disappear.
  Fast servo/balance behavior stays in its admitted local controller.
- **Observations/events:** independent clients consume cameras/sensors, platform
  run events and distinctly labeled customer progress. Source/model provenance,
  timestamps, cursors/gaps, terminal state and reconnect bounds stay explicit.
  Optional outbound callbacks need configured authenticated destinations and
  deduplication/retry limits; their delivery never drives or renews motion.
- **Results:** terminal events reference the existing authoritative job and export.
  Retrieve immutable measurements, requested/applied actions, failures, model/runtime
  identity and usage after publication. A callback summary or live preview is not
  the final evidence bundle.

### Delivery and acceptance

Package 4 describes placement and control ownership while completing the existing
remote SDK boundaries. Onsite workload submission, compute leases, callbacks and
arbitrary customer-code isolation follow in the roadmap's onsite package. Keep
optional model/GPU packages in the isolated runtime, not the core stream SDK.
CPU-only operation remains supported. No universal hardware mode, hosted runtime
API or SDK version increase is implied by this plan.

Start with a single published room-plus-workstation configuration and existing
quote/hold/booking/job authority. Pin model/runtime and room identities before
execution. Make staging/warmup finite, priced and distinct from the robot control
window; disclose which resources remain occupied while preparing. Admit GPU
capacity only after measured compatibility and isolation. Reuse the workspace
budget, reservation and settlement for the full capped quote.

Prove the first onsite workload against software simulation. Exercise model load
failure, process/GPU hang or crash, lease expiry/revocation, stale output,
competing control owners, observer disconnect/reconnect, slow or duplicated events,
deadline/cap exhaustion and final result/usage reconciliation after release.
Cancellation must terminate/fence the isolated workload through a qualified worker
design; the current cooperative asyncio timeout cannot provide that isolation.
Preserve old remote feedback/disconnection behavior; unattended continuation is
a separately admitted promise. Measure command and stream/callback latency
separately and retain failed outcomes. Actual hardware then needs its own local
tracking, stop, recovery and physical commissioning evidence.