Robot/Shape "instance handle" redesign (animation-loop API)

[petercorke]

Background

Discussed 2026-07-26 while planning the Swift frontend rebuild's
successor Python API. The current animation-loop pattern --
env.add(robot), then mutate robot.q directly and call env.step()
in a loop -- makes roboticstoolbox.Robot carry live simulation state
(.q, and transitively SceneNode/_propogate_scene_tree() world-
transform bookkeeping), which is exactly the drift RTB's own
desiderata.md already documents as unwanted ("Stateless over
stateful... no internal state arrays such as a persistent .q").

Direction agreed (implemented in large part, see below)

env.add() should return a lightweight, Swift-owned instance handle
that carries the live per-simulation state (q, plausibly base/
tool) instead of the robot model itself carrying it. Both a Shape
handle and a Robot handle satisfy a shared minimal contract -- something
like part_poses() -> list[SE3] -- trivial for a Shape (its own pose,
one part), computed via pure FK (fkine/fkine_all(q), not
SceneNode) for a Robot (N parts, one per link/gripper geometry).
Proposed as a typing.Protocol owned by swift itself, not a shared
base class, since Robot (roboticstoolbox) and Shape
(spatialgeometry) are unrelated classes from separate packages.

Default env.add(robot) holds a reference to the model (cheap,
common case); an opt-in env.add(robot, clone=True) would deep-copy
the kinematic structure for the case where one shared model needs
multiple independently-placed instances -- though if base also ends
up living on the handle rather than the model (open question, see
below), that specific use case may not need clone=True at all.

Implemented so far

• Partial progress (2026-07-26): Swift.py's per-step rendering hot
  path (_draw_all()/_step_robot()) now calls RTB's new
  Robot.fkine_geometry(robot.q, robot_alpha, collision_alpha) instead
  of _update_link_tf()/_propogate_scene_tree() -- so per-frame
  rendering no longer depends on the scene-graph mutation machinery at
  all. Confirmed working live (two robots + gripper fingers + a moving
  shape, ~90s run).
• Handle implemented (2026-07-27): env.add(robot) now returns a
  RobotHandle (swift/Handle.py) that owns q, qd, and
  control_mode for that instance -- the roboticstoolbox.Robot passed
  in stays a plain, shareable kinematic model, used functionally
  (panda.fkine(handle.q), panda.jacobe(handle.q)).
  examples/two_link_arm.py demonstrates a non-RTB object satisfying
  the same contract by hand.
  • Backward compatibility: mutating robot.q/robot.qd/robot.control_mode
    directly (the old pattern) still works -- RobotHandle._sync_legacy()
    detects the model's state diverging from a snapshot and adopts it,
    emitting one DeprecationWarning per handle. Deliberately not a
    generic RTB-wide deprecation of Robot.q/Robot.qd -- only swift's
    own old animation-loop reliance on them is deprecated.
• Assembly API generalization (2026-07-29): the handle is no longer
  robot-specific -- RobotHandle is now AssemblyHandle, constructed
  from a pose_fn(q) -> list[SE3] callable plus initial q; robot=
  is optional. Swift.add()'s single isinstance if/elif tree is now
  four explicit methods -- add_shape(), add_ui(),
  add_assembly(fk, parts, q0=, callback=, name=),
  add_robot(robot, callback=, name=, ...). env.add() still exists
  for backward compatibility, dispatching by type.
  • A per-step callback (callback=lambda t, values: ...) lets a
    scene run off a plain while True: env.step(dt) loop with no
    per-step mutation written by hand.
  • Named UI elements (add_ui(el, name=...)) push .value into
    env.values on every change, including browser-driven changes.
  • env.show() prints the current display list for debugging.
• r2q C-acceleration investigated, decided against (2026-07-29/30):
  looked into replacing sm.base.r2q (pure Python, called once per
  geometry part per step) with a C-accelerated version. Found RTB's
  fknm.r2q() is currently broken (wrong arg count in its Python
  wrapper) and swift's own phys.cpp has a dead, never-registered copy
  of the same algorithm. Benchmarked at ~23x faster per call, but not a
  measured bottleneck (~3.6ms/sec total for a 15-part robot at 60fps).
  Decided not worth pursuing: the sign-disambiguation corner cases in
  that C++ code haven't been verified against fixes spatialmath.base.r2q
  may have picked up over time, and building the corner-case test suite
  needed to trust it isn't worth it for a currently-nonexistent
  bottleneck. Separately confirmed (2026-07-30) that the actually live
  r2q in this stack -- spatialgeometry's scene_nb.cpp, a third,
  different algorithm (Shepperd's method) -- is independently correct,
  verified numerically against sm.base.r2q to float64 epsilon across
  identity/90/180/179.99-degree/near-identity/arbitrary-compound cases.

Still open

• Whether base/tool belong on the model (part of the robot's
  kinematic definition -- genuinely true for some fixed-pedestal-mounted
  robots) or on the handle (an instance-placement concern, like q).
  Needs answering before the handle redesign can be considered complete.
  Coupled with a matching open item in roboticstoolbox-python's own
  tech-debt tracking -- the two repos' redesigns should land together.
• env.add(robot, clone=True) for multiple independently-placed
  instances of one shared model is not implemented (two RobotHandles
  over one model already have independent q/qd, so simple
  multi-instance cases work today, just without a convenience clone).
• Gripper .q is still read directly off the Gripper object inside
  fkine_geometry(), not handle-owned.
• Shape does not get a matching ShapeHandle -- unlike Robot,
  Shape was never a shared model with conflated instance state, so
  env.add(shape) still returns a plain int id.
• Time slider / recorded playback (design idea, not started,
  2026-07-29): since a per-step callback is (t, values) -> pose_or_q
  and env already owns t, a pure callback can be scrubbed for free
  by re-evaluating at any t. A stateful one (e.g. one that returns
  handle.q + qd * dt) can't -- its output is a recurrence over the
  assembly's entire history. Proposed fix: record (t, q) per handle as
  the simulation runs forward, then have a time slider seek/interpolate
  into that recording rather than re-invoking the callback for past
  time -- same shape of feature as the existing recording/video capture,
  recording state instead of pixels. Two things to pin down before this
  becomes a real design: (1) passive scrub-and-look vs.
  scrub-back-and-resume-live-simulation (not generally safe -- a
  fancier controller could carry state beyond handle.q that recording
  q alone wouldn't capture); (2) whether recording is always-on or
  explicitly started like video recording.
• Playwright follow-up (not done): headless browser automation as a
  replacement for manual, live screenshot-and-console debugging -- would
  let CI (or a single local command) catch console errors, failed
  network requests, and blank/broken renders automatically. Discussed
  but not installed or wired into swift's test suite.