Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 42 additions & 2 deletions cmd/kubectl-ate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ The kind overlay installed by `hack/install-ate-kind.sh --deploy-ate-system` alr
Port-forward the Jaeger UI and invoke any command with `--trace`:
```bash
kubectl port-forward -n otel-system svc/jaeger 16686:16686 &
kubectl ate get actor my-counter-1 --trace
kubectl ate get actor my-counter-1 -a demo --trace
# open http://localhost:16686 and search for the most recent trace
```

Expand Down Expand Up @@ -108,7 +108,7 @@ kubectl ate get workers -l <label-selector>
| `ATESPACE` | The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as `actor:<atespace>:<name>`. |
| `NAME` | The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises during `ResumeGoldenActor`. |
| `TEMPLATE` | The `ActorTemplate` the actor was created from, as `<namespace>/<name>` (the template namespace is distinct from `ATESPACE`). |
| `STATUS` | One of `STATUS_RESUMING`, `STATUS_RUNNING`, `STATUS_SUSPENDING`, `STATUS_SUSPENDED`. |
| `STATUS` | One of `STATUS_RESUMING`, `STATUS_RUNNING`, `STATUS_SUSPENDING`, `STATUS_SUSPENDED`, `STATUS_PAUSING`, `STATUS_PAUSED`, `STATUS_CRASHED`, `STATUS_DELETING`. |
| `ATEOM POD` | The worker pod (namespace/name) currently hosting the actor. Empty while suspended. |
| `ATEOM IP` | The pod IP of that worker. Empty while suspended. |
| `VERSION` | Monotonic integer that increments on every state transition (resume / suspend / checkpoint). Useful for distinguishing snapshots. |
Expand Down Expand Up @@ -167,10 +167,20 @@ kubectl ate resume actor my-actor -a <atespace>
# Suspend an actor (snapshots its state to storage and frees the worker)
kubectl ate suspend actor my-actor -a <atespace>

# Pause an actor (checkpoints it, but keeps the snapshot on the node VM)
kubectl ate pause actor my-actor -a <atespace>

# Delete an actor.
kubectl ate delete actor my-actor -a <atespace>
```

> **Note:** `suspend` and `pause` are both checkpoints, but they differ in where
> the snapshot lands. `suspend` uploads it to snapshot storage, so the actor can
> resume on any node. `pause` keeps it on the node VM, and the next `resume` is
> prioritized onto that same node — cheaper and faster, but tied to one node's
> availability. See [Lifecycle](../../docs/glossary.md#lifecycle) for the full
> definitions.

### Actor Snapshots

Suspending an actor creates a durable snapshot. Tags give snapshots stable,
Expand Down Expand Up @@ -208,6 +218,36 @@ kubectl ate logs actors my-actor -a <atespace> -f

Logs are streamable only while the actor is bound to a worker (i.e., `STATUS_RUNNING`). For history across worker migrations, route through a centralized log backend (Cloud Logging, Loki, etc.); see `docs/observability.md`.

### Resource Usage

`kubectl ate top` reports CPU and memory usage of worker pods. `workers` is the
only supported resource type.

```bash
# Usage for every worker pod
kubectl ate top workers

# Same filters as `get workers`: Kubernetes namespace, the atespace of the
# assigned actor, or worker pool labels
kubectl ate top workers -n <namespace>
kubectl ate top workers -a <atespace>
kubectl ate top workers -l <label-selector>
```

Metrics come from the Kubernetes metrics API, so the cluster needs
metrics-server (or an equivalent) installed.

#### `kubectl ate top workers` output columns

| Column | Meaning |
|---|---|
| `NAME` | The worker pod name. |
| `POOL` | The `WorkerPool` the pod belongs to. |
| `STATUS` | `FREE` (idle, ready to receive an actor) or `ASSIGNED` (currently hosting an actor). |
| `ASSIGNED ACTOR` | If `STATUS=ASSIGNED`, the actor reference `<namespace>/<template>/<actor-name>`. |
| `CPU(CORES)` | CPU currently consumed by the worker pod. |
| `MEMORY(bytes)` | Memory currently consumed by the worker pod. |

### Administration & Setup
Commands for bootstrapping the Substrate control plane and debugging local environments.

Expand Down
4 changes: 2 additions & 2 deletions demos/parking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ Fill both workers by requesting two actors, leaving them `RUNNING`:
curl -s -H "Host: p1.parking.actors.resources.substrate.ate.dev" http://localhost:8000
curl -s -H "Host: p2.parking.actors.resources.substrate.ate.dev" http://localhost:8000

kubectl ate get workers # both workers are now bound to p1 and p2
kubectl ate get actors # p1,p2 RUNNING; p3,p4 SUSPENDED
kubectl ate get workers # both workers are now bound to p1 and p2
kubectl ate get actors -a parking # p1,p2 RUNNING; p3,p4 SUSPENDED
```

Now request **p3** with timing. The pool is full, so this request **parks** —
Expand Down
2 changes: 1 addition & 1 deletion hack/update/proto-fmt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ files=(

# Run clang-format on the found files.
#
# Don't reflow long lines, wince those tend to be comments which then require
# Don't reflow long lines, since those tend to be comments which then require
# re-running the proto generators, which makes "update-all" awkward.
"${clangfmt}" \
-i \
Expand Down