Merge pull request #746 from Mirantis/jell/vmpodlc

Documentation of VM pod lifecycle

Author: ivan4th

docs/README.md

@@ -4,6 +4,7 @@ This directory contains the Virtlet documentation.
 * [Cloud-init data generation](cloud-init-data-generation.md)
 * [Developer documentation](devel/README.md)
 * [Architecture overview](architecture.md)
+    * [VM pod lifecycle](vmpod-lifecycle.md)
     * [Description of networking](networking.md)
     * [Description of SyncPod workflow](sync-pod-workflow.md)
     * [Volume handling](volumes.md)

docs/vmpod-lifecycle.md

@@ -0,0 +1,97 @@
+# Lifecycle of a VM pod
+
+This document describes the lifecycle of VM pod managed by Virtlet.
+
+This description omits the details of volume setup (using
+[flexvolumes](https://kubernetes.io/docs/concepts/storage/volumes/#flexvolume)),
+handling of logs, the VM console and port forwarding (done by
+[streaming server](https://github.com/Mirantis/virtlet/tree/master/pkg/stream)),
+ or port forwarding.
+
+## Assumptions
+
+Communication between kubelet and Virtlet goes through [criproxy](https://github.com/Mirantis/criproxy)
+which directs requests to Virtlet only if the requests concern a pod that has
+Virtlet-specific annotation or an image that has Virtlet-specific prefix.
+
+## Lifecycle
+
+### VM Pod Startup
+
+ * A pod is created in Kubernetes cluster, either directly by the user or via
+   some other mechanism such as a higher-level Kubernetes object managed by
+   `kube-controller-manager` (ReplicaSet, DaemonSet etc.).
+ * Scheduler places the pod on a node based on the requested resources
+   (CPU, memory, etc.) as well as pod's nodeSelector and pod/node affinity
+   constraints, taints/tolerations and so on.
+ * `kubelet` running on the target node accepts the pod.
+ * `kubelet` invokes a [CRI](https://contributor.kubernetes.io/contributors/devel/container-runtime-interface/)
+   call RunPodSandbox to create the pod sandbox which
+   will enclose all the containers in the pod definition. Note that at this
+   point no information about the containers within the pod is passed
+   to the call. `kubelet` can later request the information about the pod
+   by means of `PodSandboxStatus` calls.
+ * If there's a Virtlet-specific annotation `kubernetes.io/target-runtime: virtlet.cloud`,
+   CRI proxy passes the call to Virtlet.
+ * Virtlet saves sandbox metadata in its internal database, sets up the
+   network namespace and then uses internal `tapmanager` mechanism to invoke
+   `ADD` operation via the CNI plugin as specified by the
+   CNI configuration on the node.
+ * The CNI plugin configures the network namespace by setting up
+   network interfaces, IP addresses, routes, iptables rules and so on,
+   and returns the network configuration information to the caller as described
+   in the [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md#result).
+ * Virtlet's [`tapmanager`](https://github.com/Mirantis/virtlet/tree/master/pkg/tapmanager)
+   mechanism adjusts the configuration of the network namespace to make it work with the VM.
+ * After creating the sandbox, kubelet starts the containers defined in
+   the pod sandbox. Currently, Virtlet supports just one container per VM pod.
+   So, the VM pod startup steps after this one describe the startup of this single container.
+ * Depending on the image pull police of the container, kubelet checks if
+   the image needs to be pulled by means of `ImageStatus` call and then uses
+   `PullImage` CRI call to pull the image if it doesn't exist or if
+   `imagePullPolicy: Always` is used.
+ * If `PullImage` is invoked, Virtlet resolves the image location based on the
+   [image name translation configuration](https://github.com/Mirantis/virtlet/blob/master/docs/image-name-translation.md),
+   then downloads the file and stores it in the image store.
+ * After the image is ready (no pull was needed or the `PullImage` call completed
+   successfully), kubelet uses `CreateContainer` CRI call to create
+   the container in the pod sandbox using the specified image.
+ * Virtlet uses the sandbox and container metadata to generate libvirt domain definition,
+   using [`vmwrapper`](https://github.com/Mirantis/virtlet/tree/master/cmd/vmwrapper)
+   binary as the emulator and without specifying any network configuration in the domain.
+ * After `CreateContainer` call completes, `kubelet` invokes `StartContainer` call
+   on the newly created container.
+ * Virtlet starts the libvirt domain. libvirt invokes `vmwrapper` as the emulator,
+   passing it the necessary command line arguments as well as environment variables
+   set by Virtlet. `vmwrapper` uses the environment variable values passed
+   to Virtlet to communicate with `tapmanager` over an Unix domain socket,
+   retrieving a file descriptor for a tap device and/or pci address of SR-IOV
+   device set up by `tapmanager`. `tapmanager` uses its own simple protocol to
+   communicate with `vmwrapper` because it needs to send file descriptors over
+   the socket. This is not usually supported by RPC libraries, see e.g.
+   [grpc/grpc#11417](https://github.com/grpc/grpc/issues/11417).
+   `vmwrapper` then updates the command line arguments to include the network
+   interface information and execs the actual emulator (`qemu`).
+
+At this point the VM is running and accessible via the network, and the pod is
+in `Running` state as well as it's only container.
+
+### Deleting the pod
+
+This sequence is initiated when the pod is deleted, either by means of `kubectl delete`
+or a controller manager action due to deletion or downscaling of a higher-level object.
+
+ * `kubelet` notices the pod being deleted.
+ * `kubelet` invokes `StopContainer` CRI calls which is getting forwared
+   to Virtlet based on the containing pod sandbox annotations.
+ * Virtlet stops the libvirt domain. libvirt sends a signal to `qemu`, which initiates
+   the shutdown. If it doesn't quit in a reasonable time determined by pod's
+   termination grace period, Virtlet will forcibly terminate the domain,
+   thus killing the `qemu` process.
+ * After all the containers in the pod (the single container in case of
+   Virtlet VM pod) are stopped, kubelet invokes `StopPodSandbox` CRI call.
+ * Virtlet asks its `tapmanager` to remove pod from the network by means of
+   `CNI DEL` command.
+ * after `StopPodSandbox` returns, the pod sandbox will be eventually GC'd
+   by `kubelet` by means of `RemovePodSandbox` CRI call.
+ * Upon `RemovePodSandbox`, Virtlet removes the pod metadata from its internal database.