The current configuration uses two types of persistent volume storage:
- Dynamic (for pre-production)
- Static (for production)
Dynamically provisioned volumes are accomplished using the iscsi-provisioner. In a pre-production environment the agility of dynamic provisioning is helpful in support of dynamic feature branch QA deployments. Since it is pre-production, we are less concerned with High Availability and so a single Targetd Storage Appliance can fit the need.
Statically provisioned volumes are pre-allocated iSCSI LUNs on an external production grade storage appliance. In production, data must be cared for and managed for High Availability and Disaster Recovery. In this case, technology like the Amazon Storage Gateway or direct Compellant allocated iSCSI LUNs are the best strategy.
By combining these two approaches we achieve a lightweight, minimalist and robust persistent volume strategy that is not subject to periodic breakage due to complex software interactions like ESXi Kernel module VIBS or 3rd party storage providers.
cluster-builder supports the automated deployment of dedicated Kubernetes Cluster Targetd storage appliances. As depicted in the diagram above, these can be paired with Kubernetes Clusters to provide controlled dynamic storage.
The Targetd Storage Appliance supports up to 255 persistent volumes per server instance. If this is insufficient, multiple provisioners can be deployed with multiple targetd storage appliance backends in the same Kubernetes cluster.
The default configuration of the appliance is a 1 TB thinly provisioned LVM volume on VMDK. This can be resized as needed but represents a solid initial storage footprint for a pre-production environment.
The Targetd Host Configuration File illustrates the settings required for deploying a targetd storage appliance (it should look familiar):
[all:vars]
cluster_type=targetd-server
cluster_name=targetd
remote_user=admin
vmware_target=esxi
overwrite_existing_vms=true
esxi_net="VM Network"
esxi_net_prefix=192.168.1
network_mask=255.255.255.0
network_gateway=192.168.1.1
network_dns=8.8.8.8
network_dns2=8.8.4.4
network_dn=onprem.idstudios.io
targetd_server=192.168.1.205
targetd_server_iqn=iqn.2003-01.org.linux-iscsi.minishift:targetd
targetd_server_volume_group=vg-targetd
targetd_server_provisioner_name=iscsi-targetd
targetd_server_account_credentials=targetd-account
targetd_server_account_username=admin
targetd_server_account_password=ciao
targetd_server_namespace=default
[targetd_server]
storage-server ansible_host=192.168.1.205 esxi_host=esxi-6 esxi_user=root
[vmware_vms]
storage-server numvcpus=4 memsize=6144 esxi_host=esxi-6 esxi_user=root esxi_ds=datastore6-ssd
targetd_server= The ip address of the targetd server (as per the anisble_host value)
targetd_server_iqn= A valid and unique iSCSI IQN
targetd_server_volume_group= vg-targetd is the default but can be modified.
targetd_server_provisioner_name= A unique name given to the iscsi provisioner
targetd_server_account_credentials= The name of the K8s secret that will store the targetd server credentials
targetd_server_account_username= The username for targetd RPC remote access (used by the provisioner)
targetd_server_account_password= The password for targetd RPC remote access (used by the provisioner)
targetd_server_namespace= This should be default, but may support additional namespaces with configuration
Once the hosts file has been prepared, deployment uses the familiar cluster-builder convention:
bash cluster-deploy <targetd package folder>
Which should result in a deployed Targetd Server Appliance with 1 TB of thinly provisioned LVM storage to be allocated dynamically as K8s PVCs are requested.
Make sure to copy the targetd configuration into the kubeadm cluster package hosts file:
targetd_server=192.168.1.205
targetd_server_iqn=iqn.2003-01.org.linux-iscsi.minishift:targetd
targetd_server_volume_group=vg-targetd
targetd_server_provisioner_name=iscsi-targetd
targetd_server_account_credentials=targetd-account
targetd_server_account_username=admin
targetd_server_account_password=ciao
These same settings will be used to create the corresponding ISCSI provisioner manifests that will bind the provisioner to the Targetd Storage Appliance.
With the Targetd Storage Appliance configuration values in our cluster configuration file we can run the cluster-builder ansible script to configure k8s for iSCSI direct:
ansible-playbook -i clusters/ids/core ansible/centos-iscsi.yml
At this stage you must setup your kubectl configuration before proceeding - ensure you can connect to your kubeadm cluster
We will then install the iSCSI Provisioner:
./iscsi-secret.sh
This will create the secret credentials that the iSCSI provisioner will use to connect to the Targetd server.
kubectl apply -f iscsi.yml
This will create the necessary roles, as well as install the iscsi-provisioner deployment and the corresponding storage class.
kubectl get sc
Will show the storage class.
kubectl get pods
Will show the running iscsi-provisioner. Check the logs and you should see something like this:
time="2018-05-31T23:30:11Z" level=debug msg="start called"
time="2018-05-31T23:30:11Z" level=debug msg="creating in cluster default kube client config"
time="2018-05-31T23:30:11Z" level=debug msg="kube client config created" config-host="https://10.3.0.1:443"
time="2018-05-31T23:30:11Z" level=debug msg="creating kube client set"
time="2018-05-31T23:30:11Z" level=debug msg="kube client set created"
time="2018-05-31T23:30:11Z" level=debug msg="targed URL http://admin:[email protected]:18700/targetrpc"
time="2018-05-31T23:30:11Z" level=debug msg="iscsi provisioner created"
The iSCSI provisioner is now ready to deploy iSCSI PVC volumes:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: iscsi-test-volume
annotations:
volume.beta.kubernetes.io/storage-class: "iscsi-targetd-vg-targetd"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
And you can run a benchmark test job on the Targetd iSCSI volumes:
kubectl apply -f iscsi-bench-pvc.yml
(wait 10 secs)
kubectl get pvc
kubectl get pv
kubectl apply -f iscsi-bench-job.yml
For better performance consider using Thickly provisioned VMware VMDK volumes.
Once your pod is running and you see the the PVCs are bound, your kubeadm cluster is ready to use dynamic iSCSI PVC provisioning and static iSCSI PVC storage.
Enjoy :)
On the Targetd Storage Appliance you can view the allocated volumes using the targetcli ls command (as root).
In the example screenshot we can see four pvc- volumes allocated in the vg-targetd volume group. These map directly to provisioned Kubernetes persistent volumes (PV) and persistent volume claims (PVC):
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
core-drupal-files-volume 10Gi RWX Retain Bound web/core-drupal-files-volume-claim 2d
ghost-files-volume 5Gi RWO Retain Bound idstudios/ghost-files-volume-claim 2d
mysql-main-iscsi-volume 10Gi RWO Retain Bound idstudios/mysql-main-iscsi-volume-claim 2d
pvc-f67ca813-65de-11e8-bcca-000c298174b4 3Gi RWO Delete Bound web/core-galera-seed-volume-claim iscsi-targetd-vg-targetd 2d
pvc-f67f1d81-65de-11e8-bcca-000c298174b4 3Gi RWO Delete Bound web/core-galera-node1-volume-claim iscsi-targetd-vg-targetd 2d
pvc-f681c73f-65de-11e8-bcca-000c298174b4 3Gi RWO Delete Bound web/core-galera-node2-volume-claim iscsi-targetd-vg-targetd 2d
pvc-f6842a73-65de-11e8-bcca-000c298174b4 3Gi RWO Delete Bound web/core-galera-node3-volume-claim iscsi-targetd-vg-targetd 2d
If we launch our benchmarking utility job, we will see an iscsi-benchmark-volume-claim created with a corresponding PV, which will map to a dynamically allocated pvc- volume we can verify with targetcli ls.
kubectl apply -f iscsi-bench.yml
And
kubectl get pvc
Will show a PVC of:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
iscsi-benchmark-target-volume-claim Bound pvc-f09f473d-6809-11e8-97f1-000c298174b4 3Gi RWO iscsi-targetd-vg-targetd 5s
With a corresponding PV of (kubectl get pv):
pvc-f09f473d-6809-11e8-97f1-000c298174b4 3Gi RWO Delete Bound default/iscsi-benchmark-target-volume-claim iscsi-targetd-vg-targetd 1m
Note the name of the volume: pvc-f09f473d-6809-11e8-97f1-000c298174b4. We see it has been allocated 3Gi, and using targetcli ls on the Targetd Storage Appliance, we can verify it has in fact been allocated (2nd from the top):
Going back to the benchmark job we launched, we can check the job (kubectl get jobs) and view the logs:
IDStudios Cluster Toolbox
-------------------------
mode: disk-bench
Performing dd performance test on target volume /target...
-----
WRITE (1st pass bs=1G count=1):
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 4.75949 s, 226 MB/s
WRITE (2nd pass):
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 4.98027 s, 216 MB/s
WRITE (1st pass bs=1M count=1024):
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 14.2731 s, 75.2 MB/s
WRITE (1st pass bs=1k count=10240):
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 89.5945 s, 117 kB/s
READ (1st pass bs=1M):
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00142444 s, 7.4 GB/s
READ (2nd pass bs=1M):
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00136206 s, 7.7 GB/s
sysctl: setting key "vm.drop_caches": Read-only file system
READ (no cache bs=1M):
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00126906 s, 8.3 GB/s
LATENCY (seconds / 1000 = latency in ms):
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.00142748 s, 359 MB/s
By using 1000 writes, the time in seconds is the latency for a single write in milliseconds.
-----
...
Run status group 0 (all jobs):
READ: bw=19.3MiB/s (20.2MB/s), 4339KiB/s-9747KiB/s (4443kB/s-9981kB/s), io=528MiB (554MB), run=3749-27388msec
WRITE: bw=9415KiB/s (9641kB/s), 4401KiB/s-8857KiB/s (4507kB/s-9070kB/s), io=272MiB (285MB), run=3749-29596msec
Disk stats (read/write):
sdf: ios=135139/64471, merge=0/971, ticks=53341/936230, in_queue=989559, util=99.71%
fio: file hash not empty on exit
-----
IOPing -R /target (disk seek rate (iops, avg))
--- /target (ext4 /dev/sdf) ioping statistics ---
10.1 k requests completed in 3.0 s, 3.4 k iops, 13.3 MiB/s
min/avg/max/mdev = 237 us / 293 us / 8.7 ms / 192 us
-----
IOPing -RL /target (disk sequential speed (MiB/s))
--- /target (ext4 /dev/sdf) ioping statistics ---
2.7 k requests completed in 3.0 s, 929 iops, 232.3 MiB/s
min/avg/max/mdev = 820 us / 1.1 ms / 9.4 ms / 464 us
-----
IOPing for 10 count...
4.0 KiB from /target (ext4 /dev/sdf): request=1 time=8.3 ms
4.0 KiB from /target (ext4 /dev/sdf): request=2 time=1.1 s
4.0 KiB from /target (ext4 /dev/sdf): request=3 time=1.5 ms
4.0 KiB from /target (ext4 /dev/sdf): request=4 time=505 us
4.0 KiB from /target (ext4 /dev/sdf): request=5 time=501 us
4.0 KiB from /target (ext4 /dev/sdf): request=6 time=2.6 ms
4.0 KiB from /target (ext4 /dev/sdf): request=7 time=940 us
4.0 KiB from /target (ext4 /dev/sdf): request=8 time=2.6 ms
4.0 KiB from /target (ext4 /dev/sdf): request=9 time=2.0 ms
4.0 KiB from /target (ext4 /dev/sdf): request=10 time=1.5 ms
--- /target (ext4 /dev/sdf) ioping statistics ---
10 requests completed in 10.1 s, 8 iops, 35.7 KiB/s
min/avg/max/mdev = 501 us / 112.2 ms / 1.1 s / 329.7 ms
-----
Benchmark complete... sleeping 300s (kill me anytime).
Now we will do as suggested, and kill (remove) our benchmark job:
kubectl delete -f iscsi-bench.yml
This will also remove the 3Gi PVC volume used for the test, which we can verify with targetcli ls:
And we see that the volume pvc-f09f473d-6809-11e8-97f1-000c298174b4 has been removed and the disk space has been reclaimed by LVM.
You can extend the Targetd Storage Appliance by adding another virtual hard disk to the VM, and then provisioning it manually:
sudo pvcreate /dev/sd_
sudo vgcreate vg-targetd-thick /dev/sd_
Where sd_ is the newly added volume. Targetd supports hosting multiple physical disks as block pools.
Edit the /etc/target/target.yaml file to support the additional block volume:
password: ciao
# defaults below; uncomment and edit
# if using a thin pool, use <volume group name>/<thin pool name>
# e.g vg-targetd/pool
# Use the multi pool syntax to support multiple volumes:
block_pools: [vg-targetd, vg-targetd-thick]
# pool_name: vg-targetd
user: admin
ssl: false
target_name: iqn.2003-01.org.linux-iscsi.minishift:targetd
And then of course you will need to restart the service:
sudo systemctl restart targetd
And validate the configuration:
sudo targetcli ls
Create a new storage class that uses the vg-targetd-thick pool (it is easiest to copy the one generated by cluster-builder as it contains the necessary initiator ids):
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: iscsi-targetd-vg-targetd-thick
provisioner: iscsi-targetd
parameters:
# this id where the iscsi server is running
targetPortal: 192.168.1.205:3260
iqn: iqn.2003-01.org.linux-iscsi.minishift:targetd
fsType: ext4
volumeGroup: vg-targetd-thick
# this is a comma separated list of initiators that will be give access to the created volumes, they must correspond to what you have configured in your nodes.
initiators: iqn.2016-04.com.iscsi:1459208749854afebb99c5953e7b2c36,iqn.2016-04.com.iscsi:21dfca155e2940b69824a35fc98ab4be,iqn.2016-04.com.iscsi:b4c735d4d7f545d4907b966947218be5,iqn.2016-04.com.iscsi:3805f9ad51fc40aca73e57c6787dab2e,iqn.2016-04.com.iscsi:4df8a3d976494f56b0405f3a1edcbdb4,iqn.2016-04.com.iscsi:95a1744390d641fe93da94a35fde654c,iqn.2016-04.com.iscsi:39839a9f9d8d480a89107b13cbce9eb1,iqn.2016-04.com.iscsi:6938c01c1b274a40948380063255f274
---
And then reference that class when creating PVCs.
The Targetd Storage Appliance has been tuned for ESXi Disk I/O as per the following article, as have all CentOS/RHEL VMs. It makes a remarkable difference.