Merge with opensource

.env

@@ -1,7 +1,16 @@
-# Can be either basic (with single instance of Zookeeper, Kafka and Redis) or high-availability (with Zookeeper, Kafka and Redis in cluster modes).
+# Platform used to bootstrap the cluster, can be either minikube or openshift or aws.
+# According to the platform corresponding kubernetes resources will be deployed (see content of the directories ./minikube, ./openshift ./gcp and ./aws for details).
+PLATFORM=
+
+# Can be either basic (with single instance of Zookeeper, Kafka and Redis) or high-availability (with Zookeeper, Kafka and Redis in cluster modes and PostgreSQL in replica mode if chosen).
 # According to the deployment type corresponding kubernetes resources will be deployed (see content of the directories ./basic and ./high-availability for details).
 DEPLOYMENT_TYPE=basic
 PLATFORM=
 # Database used by ThingsBoard, can be either postgres (PostgreSQL) or hybrid (PostgreSQL for entities database and Cassandra for timeseries database).
 # According to the database type corresponding kubernetes resources will be deployed (see postgres.yml, cassandra.yml for details).
 DATABASE=postgres
+
+# Replication factor for Cassandra database (will be ignored if PostgreSQL was selected as the database).
+# Must be less or equals to the number of Cassandra nodes which can be configured in ./common/cassandra.yml ('StatefulSet.spec.replicas' property)
+CASSANDRA_REPLICATION_FACTOR=1
+

README.md

@@ -1,189 +1,4 @@
-# Kubernetes resources configuration for ThingsBoard Microservices
+# Kubernetes resources configuration for ThingsBoard
 
-This folder containing scripts and Kubernetes resources configurations to run ThingsBoard in Microservices mode.
+Here you can find scripts for deployment on different Kubernetes platforms.
 
-## Prerequisites
-
-ThingsBoard Microservices run on the Kubernetes cluster.
-You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
-If you do not have a cluster already, you can create one by using [Minikube](https://kubernetes.io/docs/setup/minikube),
-AWS, GCP or you can choose any other available [Kubernetes cluster deployment solutions](https://unofficial-kubernetes.readthedocs.io/en/latest/setup/pick-right-solution/).
-
-### Minikube Configuration
-
-#### Enable ingress addon 
-By default ingress addon is disabled in the Minikube, and available only in cluster providers.
-To enable ingress, please execute the following command:
-
-`
-minikube addons enable ingress
-` 
-
-### AWS Configuration
-
-To configure AWS setup, please choose one of the directory and use README.md there.  After configuring AWS, you can continue the installation from this step.
-
-- ./aws/kubeone - KubeOne automates cluster operations on all your aws instances. KubeOne can install high-available (HA) master clusters as well single master clusters.
-
-- ./aws/eks - Amazon EKS is a completely AWS-managed Kubernetes service.
-
-## GCP Configuration
-
-To configure GCP setup, plesae go to the ./gcp directory and use README.md there.  After configuring GCP, you can continue the installation from this step.
-
-## Upload Docker credentials
-
-Make sure your have logged in to docker hub using command line. To upload Docker credentials, please execute next command:
-
-`
-./k8s-upload-docker-credentials.sh
-`
-
-Or you can use the following command:
-
-`
-kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=[YOUR_USERNAME] --docker-password=[YOUR_PASSWORD] --docker-email=[YOUR_EMAIL]
-`
-
-## Installation
-
-Before performing initial installation you have to select correct `PLATFORM` in `.env` file depending on the real cluster platform you are using (`minikube`, `gcp`, `aws` or `aws-eks`).
-
-Also, you can configure the type of database to be used with ThingsBoard and the type of deployment.
-In order to set database type change the value of `DATABASE` variable in `.env` file to one of the following:
-
-- `postgres` - use PostgreSQL database;
-- `hybrid` - use PostgreSQL for entities database and Cassandra for timeseries database;
-
-**NOTE**: According to the database type corresponding kubernetes resources will be deployed (see `postgres.yml`, `cassandra.yml` for details).
-
-In order to set deployment type change the value of `DEPLOYMENT_TYPE` variable in `.env` file to one of the following:
-
-- `basic` - start up with single instance of Zookeeper, Kafka and Redis;
-- `high-availability` - start up with Zookeeper, Kafka and Redis in cluster modes;
-
-**NOTE**: According to the deployment type corresponding kubernetes resources will be deployed (see the content of the directories `./basic` and `./high-availability` for details).
-
-Execute the following command to run the installation:
-
-`
-./k8s-install-tb.sh --loadDemo
-`
-
-Where:
-
-- `--loadDemo` - optional argument. Whether to load additional demo data.
-
-## Configure your license key
-
-`
-nano common/tb-node.yml
-`
-
-and put the license secret parameter 
-
-```
-tb-node StatefulSet configuration
-
-- name: TB_LICENSE_SECRET
-  value: "PUT_YOUR_LICENSE_SECRET_HERE"
-```
-
-## Running
-
-Execute the following command to deploy third-party resources:
-
-`
-./k8s-deploy-thirdparty.sh
-`
-
-Type **'yes'** when prompted, if you are running ThingsBoard in `high-availability` `DEPLOYMENT_TYPE` for the first time and don't have configured Redis cluster.
-
-Before deploying ThingsBoard resources you should configure number of pods for each service. 
-You can do it in `thingsboard.yml` by changing `spec.replicas` fields for different services. 
-It is recommended to have at least 2 `tb-node` and 10 `tb-js-executor`.
-Execute the following command to deploy resources:
-
-`
-./k8s-deploy-resources.sh
-`
-
-If you have used minikube after a while when all resources will be successfully started you can open `http://{your-cluster-ip}` in your browser (for ex. `http://192.168.99.101`).
-You should see the ThingsBoard login page.
-
-If you have used aws or gcp installations you can open ThingsBoard web interface in your browser using dns name or ip of the load balancer.
-
-You can see DNS name or ip of the loadbalancer using command:
-
-`
-kubectl get ingress -oyaml
-`
-
-Or you can see this name on the ELB page.
-
-You should see the ThingsBoard login page.
-
-Use the following default credentials:
-
-- **System Administrator**: [email protected] / sysadmin
-
-If you installed DataBase with demo data (using `--loadDemo` flag) you can also use the following credentials:
-
-- **Tenant Administrator**: [email protected] / tenant
-- **Customer User**: [email protected] / customer
-
-In case of any issues, you can examine service logs for errors.
-For example to see ThingsBoard node logs execute the following commands:
-
-1) Get the list of the running tb-node pods:
-
-`
-kubectl get pods -l app=tb-node
-`
-
-2) Fetch logs of the tb-node pod:
-
-`
-kubectl logs -f [tb-node-pod-name]
-`
-
-Where:
-
-- `tb-node-pod-name` - tb-node pod name obtained from the list of the running tb-node pods.
-
-Or use `kubectl get pods` to see the state of all the pods.
-Or use `kubectl get services` to see the state of all the services.
-Or use `kubectl get deployments` to see the state of all the deployments.
-See [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) command reference for details.
-
-Execute the following command to delete all ThingsBoard microservices:
-
-`
-./k8s-delete-resources.sh
-`
-
-Execute the following command to delete all third-party microservices:
-
-`
-./k8s-delete-thirdparty.sh
-`
-
-Execute the following command to delete all resources (including database):
-
-`
-./k8s-delete-all.sh
-`
-
-## Upgrading
-
-In case when database upgrade is needed, execute the following commands:
-
-```
-./k8s-delete-resources.sh
-./k8s-upgrade-tb.sh --fromVersion=[FROM_VERSION]
-./k8s-deploy-resources.sh
-```
-
-Where:
-
-- `FROM_VERSION` - from which version upgrade should be started. See [Upgrade Instructions](https://thingsboard.io/docs/user-guide/install/upgrade-instructions) for valid `fromVersion` values.

aws/README.md

@@ -0,0 +1,33 @@
+# AWS deployment scripts
+
+Here you can find scripts for different deployment scenarios using AWS platform:
+
+- [**monolith**](https://thingsboard.io/docs/user-guide/install/cluster/aws-monolith-setup/) - simplistic deployment of ThingsBoard monolith 
+with [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/). 
+Recommended for deployment scenarios that may sacrifice high availability to **optimize the cost**. 
+- [**microservices**](https://thingsboard.io/docs/user-guide/install/cluster/aws-microservices-setup/) - deployment of ThingsBoard microservices 
+with [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/), [Amazon MSK](https://aws.amazon.com/msk/) 
+and [ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/). Recommended for **scalable and highly available** deployments. 
+- [**custom-microservices**](https://thingsboard.io/docs/user-guide/install/cluster/aws-custom-microservices-setup/) - deployment of ThingsBoard microservices 
+alongside with self-managed PostgreSQL, Kafka and Redis.
+
+
+## Prerequisites
+
+For each of the AWS use-cases you will need to have `kubectl`, `eksctl` and `awscli` tools installed.
+Here you can find installation guides:
+
+- for [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- for [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html)
+- for [awscli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
+
+Afterwards you need to configure Access Key, Secret Key and default region. 
+To get Access and Secret keys please follow [this](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) guide.
+The default region should be the ID of the region where you'd like to deploy the cluster.
+
+```
+aws configure
+```
+
+Note that you also need to set `ACCOUNT_ID` property in `.env` file.
+[Here](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId) a guide how to find your account ID.

aws/custom-microservices/README.md

@@ -0,0 +1,5 @@
+# AWS custom microservices deployment scripts
+
+This folder containing scripts and Kubernetes resources configurations to run ThingsBoard in Custom-Microservices mode on AWS cluster.
+
+You can find the deployment guide by the [**link**](https://thingsboard.io/docs/user-guide/install/cluster/aws-custom-microservices-setup/).

aws/custom-microservices/cluster.yml

@@ -0,0 +1,20 @@
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+availabilityZones: [us-east-1a,us-east-1b,us-east-1c]
+
+metadata:
+  name: thingsboard-cluster
+  region: us-east-1
+  version: "1.20"
+
+managedNodeGroups:
+  - name: node
+    instanceType: m5.large
+    desiredCapacity: 4
+    maxSize: 4
+    minSize: 4
+    labels: { type: node }
+    ssh:
+      allow: true
+      publicKeyPath: '~/.ssh/aws_rsa.pub'

aws/custom-microservices/database-setup.yml

@@ -0,0 +1,52 @@
+#
+# Copyright © 2016-2020 The Thingsboard Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Pod
+metadata:
+  name: tb-db-setup
+  namespace: thingsboard
+spec:
+  volumes:
+    - name: tb-node-config
+      configMap:
+        name: tb-node-config
+        items:
+          - key: conf
+            path: thingsboard.conf
+          - key: logback
+            path: logback.xml
+    - name: tb-node-logs
+      emptyDir: {}
+  containers:
+    - name: tb-db-setup
+      imagePullPolicy: Always
+      image: store/thingsboard/tb-pe-node:3.2.2PE
+      env:
+        - name: TB_SERVICE_ID
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+      envFrom:
+        - configMapRef:
+            name: tb-node-db-config
+      volumeMounts:
+        - mountPath: /config
+          name: tb-node-config
+        - mountPath: /var/log/thingsboard
+          name: tb-node-logs
+      command: ['sh', '-c', 'while [ ! -f /tmp/install-finished ]; do sleep 2; done;']
+  restartPolicy: Never

aws/custom-microservices/k8s-delete-all.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Copyright © 2016-2020 The Thingsboard Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+kubectl delete -f routes.yml
+
+kubectl -n thingsboard delete svc,sts,deploy,cm,po,ing --all
+
+kubectl -n thingsboard get pvc --no-headers=true | awk '//{print $1}' | xargs kubectl -n thingsboard delete --ignore-not-found=true pvc

k8s-delete-resources.sh → aws/custom-microservices/k8s-delete-resources.sh

@@ -19,5 +19,5 @@ set -e
 
 kubectl config set-context $(kubectl config current-context) --namespace=thingsboard
 
-kubectl delete -f common/thingsboard.yml
-kubectl delete -f common/tb-node.yml
+kubectl delete -f thingsboard.yml
+kubectl delete -f tb-node.yml

aws/custom-microservices/k8s-delete-thirdparty.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright © 2016-2020 The Thingsboard Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+kubectl config set-context $(kubectl config current-context) --namespace=thingsboard
+kubectl delete -f thirdparty.yml