From a09f8e4010c6b1fd18966ae493c615824bb6fdaf Mon Sep 17 00:00:00 2001 From: atovpeko Date: Tue, 25 Feb 2025 13:58:58 +0200 Subject: [PATCH 1/4] draft --- use-timescale/integrations/striim.md | 160 +++++++++++++++++++++++++ use-timescale/page-index/page-index.js | 5 + 2 files changed, 165 insertions(+) create mode 100644 use-timescale/integrations/striim.md diff --git a/use-timescale/integrations/striim.md b/use-timescale/integrations/striim.md new file mode 100644 index 0000000000..04fa36b9f4 --- /dev/null +++ b/use-timescale/integrations/striim.md @@ -0,0 +1,160 @@ +--- +title: Integrate Striim with Timescale Cloud +excerpt: Striim is a real-time data integration platform that enables you to ingest, process, and deliver data with minimal latency. Integrate Striim with Timescale Cloud +products: [cloud, mst, self_hosted] +keywords: [Eclipse Mosquitto, Striim, IoT, MQTT, integrate] +--- + +import IntegrationPrereqs from "versionContent/_partials/_integration-prereqs.mdx"; + +# Integrate Striim with $CLOUD_LONG + +[Striim][striim] is a real-time data integration platform that enables you to ingest, process, and deliver streaming data across various systems with minimal latency. It connects data sources with targets and transforms the data on the fly. + +[Eclipse Mosquitto][mosquitto] is an open-source MQTT broker widely used for lightweight messaging in IoT and mobile applications. + +This page explains how to stream real-time IoT data from Eclipse Mosquitto to Timescale Cloud using Striim. + +## Prerequisites + + + +- Install [Eclipse Mosquitto][mosquitto-install]. +- Install [Striim Platform][striim-platform] or sign up for [Striim Cloud][striim-cloud]. + +## Simulate IoT data in Eclipse Mosquitto + +To prepare sample IoT data to stream to $CLOUD_LONG: + + + +1. **Start Eclipse Mosquitto** + + Run the following command: + + ```bash + mosquitto -v + ``` +1. **Publish sample data** + + Run the following command to publish data to the `sensor/data` MQTT topic: + + ```bash + mosquitto_pub -t sensor/data -m '{"temperature": 22.5, "humidity": 60}' + ``` + +1. **Verify the data** + + Subscribe to the topic to verify the data: + + ```bash + mosquitto_sub -t sensor/data + ``` + + You should see this: + + ```json + {"temperature": 22.5, "humidity": 60} + ``` + + + +## Prepare your $SERVICE_LONG to ingest data + +Create a table in $SERVICE_LONG to store IoT readings from Eclipse Mosquitto: + + + +1. **Connect to your $SERVICE_LONG** + + Use an [SQL editor][run-queries] in $CONSOLE. For self-hosted $TIMESCALE_DB, use [`psql`][psql]. + +1. **Create a hypertable in your $SERVICE_SHORT** + + ```sql + CREATE TABLE sensor_data ( + time TIMESTAMPTZ NOT NULL, + temperature FLOAT, + humidity FLOAT + ); + ``` + +1. **Convert the table into a hypertable** + + ```sql + SELECT create_hypertable('sensor_data', 'time'); + ``` + + + +## Create and test a data flow in Striim + +Configure Eclipse Mosquitto as the source and $CLOUD_LONG as the target, then connect them into a data flow: + + + +1. **Log in to Striim** + +1. **Create a data flow** + + 1. Navigate to `Applications` > `Create Application` > `Data Flow`. + 1. Name your application, for example, `MosquittoToTimescale`. + +1. **Configure the source** + + 1. Drag the `MQTT Source` component into the data flow. + 1. Enter the connection details for your Eclipse Mosquitto broker: `host`, `port`, and `sensor/data` for `topic`. + 1. Click `Test Connection`. + +1. **Configure the target** + + 1. Drag the `PostgreSQL Writer` component into the data flow. + 1. Enter the [connection details][connection-info] for your $SERVICE_SHORT. + 1. Choose `sensor_data` as the table name. + 1. Use the dropdowns in the `Field Mapping` section to map source fields to the destination columns. + +1. **Add a continuous query** + + 1. Drag the `Continuous Query` component between `MQTT Source` and `PostgreSQL Writer`. + 1. Open the continuous query configuration panel and write an SQL query to transform and map the data: + + ```sql + SELECT + jsonextractstring(payload, '$.time') AS time, + jsonextractfloat(payload, '$.temperature') AS temperature, + jsonextractfloat(payload, '$.humidity') AS humidity + FROM mqtt_source_stream; + ``` + +1. **Connect the source and the target** + + Draw a line from `MQTT Source` to `PostgreSQL Writer`. + +1. **Test the data flow** + + 1. Publish more sample data to the MQTT broker: + + ```bash + mosquitto_pub -t sensor/data -m '{"temperature": 24.3, "humidity": 55}' + ``` + + 1. Verify the data is streamed to your $SERVICE_LONG: + + ```sql + SELECT * FROM sensor_data; + ``` + + You should see your `sensor_data` hypertable now populated with sample readings from Eclipse Mosquitto. + + + +You have successfully created a data flow from Eclipse Mosquitto to $CLOUD_LONG using Striim. + +[connection-info]: /use-timescale/:currentVersion:/integrations/find-connection-details/ +[striim]: https://www.striim.com/ +[mosquitto]: https://mosquitto.org/documentation/ +[mosquitto-install]: https://mosquitto.org/download/ +[striim-platform]: https://www.striim.com/striim-platform/ +[striim-cloud]: https://www.striim.com/striim-cloud/ +[run-queries]: /getting-started/:currentVersion:/run-queries-from-console/ +[psql]: /use-timescale/:currentVersion:/integrations/psql/ diff --git a/use-timescale/page-index/page-index.js b/use-timescale/page-index/page-index.js index e1ff102ae8..6ea97c1767 100644 --- a/use-timescale/page-index/page-index.js +++ b/use-timescale/page-index/page-index.js @@ -848,6 +848,11 @@ module.exports = [ href: "qstudio", excerpt: "Integrate qstudio with Timescale Cloud", }, + { + title: "Striim", + href: "striim", + excerpt: "Integrate Striim with Timescale products", + }, { title: "Tableau", href: "tableau", From 9c679324b8f44c4204f596d6699887647a0a6604 Mon Sep 17 00:00:00 2001 From: atovpeko Date: Thu, 27 Feb 2025 11:32:51 +0200 Subject: [PATCH 2/4] update --- use-timescale/integrations/striim.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/use-timescale/integrations/striim.md b/use-timescale/integrations/striim.md index 04fa36b9f4..033e8fb3a4 100644 --- a/use-timescale/integrations/striim.md +++ b/use-timescale/integrations/striim.md @@ -43,20 +43,14 @@ To prepare sample IoT data to stream to $CLOUD_LONG: mosquitto_pub -t sensor/data -m '{"temperature": 22.5, "humidity": 60}' ``` -1. **Verify the data** +1. **Subscribe to topic** - Subscribe to the topic to verify the data: + Run the following command to subscribe to the `sensor/data` topic: ```bash mosquitto_sub -t sensor/data ``` - You should see this: - - ```json - {"temperature": 22.5, "humidity": 60} - ``` - ## Prepare your $SERVICE_LONG to ingest data From b019de71783606395cdf21197d53c40d155f12e9 Mon Sep 17 00:00:00 2001 From: Anagha Mittal Date: Fri, 7 Mar 2025 08:01:39 +0530 Subject: [PATCH 3/4] deployed app is running into errors --- use-timescale/integrations/striim.md | 59 ++++++++++++---------------- 1 file changed, 25 insertions(+), 34 deletions(-) diff --git a/use-timescale/integrations/striim.md b/use-timescale/integrations/striim.md index 04fa36b9f4..568907056f 100644 --- a/use-timescale/integrations/striim.md +++ b/use-timescale/integrations/striim.md @@ -11,9 +11,9 @@ import IntegrationPrereqs from "versionContent/_partials/_integration-prereqs.md [Striim][striim] is a real-time data integration platform that enables you to ingest, process, and deliver streaming data across various systems with minimal latency. It connects data sources with targets and transforms the data on the fly. -[Eclipse Mosquitto][mosquitto] is an open-source MQTT broker widely used for lightweight messaging in IoT and mobile applications. +[Eclipse Mosquitto][mosquitto] is an open-source MQTT broker widely used for lightweight messaging in IoT and mobile applications. -This page explains how to stream real-time IoT data from Eclipse Mosquitto to Timescale Cloud using Striim. +This page explains how to stream real-time IoT data from Eclipse Mosquitto to Timescale Cloud using Striim. ## Prerequisites @@ -35,28 +35,29 @@ To prepare sample IoT data to stream to $CLOUD_LONG: ```bash mosquitto -v ``` -1. **Publish sample data** - Run the following command to publish data to the `sensor/data` MQTT topic: +1. **Verify the data** + + Subscribe to the `sensor/data` MQTT topic: ```bash - mosquitto_pub -t sensor/data -m '{"temperature": 22.5, "humidity": 60}' + mosquitto_sub -t sensor/data ``` -1. **Verify the data** +1. **Publish sample data** - Subscribe to the topic to verify the data: + In a new terminal, run the following command to publish data to the `sensor/data` topic: ```bash - mosquitto_sub -t sensor/data + mosquitto_pub -t sensor/data -m '{"temperature": 22.5, "humidity": 60}' ``` - - You should see this: + + You should see the following data in the first terminal: ```json {"temperature": 22.5, "humidity": 60} ``` - + ## Prepare your $SERVICE_LONG to ingest data @@ -97,38 +98,28 @@ Configure Eclipse Mosquitto as the source and $CLOUD_LONG as the target, then co 1. **Create a data flow** - 1. Navigate to `Applications` > `Create Application` > `Data Flow`. - 1. Name your application, for example, `MosquittoToTimescale`. + 1. Navigate to `Apps` > `Create An App` > `Start from Scratch`. + 2. Name your application, for example, `MosquittoToTimescale`. 1. **Configure the source** 1. Drag the `MQTT Source` component into the data flow. - 1. Enter the connection details for your Eclipse Mosquitto broker: `host`, `port`, and `sensor/data` for `topic`. - 1. Click `Test Connection`. + 2. Enter the connection details for your Eclipse Mosquitto broker, and `sensor/data` for `topic`. + 3. Create a New Output Stream and name it `mqtt_sensor_stream`. 1. **Configure the target** - 1. Drag the `PostgreSQL Writer` component into the data flow. - 1. Enter the [connection details][connection-info] for your $SERVICE_SHORT. - 1. Choose `sensor_data` as the table name. - 1. Use the dropdowns in the `Field Mapping` section to map source fields to the destination columns. - -1. **Add a continuous query** - - 1. Drag the `Continuous Query` component between `MQTT Source` and `PostgreSQL Writer`. - 1. Open the continuous query configuration panel and write an SQL query to transform and map the data: + 1. Drag the `PostgreSQL` component into the data flow. + 2. Choose Input Stream as `mqtt_sensor_stream`. + 3. Enter the [connection details][connection-info] for your $SERVICE_SHORT. + 4. Choose `sensor_data` as the table name. - ```sql - SELECT - jsonextractstring(payload, '$.time') AS time, - jsonextractfloat(payload, '$.temperature') AS temperature, - jsonextractfloat(payload, '$.humidity') AS humidity - FROM mqtt_source_stream; - ``` +1. **Deploy and start the application** -1. **Connect the source and the target** + 1. Deploy the app by clicking on the dropdown at the top of the data flow window and selecting `Deploy App`. + 2. After the app is deployed successfully, from the same drop down, click on `Start App`. - Draw a line from `MQTT Source` to `PostgreSQL Writer`. +NOTE: Please take a look, after this setup and deploying the app, there are a few errors. 1. **Test the data flow** @@ -138,7 +129,7 @@ Configure Eclipse Mosquitto as the source and $CLOUD_LONG as the target, then co mosquitto_pub -t sensor/data -m '{"temperature": 24.3, "humidity": 55}' ``` - 1. Verify the data is streamed to your $SERVICE_LONG: + 2. Verify the data is streamed to your $SERVICE_LONG: ```sql SELECT * FROM sensor_data; From 168d48458f241f18bf2301567a3cd88075774223 Mon Sep 17 00:00:00 2001 From: Anagha Mittal Date: Fri, 7 Mar 2025 08:15:38 +0530 Subject: [PATCH 4/4] deployed app is running into errors --- use-timescale/integrations/striim.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/use-timescale/integrations/striim.md b/use-timescale/integrations/striim.md index e5b410a348..eb1457bb55 100644 --- a/use-timescale/integrations/striim.md +++ b/use-timescale/integrations/striim.md @@ -51,16 +51,8 @@ To prepare sample IoT data to stream to $CLOUD_LONG: ```bash mosquitto_pub -t sensor/data -m '{"temperature": 22.5, "humidity": 60}' ``` - -1. **Verify the data** - - Subscribe to the topic to verify the data: - - ```bash - mosquitto_sub -t sensor/data - ``` - You should see this: + You should see this in the first terminal: ```json {"temperature": 22.5, "humidity": 60}