Modify documents for Smart Data Hub

dir.yaml

@@ -298,22 +298,22 @@
     - data-integration/rule-sql-builtin-functions
     - data-integration/rule-sql-jq
     # - data-integration/rule-sql-user-defined-function
-    - title_en: Schema Registry
-      title_cn: Schema
-      path: data-integration/schema-registry
+    - title_en: Smart Data Hub
+      title_cn: 数据智能中心
+      path: data-integration/smart-data-hub
       collapsed: true
       children:
-        - data-integration/schema-registry-example-avro
-        - data-integration/schema-registry-example-protobuf
-    - title_en: Schema Validation
-      title_cn: Schema 验证
-      path: data-integration/schema-validation
-    - title_en: Message Transformation
-      title_cn: 消息转换
-      path: data-integration/message-transformation
-    - title_en: Sparkplug
-      title_cn: Sparkplug
-      path: data-integration/sparkplug
+        - title_en: Schema Registry
+          title_cn: Schema Registry
+          path: data-integration/schema-registry
+          collapsed: true
+          children:
+            - data-integration/schema-registry-example-avro
+            - data-integration/schema-registry-example-protobuf
+            - data-integration/schema-registry-example-external-http
+            - data-integration/sparkplug
+        - data-integration/schema-validation
+        - data-integration/message-transformation
 
 - title_en: Flow Designer
   title_cn: Flow 设计器
@@ -526,6 +526,7 @@
           path: dashboard/bridgeoverview
           children:
             - dashboard/rules
+        - dashboard/data-hub
         - title_en: Management
           title_cn: 管理
           path: dashboard/configuration

en_US/dashboard/bridgeoverview.md

@@ -6,10 +6,4 @@ The Integration section on the Dashboard provides functions for creating rules,
 - **Flow Designer (EMQX Enterprise feature)**: A powerful tool for visually viewing and managing the data processing and integration. For detailed introduction, see [Flow Designer](../flow-designer/introduction.md).
 - [**Rules**](./rules.md): Provides all the functions about creating rules, testing rules, adding actions to the rules, and viewing rule execution statistics.
 - **Connector**: As an indispensable component of the data integration, connectors are used to connect with external data systems. For how to create and manage connectors, see [Connector](../data-integration/connector.md).
-- **Schema Validation (EMQX Enterprise feature)**: Use validation rule to ensure that data to be published to specific topics must conform to predefined data formats. For how to create validation rules, see [Configure Schema Validation in Dashboard](../data-integration/schema-validation.md#configure-schema-validation-in-dashboard).
-- **Schema (EMQX Enterprise feature)**: You can create a schema to be used in the schema validation or SQL rules. For how to create a schema on Dashboard, see [Schema Registry Example - Avro](../data-integration/schema-registry-example-avro.md) or [Schema Registry Example - Protobuf](../data-integration/schema-registry-example-protobuf.md).
-- **Message Transform (EMQX Enterprise feature)**: Define transformation expressions to adapt data as it flows through the system. For how to create transformations, see [Configure Message Transformation in Dashboard](../data-integration/message-transformation.md#configure-message-transformation-in-dashboard).
-
-
-
 

en_US/dashboard/data-hub.md

@@ -0,0 +1,8 @@
+# Smart Data Hub
+
+The Smart Data Hub section on the Dashboard allows you to create and configure the schema, schema validation rules, and message transformation rules.
+
+- **Schema Registry**: You can create an internal or external schema to be used in the schema validation or SQL rules. For how to create a schema on Dashboard, see [Schema Registry Example - Avro](../data-integration/schema-registry-example-avro.md) or [Schema Registry Example - Protobuf](../data-integration/schema-registry-example-protobuf.md).
+- **Schema Validation**: Use validation rule to ensure that data to be published to specific topics must conform to predefined data formats. For how to create validation rules, see [Configure Schema Validation in Dashboard](../data-integration/schema-validation.md#configure-schema-validation-in-dashboard).
+- **Message Transform**: Define transformation expressions to adapt data as it flows through the system. For how to create transformations, see [Configure Message Transformation in Dashboard](../data-integration/message-transformation.md#configure-message-transformation-in-dashboard).
+

en_US/data-integration/message-transformation.md

@@ -38,7 +38,7 @@ This section demonstrates how to configure the message transformation feature an
 
 This section demonstrates how to create and configure a message transformation in the Dashboard.
 
-1. Go to Dashboard, and click **Integrations** -> **Message Transform** in the left navigation menu.
+1. Go to Dashboard, and click **Smart Data Hub** -> **Message Transform** in the left navigation menu.
 2. Click **Create** at the top right of the **Message Transform** page.
 3. On the Create Message Transform page, configure the following information:
    - **Name**: Enter the name of the transformation.

en_US/data-integration/schema-registry-example-external-http.md

@@ -0,0 +1,135 @@
+# Schema Registry Example - External HTTP Server
+
+This page demonstrates how the Schema Registry and Rule Engine support message encoding and decoding using an external HTTP server with custom logic.
+
+In some scenarios, you might need to apply custom encoding or decoding logic that EMQX does not support natively. EMQX allows you to delegate this processing to an external HTTP service by invoking it through `schema_encode` and `schema_decode` functions within a rule.
+
+## External HTTP API Specification
+
+To implement a custom External HTTP API that integrates with EMQX's `schema_encode` and `schema_decode` functions, your External HTTP server must provide a single `POST` endpoint that handles the encoding or decoding requests from EMQX.  
+
+### Request Format
+
+The request body is a JSON object with the following fields:
+
+- `payload`: Base64-encoded string value passed to the `schema_encode` or `schema_decode` function in Rule Engine.
+- `type`: Either the `encode` or the `decode` string, depending on which function is evaluated, `schema_encode` or `schema_decode`.
+- `schema_name`: A string identifying the name of this External HTTP schema configured in EMQX.
+- `opts`: An arbitrary string that can be configured in EMQX to provide further options, which is passed unaltered to the HTTP server.
+
+### Response Format
+
+- The server must respond with HTTP status code `200`.
+- The response body must contain a base64-encoded string representing the result. Note that this base64 value must not be further JSON-encoded when replying to EMQX.
+
+## Example Use Case
+
+Suppose a device publishes a binary message, and you want to encode or decode the payload using a custom XOR operation. This section demonstrates how to integrate custom encoding and decoding logic into EMQX by building a simple external HTTP service.
+
+### Build an External HTTP Service
+
+The following example demonstrates how to create and run a simple HTTP server using Python and Flask. The server receives Base64-encoded data and applies an XOR operation to the decoded payload.
+
+<details>
+<summary><strong>Code for sample External HTTP Server</strong></summary>
+
+Ensure [Flask](https://flask.palletsprojects.com/en/stable/) is installed:
+
+```sh
+pip install Flask==3.1.0
+```
+
+Sample code:
+
+```python
+from flask import Flask, request
+import base64
+
+app = Flask(__name__)
+
+@app.route("/serde", methods=['POST'])
+def serde():
+    # The input payload is base64 encoded
+    body = request.get_json(force=True)
+    print("incoming request:", body)
+    payload64 = body.get("payload")
+    payload = base64.b64decode(payload64)
+    secret = 122
+    response = bytes(b ^ secret for b in payload)
+    # The response must also be base64 encoded
+    response64 = base64.b64encode(response)
+    return response64
+```
+
+To run your server:
+
+```sh
+# This assumes your server is in the same directory in a file named `myapp.py`
+flask --app myapp --debug run -h 0.0.0.0 -p 9500
+```
+
+</details>
+
+### Create External HTTP Schema in EMQX
+
+1. Go to the Dashboard, and select **Smart Data Hub** -> **Schema Registry** from the left navigation menu.
+
+2. In the **Internal** tab page, click **Create**.
+
+3. Create an External HTTP server schema using the following parameters:
+   - **Name**: `myhttp`
+
+   - **Type**: `External HTTP`
+
+   - **URL**: The full URI where your server is running.  For example: `http://server:9500/serde`.
+
+4. Click **Create**.
+
+### Create a Rule to Apply Schema
+
+Use the EMQX rule engine to create a rule that applies your schema for message encoding and decoding.
+
+1. In the Dashboard, select **Integration** -> **Rules** from the navigation menu.
+
+2. On the **Rules** page, click **Create** at the top right corner.
+
+3. Use the schema you have just created to write the rule SQL statement:
+
+   ```sql
+   SELECT
+     schema_encode('myhttp', payload) as encoded,
+     schema_decode('myhttp', encoded) as decoded
+   FROM
+     "t/external_http"
+   ```
+
+   Both `schema_encode('myhttp', payload)` and `schema_decode('myhttp', encoded)` will call the configured External HTTP server to encode/decode the given payload.
+
+4. Click **Add Action**.  Select `Republish` from the drop-down list of the **Action** field.
+
+5. In the **Topic** field, type `external_http/out` as the destination topic.
+
+6. In the **Payload** field, type message content template: `${.}`. 
+
+7. Click **Add** to add the action to the rule.
+
+   This action sends the decoded message to the topic `external_http/out` in JSON format. `${.}` is a variable placeholder that will be replaced at runtime with the value of the whole output of the rule.
+
+8. Click **Save** to complete the rule creation.
+
+### Check Rule Execution Results
+
+1. In the Dashboard, select **Diagnose** -> **WebSocket Client**.
+2. Fill in the connection information for the current EMQX instance.
+   - If you run EMQX locally, you can use the default value.
+   - If you have changed EMQX's default configuration. For example, the configuration change on authentication can require you to type in a username and password.
+3. Click **Connect** to connect to the EMQX instance as an MQTT client.
+4. In the **Subscription** area, type `external_http/out` in the **Topic** field and click **Subscribe**.
+
+5. In the **Publish** area, type `t/external_http` in the **Topic** field, write any payload you wish, and click **Publish**.
+
+6. Check that a message with the topic `external_http/out` is received on the Websocket side.  For example, if your payload was `hello`:
+
+   ```json
+   {"encoded":"\u0012\u001F\u0016\u0016\u0015","decoded":"hello"}
+   ```

en_US/data-integration/schema-registry.md

@@ -16,11 +16,14 @@ EMQX Schema Registry currently supports codecs in the below formats:
 - [Avro](https://avro.apache.org)
 - [Protobuf](https://developers.google.com/protocol-buffers/)
 - [JSON Schema](https://json-schema.org/)
+- External HTTP server
 
 Avro and Protobuf are Schema-dependent data formats. The encoded data is binary and the decoded data is in [Map format](#rule-engine-internal-data-format-map). The decoded data can be used directly by the rule engine and other plugins. Schema Registry maintains Schema text for built-in encoding formats such as Avro and Protobuf.
 
 JSON schema can be used to validate if the input JSON object is following the schema definitions or if the JSON object output from the rule engine is valid before producing the data to downstream.
 
+External HTTP Server makes all decoding and encoding of payloads go through a configured black-box server that handles the logic.  It's useful for cases where one wishes to have custom encoding/decoding logic.
+
 The diagram below shows an example of a Schema Registry application. Multiple devices report data in different formats, which are decoded by Schema Registry into a uniform internal format and then forwarded to the backend application.
 
 <img src="./assets/schema-registry.png" alt="schema-registry" style="zoom:67%;" />
@@ -114,7 +117,7 @@ Starting with version 5.8.1, EMQX supports configuring an external Confluent Sch
 
 You can configure an external schema registry directly through the EMQX Dashboard, making it easy to manage your schema integration.
 
-Go to **Integration** -> **Schema Registry** on EMQX Dashboard. Select the **External** tab on the Schema page.
+Go to **Smart Data Hub** -> **Schema Registry** on EMQX Dashboard. Select the **External** tab on the Schema page.
 
 Click the **Create** button at the upper right corner. Configure the following fields:
 
@@ -206,7 +209,7 @@ select
 from 't'
 ```
 
-##### `schema_encode_and_tag` 
+##### `schema_encode_and_tag`
 
 This function uses a locally registered Avro schema, an external CSR schema name, and a subject to encode a payload (already in internal map format), and to tag the resulting payload with a schema ID.  The schema ID comes from registering the local schema to CSR.
 
@@ -223,7 +226,7 @@ select
 from 't'
 ```
 
-##### `schema_decode_tagged` 
+##### `schema_decode_tagged`
 
 This function uses a CSR name to decode a payload, assuming it is tagged with the schema ID retrieved from CSR.
 
@@ -235,4 +238,3 @@ select
   ) as decoded
 from 't'
 ```
-

en_US/data-integration/schema-validation.md

@@ -52,7 +52,7 @@ This section demonstrates how to configure the schema validation feature and how
 
 This section demonstrates how to create and configure a schema validator in the Dashboard.
 
-1. Click on **Integrations** -> **Schema Validation** in the left navigation of the Dashboard.
+1. Click on **Smart Data Hub** -> **Schema Validation** in the left navigation of the Dashboard.
 2. Click **Create** at the top right of the **Schema Validation** page.
 3. On the Create Schema Validation page, configure the following settings:
    - **Name**: Enter the name of the validator.

en_US/data-integration/smart-data-hub.md

@@ -0,0 +1,13 @@
+# Smart Data Hub
+
+The Smart Data Hub of EMQX is an all-in-one solution for intelligent data processing. It is designed to simplify and efficiently manage MQTT data streams. With the Smart Data Hub, you can easily manage schemas, validate data, and perform real-time data transformations as needed. Whether for data validation or message transformation, this platform enables automated and efficient data management.
+
+## Key Features
+
+The Smart Data Hub provides the following key features:
+
+- **[Schema Registry](./schema-registry.md)**: Create, modify, and delete data schemas to ensure consistency in data formats and standards.
+- **[Schema Validation](./schema-validation.md)**: Validate incoming data against predefined schemas to prevent formatting errors and data inconsistencies.
+- **[Message Transformation](./message-transformation.md)**: Transform data messages in real-time, formatting or mapping MQTT data as needed to suit various application scenarios.
+
+![data_hub](./assets/data_hub.png)

zh_CN/dashboard/bridgeoverview.md

@@ -6,9 +6,6 @@
 - **Flow 设计器**（企业版功能）：在早期版本的 Flows 可视化工具的基础上增加了创建和编辑数据处理和集成的能力。关于 Flow 设计器的介绍和使用信息，参阅 [Flow 设计器](../flow-designer/introduction.md)。
 - [**规则**](./rules.md)：本页提供了创建规则、测试规则、向规则添加动作以及查看规则执行统计等功能。
 - **连接器**：作为数据集成不可或缺的组件，连接器用于连接外部数据系统。有关如何创建和管理连接器，请参考[连接器](../data-integration/connector.md)。
-- **Schema 验证**（企业版功能）：您使用验证规则来确保发布到特定主题的数据必须符合预定义的数据格式。有关如何创建验证规则，请参考[在 Dashboard 中配置 Schema 验证](../data-integration/schema-validation.md#在-dashboard-中配置-schema-验证)。
-- **Schema**（企业版功能）: 您可以在本页创建一个 Schema 用于模式验证或 SQL 规则。有关如何创建模式，请参考[编解码举例 - Avro](../data-integration/schema-registry-example-avro.md) or [编解码举例 - Protobuf](../data-integration/schema-registry-example-protobuf.md)。
-- **消息转换**（企业版功能）：通过定义消息转换表达式，在数据流经系统时对其进行适配。有关如何创建消息转换的详细信息，请参阅[在 Dashboard 中配置消息转换](../data-integration/message-transformation.md#在-dashboard-中配置消息转换)。
 
 
 

zh_CN/dashboard/data-hub.md

@@ -0,0 +1,8 @@
+# 数据智能中心
+
+您可以通过 Dashboard 中的数据智能中心模块创建和配置 Schema、Schema 验证规则以及消息转换规则。
+
+- **Schema Registry**: 您可以在本页创建一个内部或外部的 Schema 用于模式验证或 SQL 规则。有关如何创建模式，请参考[编解码举例 - Avro](../data-integration/schema-registry-example-avro.md) or [编解码举例 - Protobuf](../data-integration/schema-registry-example-protobuf.md)。
+- **Schema 验证**：您使用验证规则来确保发布到特定主题的数据必须符合预定义的数据格式。有关如何创建验证规则，请参考[在 Dashboard 中配置 Schema 验证](../data-integration/schema-validation.md#在-dashboard-中配置-schema-验证)。
+- **消息转换**：通过定义消息转换表达式，在数据流经系统时对其进行适配。有关如何创建消息转换的详细信息，请参阅[在 Dashboard 中配置消息转换](../data-integration/message-transformation.md#在-dashboard-中配置消息转换)。
+

zh_CN/data-integration/message-transformation.md

@@ -38,7 +38,7 @@
 
 本节演示如何在 Dashboard 中创建和配置消息转换。
 
-1. 进入 Dashboard，点击左侧导航菜单中的**集成** -> **消息转换**。
+1. 进入 Dashboard，点击左侧导航菜单中的**数据智能中心** -> **消息转换**。
 2. 在**消息转换**页面右上角点击**创建**。
 3. 在创建消息转换页面，配置以下信息：
    - **名称**：输入转换的名称。
@@ -98,7 +98,7 @@ message_transformation {
 
 ### 创建解码/编码模式
 
-有关如何创建解码和编码模式的更多信息，请参阅 [Schema Registry](./schema-registry.md)部分。
+有关如何创建解码和编码模式的更多信息，请参阅 [Schema Registry](./schema-registry.md) 部分。
 
 ## 统计与指标