diff --git a/src/UserGuide/Master/Table/IoTDB-Introduction/IoTDB-Introduce_apache.md b/src/UserGuide/Master/Table/IoTDB-Introduction/IoTDB-Introduce_apache.md new file mode 100644 index 000000000..21bdfbeba --- /dev/null +++ b/src/UserGuide/Master/Table/IoTDB-Introduction/IoTDB-Introduce_apache.md @@ -0,0 +1,101 @@ + + +# Product Introduction + +Apache IoTDB is a low-cost, high-performance IoT-native time-series database. It addresses challenges faced by enterprises in managing time-series data for IoT big data platforms, including complex application scenarios, massive data volumes, high sampling frequencies, frequent out-of-order data, time-consuming data processing, diverse analytical requirements, and high storage and maintenance costs. + +- GitHub Repository: [https://github.com/apache/iotdb](https://github.com/apache/iotdb) +- Open-Source Installation Packages: [https://iotdb.apache.org/Download/](https://iotdb.apache.org/Download/) +- Installation, Deployment, and Usage Documentation: [Quick Start](../QuickStart/QuickStart_apache.md) + + +## Product Ecosystem + +The IoTDB ecosystem consists of multiple components designed to efficiently manage and analyze massive IoT-generated time-series data. + +
+ Introduction-en-apache.png +
+ + +Key components include: + +1. **Time-Series Database (Apache IoTDB)**: The core component for time-series data storage, offering high compression, rich query capabilities, real-time stream processing, high availability, and scalability. It provides security guarantees, configuration tools, multi-language APIs, and integration with external systems for building business applications. +2. **Time-Series File Format (Apache TsFile)**: A specialized storage format for time-series data, enabling efficient storage and querying. TsFile underpins IoTDB and AINode, unifying data management across collection, storage, and analysis phases. +3. **Time-Series Model Training-Inference Engine (IoTDB AINode)**: A unified engine for intelligent analysis, supporting model training, data management, and integration with machine/deep learning frameworks. + + +## TimechoDB Architecture + +The diagram below illustrates a typical IoTDB cluster deployment (3 ConfigNodes and 3 DataNodes): + + + + +## Key Features + +Apache IoTDB offers the following advantages: + +- **Flexible Deployment**: + - One-click cloud deployment + - Out-of-the-box terminal usage + - Seamless terminal-cloud synchronization + +- **Cost-Effective Storage**: + - High-compression disk storage + - Unified management of historical and real-time data + +- **Hierarchical Measurement Point Management**: + - Aligns with industrial device hierarchies + - Supports directory browsing and search + +- **High Throughput Read/Write**: + - Supports millions of devices + - Handles high-speed, out-of-order, and multi-frequency data ingestion + +- **Rich Query Capabilities**: + - Native time-series computation engine + - Timestamp alignment during queries + - Over 100 built-in aggregation and time-series functions + - AI-ready time-series feature analysis + +- **High Availability & Scalability**: + - HA distributed architecture with 24/7 uptime + - Automatic load balancing for node scaling + - Heterogeneous cluster support + +- **Low Learning Curve**: + - SQL-like query language + - Multi-language SDKs + - Comprehensive toolchain (e.g., console) + +- **Ecosystem Integration**: + - Hadoop, Spark, Grafana, ThingsBoard, DataEase, etc. + + +## TimechoDB + +Timecho Technology has developed **TimechoDB**, a commercial product built on the open-source version of Apache IoTDB, to provide enterprise-grade solutions and services for businesses and commercial clients. TimechoDB addresses the multifaceted challenges enterprises face when building IoT big data platforms for managing time-series data, including complex application scenarios, massive data volumes, high sampling frequencies, frequent out-of-order data, time-consuming data processing, diverse analytical requirements, and high storage and maintenance costs. + +Leveraging **TimechoDB**, Timecho Technology offers a broader range of product features, enhanced performance and stability, and a richer suite of efficiency tools. Additionally, it provides comprehensive enterprise services, delivering commercial clients with superior product capabilities and an optimized experience in development, operation, and usage. +- **Timecho Technology Official Website**: [https://www.timecho.com/](https://www.timecho.com/) +- **TimechoDB Documentation**: [Quick Start](https://www.timecho.com/docs/zh/UserGuide/latest/QuickStart/QuickStart_timecho.html) diff --git a/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_apache.md b/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_apache.md new file mode 100644 index 000000000..a6b17af49 --- /dev/null +++ b/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_apache.md @@ -0,0 +1,182 @@ + + +# Release History + +## V2.0.1-beta + +> Release Date: 2025.02.18 + +V2.0.1-beta introduces dual model configuration for tree and table models, along with table model support for standard SQL query syntax, multiple functions and operators, stream processing, Benchmark, and more. Additional updates include: Python client support for four new data types, database deletion in read-only mode, script tools compatible with TsFile, CSV, and SQL data import/export, Kubernetes Operator integration, and other enhancements. Key updates include: + +- **Table Model**: IoTDB now supports table models, including standard SQL query syntax (SELECT, WHERE, JOIN, GROUP BY, ORDER BY, LIMIT clauses, and subqueries). +- **Query Module**: Table model supports multiple functions and operators (logical operators, mathematical functions, and time-series-specific functions like DIFF). +- **Query Module**: Users can control UDF, PipePlugin, Trigger, and AINode to load JAR packages via URI through configuration items. +- **Storage Module**: Table model supports data writing via the Session interface, with automatic metadata creation. +- **Storage Module**: Python client adds support for four new data types: String, Blob, Date, and Timestamp. +- **Storage Module**: Optimized priority comparison rules for merge tasks of the same type. +- **Stream Processing**: Supports specifying authentication information at the receiver end. +- **Stream Processing**: TsFile Load now supports table models. +- **Stream Processing**: Stream processing plugins are adapted to table models. +- **System Module**: Improved stability of DataNode scaling down. +- **System Module**: Users can perform `DROP DATABASE` operations in read-only mode. +- **Scripts & Tools**: Benchmark tool adapted to table models. +- **Scripts & Tools**: Benchmark tool supports four new data types: String, Blob, Date, and Timestamp. +- **Scripts & Tools**: Extended `data/export-data` script to support new data types (String, Blob, Date, Timestamp). +- **Scripts & Tools**: `import-data/export-data` scripts now support TsFile, CSV, and SQL data import/export. +- **Ecosystem Integration**: Kubernetes Operator support. + +--- + +## V1.3.3 + +> Release Date: 2024.11.20 + +V1.3.3 introduces String, Blob, Date, and Timestamp data types, data subscription, DataNode auto-monitoring and TsFile loading, observability metrics, sender-to-receiver file transfer automation, configuration file consolidation, client query load balancing, and stability improvements. Key updates include: + +- **Storage Module**: Added String, Blob, Date, and Timestamp data types. +- **Storage Module**: Optimized memory control for merge operations. +- **Query Module**: Client query load balancing enhancements. +- **Query Module**: Added active metadata statistics queries. +- **Query Module**: Filter performance optimization for faster aggregation and WHERE queries. +- **Data Sync**: Sender supports transferring files to a specified directory, after which the receiver automatically loads them into IoTDB. +- **Data Sync**: Receiver adds automatic data type conversion. +- **Data Subscription**: Added subscription capabilities (data points or TsFile-based). +- **Data Loading**: DataNode actively monitors and loads TsFiles, with observability metrics. +- **Stream Processing**: `ALTER PIPE` supports `ALTER SOURCE`. +- **System Module**: Simplified configuration files (merged three into one). +- **System Module**: Added configuration item APIs. +- **System Module**: Improved restart recovery performance. +- **Scripts & Tools**: Metadata import/export scripts. +- **Scripts & Tools**: Kubernetes Helm support. + +--- + +## V1.3.2 + +> Release Date: 2024.07.01 + +V1.3.2 introduces `EXPLAIN ANALYZE` for SQL query profiling, UDAF framework, metadata sync, data point counting, SQL import/export scripts, rolling upgrades, and stability improvements. Key updates include: + +- **Storage Module**: Optimized `insertRecords` write performance. +- **Query Module**: Added `EXPLAIN ANALYZE` for query stage time analysis. +- **Query Module**: Added UDAF (User-Defined Aggregation Function) framework. +- **Query Module**: Added `MAX_BY`/`MIN_BY` functions to return timestamps with max/min values. +- **Query Module**: Value-filter query performance improvements. +- **Data Sync**: Wildcard support for path matching. +- **Data Sync**: Metadata synchronization (time series, attributes, permissions). +- **Stream Processing**: Added `ALTER PIPE` for hot-updating plugins. +- **System Module**: TsFile load data included in data point statistics. +- **Scripts & Tools**: Local upgrade backup tool (hard-link-based backups). +- **Scripts & Tools**: `export-data`/`import-data` scripts for CSV, TsFile, and SQL formats. +- **Scripts & Tools**: Windows support for ConfigNode/DataNode/Cli window naming. + +--- + +## V1.3.1 + +> Release Date: 2024.04.22 + +V1.3.1 adds one-click cluster start/stop scripts, info collection scripts, built-in functions, sync/log/query optimizations, and observability improvements. Key updates include: + +- Added one-click cluster start/stop scripts (`start-all/stop-all.sh` and `.bat`). +- Added instance info collection scripts (`collect-info.sh` and `.bat`). +- Added `STDDEV`, `VARIANCE` aggregation functions. +- Added TsFile repair command. +- `FILL` clause supports timeout thresholds. +- Simplified data sync time range configuration. +- Enhanced observability (cluster node divergence monitoring, task scheduling metrics). +- Optimized default logging strategy. +- Improved memory control for TsFile loading. +- REST API (v2) returns column types. +- Optimized query execution. +- Client auto-fetches available DataNode lists. + +--- + +## V1.3.0 + +> Release Date: 2024.01.01 + +V1.3.0 adds SSL encryption, sync monitoring, permission syntax optimizations, metrics performance improvements, and query optimizations. Key updates include: + +- **Security**: Time-series-level permission control. +- **Security**: SSL support for client-server communication. +- **Query Module**: Calculation views support `LAST` queries. +- **Stream Processing**: Added Pipe monitoring metrics. +- **Storage Module**: Support for negative timestamps. +- **Scripts & Tools**: `LOAD` script data included in monitoring. +- **Client**: Optimized Python client performance. +- **Query Module**: Fixed slow `SHOW PATH` responses. +- **Query Module**: Improved `EXPLAIN` result formatting. +- **System Module**: Added `MEMORY_SIZE` configuration. +- **System Module**: Renamed config items (e.g., `target_config_node_list` → `seed_config_node`). + +--- + +## V1.2.0 + +> Release Date: 2023.06.30 + +V1.2.0 introduces stream processing, dynamic templates, `SUBSTRING`/`REPLACE`/`ROUND` functions, and optimizations. Key updates include: + +- **Stream Processing**: Framework added. +- **Metadata**: Dynamic template expansion. +- **Storage**: Added SPRINTZ/RLBE encodings and LZMA2 compression. +- **Query Module**: `CAST`, `ROUND`, `SUBSTR`, `REPLACE` functions. +- **Query Module**: `TIME_DURATION`, `MODE` aggregation functions. +- **Query Module**: `CASE WHEN` and `ORDER BY` expression support. +- **Client**: Python API supports multi-node distributed connections. +- **Client**: Python write redirection. +- **Session API**: Batch time-series creation via templates. + +--- + +## V1.1.0 + +> Release Date: 2023.04.03 + +V1.1.0 adds `GROUP BY VARIATION/CONDITION`, `DIFF`, `COUNT_IF`, and pipeline engine optimizations. Key fixes include metadata template errors and query issues. Key updates: + +- **Query Module**: `ALIGN BY DEVICE` supports `ORDER BY TIME`. +- **Query Module**: `SHOW QUERIES` and `KILL QUERY` commands. +- **System Module**: `SHOW REGIONS` with database filtering. +- **System Module**: `SHOW VARIABLES` for cluster parameters. +- **Query Module**: `GROUP BY VARIATION` and `GROUP BY CONDITION`. +- **Query Module**: `DIFF` scalar function and `COUNT_IF` aggregation. +- **System Module**: `dn_rpc_port` and `dn_rpc_address` configurability. + +--- + +## V1.0.0 + +> Release Date: 2022.12.03 + +V1.0.0 fixes partition calculation, query execution, and memory issues. Adds HA architecture, multi-replica storage, cluster management SQL, and MPP framework. Key updates: + +- **System Module**: Distributed high-availability architecture. +- **System Module**: Multi-replica storage. +- **System Module**: Port conflict detection during startup. +- **System Module**: Cluster management SQL (start/stop/remove nodes). +- **System Module**: Configurable consensus protocols (Simple, IoTConsensus, Ratis). +- **Query Module**: MPP framework for distributed read/write. +- **Stream Processing**: Framework and cross-cluster sync. + diff --git a/src/UserGuide/Master/Table/Tools-System/Maitenance-Tool_apache.md b/src/UserGuide/Master/Table/Tools-System/Maitenance-Tool_apache.md new file mode 100644 index 000000000..1194c7883 --- /dev/null +++ b/src/UserGuide/Master/Table/Tools-System/Maitenance-Tool_apache.md @@ -0,0 +1,229 @@ + + +# Cluster Management Tools + +## Data Directory Overview Tool + +The IoTDB data directory overview tool is designed to print out the structural overview information of the data directory. The tool is located at `tools/tsfile/print-iotdb-data-dir`. + +### Usage + +- **Windows:** + +```bash +.\print-iotdb-data-dir.bat () +``` + +- **Linux or MacOS:** + +```shell +./print-iotdb-data-dir.sh () +``` + +**Note:** If the output result storage path is not set, the relative path `"IoTDB_data_dir_overview.txt"` will be used as the default. + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-iotdb-data-dir.bat D:\github\master\iotdb\data\datanode\data +```````````````````````` +Starting Printing the IoTDB Data Directory Overview +```````````````````````` +output save path: IoTDB_data_dir_overview.txt +data dir num: 1 +143 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +|============================================================== +|D:\github\master\iotdb\data\datanode\data +|--sequence +| |--root.redirect0 +| | |--1 +| | | |--0 +| |--root.redirect1 +| | |--2 +| | | |--0 +| |--root.redirect2 +| | |--3 +| | | |--0 +| |--root.redirect3 +| | |--4 +| | | |--0 +| |--root.redirect4 +| | |--5 +| | | |--0 +| |--root.redirect5 +| | |--6 +| | | |--0 +| |--root.sg1 +| | |--0 +| | | |--0 +| | | |--2760 +|--unsequence +|============================================================== +````````````````````````` + +## TsFile Overview Tool + +The TsFile overview tool is used to print the content of a TsFile in a sketch mode. The tool is located at `tools/tsfile/print-tsfile`. + +### Usage + +- **Windows:** + +```bash +.\print-tsfile-sketch.bat () +``` + +- **Linux or MacOS:** + +```shell +./print-tsfile-sketch.sh () +``` + +**Note:** If the output result storage path is not set, the relative path `"TsFile_sketch_view.txt"` will be used as the default. + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-tsfile.bat D:\github\master\1669359533965-1-0-0.tsfile D:\github\master\sketch.txt +```````````````````````` +Starting Printing the TsFile Sketch +```````````````````````` +TsFile path: D:\github\master\1669359533965-1-0-0.tsfile +Sketch save path: D:\github\master\sketch.txt +148 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +-------------------------------- TsFile Sketch -------------------------------- +file path: D:\github\master\1669359533965-1-0-0.tsfile +file length: 2974 + + POSITION| CONTENT + -------- ------- + 0| [magic head] TsFile + 6| [version number] 3 +||||||||||||||||||||| [Chunk Group] of root.sg1.d1, num of Chunks:3 + 7| [Chunk Group Header] + | [marker] 0 + | [deviceID] root.sg1.d1 + 20| [Chunk] of root.sg1.d1.s1, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9032452783138882770,maxValue:9117677033041335123,firstValue:7068645577795875906,lastValue:-5833792328174747265,sumValue:5.795959009889246E19] + | [chunk header] marker=5, measurementID=s1, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 + 893| [Chunk] of root.sg1.d1.s2, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-8806861312244965718,maxValue:9192550740609853234,firstValue:1150295375739457693,lastValue:-2839553973758938646,sumValue:8.2822564314572677E18] + | [chunk header] marker=5, measurementID=s2, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 + 1766| [Chunk] of root.sg1.d1.s3, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9076669333460323191,maxValue:9175278522960949594,firstValue:2537897870994797700,lastValue:7194625271253769397,sumValue:-2.126008424849926E19] + | [chunk header] marker=5, measurementID=s3, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 +||||||||||||||||||||| [Chunk Group] of root.sg1.d1 ends + 2656| [marker] 2 + 2657| [TimeseriesIndex] of root.sg1.d1.s1, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9032452783138882770,maxValue:9117677033041335123,firstValue:7068645577795875906,lastValue:-5833792328174747265,sumValue:5.795959009889246E19] + | [ChunkIndex] offset=20 + 2728| [TimeseriesIndex] of root.sg1.d1.s2, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-8806861312244965718,maxValue:9192550740609853234,firstValue:1150295375739457693,lastValue:-2839553973758938646,sumValue:8.2822564314572677E18] + | [ChunkIndex] offset=893 + 2799| [TimeseriesIndex] of root.sg1.d1.s3, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9076669333460323191,maxValue:9175278522960949594,firstValue:2537897870994797700,lastValue:7194625271253769397,sumValue:-2.126008424849926E19] + | [ChunkIndex] offset=1766 + 2870| [IndexOfTimerseriesIndex Node] type=LEAF_MEASUREMENT + | + | +||||||||||||||||||||| [TsFileMetadata] begins + 2891| [IndexOfTimerseriesIndex Node] type=LEAF_DEVICE + | + | + | [meta offset] 2656 + | [bloom filter] bit vector byte array length=31, filterSize=256, hashFunctionSize=5 +||||||||||||||||||||| [TsFileMetadata] ends + 2964| [TsFileMetadataSize] 73 + 2968| [magic tail] TsFile + 2974| END of TsFile +---------------------------- IndexOfTimerseriesIndex Tree ----------------------------- + [MetadataIndex:LEAF_DEVICE] + └──────[root.sg1.d1,2870] + [MetadataIndex:LEAF_MEASUREMENT] + └──────[s1,2657] +---------------------------------- TsFile Sketch End ---------------------------------- +````````````````````````` + +**Explanation:** + +- The left side of the "|" separator represents the actual position in the TsFile, and the right side represents the sketch content. +- The "|||||||||||||||||||||" is added for enhanced readability and is not actual data stored in the TsFile. +- The final printed "IndexOfTimerseriesIndex Tree" is a reorganized print of the metadata index tree at the end of the TsFile for intuitive understanding and is not the actual data stored in the TsFile. + +## TsFile Resource Overview Tool + +The TsFile resource overview tool is used to print out the content of TsFile resource files. The tool is located at `tools/tsfile/print-tsfile-resource-files`. + +### Usage + +- **Windows:** + +```bash +.\print-tsfile-resource-files.bat +``` + +- **Linux or MacOS:** + +```shell +./print-tsfile-resource-files.sh +``` + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-tsfile-resource-files.bat D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0 +```````````````````````` +Starting Printing the TsFileResources +```````````````````````` +147 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +230 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +231 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +233 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +237 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +Analyzing D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile ... + +Resource plan index range [9223372036854775807, -9223372036854775808] +device root.sg1.d1, start time 0 (1970-01-01T08:00+08:00[GMT+08:00]), end time 99 (1970-01-01T08:00:00.099+08:00[GMT+08:00]) + +Analyzing the resource file folder D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0 finished. +````````````````````````` + +`````````````````````````bash +.\print-tsfile-resource-files.bat D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile.resource +```````````````````````` +Starting Printing the TsFileResources +```````````````````````` +178 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +186 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +187 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +188 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +192 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +Analyzing D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile ... + +Resource plan index range [9223372036854775807, -9223372036854775808] +device root.sg1.d1, start time 0 (1970-01-01T08:00+08:00[GMT+08:00]), end time 99 (1970-01-01T08:00:00.099+08:00[GMT+08:00]) + +Analyzing the resource file D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile.resource finished. +````````````````````````` \ No newline at end of file diff --git a/src/UserGuide/Master/Table/Tools-System/Monitor-Tool_apache.md b/src/UserGuide/Master/Table/Tools-System/Monitor-Tool_apache.md new file mode 100644 index 000000000..5e29a0fa1 --- /dev/null +++ b/src/UserGuide/Master/Table/Tools-System/Monitor-Tool_apache.md @@ -0,0 +1,168 @@ + + +# Prometheus + +The deployment of monitoring tools can be referenced in the document [Monitoring Panel Deployment](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) chapter. + +## Prometheus Mapping Relationship for Monitoring Metrics + +> For a monitoring metric with Metric Name as `name`, Tags as `K1=V1, ..., Kn=Vn`, the following mapping applies, where `value` is the specific value. + +| Monitoring Metric Type | Mapping Relationship | +| ---------------------- | ------------------------------------------------------------ | +| Counter | `name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value` | +| AutoGauge, Gauge | `name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value` | +| Histogram | `name_max{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_sum{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_count{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.5"} value`
`name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.99"} value` | +| Rate | `name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m1"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m5"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m15"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="mean"} value` | +| Timer | `name_seconds_max{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds_sum{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds_count{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.5"} value`
`name_seconds{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.99"} value` | + +## Modifying Configuration Files + +1) Taking DataNode as an example, modify the `iotdb-system.properties` configuration file as follows: + +```properties +dn_metric_reporter_list=PROMETHEUS +dn_metric_level=CORE +dn_metric_prometheus_reporter_port=9091 +``` + +2) Start the IoTDB DataNode. + +3) Open a browser or use `curl` to access `http://server_ip:9091/metrics`, and you will get metric data as follows: + +``` +... +# HELP file_count +# TYPE file_count gauge +file_count{name="wal",} 0.0 +file_count{name="unseq",} 0.0 +file_count{name="seq",} 2.0 +... +``` + +## Prometheus + Grafana + +As shown above, IoTDB exposes monitoring metrics in the standard Prometheus format. You can use Prometheus to collect and store these metrics and Grafana to visualize them. + +The relationship between IoTDB, Prometheus, and Grafana is illustrated below: + +![iotdb_prometheus_grafana](/img/UserGuide/System-Tools/Metrics/iotdb_prometheus_grafana.png) + +1. IoTDB continuously collects monitoring metrics during operation. +2. Prometheus pulls monitoring metrics from IoTDB's HTTP interface at fixed intervals (configurable). +3. Prometheus stores the pulled monitoring metrics in its TSDB. +4. Grafana queries monitoring metrics from Prometheus at fixed intervals (configurable) and visualizes them. + +From the interaction flow, it is clear that additional work is required to deploy and configure Prometheus and Grafana. + +For example, you can configure Prometheus as follows (some parameters can be adjusted as needed) to pull metrics from IoTDB: + +```yaml +job_name: pull-metrics +honor_labels: true +honor_timestamps: true +scrape_interval: 15s +scrape_timeout: 10s +metrics_path: /metrics +scheme: http +follow_redirects: true +static_configs: + - targets: + - localhost:9091 +``` + +For more details, refer to the following documents: + +- [Prometheus Installation and Usage Documentation](https://prometheus.io/docs/prometheus/latest/getting_started/) +- [Prometheus Configuration for Pulling Metrics from HTTP Interface](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) +- [Grafana Installation and Usage Documentation](https://grafana.com/docs/grafana/latest/getting-started/getting-started/) +- [Grafana Querying Data from Prometheus and Plotting Documentation](https://prometheus.io/docs/visualization/grafana/#grafana-support-for-prometheus) + +## Apache IoTDB Dashboard + +The `Apache IoTDB Dashboard` is a companion product of IoTDB Enterprise Edition, supporting unified centralized operation and maintenance management. It allows monitoring multiple clusters through a single monitoring panel. You can contact the business team to obtain the Dashboard's JSON file. + +![Apache IoTDB Dashboard](/img/%E7%9B%91%E6%8E%A7%20default%20cluster.png) + +![Apache IoTDB Dashboard](/img/%E7%9B%91%E6%8E%A7%20cluster2.png) + +### Cluster Overview + +You can monitor, but not limited to: + +- Total CPU cores, total memory space, total disk space of the cluster. +- Number of ConfigNodes and DataNodes in the cluster. +- Cluster uptime. +- Cluster write speed. +- Current CPU, memory, and disk usage of each node in the cluster. +- Node-specific information. + +![](/img/%E7%9B%91%E6%8E%A7%20%E6%A6%82%E8%A7%88.png) + +### Data Writing + +You can monitor, but not limited to: + +- Average write latency, median latency, 99th percentile latency. +- Number and size of WAL files. +- Node WAL flush SyncBuffer latency. + +![](/img/%E7%9B%91%E6%8E%A7%20%E5%86%99%E5%85%A5.png) + +### Data Query + +You can monitor, but not limited to: + +- Node query loading time series metadata latency. +- Node query reading time series latency. +- Node query modifying time series metadata latency. +- Node query loading Chunk metadata list latency. +- Node query modifying Chunk metadata latency. +- Node query filtering by Chunk metadata latency. +- Node query constructing Chunk Reader latency average. + +![](/img/%E7%9B%91%E6%8E%A7%20%E6%9F%A5%E8%AF%A2.png) + +### Storage Engine + +You can monitor, but not limited to: + +- Number and size of files by type. +- Number and size of TsFiles in various stages. +- Number and latency of various tasks. + +![](/img/%E7%9B%91%E6%8E%A7%20%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E.png) + +### System Monitoring + +You can monitor, but not limited to: + +- System memory, swap memory, process memory. +- Disk space, file count, file size. +- JVM GC time ratio, GC count by type, GC data volume, heap memory usage by generation. +- Network transmission rate, packet sending rate. + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9F%20%E5%86%85%E5%AD%98%E4%B8%8E%E7%A1%AC%E7%9B%98.png) + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9Fjvm.png) + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9F%20%E7%BD%91%E7%BB%9C.png) \ No newline at end of file diff --git a/src/UserGuide/latest-Table/IoTDB-Introduction/IoTDB-Introduce_apache.md b/src/UserGuide/latest-Table/IoTDB-Introduction/IoTDB-Introduce_apache.md new file mode 100644 index 000000000..21bdfbeba --- /dev/null +++ b/src/UserGuide/latest-Table/IoTDB-Introduction/IoTDB-Introduce_apache.md @@ -0,0 +1,101 @@ + + +# Product Introduction + +Apache IoTDB is a low-cost, high-performance IoT-native time-series database. It addresses challenges faced by enterprises in managing time-series data for IoT big data platforms, including complex application scenarios, massive data volumes, high sampling frequencies, frequent out-of-order data, time-consuming data processing, diverse analytical requirements, and high storage and maintenance costs. + +- GitHub Repository: [https://github.com/apache/iotdb](https://github.com/apache/iotdb) +- Open-Source Installation Packages: [https://iotdb.apache.org/Download/](https://iotdb.apache.org/Download/) +- Installation, Deployment, and Usage Documentation: [Quick Start](../QuickStart/QuickStart_apache.md) + + +## Product Ecosystem + +The IoTDB ecosystem consists of multiple components designed to efficiently manage and analyze massive IoT-generated time-series data. + +
+ Introduction-en-apache.png +
+ + +Key components include: + +1. **Time-Series Database (Apache IoTDB)**: The core component for time-series data storage, offering high compression, rich query capabilities, real-time stream processing, high availability, and scalability. It provides security guarantees, configuration tools, multi-language APIs, and integration with external systems for building business applications. +2. **Time-Series File Format (Apache TsFile)**: A specialized storage format for time-series data, enabling efficient storage and querying. TsFile underpins IoTDB and AINode, unifying data management across collection, storage, and analysis phases. +3. **Time-Series Model Training-Inference Engine (IoTDB AINode)**: A unified engine for intelligent analysis, supporting model training, data management, and integration with machine/deep learning frameworks. + + +## TimechoDB Architecture + +The diagram below illustrates a typical IoTDB cluster deployment (3 ConfigNodes and 3 DataNodes): + + + + +## Key Features + +Apache IoTDB offers the following advantages: + +- **Flexible Deployment**: + - One-click cloud deployment + - Out-of-the-box terminal usage + - Seamless terminal-cloud synchronization + +- **Cost-Effective Storage**: + - High-compression disk storage + - Unified management of historical and real-time data + +- **Hierarchical Measurement Point Management**: + - Aligns with industrial device hierarchies + - Supports directory browsing and search + +- **High Throughput Read/Write**: + - Supports millions of devices + - Handles high-speed, out-of-order, and multi-frequency data ingestion + +- **Rich Query Capabilities**: + - Native time-series computation engine + - Timestamp alignment during queries + - Over 100 built-in aggregation and time-series functions + - AI-ready time-series feature analysis + +- **High Availability & Scalability**: + - HA distributed architecture with 24/7 uptime + - Automatic load balancing for node scaling + - Heterogeneous cluster support + +- **Low Learning Curve**: + - SQL-like query language + - Multi-language SDKs + - Comprehensive toolchain (e.g., console) + +- **Ecosystem Integration**: + - Hadoop, Spark, Grafana, ThingsBoard, DataEase, etc. + + +## TimechoDB + +Timecho Technology has developed **TimechoDB**, a commercial product built on the open-source version of Apache IoTDB, to provide enterprise-grade solutions and services for businesses and commercial clients. TimechoDB addresses the multifaceted challenges enterprises face when building IoT big data platforms for managing time-series data, including complex application scenarios, massive data volumes, high sampling frequencies, frequent out-of-order data, time-consuming data processing, diverse analytical requirements, and high storage and maintenance costs. + +Leveraging **TimechoDB**, Timecho Technology offers a broader range of product features, enhanced performance and stability, and a richer suite of efficiency tools. Additionally, it provides comprehensive enterprise services, delivering commercial clients with superior product capabilities and an optimized experience in development, operation, and usage. +- **Timecho Technology Official Website**: [https://www.timecho.com/](https://www.timecho.com/) +- **TimechoDB Documentation**: [Quick Start](https://www.timecho.com/docs/zh/UserGuide/latest/QuickStart/QuickStart_timecho.html) diff --git a/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_apache.md b/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_apache.md new file mode 100644 index 000000000..a6b17af49 --- /dev/null +++ b/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_apache.md @@ -0,0 +1,182 @@ + + +# Release History + +## V2.0.1-beta + +> Release Date: 2025.02.18 + +V2.0.1-beta introduces dual model configuration for tree and table models, along with table model support for standard SQL query syntax, multiple functions and operators, stream processing, Benchmark, and more. Additional updates include: Python client support for four new data types, database deletion in read-only mode, script tools compatible with TsFile, CSV, and SQL data import/export, Kubernetes Operator integration, and other enhancements. Key updates include: + +- **Table Model**: IoTDB now supports table models, including standard SQL query syntax (SELECT, WHERE, JOIN, GROUP BY, ORDER BY, LIMIT clauses, and subqueries). +- **Query Module**: Table model supports multiple functions and operators (logical operators, mathematical functions, and time-series-specific functions like DIFF). +- **Query Module**: Users can control UDF, PipePlugin, Trigger, and AINode to load JAR packages via URI through configuration items. +- **Storage Module**: Table model supports data writing via the Session interface, with automatic metadata creation. +- **Storage Module**: Python client adds support for four new data types: String, Blob, Date, and Timestamp. +- **Storage Module**: Optimized priority comparison rules for merge tasks of the same type. +- **Stream Processing**: Supports specifying authentication information at the receiver end. +- **Stream Processing**: TsFile Load now supports table models. +- **Stream Processing**: Stream processing plugins are adapted to table models. +- **System Module**: Improved stability of DataNode scaling down. +- **System Module**: Users can perform `DROP DATABASE` operations in read-only mode. +- **Scripts & Tools**: Benchmark tool adapted to table models. +- **Scripts & Tools**: Benchmark tool supports four new data types: String, Blob, Date, and Timestamp. +- **Scripts & Tools**: Extended `data/export-data` script to support new data types (String, Blob, Date, Timestamp). +- **Scripts & Tools**: `import-data/export-data` scripts now support TsFile, CSV, and SQL data import/export. +- **Ecosystem Integration**: Kubernetes Operator support. + +--- + +## V1.3.3 + +> Release Date: 2024.11.20 + +V1.3.3 introduces String, Blob, Date, and Timestamp data types, data subscription, DataNode auto-monitoring and TsFile loading, observability metrics, sender-to-receiver file transfer automation, configuration file consolidation, client query load balancing, and stability improvements. Key updates include: + +- **Storage Module**: Added String, Blob, Date, and Timestamp data types. +- **Storage Module**: Optimized memory control for merge operations. +- **Query Module**: Client query load balancing enhancements. +- **Query Module**: Added active metadata statistics queries. +- **Query Module**: Filter performance optimization for faster aggregation and WHERE queries. +- **Data Sync**: Sender supports transferring files to a specified directory, after which the receiver automatically loads them into IoTDB. +- **Data Sync**: Receiver adds automatic data type conversion. +- **Data Subscription**: Added subscription capabilities (data points or TsFile-based). +- **Data Loading**: DataNode actively monitors and loads TsFiles, with observability metrics. +- **Stream Processing**: `ALTER PIPE` supports `ALTER SOURCE`. +- **System Module**: Simplified configuration files (merged three into one). +- **System Module**: Added configuration item APIs. +- **System Module**: Improved restart recovery performance. +- **Scripts & Tools**: Metadata import/export scripts. +- **Scripts & Tools**: Kubernetes Helm support. + +--- + +## V1.3.2 + +> Release Date: 2024.07.01 + +V1.3.2 introduces `EXPLAIN ANALYZE` for SQL query profiling, UDAF framework, metadata sync, data point counting, SQL import/export scripts, rolling upgrades, and stability improvements. Key updates include: + +- **Storage Module**: Optimized `insertRecords` write performance. +- **Query Module**: Added `EXPLAIN ANALYZE` for query stage time analysis. +- **Query Module**: Added UDAF (User-Defined Aggregation Function) framework. +- **Query Module**: Added `MAX_BY`/`MIN_BY` functions to return timestamps with max/min values. +- **Query Module**: Value-filter query performance improvements. +- **Data Sync**: Wildcard support for path matching. +- **Data Sync**: Metadata synchronization (time series, attributes, permissions). +- **Stream Processing**: Added `ALTER PIPE` for hot-updating plugins. +- **System Module**: TsFile load data included in data point statistics. +- **Scripts & Tools**: Local upgrade backup tool (hard-link-based backups). +- **Scripts & Tools**: `export-data`/`import-data` scripts for CSV, TsFile, and SQL formats. +- **Scripts & Tools**: Windows support for ConfigNode/DataNode/Cli window naming. + +--- + +## V1.3.1 + +> Release Date: 2024.04.22 + +V1.3.1 adds one-click cluster start/stop scripts, info collection scripts, built-in functions, sync/log/query optimizations, and observability improvements. Key updates include: + +- Added one-click cluster start/stop scripts (`start-all/stop-all.sh` and `.bat`). +- Added instance info collection scripts (`collect-info.sh` and `.bat`). +- Added `STDDEV`, `VARIANCE` aggregation functions. +- Added TsFile repair command. +- `FILL` clause supports timeout thresholds. +- Simplified data sync time range configuration. +- Enhanced observability (cluster node divergence monitoring, task scheduling metrics). +- Optimized default logging strategy. +- Improved memory control for TsFile loading. +- REST API (v2) returns column types. +- Optimized query execution. +- Client auto-fetches available DataNode lists. + +--- + +## V1.3.0 + +> Release Date: 2024.01.01 + +V1.3.0 adds SSL encryption, sync monitoring, permission syntax optimizations, metrics performance improvements, and query optimizations. Key updates include: + +- **Security**: Time-series-level permission control. +- **Security**: SSL support for client-server communication. +- **Query Module**: Calculation views support `LAST` queries. +- **Stream Processing**: Added Pipe monitoring metrics. +- **Storage Module**: Support for negative timestamps. +- **Scripts & Tools**: `LOAD` script data included in monitoring. +- **Client**: Optimized Python client performance. +- **Query Module**: Fixed slow `SHOW PATH` responses. +- **Query Module**: Improved `EXPLAIN` result formatting. +- **System Module**: Added `MEMORY_SIZE` configuration. +- **System Module**: Renamed config items (e.g., `target_config_node_list` → `seed_config_node`). + +--- + +## V1.2.0 + +> Release Date: 2023.06.30 + +V1.2.0 introduces stream processing, dynamic templates, `SUBSTRING`/`REPLACE`/`ROUND` functions, and optimizations. Key updates include: + +- **Stream Processing**: Framework added. +- **Metadata**: Dynamic template expansion. +- **Storage**: Added SPRINTZ/RLBE encodings and LZMA2 compression. +- **Query Module**: `CAST`, `ROUND`, `SUBSTR`, `REPLACE` functions. +- **Query Module**: `TIME_DURATION`, `MODE` aggregation functions. +- **Query Module**: `CASE WHEN` and `ORDER BY` expression support. +- **Client**: Python API supports multi-node distributed connections. +- **Client**: Python write redirection. +- **Session API**: Batch time-series creation via templates. + +--- + +## V1.1.0 + +> Release Date: 2023.04.03 + +V1.1.0 adds `GROUP BY VARIATION/CONDITION`, `DIFF`, `COUNT_IF`, and pipeline engine optimizations. Key fixes include metadata template errors and query issues. Key updates: + +- **Query Module**: `ALIGN BY DEVICE` supports `ORDER BY TIME`. +- **Query Module**: `SHOW QUERIES` and `KILL QUERY` commands. +- **System Module**: `SHOW REGIONS` with database filtering. +- **System Module**: `SHOW VARIABLES` for cluster parameters. +- **Query Module**: `GROUP BY VARIATION` and `GROUP BY CONDITION`. +- **Query Module**: `DIFF` scalar function and `COUNT_IF` aggregation. +- **System Module**: `dn_rpc_port` and `dn_rpc_address` configurability. + +--- + +## V1.0.0 + +> Release Date: 2022.12.03 + +V1.0.0 fixes partition calculation, query execution, and memory issues. Adds HA architecture, multi-replica storage, cluster management SQL, and MPP framework. Key updates: + +- **System Module**: Distributed high-availability architecture. +- **System Module**: Multi-replica storage. +- **System Module**: Port conflict detection during startup. +- **System Module**: Cluster management SQL (start/stop/remove nodes). +- **System Module**: Configurable consensus protocols (Simple, IoTConsensus, Ratis). +- **Query Module**: MPP framework for distributed read/write. +- **Stream Processing**: Framework and cross-cluster sync. + diff --git a/src/UserGuide/latest-Table/Tools-System/Maitenance-Tool_apache.md b/src/UserGuide/latest-Table/Tools-System/Maitenance-Tool_apache.md new file mode 100644 index 000000000..1194c7883 --- /dev/null +++ b/src/UserGuide/latest-Table/Tools-System/Maitenance-Tool_apache.md @@ -0,0 +1,229 @@ + + +# Cluster Management Tools + +## Data Directory Overview Tool + +The IoTDB data directory overview tool is designed to print out the structural overview information of the data directory. The tool is located at `tools/tsfile/print-iotdb-data-dir`. + +### Usage + +- **Windows:** + +```bash +.\print-iotdb-data-dir.bat () +``` + +- **Linux or MacOS:** + +```shell +./print-iotdb-data-dir.sh () +``` + +**Note:** If the output result storage path is not set, the relative path `"IoTDB_data_dir_overview.txt"` will be used as the default. + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-iotdb-data-dir.bat D:\github\master\iotdb\data\datanode\data +```````````````````````` +Starting Printing the IoTDB Data Directory Overview +```````````````````````` +output save path: IoTDB_data_dir_overview.txt +data dir num: 1 +143 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +|============================================================== +|D:\github\master\iotdb\data\datanode\data +|--sequence +| |--root.redirect0 +| | |--1 +| | | |--0 +| |--root.redirect1 +| | |--2 +| | | |--0 +| |--root.redirect2 +| | |--3 +| | | |--0 +| |--root.redirect3 +| | |--4 +| | | |--0 +| |--root.redirect4 +| | |--5 +| | | |--0 +| |--root.redirect5 +| | |--6 +| | | |--0 +| |--root.sg1 +| | |--0 +| | | |--0 +| | | |--2760 +|--unsequence +|============================================================== +````````````````````````` + +## TsFile Overview Tool + +The TsFile overview tool is used to print the content of a TsFile in a sketch mode. The tool is located at `tools/tsfile/print-tsfile`. + +### Usage + +- **Windows:** + +```bash +.\print-tsfile-sketch.bat () +``` + +- **Linux or MacOS:** + +```shell +./print-tsfile-sketch.sh () +``` + +**Note:** If the output result storage path is not set, the relative path `"TsFile_sketch_view.txt"` will be used as the default. + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-tsfile.bat D:\github\master\1669359533965-1-0-0.tsfile D:\github\master\sketch.txt +```````````````````````` +Starting Printing the TsFile Sketch +```````````````````````` +TsFile path: D:\github\master\1669359533965-1-0-0.tsfile +Sketch save path: D:\github\master\sketch.txt +148 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +-------------------------------- TsFile Sketch -------------------------------- +file path: D:\github\master\1669359533965-1-0-0.tsfile +file length: 2974 + + POSITION| CONTENT + -------- ------- + 0| [magic head] TsFile + 6| [version number] 3 +||||||||||||||||||||| [Chunk Group] of root.sg1.d1, num of Chunks:3 + 7| [Chunk Group Header] + | [marker] 0 + | [deviceID] root.sg1.d1 + 20| [Chunk] of root.sg1.d1.s1, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9032452783138882770,maxValue:9117677033041335123,firstValue:7068645577795875906,lastValue:-5833792328174747265,sumValue:5.795959009889246E19] + | [chunk header] marker=5, measurementID=s1, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 + 893| [Chunk] of root.sg1.d1.s2, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-8806861312244965718,maxValue:9192550740609853234,firstValue:1150295375739457693,lastValue:-2839553973758938646,sumValue:8.2822564314572677E18] + | [chunk header] marker=5, measurementID=s2, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 + 1766| [Chunk] of root.sg1.d1.s3, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9076669333460323191,maxValue:9175278522960949594,firstValue:2537897870994797700,lastValue:7194625271253769397,sumValue:-2.126008424849926E19] + | [chunk header] marker=5, measurementID=s3, dataSize=864, dataType=INT64, compressionType=SNAPPY, encodingType=RLE + | [page] UncompressedSize:862, CompressedSize:860 +||||||||||||||||||||| [Chunk Group] of root.sg1.d1 ends + 2656| [marker] 2 + 2657| [TimeseriesIndex] of root.sg1.d1.s1, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9032452783138882770,maxValue:9117677033041335123,firstValue:7068645577795875906,lastValue:-5833792328174747265,sumValue:5.795959009889246E19] + | [ChunkIndex] offset=20 + 2728| [TimeseriesIndex] of root.sg1.d1.s2, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-8806861312244965718,maxValue:9192550740609853234,firstValue:1150295375739457693,lastValue:-2839553973758938646,sumValue:8.2822564314572677E18] + | [ChunkIndex] offset=893 + 2799| [TimeseriesIndex] of root.sg1.d1.s3, tsDataType:INT64, startTime: 1669359533948 endTime: 1669359534047 count: 100 [minValue:-9076669333460323191,maxValue:9175278522960949594,firstValue:2537897870994797700,lastValue:7194625271253769397,sumValue:-2.126008424849926E19] + | [ChunkIndex] offset=1766 + 2870| [IndexOfTimerseriesIndex Node] type=LEAF_MEASUREMENT + | + | +||||||||||||||||||||| [TsFileMetadata] begins + 2891| [IndexOfTimerseriesIndex Node] type=LEAF_DEVICE + | + | + | [meta offset] 2656 + | [bloom filter] bit vector byte array length=31, filterSize=256, hashFunctionSize=5 +||||||||||||||||||||| [TsFileMetadata] ends + 2964| [TsFileMetadataSize] 73 + 2968| [magic tail] TsFile + 2974| END of TsFile +---------------------------- IndexOfTimerseriesIndex Tree ----------------------------- + [MetadataIndex:LEAF_DEVICE] + └──────[root.sg1.d1,2870] + [MetadataIndex:LEAF_MEASUREMENT] + └──────[s1,2657] +---------------------------------- TsFile Sketch End ---------------------------------- +````````````````````````` + +**Explanation:** + +- The left side of the "|" separator represents the actual position in the TsFile, and the right side represents the sketch content. +- The "|||||||||||||||||||||" is added for enhanced readability and is not actual data stored in the TsFile. +- The final printed "IndexOfTimerseriesIndex Tree" is a reorganized print of the metadata index tree at the end of the TsFile for intuitive understanding and is not the actual data stored in the TsFile. + +## TsFile Resource Overview Tool + +The TsFile resource overview tool is used to print out the content of TsFile resource files. The tool is located at `tools/tsfile/print-tsfile-resource-files`. + +### Usage + +- **Windows:** + +```bash +.\print-tsfile-resource-files.bat +``` + +- **Linux or MacOS:** + +```shell +./print-tsfile-resource-files.sh +``` + +### Example + +Taking Windows system as an example: + +`````````````````````````bash +.\print-tsfile-resource-files.bat D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0 +```````````````````````` +Starting Printing the TsFileResources +```````````````````````` +147 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +230 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +231 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +233 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +237 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +Analyzing D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile ... + +Resource plan index range [9223372036854775807, -9223372036854775808] +device root.sg1.d1, start time 0 (1970-01-01T08:00+08:00[GMT+08:00]), end time 99 (1970-01-01T08:00:00.099+08:00[GMT+08:00]) + +Analyzing the resource file folder D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0 finished. +````````````````````````` + +`````````````````````````bash +.\print-tsfile-resource-files.bat D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile.resource +```````````````````````` +Starting Printing the TsFileResources +```````````````````````` +178 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +186 [main] WARN o.a.i.t.c.conf.TSFileDescriptor - not found iotdb-system.properties, use the default configs. +187 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +188 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-system.properties, use default configuration +192 [main] WARN o.a.iotdb.db.conf.IoTDBDescriptor - Couldn't load the configuration iotdb-system.properties from any of the known sources. +Analyzing D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile ... + +Resource plan index range [9223372036854775807, -9223372036854775808] +device root.sg1.d1, start time 0 (1970-01-01T08:00+08:00[GMT+08:00]), end time 99 (1970-01-01T08:00:00.099+08:00[GMT+08:00]) + +Analyzing the resource file D:\github\master\iotdb\data\datanode\data\sequence\root.sg1\0\0\1669359533489-1-0-0.tsfile.resource finished. +````````````````````````` \ No newline at end of file diff --git a/src/UserGuide/latest-Table/Tools-System/Monitor-Tool_apache.md b/src/UserGuide/latest-Table/Tools-System/Monitor-Tool_apache.md new file mode 100644 index 000000000..5e29a0fa1 --- /dev/null +++ b/src/UserGuide/latest-Table/Tools-System/Monitor-Tool_apache.md @@ -0,0 +1,168 @@ + + +# Prometheus + +The deployment of monitoring tools can be referenced in the document [Monitoring Panel Deployment](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) chapter. + +## Prometheus Mapping Relationship for Monitoring Metrics + +> For a monitoring metric with Metric Name as `name`, Tags as `K1=V1, ..., Kn=Vn`, the following mapping applies, where `value` is the specific value. + +| Monitoring Metric Type | Mapping Relationship | +| ---------------------- | ------------------------------------------------------------ | +| Counter | `name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value` | +| AutoGauge, Gauge | `name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value` | +| Histogram | `name_max{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_sum{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_count{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.5"} value`
`name{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.99"} value` | +| Rate | `name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m1"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m5"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="m15"} value`
`name_total{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", rate="mean"} value` | +| Timer | `name_seconds_max{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds_sum{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds_count{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn"} value`
`name_seconds{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.5"} value`
`name_seconds{cluster="clusterName", nodeType="nodeType", nodeId="nodeId",k1="V1" , ..., Kn="Vn", quantile="0.99"} value` | + +## Modifying Configuration Files + +1) Taking DataNode as an example, modify the `iotdb-system.properties` configuration file as follows: + +```properties +dn_metric_reporter_list=PROMETHEUS +dn_metric_level=CORE +dn_metric_prometheus_reporter_port=9091 +``` + +2) Start the IoTDB DataNode. + +3) Open a browser or use `curl` to access `http://server_ip:9091/metrics`, and you will get metric data as follows: + +``` +... +# HELP file_count +# TYPE file_count gauge +file_count{name="wal",} 0.0 +file_count{name="unseq",} 0.0 +file_count{name="seq",} 2.0 +... +``` + +## Prometheus + Grafana + +As shown above, IoTDB exposes monitoring metrics in the standard Prometheus format. You can use Prometheus to collect and store these metrics and Grafana to visualize them. + +The relationship between IoTDB, Prometheus, and Grafana is illustrated below: + +![iotdb_prometheus_grafana](/img/UserGuide/System-Tools/Metrics/iotdb_prometheus_grafana.png) + +1. IoTDB continuously collects monitoring metrics during operation. +2. Prometheus pulls monitoring metrics from IoTDB's HTTP interface at fixed intervals (configurable). +3. Prometheus stores the pulled monitoring metrics in its TSDB. +4. Grafana queries monitoring metrics from Prometheus at fixed intervals (configurable) and visualizes them. + +From the interaction flow, it is clear that additional work is required to deploy and configure Prometheus and Grafana. + +For example, you can configure Prometheus as follows (some parameters can be adjusted as needed) to pull metrics from IoTDB: + +```yaml +job_name: pull-metrics +honor_labels: true +honor_timestamps: true +scrape_interval: 15s +scrape_timeout: 10s +metrics_path: /metrics +scheme: http +follow_redirects: true +static_configs: + - targets: + - localhost:9091 +``` + +For more details, refer to the following documents: + +- [Prometheus Installation and Usage Documentation](https://prometheus.io/docs/prometheus/latest/getting_started/) +- [Prometheus Configuration for Pulling Metrics from HTTP Interface](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) +- [Grafana Installation and Usage Documentation](https://grafana.com/docs/grafana/latest/getting-started/getting-started/) +- [Grafana Querying Data from Prometheus and Plotting Documentation](https://prometheus.io/docs/visualization/grafana/#grafana-support-for-prometheus) + +## Apache IoTDB Dashboard + +The `Apache IoTDB Dashboard` is a companion product of IoTDB Enterprise Edition, supporting unified centralized operation and maintenance management. It allows monitoring multiple clusters through a single monitoring panel. You can contact the business team to obtain the Dashboard's JSON file. + +![Apache IoTDB Dashboard](/img/%E7%9B%91%E6%8E%A7%20default%20cluster.png) + +![Apache IoTDB Dashboard](/img/%E7%9B%91%E6%8E%A7%20cluster2.png) + +### Cluster Overview + +You can monitor, but not limited to: + +- Total CPU cores, total memory space, total disk space of the cluster. +- Number of ConfigNodes and DataNodes in the cluster. +- Cluster uptime. +- Cluster write speed. +- Current CPU, memory, and disk usage of each node in the cluster. +- Node-specific information. + +![](/img/%E7%9B%91%E6%8E%A7%20%E6%A6%82%E8%A7%88.png) + +### Data Writing + +You can monitor, but not limited to: + +- Average write latency, median latency, 99th percentile latency. +- Number and size of WAL files. +- Node WAL flush SyncBuffer latency. + +![](/img/%E7%9B%91%E6%8E%A7%20%E5%86%99%E5%85%A5.png) + +### Data Query + +You can monitor, but not limited to: + +- Node query loading time series metadata latency. +- Node query reading time series latency. +- Node query modifying time series metadata latency. +- Node query loading Chunk metadata list latency. +- Node query modifying Chunk metadata latency. +- Node query filtering by Chunk metadata latency. +- Node query constructing Chunk Reader latency average. + +![](/img/%E7%9B%91%E6%8E%A7%20%E6%9F%A5%E8%AF%A2.png) + +### Storage Engine + +You can monitor, but not limited to: + +- Number and size of files by type. +- Number and size of TsFiles in various stages. +- Number and latency of various tasks. + +![](/img/%E7%9B%91%E6%8E%A7%20%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E.png) + +### System Monitoring + +You can monitor, but not limited to: + +- System memory, swap memory, process memory. +- Disk space, file count, file size. +- JVM GC time ratio, GC count by type, GC data volume, heap memory usage by generation. +- Network transmission rate, packet sending rate. + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9F%20%E5%86%85%E5%AD%98%E4%B8%8E%E7%A1%AC%E7%9B%98.png) + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9Fjvm.png) + +![](/img/%E7%9B%91%E6%8E%A7%20%E7%B3%BB%E7%BB%9F%20%E7%BD%91%E7%BB%9C.png) \ No newline at end of file