diff --git a/src/UserGuide/Master/Reference/Common-Config-Manual.md b/src/UserGuide/Master/Reference/Common-Config-Manual.md index ca3292388..97a9414fe 100644 --- a/src/UserGuide/Master/Reference/Common-Config-Manual.md +++ b/src/UserGuide/Master/Reference/Common-Config-Manual.md @@ -59,7 +59,7 @@ Different configuration parameters take effect in the following three ways: | Name | schema\_region\_consensus\_protocol\_class | |:-----------:|:--------------------------------------------------------------------------------------------------------------------------------------------:| -| Description | Consensus protocol of schema replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could only use RatisConsensus | +| Description | Consensus protocol of schema replicas，larger than 1 replicas could only use RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.ratis.RatisConsensus | | Effective | Only allowed to be modified in first start up | @@ -77,7 +77,7 @@ Different configuration parameters take effect in the following three ways: | Name | data\_region\_consensus\_protocol\_class | |:-----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------| -| Description | Consensus protocol of data replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could use IoTConsensus or RatisConsensus | +| Description | Consensus protocol of data replicas，larger than 1 replicas could use IoTConsensus or RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.simple.SimpleConsensus | | Effective | Only allowed to be modified in first start up | @@ -147,11 +147,11 @@ Different configuration parameters take effect in the following three ways: | Default | 1 | | Effective | After restarting system | -* data\_region\_per\_processor +* data\_region\_per\_data\_node -| Name | data\_region\_per\_processor | +| Name | data\_region\_per\_data\_node | |:-----------:|:--------------------------------------------------------------------------| -| Description | The maximum number of DataRegion expected to be managed by each processor | +| Description | The maximum number of DataRegion expected to be managed by each DataNode | | Type | double | | Default | 1.0 | | Effective | After restarting system | diff --git a/src/UserGuide/Master/Reference/ConfigNode-Config-Manual.md b/src/UserGuide/Master/Reference/ConfigNode-Config-Manual.md index 563efe253..80c2cbaf7 100644 --- a/src/UserGuide/Master/Reference/ConfigNode-Config-Manual.md +++ b/src/UserGuide/Master/Reference/ConfigNode-Config-Manual.md @@ -96,13 +96,13 @@ The global configuration of cluster is in ConfigNode. |Default| 10720 | |Effective|Only allowed to be modified in first start up| -### Target Config Nodes +### SeedConfigNode * cn\_seed\_config\_node |Name| cn\_seed\_config\_node | |:---:|:----------------------------------------------------------------------| -|Description| Target ConfigNode address, for current ConfigNode to join the cluster | +|Description| Seed ConfigNode's address for current ConfigNode to join the cluster. This parameter is corresponding to cn\_target\_config\_node\_list before V1.2.2 | |Type| String | |Default| 127.0.0.1:10710 | |Effective| Only allowed to be modified in first start up | diff --git a/src/UserGuide/Master/Reference/DataNode-Config-Manual.md b/src/UserGuide/Master/Reference/DataNode-Config-Manual.md index be0478dda..947a16546 100644 --- a/src/UserGuide/Master/Reference/DataNode-Config-Manual.md +++ b/src/UserGuide/Master/Reference/DataNode-Config-Manual.md @@ -216,13 +216,13 @@ The permission definitions are in ${IOTDB\_CONF}/conf/jmx.access. |Default| "" | |Effective| After restarting system | -### Target Config Nodes +### SeedConfigNode * dn\_seed\_config\_node |Name| dn\_seed\_config\_node | |:---:|:------------------------------------------------| -|Description| ConfigNode Address for DataNode to join cluster | +|Description| ConfigNode Address for DataNode to join cluster. This parameter is corresponding to dn\_target\_config\_node\_list before V1.2.2 | |Type| String | |Default| 127.0.0.1:10710 | |Effective| Only allowed to be modified in first start up | diff --git a/src/UserGuide/Master/Reference/UDF-development.md b/src/UserGuide/Master/Reference/UDF-development.md index 8e14f31db..057aabbe1 100644 --- a/src/UserGuide/Master/Reference/UDF-development.md +++ b/src/UserGuide/Master/Reference/UDF-development.md @@ -1,8 +1,8 @@ # UDF development -## UDF development +## 1. UDF development -### UDF Development Dependencies +### 1.1 UDF Development Dependencies If you use [Maven](http://search.maven.org/), you can search for the development dependencies listed below from the [Maven repository](http://search.maven.org/) . Please note that you must select the same dependency version as the target IoTDB server version for development. @@ -15,7 +15,7 @@ If you use [Maven](http://search.maven.org/), you can search for the development ``` -## UDTF（User Defined Timeseries Generating Function） +## 1.2 UDTF（User Defined Timeseries Generating Function） To write a UDTF, you need to inherit the `org.apache.iotdb.udf.api.UDTF` class, and at least implement the `beforeStart` method and a `transform` method. @@ -25,6 +25,8 @@ To write a UDTF, you need to inherit the `org.apache.iotdb.udf.api.UDTF` class, | :----------------------------------------------------------- | :----------------------------------------------------------- | ----------------------------------------------------- | | void validate(UDFParameterValidator validator) throws Exception | This method is mainly used to validate `UDFParameters` and it is executed before `beforeStart(UDFParameters, UDTFConfigurations)` is called. | Optional | | void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | The initialization method to call the user-defined initialization behavior before a UDTF processes the input data. Every time a user executes a UDTF query, the framework will construct a new UDF instance, and `beforeStart` will be called. | Required | +| Object transform(Row row) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `MappableRowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Row`, and the transformation result should be returned. | Required to implement at least one `transform` method | +| void transform(Column[] columns, ColumnBuilder builder) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `MappableRowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Column[]`, and the transformation result should be output by `ColumnBuilder`. You need to call the data collection method provided by `builder` to determine the output data. | Required to implement at least one `transform` method | | void transform(Row row, PointCollector collector) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `RowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Row`, and the transformation result should be output by `PointCollector`. You need to call the data collection method provided by `collector` to determine the output data. | Required to implement at least one `transform` method | | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `SlidingSizeWindowAccessStrategy` or `SlidingTimeWindowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `RowWindow`, and the transformation result should be output by `PointCollector`. You need to call the data collection method provided by `collector` to determine the output data. | Required to implement at least one `transform` method | | void terminate(PointCollector collector) throws Exception | This method is called by the framework. This method will be called once after all `transform` calls have been executed. In a single UDF query, this method will and will only be called once. You need to call the data collection method provided by `collector` to determine the output data. | Optional | @@ -34,7 +36,7 @@ In the life cycle of a UDTF instance, the calling sequence of each method is as 1. void validate(UDFParameterValidator validator) throws Exception 2. void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception -3. void transform(Row row, PointCollector collector) throws Exception or void transform(RowWindow rowWindow, PointCollector collector) throws Exception +3. `Object transform(Row row) throws Exception` or `void transform(Column[] columns, ColumnBuilder builder) throws Exception` or `void transform(Row row, PointCollector collector) throws Exception` or `void transform(RowWindow rowWindow, PointCollector collector) throws Exception` 4. void terminate(PointCollector collector) throws Exception 5. void beforeDestroy() @@ -116,7 +118,7 @@ The following are the strategies you can set: | Interface definition | Description | The `transform` Method to Call | | :-------------------------------- | :----------------------------------------------------------- | ------------------------------------------------------------ | -| MappableRowByRow | Custom scalar function
The framework will call the `transform` method once for each row of raw data input, with k columns of time series and 1 row of data input, and 1 column of time series and 1 row of data output. It can be used in any clause and expression where scalar functions appear, such as select clauses, where clauses, etc. | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | +| MappableRowByRowStrategy | Custom scalar function
The framework will call the `transform` method once for each row of raw data input, with k columns of time series and 1 row of data input, and 1 column of time series and 1 row of data output. It can be used in any clause and expression where scalar functions appear, such as select clauses, where clauses, etc. | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | | RowByRowAccessStrategy | Customize time series generation function to process raw data line by line.
The framework will call the `transform` method once for each row of raw data input, inputting k columns of time series and 1 row of data, and outputting 1 column of time series and n rows of data.
When a sequence is input, the row serves as a data point for the input sequence.
When multiple sequences are input, after aligning the input sequences in time, each row serves as a data point for the input sequence.
（In a row of data, there may be a column with a `null` value, but not all columns are `null`） | void transform(Row row, PointCollector collector) throws Exception | | SlidingTimeWindowAccessStrategy | Customize time series generation functions to process raw data in a sliding time window manner.
The framework will call the `transform` method once for each raw data input window, input k columns of time series m rows of data, and output 1 column of time series n rows of data.
A window may contain multiple rows of data, and after aligning the input sequence in time, each window serves as a data point for the input sequence.
(Each window may have i rows, and each row of data may have a column with a `null` value, but not all of them are `null`) | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | | SlidingSizeWindowAccessStrategy | Customize the time series generation function to process raw data in a fixed number of rows, meaning that each data processing window will contain a fixed number of rows of data (except for the last window).
The framework will call the `transform` method once for each raw data input window, input k columns of time series m rows of data, and output 1 column of time series n rows of data.
A window may contain multiple rows of data, and after aligning the input sequence in time, each window serves as a data point for the input sequence.
(Each window may have i rows, and each row of data may have a column with a `null` value, but not all of them are `null`) | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | @@ -126,7 +128,7 @@ The following are the strategies you can set: #### Interface Description: -- `RowByRowAccessStrategy`: The construction of `RowByRowAccessStrategy` does not require any parameters. +- `MappableRowByRowStrategy` and `RowByRowAccessStrategy`: The construction of `RowByRowAccessStrategy` does not require any parameters. - `SlidingTimeWindowAccessStrategy` @@ -219,7 +221,100 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th } ``` -3. **void transform(Row row, PointCollector collector) throws Exception** +3. **Object transform(Row row) throws Exception** + +You need to implement this method or `transform(Column[] columns, ColumnBuilder builder) throws Exception` when you specify the strategy of UDF to read the original data as `MappableRowByRowAccessStrategy`. + +This method processes the raw data one row at a time. The raw data is input from `Row` and output by its return object. You must return only one object based on each input data point in a single `transform` method call, i.e., input and output are one-to-one. It should be noted that the type of output data points must be the same as you set in the `beforeStart` method, and the timestamps of output data points must be strictly monotonically increasing. + +The following is a complete UDF example that implements the `Object transform(Row row) throws Exception` method. It is an adder that receives two columns of time series as input. + +```java +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.access.Row; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type dataType; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + dataType = parameters.getDataType(0); + configurations + .setAccessStrategy(new MappableRowByRowAccessStrategy()) + .setOutputDataType(dataType); + } + + @Override + public Object transform(Row row) throws Exception { + return row.getLong(0) + row.getLong(1); + } +} +``` + + + +4. **void transform(Column[] columns, ColumnBuilder builder) throws Exception** + +You need to implement this method or `Object transform(Row row) throws Exception` when you specify the strategy of UDF to read the original data as `MappableRowByRowAccessStrategy`. + +This method processes the raw data multiple rows at a time. After performance tests, we found that UDTF that process multiple rows at once perform better than those UDTF that process one data point at a time. The raw data is input from `Column[]` and output by `ColumnBuilder`. You must output a corresponding data point based on each input data point in a single `transform` method call, i.e., input and output are still one-to-one. It should be noted that the type of output data points must be the same as you set in the `beforeStart` method, and the timestamps of output data points must be strictly monotonically increasing. + +The following is a complete UDF example that implements the `void transform(Column[] columns, ColumnBuilder builder) throws Exception` method. It is an adder that receives two columns of time series as input. + +```java +import org.apache.iotdb.tsfile.read.common.block.column.Column; +import org.apache.iotdb.tsfile.read.common.block.column.ColumnBuilder; +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type type; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + type = parameters.getDataType(0); + configurations.setAccessStrategy(new MappableRowByRowAccessStrategy()).setOutputDataType(type); + } + + @Override + public void transform(Column[] columns, ColumnBuilder builder) throws Exception { + long[] inputs1 = columns[0].getLongs(); + long[] inputs2 = columns[1].getLongs(); + + int count = columns[0].getPositionCount(); + for (int i = 0; i < count; i++) { + builder.writeLong(inputs1[i] + inputs2[i]); + } + } +} +``` + +5. **void transform(Row row, PointCollector collector) throws Exception** You need to implement this method when you specify the strategy of UDF to read the original data as `RowByRowAccessStrategy`. @@ -255,7 +350,7 @@ public class Adder implements UDTF { } ``` -4. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** +6. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** You need to implement this method when you specify the strategy of UDF to read the original data as `SlidingTimeWindowAccessStrategy` or `SlidingSizeWindowAccessStrategy`. @@ -296,7 +391,7 @@ public class Counter implements UDTF { } ``` -5. **void terminate(PointCollector collector) throws Exception** +7. **void terminate(PointCollector collector) throws Exception** In some scenarios, a UDF needs to traverse all the original data to calculate the final output data points. The `terminate` interface provides support for those scenarios. @@ -349,7 +444,7 @@ public class Max implements UDTF { } ``` -6. **void beforeDestroy()** +8. **void beforeDestroy()** The method for terminating a UDF. @@ -357,7 +452,7 @@ This method is called by the framework. For a UDF instance, `beforeDestroy` will -### UDAF (User Defined Aggregation Function) +### 1.3 UDAF (User Defined Aggregation Function) A complete definition of UDAF involves two classes, `State` and `UDAF`. @@ -594,16 +689,16 @@ The method for terminating a UDF. This method is called by the framework. For a UDF instance, `beforeDestroy` will be called after the last record is processed. In the entire life cycle of the instance, `beforeDestroy` will only be called once. -### Maven Project Example +### 1.4 Maven Project Example If you use Maven, you can build your own UDF project referring to our **udf-example** module. You can find the project [here](https://github.com/apache/iotdb/tree/master/example/udf). -## Contribute universal built-in UDF functions to iotdb +## 2. Contribute universal built-in UDF functions to iotdb This part mainly introduces how external users can contribute their own UDFs to the IoTDB community. -#### Prerequisites +#### 2.1 Prerequisites 1. UDFs must be universal. @@ -614,31 +709,33 @@ This part mainly introduces how external users can contribute their own UDFs to 2. The UDF you are going to contribute has been well tested and can run normally in the production environment. -#### What you need to prepare +#### 2.2 What you need to prepare 1. UDF source code 2. Test cases 3. Instructions -#### UDF Source Code +### 2.3 Contribution Content + +#### 2.3.1 UDF Source Code 1. Create the UDF main class and related classes in `iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin` or in its subfolders. 2. Register your UDF in `iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/BuiltinTimeSeriesGeneratingFunction.java`. -#### Test Cases +#### 2.3.2 Test Cases At a minimum, you need to write integration tests for the UDF. You can add a test class in `integration-test/src/test/java/org/apache/iotdb/db/it/udf`. -#### Instructions +#### 2.3.3 Instructions The instructions need to include: the name and the function of the UDF, the attribute parameters that must be provided when the UDF is executed, the applicable scenarios, and the usage examples, etc. The instructions for use should include both Chinese and English versions. Instructions for use should be added separately in `docs/zh/UserGuide/Operation Manual/DML Data Manipulation Language.md` and `docs/UserGuide/Operation Manual/DML Data Manipulation Language.md`. -#### Submit a PR +#### 2.3.4 Submit a PR When you have prepared the UDF source code, test cases, and instructions, you are ready to submit a Pull Request (PR) on [Github](https://github.com/apache/iotdb). You can refer to our code contribution guide to submit a PR: [Development Guide](https://iotdb.apache.org/Community/Development-Guide.html). diff --git a/src/UserGuide/Master/SQL-Manual/SQL-Manual.md b/src/UserGuide/Master/SQL-Manual/SQL-Manual.md index e7d53ea2f..66f9115fa 100644 --- a/src/UserGuide/Master/SQL-Manual/SQL-Manual.md +++ b/src/UserGuide/Master/SQL-Manual/SQL-Manual.md @@ -450,22 +450,17 @@ For more details, see document [Import-Export-Tool](../Tools-System/TsFile-Impor 1. Load a single tsfile by specifying a file path (absolute path). - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` #### Load with Script diff --git a/src/UserGuide/Master/User-Manual/Operator-and-Expression.md b/src/UserGuide/Master/User-Manual/Operator-and-Expression.md index 152af176e..0f094c982 100644 --- a/src/UserGuide/Master/User-Manual/Operator-and-Expression.md +++ b/src/UserGuide/Master/User-Manual/Operator-and-Expression.md @@ -104,6 +104,12 @@ The built-in functions can be used in IoTDB without registration, and the functi | SUM | Summation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | | COUNT | Counts the number of data points. | All types | / | INT | | AVG | Average. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV | Alias for STDDEV_SAMP. Return the sample standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV_POP | Return the population standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV_SAMP | Return the sample standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VARIANCE | Alias for VAR_SAMP. Return the sample variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VAR_POP | Return the population variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VAR_SAMP | Return the sample variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | | EXTREME | Finds the value with the largest absolute value. Returns a positive value if the maximum absolute value of positive and negative values is equal. | INT32 INT64 FLOAT DOUBLE | / | Consistent with the input data type | | MAX_VALUE | Find the maximum value. | INT32 INT64 FLOAT DOUBLE STRING TIMESTAMP DATE | / | Consistent with the input data type | | MIN_VALUE | Find the minimum value. | INT32 INT64 FLOAT DOUBLE STRING TIMESTAMP DATE | / | Consistent with the input data type | diff --git a/src/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md b/src/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md index 9d24ef130..8915e0425 100644 --- a/src/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md +++ b/src/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md @@ -31,20 +31,18 @@ The user sends specified commands to the Apache IoTDB system through the Cli too ### load tsfiles -The command to load tsfiles is `load [autoregister=true/false][,sglevel=int][,verify=true/false]`. +The command to load tsfiles is `load [autoregister=true/false][,sglevel=int]`. This command has two usages: 1. Load a single tsfile by specifying a file path (absolute path). -The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{in_space_compaction_num}-{cross_space_compaction_num}.tsfile`. This command has three options: autoregister, sglevel and verify. +The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{in_space_compaction_num}-{cross_space_compaction_num}.tsfile`. This command has 2 options: autoregister and sglevel. AUTOREGISTER option. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If this parameter is true, the schema is created automatically. If it is false, the schema will not be created. By default, the schema will be created. SGLEVEL option. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. Examples: @@ -53,11 +51,8 @@ Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=true` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=true,sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false,sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false,verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false,sglevel=1,verify=true` + 2. Load a batch of files by specifying a folder path (absolute path). @@ -69,7 +64,6 @@ Examples: * `load '/Users/Desktop/data' autoregister=false` * `load '/Users/Desktop/data' autoregister=true` * `load '/Users/Desktop/data' autoregister=true,sglevel=1` -* `load '/Users/Desktop/data' autoregister=false,sglevel=1,verify=true` #### Remote Load TsFile diff --git a/src/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md b/src/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md index 112b42aba..a281eec25 100644 --- a/src/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md +++ b/src/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md @@ -31,18 +31,16 @@ The user sends specified commands to the Apache IoTDB system through the Cli too ### load tsfiles -The command to load tsfiles is `load [sglevel=int][verify=true/false][onSuccess=delete/none]`. +The command to load tsfiles is `load [sglevel=int][onSuccess=delete/none]`. This command has two usages: 1. Load a single tsfile by specifying a file path (absolute path). -The first parameter indicates the path of the tsfile to be loaded. This command has three options: sglevel, verify, onSuccess. +The first parameter indicates the path of the tsfile to be loaded. This command has 2 options: sglevel, onSuccess. SGLEVEL option. If the database correspond to the tsfile does not exist, the user can set the level of database through the fourth parameter. By default, it uses the database level which is set in `iotdb-datanode.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - ONSUCCESS option. The default value is DELETE, which means the processing method of successfully loaded tsfiles, and DELETE means after the tsfile is successfully loaded, it will be deleted. NONE means after the tsfile is successfully loaded, it will be remained in the origin dir. If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. @@ -50,14 +48,10 @@ If the `.resource` file corresponding to the file exists, it will be loaded into Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). @@ -66,26 +60,23 @@ The first parameter indicates the path of the tsfile to be loaded. The options a Examples: * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **NOTICE**: When `$IOTDB_HOME$/conf/iotdb-datanode.properties` has `enable_auto_create_schema=true`, it will automatically create metadata in TSFILE, otherwise it will not be created automatically. ## Load with Script -Run rewrite-tsfile.bat if you are in a Windows environment, or rewrite-tsfile.sh if you are on Linux or Unix. +If you are in a Windows environment, please run `$IOTDB_HOME\tools\load-tsfile.bat`. If you are in Linux or Unix, please run `load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +.\load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f File/Directory to be load, required -h IoTDB Host address, optional field, 127.0.0.1 by default -p IoTDB port, optional field, 6667 by default -u IoTDB user name, optional field, root by default -pw IoTDB password, optional field, root by default --sgLevel Sg level of loading Tsfile, optional field, default_storage_group_level in iotdb-common.properties by default ---verify Verify schema or not, optional field, True by default --onSuccess Delete or remain origin TsFile after loading, optional field, none by default ``` diff --git a/src/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md b/src/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md index 1e6657c65..57d5747de 100644 --- a/src/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md +++ b/src/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md @@ -32,18 +32,16 @@ The user sends specified commands to the Apache IoTDB system through the Cli too #### load tsfiles -The command to load tsfiles is `load [sglevel=int][verify=true/false][onSuccess=delete/none]`. +The command to load tsfiles is `load [sglevel=int][onSuccess=delete/none]`. This command has two usages: 1. Load a single tsfile by specifying a file path (absolute path). -The first parameter indicates the path of the tsfile to be loaded. This command has three options: sglevel, verify, onSuccess. +The first parameter indicates the path of the tsfile to be loaded. This command has 2 options: sglevel, onSuccess. SGLEVEL option. If the database correspond to the tsfile does not exist, the user can set the level of database through the fourth parameter. By default, it uses the database level which is set in `iotdb-datanode.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - ONSUCCESS option. The default value is DELETE, which means the processing method of successfully loaded tsfiles, and DELETE means after the tsfile is successfully loaded, it will be deleted. NONE means after the tsfile is successfully loaded, it will be remained in the origin dir. If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. @@ -51,14 +49,11 @@ If the `.resource` file corresponding to the file exists, it will be loaded into Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). @@ -67,10 +62,8 @@ The first parameter indicates the path of the tsfile to be loaded. The options a Examples: * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **NOTICE**: When `$IOTDB_HOME$/conf/iotdb-datanode.properties` has `enable_auto_create_schema=true`, it will automatically create metadata in TSFILE, otherwise it will not be created automatically. @@ -79,14 +72,13 @@ Examples: Run rewrite-tsfile.bat if you are in a Windows environment, or rewrite-tsfile.sh if you are on Linux or Unix. ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f File/Directory to be load, required -h IoTDB Host address, optional field, 127.0.0.1 by default -p IoTDB port, optional field, 6667 by default -u IoTDB user name, optional field, root by default -pw IoTDB password, optional field, root by default --sgLevel Sg level of loading Tsfile, optional field, default_storage_group_level in iotdb-common.properties by default ---verify Verify schema or not, optional field, True by default --onSuccess Delete or remain origin TsFile after loading, optional field, none by default ``` diff --git a/src/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md b/src/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md index 112b42aba..a281eec25 100644 --- a/src/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md +++ b/src/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md @@ -31,18 +31,16 @@ The user sends specified commands to the Apache IoTDB system through the Cli too ### load tsfiles -The command to load tsfiles is `load [sglevel=int][verify=true/false][onSuccess=delete/none]`. +The command to load tsfiles is `load [sglevel=int][onSuccess=delete/none]`. This command has two usages: 1. Load a single tsfile by specifying a file path (absolute path). -The first parameter indicates the path of the tsfile to be loaded. This command has three options: sglevel, verify, onSuccess. +The first parameter indicates the path of the tsfile to be loaded. This command has 2 options: sglevel, onSuccess. SGLEVEL option. If the database correspond to the tsfile does not exist, the user can set the level of database through the fourth parameter. By default, it uses the database level which is set in `iotdb-datanode.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - ONSUCCESS option. The default value is DELETE, which means the processing method of successfully loaded tsfiles, and DELETE means after the tsfile is successfully loaded, it will be deleted. NONE means after the tsfile is successfully loaded, it will be remained in the origin dir. If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. @@ -50,14 +48,10 @@ If the `.resource` file corresponding to the file exists, it will be loaded into Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). @@ -66,26 +60,23 @@ The first parameter indicates the path of the tsfile to be loaded. The options a Examples: * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **NOTICE**: When `$IOTDB_HOME$/conf/iotdb-datanode.properties` has `enable_auto_create_schema=true`, it will automatically create metadata in TSFILE, otherwise it will not be created automatically. ## Load with Script -Run rewrite-tsfile.bat if you are in a Windows environment, or rewrite-tsfile.sh if you are on Linux or Unix. +If you are in a Windows environment, please run `$IOTDB_HOME\tools\load-tsfile.bat`. If you are in Linux or Unix, please run `load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +.\load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f File/Directory to be load, required -h IoTDB Host address, optional field, 127.0.0.1 by default -p IoTDB port, optional field, 6667 by default -u IoTDB user name, optional field, root by default -pw IoTDB password, optional field, root by default --sgLevel Sg level of loading Tsfile, optional field, default_storage_group_level in iotdb-common.properties by default ---verify Verify schema or not, optional field, True by default --onSuccess Delete or remain origin TsFile after loading, optional field, none by default ``` diff --git a/src/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md b/src/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md index 1e6657c65..818982357 100644 --- a/src/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md +++ b/src/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md @@ -32,18 +32,16 @@ The user sends specified commands to the Apache IoTDB system through the Cli too #### load tsfiles -The command to load tsfiles is `load [sglevel=int][verify=true/false][onSuccess=delete/none]`. +The command to load tsfiles is `load [sglevel=int][onSuccess=delete/none]`. This command has two usages: 1. Load a single tsfile by specifying a file path (absolute path). -The first parameter indicates the path of the tsfile to be loaded. This command has three options: sglevel, verify, onSuccess. +The first parameter indicates the path of the tsfile to be loaded. This command has 2 options: sglevel, onSuccess. SGLEVEL option. If the database correspond to the tsfile does not exist, the user can set the level of database through the fourth parameter. By default, it uses the database level which is set in `iotdb-datanode.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - ONSUCCESS option. The default value is DELETE, which means the processing method of successfully loaded tsfiles, and DELETE means after the tsfile is successfully loaded, it will be deleted. NONE means after the tsfile is successfully loaded, it will be remained in the origin dir. If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. @@ -51,14 +49,10 @@ If the `.resource` file corresponding to the file exists, it will be loaded into Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). @@ -67,10 +61,8 @@ The first parameter indicates the path of the tsfile to be loaded. The options a Examples: * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **NOTICE**: When `$IOTDB_HOME$/conf/iotdb-datanode.properties` has `enable_auto_create_schema=true`, it will automatically create metadata in TSFILE, otherwise it will not be created automatically. @@ -79,14 +71,13 @@ Examples: Run rewrite-tsfile.bat if you are in a Windows environment, or rewrite-tsfile.sh if you are on Linux or Unix. ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f File/Directory to be load, required -h IoTDB Host address, optional field, 127.0.0.1 by default -p IoTDB port, optional field, 6667 by default -u IoTDB user name, optional field, root by default -pw IoTDB password, optional field, root by default --sgLevel Sg level of loading Tsfile, optional field, default_storage_group_level in iotdb-common.properties by default ---verify Verify schema or not, optional field, True by default --onSuccess Delete or remain origin TsFile after loading, optional field, none by default ``` diff --git a/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide.md b/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide.md index 05f76c01e..c965f296d 100644 --- a/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide.md +++ b/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide.md @@ -77,7 +77,6 @@ Users can start IoTDB standalone mode by the start-standalone script under the s ``` Note: Currently, To run standalone mode, you need to ensure that all addresses are set to 127.0.0.1, If you need to access the IoTDB from a machine different from the one where the IoTDB is located, please change the configuration item `dn_rpc_address` to the IP of the machine where the IoTDB lives. And replication factors set to 1, which is by now the default setting. -Besides, it's recommended to use SimpleConsensus in this mode, since it brings additional efficiency. ## Cluster Deployment This article uses a local environment as an example to diff --git a/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide_timecho.md b/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide_timecho.md index cfd1a141f..bd323615c 100644 --- a/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide_timecho.md +++ b/src/UserGuide/V1.2.x/Deployment-and-Maintenance/Deployment-Guide_timecho.md @@ -77,7 +77,6 @@ Users can start IoTDB standalone mode by the start-standalone script under the s ``` Note: Currently, To run standalone mode, you need to ensure that all addresses are set to 127.0.0.1, If you need to access the IoTDB from a machine different from the one where the IoTDB is located, please change the configuration item `dn_rpc_address` to the IP of the machine where the IoTDB lives. And replication factors set to 1, which is by now the default setting. -Besides, it's recommended to use SimpleConsensus in this mode, since it brings additional efficiency. ## Cluster deployment(Cluster management tool) diff --git a/src/UserGuide/V1.2.x/QuickStart/QuickStart.md b/src/UserGuide/V1.2.x/QuickStart/QuickStart.md index 7d08ee402..8ac4b45c0 100644 --- a/src/UserGuide/V1.2.x/QuickStart/QuickStart.md +++ b/src/UserGuide/V1.2.x/QuickStart/QuickStart.md @@ -76,7 +76,6 @@ Users can start IoTDB in standalone mode by using the `start-standalone` script Note: In order to run IoTDB in standalone mode, you need to ensure that all addresses are set to 127.0.0.1. If you need to access the IoTDB from a different machine, please change the configuration item `dn_rpc_address` to the public IP of the machine where IoTDB is running and be sure to set `replication factors` to 1, which is currently the default setting. -It is recommended to use `SimpleConsensus` in this mode, as this has performance advantages in this mode of operation. ### Using the Command Line Interfave (CLI) diff --git a/src/UserGuide/V1.2.x/Reference/Common-Config-Manual.md b/src/UserGuide/V1.2.x/Reference/Common-Config-Manual.md index a053da136..d227fd87e 100644 --- a/src/UserGuide/V1.2.x/Reference/Common-Config-Manual.md +++ b/src/UserGuide/V1.2.x/Reference/Common-Config-Manual.md @@ -59,7 +59,7 @@ Different configuration parameters take effect in the following three ways: | Name | schema\_region\_consensus\_protocol\_class | |:-----------:|:--------------------------------------------------------------------------------------------------------------------------------------------:| -| Description | Consensus protocol of schema replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could only use RatisConsensus | +| Description | Consensus protocol of schema replicas，larger than 1 replicas could only use RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.ratis.RatisConsensus | | Effective | Only allowed to be modified in first start up | @@ -77,7 +77,7 @@ Different configuration parameters take effect in the following three ways: | Name | data\_region\_consensus\_protocol\_class | |:-----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------| -| Description | Consensus protocol of data replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could use IoTConsensus or RatisConsensus | +| Description | Consensus protocol of data replicasa，larger than 1 replicas could use IoTConsensus or RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.simple.SimpleConsensus | | Effective | Only allowed to be modified in first start up | diff --git a/src/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md b/src/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md index e753cae93..e2980a9ea 100644 --- a/src/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md +++ b/src/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md @@ -458,22 +458,17 @@ For more details, see document [Import-Export-Tool](../Tools-System/Import-Expor 1. Load a single tsfile by specifying a file path (absolute path). - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` #### Load with Script diff --git a/src/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md b/src/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md index 381aaf9ee..8c8dde0ce 100644 --- a/src/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md +++ b/src/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md @@ -37,18 +37,16 @@ The user sends specified commands to the Apache IoTDB system through the Cli too ##### Load Tsfiles -The command to load tsfiles is `load [sglevel=int][verify=true/false][onSuccess=delete/none]`. +The command to load tsfiles is `load [sglevel=int][onSuccess=delete/none]`. This command has two usages: -1. Load a single tsfile by specifying a file path (absolute path). +1. Load a single tsfile by specifying a file path (absolute path). -The first parameter indicates the path of the tsfile to be loaded. This command has three options: sglevel, verify, onSuccess. +The first parameter indicates the path of the tsfile to be loaded. This command has 2 options: sglevel, onSuccess. SGLEVEL option. If the database correspond to the tsfile does not exist, the user can set the level of database through the fourth parameter. By default, it uses the database level which is set in `iotdb-datanode.properties`. -VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice. - ONSUCCESS option. The default value is DELETE, which means the processing method of successfully loaded tsfiles, and DELETE means after the tsfile is successfully loaded, it will be deleted. NONE means after the tsfile is successfully loaded, it will be remained in the origin dir. If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file. @@ -56,26 +54,20 @@ If the `.resource` file corresponding to the file exists, it will be loaded into Examples: * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + -2. Load a batch of files by specifying a folder path (absolute path). +2. Load a batch of files by specifying a folder path (absolute path). The first parameter indicates the path of the tsfile to be loaded. The options above also works for this command. Examples: * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **NOTICE**: When `$IOTDB_HOME$/conf/iotdb-datanode.properties` has `enable_auto_create_schema=true`, it will automatically create metadata in TSFILE, otherwise it will not be created automatically. @@ -84,14 +76,13 @@ Examples: Run rewrite-tsfile.bat if you are in a Windows environment, or rewrite-tsfile.sh if you are on Linux or Unix. ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f File/Directory to be load, required -h IoTDB Host address, optional field, 127.0.0.1 by default -p IoTDB port, optional field, 6667 by default -u IoTDB user name, optional field, root by default -pw IoTDB password, optional field, root by default --sgLevel Sg level of loading Tsfile, optional field, default_storage_group_level in iotdb-common.properties by default ---verify Verify schema or not, optional field, True by default --onSuccess Delete or remain origin TsFile after loading, optional field, none by default ``` diff --git a/src/UserGuide/latest/Reference/Common-Config-Manual.md b/src/UserGuide/latest/Reference/Common-Config-Manual.md index a9d702ee0..38e1d5d2b 100644 --- a/src/UserGuide/latest/Reference/Common-Config-Manual.md +++ b/src/UserGuide/latest/Reference/Common-Config-Manual.md @@ -59,7 +59,7 @@ Different configuration parameters take effect in the following three ways: | Name | schema\_region\_consensus\_protocol\_class | |:-----------:|:--------------------------------------------------------------------------------------------------------------------------------------------:| -| Description | Consensus protocol of schema replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could only use RatisConsensus | +| Description | Consensus protocol of schema replicas，larger than 1 replicas could only use RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.ratis.RatisConsensus | | Effective | Only allowed to be modified in first start up | @@ -77,7 +77,7 @@ Different configuration parameters take effect in the following three ways: | Name | data\_region\_consensus\_protocol\_class | |:-----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------| -| Description | Consensus protocol of data replicas, SimpleConsensus could only be used in 1 replica，larger than 1 replicas could use IoTConsensus or RatisConsensus | +| Description | Consensus protocol of data replicasa，larger than 1 replicas could use IoTConsensus or RatisConsensus | | Type | String | | Default | org.apache.iotdb.consensus.simple.SimpleConsensus | | Effective | Only allowed to be modified in first start up | diff --git a/src/UserGuide/latest/Reference/UDF-development.md b/src/UserGuide/latest/Reference/UDF-development.md index 8e14f31db..057aabbe1 100644 --- a/src/UserGuide/latest/Reference/UDF-development.md +++ b/src/UserGuide/latest/Reference/UDF-development.md @@ -1,8 +1,8 @@ # UDF development -## UDF development +## 1. UDF development -### UDF Development Dependencies +### 1.1 UDF Development Dependencies If you use [Maven](http://search.maven.org/), you can search for the development dependencies listed below from the [Maven repository](http://search.maven.org/) . Please note that you must select the same dependency version as the target IoTDB server version for development. @@ -15,7 +15,7 @@ If you use [Maven](http://search.maven.org/), you can search for the development ``` -## UDTF（User Defined Timeseries Generating Function） +## 1.2 UDTF（User Defined Timeseries Generating Function） To write a UDTF, you need to inherit the `org.apache.iotdb.udf.api.UDTF` class, and at least implement the `beforeStart` method and a `transform` method. @@ -25,6 +25,8 @@ To write a UDTF, you need to inherit the `org.apache.iotdb.udf.api.UDTF` class, | :----------------------------------------------------------- | :----------------------------------------------------------- | ----------------------------------------------------- | | void validate(UDFParameterValidator validator) throws Exception | This method is mainly used to validate `UDFParameters` and it is executed before `beforeStart(UDFParameters, UDTFConfigurations)` is called. | Optional | | void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | The initialization method to call the user-defined initialization behavior before a UDTF processes the input data. Every time a user executes a UDTF query, the framework will construct a new UDF instance, and `beforeStart` will be called. | Required | +| Object transform(Row row) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `MappableRowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Row`, and the transformation result should be returned. | Required to implement at least one `transform` method | +| void transform(Column[] columns, ColumnBuilder builder) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `MappableRowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Column[]`, and the transformation result should be output by `ColumnBuilder`. You need to call the data collection method provided by `builder` to determine the output data. | Required to implement at least one `transform` method | | void transform(Row row, PointCollector collector) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `RowByRowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `Row`, and the transformation result should be output by `PointCollector`. You need to call the data collection method provided by `collector` to determine the output data. | Required to implement at least one `transform` method | | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | This method is called by the framework. This data processing method will be called when you choose to use the `SlidingSizeWindowAccessStrategy` or `SlidingTimeWindowAccessStrategy` strategy (set in `beforeStart`) to consume raw data. Input data is passed in by `RowWindow`, and the transformation result should be output by `PointCollector`. You need to call the data collection method provided by `collector` to determine the output data. | Required to implement at least one `transform` method | | void terminate(PointCollector collector) throws Exception | This method is called by the framework. This method will be called once after all `transform` calls have been executed. In a single UDF query, this method will and will only be called once. You need to call the data collection method provided by `collector` to determine the output data. | Optional | @@ -34,7 +36,7 @@ In the life cycle of a UDTF instance, the calling sequence of each method is as 1. void validate(UDFParameterValidator validator) throws Exception 2. void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception -3. void transform(Row row, PointCollector collector) throws Exception or void transform(RowWindow rowWindow, PointCollector collector) throws Exception +3. `Object transform(Row row) throws Exception` or `void transform(Column[] columns, ColumnBuilder builder) throws Exception` or `void transform(Row row, PointCollector collector) throws Exception` or `void transform(RowWindow rowWindow, PointCollector collector) throws Exception` 4. void terminate(PointCollector collector) throws Exception 5. void beforeDestroy() @@ -116,7 +118,7 @@ The following are the strategies you can set: | Interface definition | Description | The `transform` Method to Call | | :-------------------------------- | :----------------------------------------------------------- | ------------------------------------------------------------ | -| MappableRowByRow | Custom scalar function
The framework will call the `transform` method once for each row of raw data input, with k columns of time series and 1 row of data input, and 1 column of time series and 1 row of data output. It can be used in any clause and expression where scalar functions appear, such as select clauses, where clauses, etc. | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | +| MappableRowByRowStrategy | Custom scalar function
The framework will call the `transform` method once for each row of raw data input, with k columns of time series and 1 row of data input, and 1 column of time series and 1 row of data output. It can be used in any clause and expression where scalar functions appear, such as select clauses, where clauses, etc. | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | | RowByRowAccessStrategy | Customize time series generation function to process raw data line by line.
The framework will call the `transform` method once for each row of raw data input, inputting k columns of time series and 1 row of data, and outputting 1 column of time series and n rows of data.
When a sequence is input, the row serves as a data point for the input sequence.
When multiple sequences are input, after aligning the input sequences in time, each row serves as a data point for the input sequence.
（In a row of data, there may be a column with a `null` value, but not all columns are `null`） | void transform(Row row, PointCollector collector) throws Exception | | SlidingTimeWindowAccessStrategy | Customize time series generation functions to process raw data in a sliding time window manner.
The framework will call the `transform` method once for each raw data input window, input k columns of time series m rows of data, and output 1 column of time series n rows of data.
A window may contain multiple rows of data, and after aligning the input sequence in time, each window serves as a data point for the input sequence.
(Each window may have i rows, and each row of data may have a column with a `null` value, but not all of them are `null`) | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | | SlidingSizeWindowAccessStrategy | Customize the time series generation function to process raw data in a fixed number of rows, meaning that each data processing window will contain a fixed number of rows of data (except for the last window).
The framework will call the `transform` method once for each raw data input window, input k columns of time series m rows of data, and output 1 column of time series n rows of data.
A window may contain multiple rows of data, and after aligning the input sequence in time, each window serves as a data point for the input sequence.
(Each window may have i rows, and each row of data may have a column with a `null` value, but not all of them are `null`) | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | @@ -126,7 +128,7 @@ The following are the strategies you can set: #### Interface Description: -- `RowByRowAccessStrategy`: The construction of `RowByRowAccessStrategy` does not require any parameters. +- `MappableRowByRowStrategy` and `RowByRowAccessStrategy`: The construction of `RowByRowAccessStrategy` does not require any parameters. - `SlidingTimeWindowAccessStrategy` @@ -219,7 +221,100 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th } ``` -3. **void transform(Row row, PointCollector collector) throws Exception** +3. **Object transform(Row row) throws Exception** + +You need to implement this method or `transform(Column[] columns, ColumnBuilder builder) throws Exception` when you specify the strategy of UDF to read the original data as `MappableRowByRowAccessStrategy`. + +This method processes the raw data one row at a time. The raw data is input from `Row` and output by its return object. You must return only one object based on each input data point in a single `transform` method call, i.e., input and output are one-to-one. It should be noted that the type of output data points must be the same as you set in the `beforeStart` method, and the timestamps of output data points must be strictly monotonically increasing. + +The following is a complete UDF example that implements the `Object transform(Row row) throws Exception` method. It is an adder that receives two columns of time series as input. + +```java +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.access.Row; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type dataType; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + dataType = parameters.getDataType(0); + configurations + .setAccessStrategy(new MappableRowByRowAccessStrategy()) + .setOutputDataType(dataType); + } + + @Override + public Object transform(Row row) throws Exception { + return row.getLong(0) + row.getLong(1); + } +} +``` + + + +4. **void transform(Column[] columns, ColumnBuilder builder) throws Exception** + +You need to implement this method or `Object transform(Row row) throws Exception` when you specify the strategy of UDF to read the original data as `MappableRowByRowAccessStrategy`. + +This method processes the raw data multiple rows at a time. After performance tests, we found that UDTF that process multiple rows at once perform better than those UDTF that process one data point at a time. The raw data is input from `Column[]` and output by `ColumnBuilder`. You must output a corresponding data point based on each input data point in a single `transform` method call, i.e., input and output are still one-to-one. It should be noted that the type of output data points must be the same as you set in the `beforeStart` method, and the timestamps of output data points must be strictly monotonically increasing. + +The following is a complete UDF example that implements the `void transform(Column[] columns, ColumnBuilder builder) throws Exception` method. It is an adder that receives two columns of time series as input. + +```java +import org.apache.iotdb.tsfile.read.common.block.column.Column; +import org.apache.iotdb.tsfile.read.common.block.column.ColumnBuilder; +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type type; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + type = parameters.getDataType(0); + configurations.setAccessStrategy(new MappableRowByRowAccessStrategy()).setOutputDataType(type); + } + + @Override + public void transform(Column[] columns, ColumnBuilder builder) throws Exception { + long[] inputs1 = columns[0].getLongs(); + long[] inputs2 = columns[1].getLongs(); + + int count = columns[0].getPositionCount(); + for (int i = 0; i < count; i++) { + builder.writeLong(inputs1[i] + inputs2[i]); + } + } +} +``` + +5. **void transform(Row row, PointCollector collector) throws Exception** You need to implement this method when you specify the strategy of UDF to read the original data as `RowByRowAccessStrategy`. @@ -255,7 +350,7 @@ public class Adder implements UDTF { } ``` -4. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** +6. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** You need to implement this method when you specify the strategy of UDF to read the original data as `SlidingTimeWindowAccessStrategy` or `SlidingSizeWindowAccessStrategy`. @@ -296,7 +391,7 @@ public class Counter implements UDTF { } ``` -5. **void terminate(PointCollector collector) throws Exception** +7. **void terminate(PointCollector collector) throws Exception** In some scenarios, a UDF needs to traverse all the original data to calculate the final output data points. The `terminate` interface provides support for those scenarios. @@ -349,7 +444,7 @@ public class Max implements UDTF { } ``` -6. **void beforeDestroy()** +8. **void beforeDestroy()** The method for terminating a UDF. @@ -357,7 +452,7 @@ This method is called by the framework. For a UDF instance, `beforeDestroy` will -### UDAF (User Defined Aggregation Function) +### 1.3 UDAF (User Defined Aggregation Function) A complete definition of UDAF involves two classes, `State` and `UDAF`. @@ -594,16 +689,16 @@ The method for terminating a UDF. This method is called by the framework. For a UDF instance, `beforeDestroy` will be called after the last record is processed. In the entire life cycle of the instance, `beforeDestroy` will only be called once. -### Maven Project Example +### 1.4 Maven Project Example If you use Maven, you can build your own UDF project referring to our **udf-example** module. You can find the project [here](https://github.com/apache/iotdb/tree/master/example/udf). -## Contribute universal built-in UDF functions to iotdb +## 2. Contribute universal built-in UDF functions to iotdb This part mainly introduces how external users can contribute their own UDFs to the IoTDB community. -#### Prerequisites +#### 2.1 Prerequisites 1. UDFs must be universal. @@ -614,31 +709,33 @@ This part mainly introduces how external users can contribute their own UDFs to 2. The UDF you are going to contribute has been well tested and can run normally in the production environment. -#### What you need to prepare +#### 2.2 What you need to prepare 1. UDF source code 2. Test cases 3. Instructions -#### UDF Source Code +### 2.3 Contribution Content + +#### 2.3.1 UDF Source Code 1. Create the UDF main class and related classes in `iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin` or in its subfolders. 2. Register your UDF in `iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/BuiltinTimeSeriesGeneratingFunction.java`. -#### Test Cases +#### 2.3.2 Test Cases At a minimum, you need to write integration tests for the UDF. You can add a test class in `integration-test/src/test/java/org/apache/iotdb/db/it/udf`. -#### Instructions +#### 2.3.3 Instructions The instructions need to include: the name and the function of the UDF, the attribute parameters that must be provided when the UDF is executed, the applicable scenarios, and the usage examples, etc. The instructions for use should include both Chinese and English versions. Instructions for use should be added separately in `docs/zh/UserGuide/Operation Manual/DML Data Manipulation Language.md` and `docs/UserGuide/Operation Manual/DML Data Manipulation Language.md`. -#### Submit a PR +#### 2.3.4 Submit a PR When you have prepared the UDF source code, test cases, and instructions, you are ready to submit a Pull Request (PR) on [Github](https://github.com/apache/iotdb). You can refer to our code contribution guide to submit a PR: [Development Guide](https://iotdb.apache.org/Community/Development-Guide.html). diff --git a/src/UserGuide/latest/SQL-Manual/SQL-Manual.md b/src/UserGuide/latest/SQL-Manual/SQL-Manual.md index 6f07e9b30..bd49782cb 100644 --- a/src/UserGuide/latest/SQL-Manual/SQL-Manual.md +++ b/src/UserGuide/latest/SQL-Manual/SQL-Manual.md @@ -449,22 +449,17 @@ For more details, see document [Import-Export-Tool](../Tools-System/TsFile-Impor 1. Load a single tsfile by specifying a file path (absolute path). - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. Load a batch of files by specifying a folder path (absolute path). - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` #### Load with Script diff --git a/src/UserGuide/latest/User-Manual/Operator-and-Expression.md b/src/UserGuide/latest/User-Manual/Operator-and-Expression.md index a1aa45576..64626eaf5 100644 --- a/src/UserGuide/latest/User-Manual/Operator-and-Expression.md +++ b/src/UserGuide/latest/User-Manual/Operator-and-Expression.md @@ -104,6 +104,12 @@ The built-in functions can be used in IoTDB without registration, and the functi | SUM | Summation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | | COUNT | Counts the number of data points. | All types | / | INT | | AVG | Average. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV | Alias for STDDEV_SAMP. Return the sample standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV_POP | Return the population standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| STDDEV_SAMP | Return the sample standard deviation. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VARIANCE | Alias for VAR_SAMP. Return the sample variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VAR_POP | Return the population variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | +| VAR_SAMP | Return the sample variance. | INT32 INT64 FLOAT DOUBLE | / | DOUBLE | | EXTREME | Finds the value with the largest absolute value. Returns a positive value if the maximum absolute value of positive and negative values is equal. | INT32 INT64 FLOAT DOUBLE | / | Consistent with the input data type | | MAX_VALUE | Find the maximum value. | INT32 INT64 FLOAT DOUBLE | / | Consistent with the input data type | | MIN_VALUE | Find the minimum value. | INT32 INT64 FLOAT DOUBLE | / | Consistent with the input data type | diff --git a/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md b/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md index 02bdd56a1..fd76cb4e3 100644 --- a/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md +++ b/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md @@ -61,7 +61,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | schema\_region\_consensus\_protocol\_class | |:------:|:----------------------------------------------------------------| -| 描述 | 元数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时只能使用 RatisConsensus | +| 描述 | 元数据副本的共识协议，多副本时只能使用 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.ratis.RatisConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | @@ -79,7 +79,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | data\_region\_consensus\_protocol\_class | |:------:|:------------------------------------------------------------------------------| -| 描述 | 数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | +| 描述 | 数据副本的共识协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.iot.IoTConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | @@ -149,11 +149,11 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 默认值 | 2 | | 改后生效方式 | 重启生效 | -* data\_region\_per\_processor +* data\_region\_per\_data\_node -| 名字 | data\_region\_per\_processor | +| 名字 | data\_region\_per\_data\_node| |:------:|:-----------------------------| -| 描述 | 期望每个处理器可管理的 DataRegion 的最大数量 | +| 描述 | 期望每个 DataNode 可管理的 DataRegion 的最大数量 | | 类型 | double | | 默认值 | 1.0 | | 改后生效方式 | 重启生效 | diff --git a/src/zh/UserGuide/Master/Reference/ConfigNode-Config-Manual.md b/src/zh/UserGuide/Master/Reference/ConfigNode-Config-Manual.md index b0d8c6fc1..4e08f9e01 100644 --- a/src/zh/UserGuide/Master/Reference/ConfigNode-Config-Manual.md +++ b/src/zh/UserGuide/Master/Reference/ConfigNode-Config-Manual.md @@ -93,13 +93,13 @@ IoTDB 集群的全局配置通过 ConfigNode 配置。 | 默认值 | 10720 | | 改后生效方式 | 仅允许在第一次启动服务前修改 | -### 目标 Config Node 配置 +### SeedConfigNode 配置 * cn\_seed\_config\_node | 名字 | cn\_seed\_config\_node | |:------:|:--------------------------------------| -| 描述 | 目标 ConfigNode 地址，ConfigNode 通过此地址加入集群 | +| 描述 | 目标 ConfigNode 地址，ConfigNode 通过此地址加入集群，推荐使用 SeedConfigNode。V1.2.2 及以前曾用名是 cn\_target\_config\_node\_list | | 类型 | String | | 默认值 | 127.0.0.1:10710 | | 改后生效方式 | 仅允许在第一次启动服务前修改 | diff --git a/src/zh/UserGuide/Master/Reference/DataNode-Config-Manual.md b/src/zh/UserGuide/Master/Reference/DataNode-Config-Manual.md index 01c0ee937..434744aeb 100644 --- a/src/zh/UserGuide/Master/Reference/DataNode-Config-Manual.md +++ b/src/zh/UserGuide/Master/Reference/DataNode-Config-Manual.md @@ -201,13 +201,13 @@ IoTDB DataNode 与 Standalone 模式共用一套配置文件，均位于 IoTDB |改后生效方式| 重启服务生效 | -### 目标 Config Nodes 配置 +### SeedConfigNode 配置 * dn\_seed\_config\_node |名字| dn\_seed\_config\_node | |:---:|:------------------------------------| -|描述| ConfigNode 地址，DataNode 启动时通过此地址加入集群 | +|描述| ConfigNode 地址，DataNode 启动时通过此地址加入集群，推荐使用 SeedConfigNode。V1.2.2 及以前曾用名是 dn\_target\_config\_node\_list | |类型| String | |默认值| 127.0.0.1:10710 | |改后生效方式| 仅允许在第一次启动服务前修改 | diff --git a/src/zh/UserGuide/Master/Reference/Function-and-Expression.md b/src/zh/UserGuide/Master/Reference/Function-and-Expression.md index 8775e99f7..2de545af7 100644 --- a/src/zh/UserGuide/Master/Reference/Function-and-Expression.md +++ b/src/zh/UserGuide/Master/Reference/Function-and-Expression.md @@ -29,23 +29,28 @@ IoTDB 支持的聚合函数如下： -| 函数名 | 功能描述 | 允许的输入类型 | 必要的属性参数 | 输出类型 | -|---------------|----------------------------------------------------------------------------------------------------------|-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| -| SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| COUNT | 计算数据点数。 | 所有类型 | 无 | INT64 | -| AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| FIRST_VALUE | 求时间戳最小的值。 | 所有类型 | 无 | 与输入类型一致 | -| LAST_VALUE | 求时间戳最大的值。 | 所有类型 | 无 | 与输入类型一致 | -| MAX_TIME | 求最大时间戳。 | 所有类型 | 无 | Timestamp | -| MIN_TIME | 求最小时间戳。 | 所有类型 | 无 | Timestamp | -| COUNT_IF | 求数据点连续满足某一给定条件，且满足条件的数据点个数（用keep表示）满足指定阈值的次数。 | BOOLEAN | `[keep >=/>/=/!=/= threshold`,`threshold`类型为`INT64`
`ignoreNull`：可选，默认为`true`；为`true`表示忽略null值，即如果中间出现null值，直接忽略，不会打断连续性；为`false`表示不忽略null值，即如果中间出现null值，会打断连续性 | INT64 | -| TIME_DURATION | 求某一列最大一个不为NULL的值所在时间戳与最小一个不为NULL的值所在时间戳的时间戳差 | 所有类型 | 无 | INT64 | -| MODE | 求众数。注意：
1.输入序列的不同值个数过多时会有内存异常风险;
2.如果所有元素出现的频次相同，即没有众数，则返回对应时间戳最小的值;
3.如果有多个众数，则返回对应时间戳最小的众数。 | 所有类型 | 无 | 与输入类型一致 | -| STDDEV | 求数据的总体标准差。注意：
数据中的空值、缺失值和`NaN`将会被忽略。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| COUNT_TIME | 查询结果集的时间戳的数量。与 align by device 搭配使用时，得到的结果是每个设备的结果集的时间戳的数量。 | 所有类型，输入参数只能为* | 无 | INT64 | +| 函数名 | 功能描述 | 允许的输入类型 | 必要的属性参数 | 输出类型 | +| ------------- | ------------------------------------------------------------ | ------------------------- | ------------------------------------------------------------ | -------------- | +| SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| COUNT | 计算数据点数。 | 所有类型 | 无 | INT64 | +| AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV | STDDEV_SAMP 的别名，求样本标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV_POP | 求总体标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV_SAMP | 求样本标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VARIANCE | VAR_SAMP 的别名，求样本方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VAR_POP | 求总体方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VAR_SAMP | 求样本方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| FIRST_VALUE | 求时间戳最小的值。 | 所有类型 | 无 | 与输入类型一致 | +| LAST_VALUE | 求时间戳最大的值。 | 所有类型 | 无 | 与输入类型一致 | +| MAX_TIME | 求最大时间戳。 | 所有类型 | 无 | Timestamp | +| MIN_TIME | 求最小时间戳。 | 所有类型 | 无 | Timestamp | +| COUNT_IF | 求数据点连续满足某一给定条件，且满足条件的数据点个数（用keep表示）满足指定阈值的次数。 | BOOLEAN | `[keep >=/>/=/!=/= threshold`,`threshold`类型为`INT64`
`ignoreNull`：可选，默认为`true`；为`true`表示忽略null值，即如果中间出现null值，直接忽略，不会打断连续性；为`false`表示不忽略null值，即如果中间出现null值，会打断连续性 | INT64 | +| TIME_DURATION | 求某一列最大一个不为NULL的值所在时间戳与最小一个不为NULL的值所在时间戳的时间戳差 | 所有类型 | 无 | INT64 | +| MODE | 求众数。注意：
1.输入序列的不同值个数过多时会有内存异常风险;
2.如果所有元素出现的频次相同，即没有众数，则返回对应时间戳最小的值;
3.如果有多个众数，则返回对应时间戳最小的众数。 | 所有类型 | 无 | 与输入类型一致 | +| COUNT_TIME | 查询结果集的时间戳的数量。与 align by device 搭配使用时，得到的结果是每个设备的结果集的时间戳的数量。 | 所有类型，输入参数只能为* | 无 | INT64 | | MAX_BY | MAX_BY(x, y) 求二元输入 x 和 y 在 y 最大时对应的 x 的值。MAX_BY(time, x) 返回 x 取最大值时对应的时间戳。 | 第一个输入 x 可以是任意类型，第二个输入 y 只能是 INT32 INT64 FLOAT DOUBLE | 无 | 与第一个输入 x 的数据类型一致 | | MIN_BY | MIN_BY(x, y) 求二元输入 x 和 y 在 y 最小时对应的 x 的值。MIN_BY(time, x) 返回 x 取最小值时对应的时间戳。 | 第一个输入 x 可以是任意类型，第二个输入 y 只能是 INT32 INT64 FLOAT DOUBLE | 无 | 与第一个输入 x 的数据类型一致 | diff --git a/src/zh/UserGuide/Master/Reference/UDF-development.md b/src/zh/UserGuide/Master/Reference/UDF-development.md index 975744d4d..d2ecb5dcd 100644 --- a/src/zh/UserGuide/Master/Reference/UDF-development.md +++ b/src/zh/UserGuide/Master/Reference/UDF-development.md @@ -21,20 +21,22 @@ #### 接口说明: -| 接口定义 | 描述 | 是否必须 | -| :----------------------------------------------------------- | :----------------------------------------------------------- | ------------------ | -| void validate(UDFParameterValidator validator) throws Exception | 在初始化方法`beforeStart`调用前执行，用于检测`UDFParameters`中用户输入的参数是否合法。 | 否 | -| void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | 初始化方法，在 UDTF 处理输入数据前，调用用户自定义的初始化行为。用户每执行一次 UDTF 查询，框架就会构造一个新的 UDF 类实例，该方法在每个 UDF 类实例被初始化时调用一次。在每一个 UDF 类实例的生命周期内，该方法只会被调用一次。 | 是 | -| void transform(Row row, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`RowByRowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`Row`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 与下面的方法二选一 | -| void transform(RowWindow rowWindow, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`SlidingSizeWindowAccessStrategy`或者`SlidingTimeWindowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`RowWindow`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 与上面的方法二选一 | -| void terminate(PointCollector collector) throws Exception | 这个方法由框架调用。该方法会在所有的`transform`调用执行完成后，在`beforeDestory`方法执行前被调用。在一个 UDF 查询过程中，该方法会且只会调用一次。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 否 | -| void beforeDestroy() | UDTF 的结束方法。此方法由框架调用，并且只会被调用一次，即在处理完最后一条记录之后被调用。 | 否 | +| 接口定义 | 描述 | 是否必须 | +| :----------------------------------------------------------- | :----------------------------------------------------------- | ------------------------- | +| void validate(UDFParameterValidator validator) throws Exception | 在初始化方法`beforeStart`调用前执行，用于检测`UDFParameters`中用户输入的参数是否合法。 | 否 | +| void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | 初始化方法，在 UDTF 处理输入数据前，调用用户自定义的初始化行为。用户每执行一次 UDTF 查询，框架就会构造一个新的 UDF 类实例，该方法在每个 UDF 类实例被初始化时调用一次。在每一个 UDF 类实例的生命周期内，该方法只会被调用一次。 | 是 | +| Object transform(Row row) throws Exception` | 这个方法由框架调用。当您在`beforeStart`中选择以`MappableRowByRowAccessStrategy`的策略消费原始数据时，可以选用该方法进行数据处理。输入参数以`Row`的形式传入，输出结果通过返回值`Object`输出。 | 所有`transform`方法四选一 | +| void transform(Column[] columns, ColumnBuilder builder) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`MappableRowByRowAccessStrategy`的策略消费原始数据时，可以选用该方法进行数据处理。输入参数以`Column[]`的形式传入，输出结果通过`ColumnBuilder`输出。您需要在该方法内自行调用`builder`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void transform(Row row, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`RowByRowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`Row`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void transform(RowWindow rowWindow, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`SlidingSizeWindowAccessStrategy`或者`SlidingTimeWindowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`RowWindow`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void terminate(PointCollector collector) throws Exception | 这个方法由框架调用。该方法会在所有的`transform`调用执行完成后，在`beforeDestory`方法执行前被调用。在一个 UDF 查询过程中，该方法会且只会调用一次。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 否 | +| void beforeDestroy() | UDTF 的结束方法。此方法由框架调用，并且只会被调用一次，即在处理完最后一条记录之后被调用。 | 否 | 在一个完整的 UDTF 实例生命周期中，各个方法的调用顺序如下： 1. void validate(UDFParameterValidator validator) throws Exception 2. void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception -3. void transform(Row row, PointCollector collector) throws Exception 或者 void transform(RowWindow rowWindow, PointCollector collector) throws Exception +3. Object transform(Row row) throws Exception 或着 void transform(Column[] columns, ColumnBuilder builder) throws Exception 或者 void transform(Row row, PointCollector collector) throws Exception 或者 void transform(RowWindow rowWindow, PointCollector collector) throws Exception 4. void terminate(PointCollector collector) throws Exception 5. void beforeDestroy() @@ -110,7 +112,7 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th | 接口定义 | 描述 | 调用的`transform`方法 | | ------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| MappableRowByRow | 自定义标量函数
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 1 行数据，可用于标量函数出现的任何子句和表达式中，如select子句、where子句等。 | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | +| MappableRowByRowStrategy | 自定义标量函数
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 1 行数据，可用于标量函数出现的任何子句和表达式中，如select子句、where子句等。 | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | | RowByRowAccessStrategy | 自定义时间序列生成函数，逐行地处理原始数据。
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 n 行数据。
当输入一个序列时，该行就作为输入序列的一个数据点。
当输入多个序列时，输入序列按时间对齐后，每一行作为的输入序列的一个数据点。
（一行数据中，可能存在某一列为`null`值，但不会全部都是`null`） | void transform(Row row, PointCollector collector) throws Exception | | SlidingTimeWindowAccessStrategy | 自定义时间序列生成函数，以滑动时间窗口的方式处理原始数据。
框架会为每一个原始数据输入窗口调用一次`transform`方法，输入 k 列时间序列 m 行数据，输出 1 列时间序列 n 行数据。
一个窗口可能存在多行数据，输入序列按时间对齐后，每个窗口作为的输入序列的一个数据点。
（每个窗口可能存在 i 行，每行数据可能存在某一列为`null`值，但不会全部都是`null`） | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | | SlidingSizeWindowAccessStrategy | 自定义时间序列生成函数，以固定行数的方式处理原始数据，即每个数据处理窗口都会包含固定行数的数据（最后一个窗口除外）。
框架会为每一个原始数据输入窗口调用一次`transform`方法，输入 k 列时间序列 m 行数据，输出 1 列时间序列 n 行数据。
一个窗口可能存在多行数据，输入序列按时间对齐后，每个窗口作为的输入序列的一个数据点。
（每个窗口可能存在 i 行，每行数据可能存在某一列为`null`值，但不会全部都是`null`） | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | @@ -119,7 +121,7 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th #### 接口详情： -- `RowByRowAccessStrategy`的构造不需要任何参数。 +- `MappableRowByRowStrategy` 和 `RowByRowAccessStrategy`的构造不需要任何参数。 - `SlidingTimeWindowAccessStrategy` @@ -212,7 +214,98 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th } ``` -3. **void transform(Row row, PointCollector collector) throws Exception** +3. **Object transform(Row row) throws Exception** + +当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `MappableRowByRowAccessStrategy`，您就需要该方法和下面的`void transform(Column[] columns, ColumnBuilder builder) throws Exception` 二选一来实现，在该方法中增加对原始数据处理的逻辑。 + +该方法每次处理原始数据的一行。原始数据由`Row`读入，由返回值输出。您必须在一次`transform`方法调用中，根据每个输入的数据点输出一个对应的数据点，即输入和输出依然是一对一的。需要注意的是，输出数据点的类型必须与您在`beforeStart`方法中设置的一致，而输出数据点的时间戳必须是严格单调递增的。 + +下面是一个实现了`Object transform(Row row) throws Exception`方法的完整 UDF 示例。它是一个加法器，接收两列时间序列输入，输出这两个数据点的代数和。 + +```java +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.access.Row; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type dataType; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + dataType = parameters.getDataType(0); + configurations + .setAccessStrategy(new MappableRowByRowAccessStrategy()) + .setOutputDataType(dataType); + } + + @Override + public Object transform(Row row) throws Exception { + return row.getLong(0) + row.getLong(1); + } +} +``` + +4. **void transform(Column[] columns, ColumnBuilder builder) throws Exception** + +当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `MappableRowByRowAccessStrategy`，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 + +该方法每次处理原始数据的多行，经过性能测试，我们发现一次性处理多行的 UDTF 比一次处理一行的 UDTF 性能更好。原始数据由`Column[]`读入，由`ColumnBuilder`输出。您必须在一次`transform`方法调用中，根据每个输入的数据点输出一个对应的数据点，即输入和输出依然是一对一的。需要注意的是，输出数据点的类型必须与您在`beforeStart`方法中设置的一致，而输出数据点的时间戳必须是严格单调递增的。 + +下面是一个实现了`void transform(Column[] columns, ColumnBuilder builder) throws Exceptionn`方法的完整 UDF 示例。它是一个加法器，接收两列时间序列输入，输出这两个数据点的代数和。 + +``` java +import org.apache.iotdb.tsfile.read.common.block.column.Column; +import org.apache.iotdb.tsfile.read.common.block.column.ColumnBuilder; +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type type; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + type = parameters.getDataType(0); + configurations.setAccessStrategy(new MappableRowByRowAccessStrategy()).setOutputDataType(type); + } + + @Override + public void transform(Column[] columns, ColumnBuilder builder) throws Exception { + long[] inputs1 = columns[0].getLongs(); + long[] inputs2 = columns[1].getLongs(); + + int count = columns[0].getPositionCount(); + for (int i = 0; i < count; i++) { + builder.writeLong(inputs1[i] + inputs2[i]); + } + } +} +``` + +5. **void transform(Row row, PointCollector collector) throws Exception** 当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `RowByRowAccessStrategy`，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 @@ -248,7 +341,7 @@ public class Adder implements UDTF { } ``` -4. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** +6. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** 当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `SlidingTimeWindowAccessStrategy`或者`SlidingSizeWindowAccessStrategy`时，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 @@ -288,7 +381,7 @@ public class Counter implements UDTF { } ``` -5. **void terminate(PointCollector collector) throws Exception** +7. **void terminate(PointCollector collector) throws Exception** 在一些场景下，UDF 需要遍历完所有的原始数据后才能得到最后的输出结果。`terminate`接口为这类 UDF 提供了支持。 @@ -341,7 +434,7 @@ public class Max implements UDTF { } ``` -6. **void beforeDestroy()** +8. **void beforeDestroy()** UDTF 的结束方法，您可以在此方法中进行一些资源释放等的操作。 diff --git a/src/zh/UserGuide/Master/SQL-Manual/SQL-Manual.md b/src/zh/UserGuide/Master/SQL-Manual/SQL-Manual.md index 0f5bffeff..9779f8972 100644 --- a/src/zh/UserGuide/Master/SQL-Manual/SQL-Manual.md +++ b/src/zh/UserGuide/Master/SQL-Manual/SQL-Manual.md @@ -392,27 +392,22 @@ IoTDB > select * from root.sg1.d1 ``` ### 加载 TsFile 文件数据 -load '' [sglevel=int][verify=true/false][onSuccess=delete/none] +load '' [sglevel=int][onSuccess=delete/none] #### 通过指定文件路径(绝对路径)加载单 tsfile 文件 - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + #### 通过指定文件夹路径(绝对路径)批量加载文件 - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` ## 删除数据 diff --git a/src/zh/UserGuide/Master/Tools-System/Maintenance-Tool_timecho.md b/src/zh/UserGuide/Master/Tools-System/Maintenance-Tool_timecho.md index 831f5b2e5..0ef56dcec 100644 --- a/src/zh/UserGuide/Master/Tools-System/Maintenance-Tool_timecho.md +++ b/src/zh/UserGuide/Master/Tools-System/Maintenance-Tool_timecho.md @@ -80,6 +80,7 @@ IoTDB集群管理工具主要由config、logs、doc、sbin目录组成。 更多详细命令请参考下面命令列表。 + | 参数 | 说明 | 是否必填 | |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------| | iotdb\_zip\_dir | IoTDB 部署分发目录，如果值为空则从`iotdb_download_url`指定地址下载 | 非必填 | @@ -108,6 +109,7 @@ extra_opts: | IOTDB_JMX_OPTS="$IOTDB_JMX_OPTS -XX:+UseG1GC" IOTDB_JMX_OPTS="$IOTDB_JMX_OPTS -XX:MaxGCPauseMillis=200" + * confignode_servers 是部署IoTDB Confignodes配置，里面可以配置多个Confignode 默认将第一个启动的ConfigNode节点node1当作Seed-ConfigNode @@ -124,6 +126,17 @@ IOTDB_JMX_OPTS="$IOTDB_JMX_OPTS -XX:MaxGCPauseMillis=200" * datanode_servers 是部署IoTDB Datanodes配置，里面可以配置多个Datanode +| 参数 | 说明 | 是否必填 | +| -------------------------- | ------------------------------------------------------------ | -------- | +| name | Datanode 名称 | 必填 | +| deploy_dir | IoTDB data node 部署目录，注:该目录不能与下面的IoTDB config node部署目录相同 | 必填 | +| dn_rpc_address | datanode rpc 地址对应`iotdb/config/iotdb-system.properties`中的`dn_rpc_address` | 必填 | +| dn_internal_address | 内部通信地址，对应`iotdb/config/iotdb-system.properties`中的`dn_internal_address` | 必填 | +| dn_seed_config_node | 集群配置地址指向存活的ConfigNode,默认指向confignode_x，在`global`与`datanode_servers`同时配置值时优先使用`datanode_servers`中的值，对应`iotdb/config/iotdb-datanode.properties`中的`dn_seed_config_node`，推荐使用 SeedConfigNode | 必填 | +| dn_rpc_port | datanode rpc端口地址，对应`iotdb/config/iotdb-system.properties`中的`dn_rpc_port` | 必填 | +| dn_internal_port | 内部通信端口，对应`iotdb/config/iotdb-system.properties`中的`dn_internal_port` | 必填 | +| iotdb-system.properties | 对应`iotdb/config/iotdb-system.properties`在`global`与`datanode_servers`同时配置值优先使用`datanode_servers`中的值 | 非必填 | + | 参数 | 说明 |是否必填| |---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|--- | diff --git a/src/zh/UserGuide/Master/User-Manual/Operator-and-Expression.md b/src/zh/UserGuide/Master/User-Manual/Operator-and-Expression.md index 3f81e3ead..2818fd7cd 100644 --- a/src/zh/UserGuide/Master/User-Manual/Operator-and-Expression.md +++ b/src/zh/UserGuide/Master/User-Manual/Operator-and-Expression.md @@ -94,6 +94,12 @@ OR, |, || | SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | | COUNT | 计算数据点数。 | 所有类型 | INT | | AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV | STDDEV_SAMP 的别名，求样本标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV_POP | 求总体标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV_SAMP | 求样本标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VARIANCE | VAR_SAMP 的别名，求样本方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VAR_POP | 求总体方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VAR_SAMP | 求样本方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | | EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 与输入类型一致 | | MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE STRING TIMESTAMP DATE | 与输入类型一致 | | MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE STRING TIMESTAMP DATE | 与输入类型一致 | diff --git a/src/zh/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md b/src/zh/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md index 65cbb2f76..057434ccc 100644 --- a/src/zh/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md +++ b/src/zh/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.md @@ -29,20 +29,19 @@ #### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [autoregister=true/false][,sglevel=int][,verify=true/false]` +加载 tsfile 文件的指令为：`load '' [autoregister=true/false][,sglevel=int]` + 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第二个参数表示待加载的 tsfile 文件的路径，其中文件名称需要符合 tsfile 的命名规范，即`{systemTime}-{versionNum}-{in_space_compaction_num}-{cross_space_compaction_num}.tsfile`。load 命令有三个可选项，分别是 autoregister，值域为 true/false，sglevel，值域为整数，verify，值域为 true/false。不同选项之间用逗号连接，选项之间无顺序要求。 +第二个参数表示待加载的 tsfile 文件的路径，其中文件名称需要符合 tsfile 的命名规范，即`{systemTime}-{versionNum}-{in_space_compaction_num}-{cross_space_compaction_num}.tsfile`。load 命令有 2 个可选项，分别是 autoregister，值域为 true/false，sglevel，值域为整数。选项之间无顺序要求。 AUTOREGISTER 选项表示当待加载的 tsfile 文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建 schema ，参数为 true 表示自动创建 schema，相反 false 表示不创建，缺省时默认创建 schema。 SGLEVEL 选项，当 tsfile 对应的存储组不存在时，用户可以通过 sglevel 参数的值来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是存储组，即若存在设备 root.sg.d1.s1，此时 root.sg 被指定为存储组。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 示例： @@ -51,11 +50,10 @@ VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元 * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=true` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=true,sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false,sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false,verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false,sglevel=1,verify=true` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' autoregister=false,sglevel=1` + 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -68,7 +66,6 @@ VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元 * `load '/Users/Desktop/data' autoregister=false` * `load '/Users/Desktop/data' autoregister=true` * `load '/Users/Desktop/data' autoregister=true,sglevel=1` -* `load '/Users/Desktop/data' autoregister=false,sglevel=1,verify=true` #### 远程加载 diff --git a/src/zh/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md b/src/zh/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md index fba206399..2b61901ac 100644 --- a/src/zh/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md +++ b/src/zh/UserGuide/V1.0.x/Maintenance-Tools/Load-Tsfile.md @@ -29,18 +29,14 @@ ### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [sglevel=int][verify=true/false][onSuccess=delete/none]` +加载 tsfile 文件的指令为：`load '' [sglevel=int][onSuccess=delete/none]` 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第一个参数表示待加载的 tsfile 文件的路径。load 命令有三个可选项，分别是 sglevel，值域为整数，verify，值域为 true/false，onSuccess，值域为delete/none。不同选项之间用空格隔开，选项之间无顺序要求。 - SGLEVEL 选项，当 tsfile 对应的 database 不存在时，用户可以通过 sglevel 参数的值来制定 database 的级别，默认为`iotdb-datanode.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database，即若存在设备 root.sg.d1.s1，此时 root.sg 被指定为 database。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delete，即tsfile成功加载后将被删除，如果是none表明tsfile成功加载之后依然被保留在源文件夹。 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 @@ -48,14 +44,9 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -65,26 +56,24 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **注意**，如果`$IOTDB_HOME$/conf/iotdb-datanode.properties`中`enable_auto_create_schema=true`时会在加载tsfile的时候自动创建tsfile中的元数据，否则不会自动创建。 ## 使用脚本加载 -若您在Windows环境中，请运行`$IOTDB_HOME/tools/load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` +若您在Windows环境中，请运行`$IOTDB_HOME\tools\load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +.\load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f 待加载的文件或文件夹路径，必要字段 -h IoTDB的Host地址，可选，默认127.0.0.1 -p IoTDB的端口，可选，默认6667 -u IoTDb登录用户名，可选，默认root -pw IoTDB登录密码，可选，默认root --sgLevel 加载TsFile自动创建Database的路径层级，可选，默认值为iotdb-common.properties指定值 ---verify 是否对加载TsFile进行元数据校验，可选，默认为True --onSuccess 对成功加载的TsFile的处理方法，可选，默认为delete，成功加载之后删除源TsFile，设为none时会 保留源TsFile ``` diff --git a/src/zh/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md b/src/zh/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md index 74b4f2c02..82fbee966 100644 --- a/src/zh/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md +++ b/src/zh/UserGuide/V1.0.x/Maintenance-Tools/TsFile-Load-Export-Tool.md @@ -29,18 +29,14 @@ ### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [sglevel=int][verify=true/false][onSuccess=delete/none]` +加载 tsfile 文件的指令为：`load '' [sglevel=int][onSuccess=delete/none]` 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第一个参数表示待加载的 tsfile 文件的路径。load 命令有三个可选项，分别是 sglevel，值域为整数，verify，值域为 true/false，onSuccess，值域为delete/none。不同选项之间用空格隔开，选项之间无顺序要求。 - SGLEVEL 选项，当 tsfile 对应的 database 不存在时，用户可以通过 sglevel 参数的值来制定 database 的级别，默认为`iotdb-datanode.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database，即若存在设备 root.sg.d1.s1，此时 root.sg 被指定为 database。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delete，即tsfile成功加载后将被删除，如果是none表明tsfile成功加载之后依然被保留在源文件夹。 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 @@ -48,14 +44,9 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -65,10 +56,8 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **注意**，如果`$IOTDB_HOME$/conf/iotdb-datanode.properties`中`enable_auto_create_schema=true`时会在加载tsfile的时候自动创建tsfile中的元数据，否则不会自动创建。 @@ -77,14 +66,13 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 若您在Windows环境中，请运行`$IOTDB_HOME/tools/load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f 待加载的文件或文件夹路径，必要字段 -h IoTDB的Host地址，可选，默认127.0.0.1 -p IoTDB的端口，可选，默认6667 -u IoTDb登录用户名，可选，默认root -pw IoTDB登录密码，可选，默认root --sgLevel 加载TsFile自动创建Database的路径层级，可选，默认值为iotdb-common.properties指定值 ---verify 是否对加载TsFile进行元数据校验，可选，默认为True --onSuccess 对成功加载的TsFile的处理方法，可选，默认为delete，成功加载之后删除源TsFile，设为none时会 保留源TsFile ``` diff --git a/src/zh/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md b/src/zh/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md index f10b9dc57..a3516b71e 100644 --- a/src/zh/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md +++ b/src/zh/UserGuide/V1.1.x/Maintenance-Tools/Load-Tsfile.md @@ -29,18 +29,14 @@ #### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [sglevel=int][verify=true/false][onSuccess=delete/none]` +加载 tsfile 文件的指令为：`load '' [sglevel=int][onSuccess=delete/none]` 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第一个参数表示待加载的 tsfile 文件的路径。load 命令有三个可选项，分别是 sglevel，值域为整数，verify，值域为 true/false，onSuccess，值域为delete/none。不同选项之间用空格隔开，选项之间无顺序要求。 - SGLEVEL 选项，当 tsfile 对应的 database 不存在时，用户可以通过 sglevel 参数的值来制定 database 的级别，默认为`iotdb-datanode.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database，即若存在设备 root.sg.d1.s1，此时 root.sg 被指定为 database。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delete，即tsfile成功加载后将被删除，如果是none表明tsfile成功加载之后依然被保留在源文件夹。 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 @@ -48,14 +44,9 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -65,26 +56,24 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **注意**，如果`$IOTDB_HOME$/conf/iotdb-datanode.properties`中`enable_auto_create_schema=true`时会在加载tsfile的时候自动创建tsfile中的元数据，否则不会自动创建。 ### 使用脚本加载 -若您在Windows环境中，请运行`$IOTDB_HOME/tools/load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` + +若您在Windows环境中，请运行`$IOTDB_HOME\tools\load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +.\load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f 待加载的文件或文件夹路径，必要字段 -h IoTDB的Host地址，可选，默认127.0.0.1 -p IoTDB的端口，可选，默认6667 -u IoTDb登录用户名，可选，默认root -pw IoTDB登录密码，可选，默认root --sgLevel 加载TsFile自动创建Database的路径层级，可选，默认值为iotdb-common.properties指定值 ---verify 是否对加载TsFile进行元数据校验，可选，默认为True --onSuccess 对成功加载的TsFile的处理方法，可选，默认为delete，成功加载之后删除源TsFile，设为none时会 保留源TsFile ``` diff --git a/src/zh/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md b/src/zh/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md index 80eb6b047..37c731f81 100644 --- a/src/zh/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md +++ b/src/zh/UserGuide/V1.1.x/Maintenance-Tools/TsFile-Load-Export-Tool.md @@ -29,18 +29,14 @@ #### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [sglevel=int][verify=true/false][onSuccess=delete/none]` +加载 tsfile 文件的指令为：`load '' [sglevel=int][onSuccess=delete/none]` 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第一个参数表示待加载的 tsfile 文件的路径。load 命令有三个可选项，分别是 sglevel，值域为整数，verify，值域为 true/false，onSuccess，值域为delete/none。不同选项之间用空格隔开，选项之间无顺序要求。 - SGLEVEL 选项，当 tsfile 对应的 database 不存在时，用户可以通过 sglevel 参数的值来制定 database 的级别，默认为`iotdb-datanode.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database，即若存在设备 root.sg.d1.s1，此时 root.sg 被指定为 database。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delete，即tsfile成功加载后将被删除，如果是none表明tsfile成功加载之后依然被保留在源文件夹。 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 @@ -48,14 +44,10 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -65,10 +57,8 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` **注意**，如果`$IOTDB_HOME$/conf/iotdb-datanode.properties`中`enable_auto_create_schema=true`时会在加载tsfile的时候自动创建tsfile中的元数据，否则不会自动创建。 @@ -77,14 +67,13 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 若您在Windows环境中，请运行`$IOTDB_HOME/tools/load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f 待加载的文件或文件夹路径，必要字段 -h IoTDB的Host地址，可选，默认127.0.0.1 -p IoTDB的端口，可选，默认6667 -u IoTDb登录用户名，可选，默认root -pw IoTDB登录密码，可选，默认root --sgLevel 加载TsFile自动创建Database的路径层级，可选，默认值为iotdb-common.properties指定值 ---verify 是否对加载TsFile进行元数据校验，可选，默认为True --onSuccess 对成功加载的TsFile的处理方法，可选，默认为delete，成功加载之后删除源TsFile，设为none时会 保留源TsFile ``` diff --git a/src/zh/UserGuide/V1.2.x/QuickStart/QuickStart.md b/src/zh/UserGuide/V1.2.x/QuickStart/QuickStart.md index cfff89192..22d619ba6 100644 --- a/src/zh/UserGuide/V1.2.x/QuickStart/QuickStart.md +++ b/src/zh/UserGuide/V1.2.x/QuickStart/QuickStart.md @@ -67,7 +67,8 @@ Windows 系统启动命令如下： > sbin\start-standalone.bat ``` -注意：目前，要使用单机模式，你需要保证所有的地址设置为 127.0.0.1，如果需要从非 IoTDB 所在的机器访问此IoTDB，请将配置项 `dn_rpc_address` 修改为 IoTDB 所在的机器 IP。副本数设置为1。并且，推荐使用 SimpleConsensus，因为这会带来额外的效率。这些现在都是默认配置。 +注意：目前，要使用单机模式，你需要保证所有的地址设置为 127.0.0.1，如果需要从非 IoTDB 所在的机器访问此IoTDB，请将配置项 `dn_rpc_address` 修改为 IoTDB 所在的机器 IP。副本数设置为1。这些现在都是默认配置。 + ### 使用 Cli 工具 diff --git a/src/zh/UserGuide/V1.2.x/Reference/Common-Config-Manual.md b/src/zh/UserGuide/V1.2.x/Reference/Common-Config-Manual.md index d960662a4..d3abf55a9 100644 --- a/src/zh/UserGuide/V1.2.x/Reference/Common-Config-Manual.md +++ b/src/zh/UserGuide/V1.2.x/Reference/Common-Config-Manual.md @@ -61,7 +61,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | schema\_region\_consensus\_protocol\_class | |:------:|:----------------------------------------------------------------| -| 描述 | 元数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时只能使用 RatisConsensus | +| 描述 | 元数据副本的共识协议，多副本时只能使用 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.ratis.RatisConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | @@ -79,7 +79,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | data\_region\_consensus\_protocol\_class | |:------:|:------------------------------------------------------------------------------| -| 描述 | 数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | +| 描述 | 数据副本的共识协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.iot.IoTConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | diff --git a/src/zh/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md b/src/zh/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md index 59bcd0205..d9ba5ca1c 100644 --- a/src/zh/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md +++ b/src/zh/UserGuide/V1.2.x/SQL-Manual/SQL-Manual.md @@ -385,27 +385,22 @@ IoTDB > select * from root.sg1.d1 ``` ### 5、加载 TsFile 文件数据 ``` -load '' [sglevel=int][verify=true/false][onSuccess=delete/none] +load '' [sglevel=int][onSuccess=delete/none] ``` #### 通过指定文件路径(绝对路径)加载单 tsfile 文件 - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + #### 通过指定文件夹路径(绝对路径)批量加载文件 - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` ## 删除数据 diff --git a/src/zh/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md b/src/zh/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md index 0b4d467fa..cf973e798 100644 --- a/src/zh/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md +++ b/src/zh/UserGuide/V1.2.x/Tools-System/Import-Export-Tool.md @@ -34,18 +34,14 @@ TsFile 是在 IoTDB 中使用的时间序列的文件格式，您可以通过CLI #### 加载 tsfile 文件 -加载 tsfile 文件的指令为：`load '' [sglevel=int][verify=true/false][onSuccess=delete/none]` +加载 tsfile 文件的指令为：`load '' [sglevel=int][onSuccess=delete/none]` 该指令有两种用法： 1. 通过指定文件路径(绝对路径)加载单 tsfile 文件。 -第一个参数表示待加载的 tsfile 文件的路径。load 命令有三个可选项，分别是 sglevel，值域为整数，verify，值域为 true/false，onSuccess，值域为delete/none。不同选项之间用空格隔开，选项之间无顺序要求。 - SGLEVEL 选项，当 tsfile 对应的 database 不存在时，用户可以通过 sglevel 参数的值来制定 database 的级别，默认为`iotdb-datanode.properties`中设置的级别。例如当设置 level 参数为1时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database，即若存在设备 root.sg.d1.s1，此时 `root.sg` 被指定为 database。 -VERIFY 选项表示是否对载入的 tsfile 中的所有时间序列进行元数据检查，默认为 true。开启时，若载入的 tsfile 中的时间序列在当前 iotdb 中也存在，则会比较该时间序列的所有 Measurement 的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。 - ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delete，即tsfile成功加载后将被删除，如果是none表明tsfile成功加载之后依然被保留在源文件夹。 若待加载的 tsfile 文件对应的`.resource`文件存在，会被一并加载至 Apache IoTDB 数据文件的目录和引擎中，否则将通过 tsfile 文件重新生成对应的`.resource`文件，即加载的 tsfile 文件所对应的`.resource`文件不是必要的。 @@ -53,14 +49,9 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` * `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` 2. 通过指定文件夹路径(绝对路径)批量加载文件。 @@ -70,10 +61,9 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 示例： * `load '/Users/Desktop/data'` -* `load '/Users/Desktop/data' verify=false` -* `load '/Users/Desktop/data' verify=true` -* `load '/Users/Desktop/data' verify=true sglevel=1` -* `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +* `load '/Users/Desktop/data' sglevel=1` +* `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` + **注意**，如果`$IOTDB_HOME$/conf/iotdb-datanode.properties`中`enable_auto_create_schema=true`时会在加载tsfile的时候自动创建tsfile中的元数据，否则不会自动创建。 @@ -82,14 +72,13 @@ ONSUCCESS选项表示对于成功载入的tsfile的处置方式，默认为delet 若您在Windows环境中，请运行`$IOTDB_HOME/tools/load-tsfile.bat`，若为Linux或Unix，请运行`load-tsfile.sh` ```bash -./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--verify true/false] [--onSuccess none/delete] +./load-tsfile.bat -f filePath [-h host] [-p port] [-u username] [-pw password] [--sgLevel int] [--onSuccess none/delete] -f 待加载的文件或文件夹路径，必要字段 -h IoTDB的Host地址，可选，默认127.0.0.1 -p IoTDB的端口，可选，默认6667 -u IoTDb登录用户名，可选，默认root -pw IoTDB登录密码，可选，默认root --sgLevel 加载TsFile自动创建Database的路径层级，可选，默认值为iotdb-common.properties指定值 ---verify 是否对加载TsFile进行元数据校验，可选，默认为True --onSuccess 对成功加载的TsFile的处理方法，可选，默认为delete，成功加载之后删除源TsFile，设为none时会 保留源TsFile ``` diff --git a/src/zh/UserGuide/latest/Reference/Common-Config-Manual.md b/src/zh/UserGuide/latest/Reference/Common-Config-Manual.md index 575be0239..01d48e57a 100644 --- a/src/zh/UserGuide/latest/Reference/Common-Config-Manual.md +++ b/src/zh/UserGuide/latest/Reference/Common-Config-Manual.md @@ -66,7 +66,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | schema\_region\_consensus\_protocol\_class | |:------:|:----------------------------------------------------------------| -| 描述 | 元数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时只能使用 RatisConsensus | +| 描述 | 元数据副本的共识协议，多副本时只能使用 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.ratis.RatisConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | @@ -84,7 +84,7 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。 | 名字 | data\_region\_consensus\_protocol\_class | |:------:|:------------------------------------------------------------------------------| -| 描述 | 数据副本的共识协议，1 副本时可以使用 SimpleConsensus 协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | +| 描述 | 数据副本的共识协议，多副本时可以使用 IoTConsensus 或 RatisConsensus | | 类型 | String | | 默认值 | org.apache.iotdb.consensus.iot.IoTConsensus | | 改后生效方式 | 仅允许在第一次启动服务前修改 | diff --git a/src/zh/UserGuide/latest/Reference/Function-and-Expression.md b/src/zh/UserGuide/latest/Reference/Function-and-Expression.md index 816ebb0c5..92b8df136 100644 --- a/src/zh/UserGuide/latest/Reference/Function-and-Expression.md +++ b/src/zh/UserGuide/latest/Reference/Function-and-Expression.md @@ -29,23 +29,28 @@ IoTDB 支持的聚合函数如下： -| 函数名 | 功能描述 | 允许的输入类型 | 必要的属性参数 | 输出类型 | -|---------------|----------------------------------------------------------------------------------------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| COUNT | 计算数据点数。 | 所有类型 | 无 | INT64 | -| AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | -| FIRST_VALUE | 求时间戳最小的值。 | 所有类型 | 无 | 与输入类型一致 | -| LAST_VALUE | 求时间戳最大的值。 | 所有类型 | 无 | 与输入类型一致 | -| MAX_TIME | 求最大时间戳。 | 所有类型 | 无 | Timestamp | -| MIN_TIME | 求最小时间戳。 | 所有类型 | 无 | Timestamp | -| COUNT_IF | 求数据点连续满足某一给定条件，且满足条件的数据点个数（用keep表示）满足指定阈值的次数。 | BOOLEAN | `[keep >=/>/=/!=/= threshold`,`threshold`类型为`INT64`
`ignoreNull`：可选，默认为`true`；为`true`表示忽略null值，即如果中间出现null值，直接忽略，不会打断连续性；为`false`表示不忽略null值，即如果中间出现null值，会打断连续性 | INT64 | -| TIME_DURATION | 求某一列最大一个不为NULL的值所在时间戳与最小一个不为NULL的值所在时间戳的时间戳差 | 所有类型 | 无 | INT64 | -| MODE | 求众数。注意：
1.输入序列的不同值个数过多时会有内存异常风险;
2.如果所有元素出现的频次相同，即没有众数，则返回对应时间戳最小的值;
3.如果有多个众数，则返回对应时间戳最小的众数。 | 所有类型 | 无 | 与输入类型一致 | -| STDDEV | 求数据的总体标准差。注意：
数据中的空值、缺失值和`NaN`将会被忽略。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | -| COUNT_TIME | 查询结果集的时间戳的数量。与 align by device 搭配使用时，得到的结果是每个设备的结果集的时间戳的数量。 | 所有类型，输入参数只能为* | 无 | INT64 | +| 函数名 | 功能描述 | 允许的输入类型 | 必要的属性参数 | 输出类型 | +| ------------- | ------------------------------------------------------------ | ------------------------- | ------------------------------------------------------------ | -------------- | +| SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| COUNT | 计算数据点数。 | 所有类型 | 无 | INT64 | +| AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV | STDDEV_SAMP 的别名，求样本标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV_POP | 求总体标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| STDDEV_SAMP | 求样本标准差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VARIANCE | VAR_SAMP 的别名，求样本方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VAR_POP | 求总体方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| VAR_SAMP | 求样本方差。 | INT32 INT64 FLOAT DOUBLE | 无 | DOUBLE | +| EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE | 无 | 与输入类型一致 | +| FIRST_VALUE | 求时间戳最小的值。 | 所有类型 | 无 | 与输入类型一致 | +| LAST_VALUE | 求时间戳最大的值。 | 所有类型 | 无 | 与输入类型一致 | +| MAX_TIME | 求最大时间戳。 | 所有类型 | 无 | Timestamp | +| MIN_TIME | 求最小时间戳。 | 所有类型 | 无 | Timestamp | +| COUNT_IF | 求数据点连续满足某一给定条件，且满足条件的数据点个数（用keep表示）满足指定阈值的次数。 | BOOLEAN | `[keep >=/>/=/!=/= threshold`,`threshold`类型为`INT64`
`ignoreNull`：可选，默认为`true`；为`true`表示忽略null值，即如果中间出现null值，直接忽略，不会打断连续性；为`false`表示不忽略null值，即如果中间出现null值，会打断连续性 | INT64 | +| TIME_DURATION | 求某一列最大一个不为NULL的值所在时间戳与最小一个不为NULL的值所在时间戳的时间戳差 | 所有类型 | 无 | INT64 | +| MODE | 求众数。注意：
1.输入序列的不同值个数过多时会有内存异常风险;
2.如果所有元素出现的频次相同，即没有众数，则返回对应时间戳最小的值;
3.如果有多个众数，则返回对应时间戳最小的众数。 | 所有类型 | 无 | 与输入类型一致 | +| COUNT_TIME | 查询结果集的时间戳的数量。与 align by device 搭配使用时，得到的结果是每个设备的结果集的时间戳的数量。 | 所有类型，输入参数只能为* | 无 | INT64 | | MAX_BY | MAX_BY(x, y) 求二元输入 x 和 y 在 y 最大时对应的 x 的值。MAX_BY(time, x) 返回 x 取最大值时对应的时间戳。 | 第一个输入 x 可以是任意类型，第二个输入 y 只能是 INT32 INT64 FLOAT DOUBLE | 无 | 与第一个输入 x 的数据类型一致 | | MIN_BY | MIN_BY(x, y) 求二元输入 x 和 y 在 y 最小时对应的 x 的值。MIN_BY(time, x) 返回 x 取最小值时对应的时间戳。 | 第一个输入 x 可以是任意类型，第二个输入 y 只能是 INT32 INT64 FLOAT DOUBLE | 无 | 与第一个输入 x 的数据类型一致 | diff --git a/src/zh/UserGuide/latest/Reference/UDF-development.md b/src/zh/UserGuide/latest/Reference/UDF-development.md index 5e0d4ec76..d2ecb5dcd 100644 --- a/src/zh/UserGuide/latest/Reference/UDF-development.md +++ b/src/zh/UserGuide/latest/Reference/UDF-development.md @@ -1,8 +1,8 @@ # UDF 开发 -## UDF 开发 +## 1. UDF 开发 -### UDF 依赖 +### 1.1 UDF 依赖 如果您使用 [Maven](http://search.maven.org/) ，可以从 [Maven 库](http://search.maven.org/) 中搜索下面示例中的依赖。请注意选择和目标 IoTDB 服务器版本相同的依赖版本。 @@ -15,26 +15,28 @@ ``` -### UDTF（User Defined Timeseries Generating Function） +### 1.2 UDTF（User Defined Timeseries Generating Function） 编写一个 UDTF 需要继承`org.apache.iotdb.udf.api.UDTF`类，并至少实现`beforeStart`方法和一种`transform`方法。 #### 接口说明: -| 接口定义 | 描述 | 是否必须 | -| :----------------------------------------------------------- | :----------------------------------------------------------- | ------------------ | -| void validate(UDFParameterValidator validator) throws Exception | 在初始化方法`beforeStart`调用前执行，用于检测`UDFParameters`中用户输入的参数是否合法。 | 否 | -| void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | 初始化方法，在 UDTF 处理输入数据前，调用用户自定义的初始化行为。用户每执行一次 UDTF 查询，框架就会构造一个新的 UDF 类实例，该方法在每个 UDF 类实例被初始化时调用一次。在每一个 UDF 类实例的生命周期内，该方法只会被调用一次。 | 是 | -| void transform(Row row, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`RowByRowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`Row`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 与下面的方法二选一 | -| void transform(RowWindow rowWindow, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`SlidingSizeWindowAccessStrategy`或者`SlidingTimeWindowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`RowWindow`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 与上面的方法二选一 | -| void terminate(PointCollector collector) throws Exception | 这个方法由框架调用。该方法会在所有的`transform`调用执行完成后，在`beforeDestory`方法执行前被调用。在一个 UDF 查询过程中，该方法会且只会调用一次。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 否 | -| void beforeDestroy() | UDTF 的结束方法。此方法由框架调用，并且只会被调用一次，即在处理完最后一条记录之后被调用。 | 否 | +| 接口定义 | 描述 | 是否必须 | +| :----------------------------------------------------------- | :----------------------------------------------------------- | ------------------------- | +| void validate(UDFParameterValidator validator) throws Exception | 在初始化方法`beforeStart`调用前执行，用于检测`UDFParameters`中用户输入的参数是否合法。 | 否 | +| void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception | 初始化方法，在 UDTF 处理输入数据前，调用用户自定义的初始化行为。用户每执行一次 UDTF 查询，框架就会构造一个新的 UDF 类实例，该方法在每个 UDF 类实例被初始化时调用一次。在每一个 UDF 类实例的生命周期内，该方法只会被调用一次。 | 是 | +| Object transform(Row row) throws Exception` | 这个方法由框架调用。当您在`beforeStart`中选择以`MappableRowByRowAccessStrategy`的策略消费原始数据时，可以选用该方法进行数据处理。输入参数以`Row`的形式传入，输出结果通过返回值`Object`输出。 | 所有`transform`方法四选一 | +| void transform(Column[] columns, ColumnBuilder builder) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`MappableRowByRowAccessStrategy`的策略消费原始数据时，可以选用该方法进行数据处理。输入参数以`Column[]`的形式传入，输出结果通过`ColumnBuilder`输出。您需要在该方法内自行调用`builder`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void transform(Row row, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`RowByRowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`Row`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void transform(RowWindow rowWindow, PointCollector collector) throws Exception | 这个方法由框架调用。当您在`beforeStart`中选择以`SlidingSizeWindowAccessStrategy`或者`SlidingTimeWindowAccessStrategy`的策略消费原始数据时，这个数据处理方法就会被调用。输入参数以`RowWindow`的形式传入，输出结果通过`PointCollector`输出。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 所有`transform`方法四选一 | +| void terminate(PointCollector collector) throws Exception | 这个方法由框架调用。该方法会在所有的`transform`调用执行完成后，在`beforeDestory`方法执行前被调用。在一个 UDF 查询过程中，该方法会且只会调用一次。您需要在该方法内自行调用`collector`提供的数据收集方法，以决定最终的输出数据。 | 否 | +| void beforeDestroy() | UDTF 的结束方法。此方法由框架调用，并且只会被调用一次，即在处理完最后一条记录之后被调用。 | 否 | 在一个完整的 UDTF 实例生命周期中，各个方法的调用顺序如下： 1. void validate(UDFParameterValidator validator) throws Exception 2. void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) throws Exception -3. void transform(Row row, PointCollector collector) throws Exception 或者 void transform(RowWindow rowWindow, PointCollector collector) throws Exception +3. Object transform(Row row) throws Exception 或着 void transform(Column[] columns, ColumnBuilder builder) throws Exception 或者 void transform(Row row, PointCollector collector) throws Exception 或者 void transform(RowWindow rowWindow, PointCollector collector) throws Exception 4. void terminate(PointCollector collector) throws Exception 5. void beforeDestroy() @@ -110,7 +112,7 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th | 接口定义 | 描述 | 调用的`transform`方法 | | ------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| MappableRowByRow | 自定义标量函数
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 1 行数据，可用于标量函数出现的任何子句和表达式中，如select子句、where子句等。 | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | +| MappableRowByRowStrategy | 自定义标量函数
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 1 行数据，可用于标量函数出现的任何子句和表达式中，如select子句、where子句等。 | void transform(Column[] columns, ColumnBuilder builder) throws ExceptionObject transform(Row row) throws Exception | | RowByRowAccessStrategy | 自定义时间序列生成函数，逐行地处理原始数据。
框架会为每一行原始数据输入调用一次`transform`方法，输入 k 列时间序列 1 行数据，输出 1 列时间序列 n 行数据。
当输入一个序列时，该行就作为输入序列的一个数据点。
当输入多个序列时，输入序列按时间对齐后，每一行作为的输入序列的一个数据点。
（一行数据中，可能存在某一列为`null`值，但不会全部都是`null`） | void transform(Row row, PointCollector collector) throws Exception | | SlidingTimeWindowAccessStrategy | 自定义时间序列生成函数，以滑动时间窗口的方式处理原始数据。
框架会为每一个原始数据输入窗口调用一次`transform`方法，输入 k 列时间序列 m 行数据，输出 1 列时间序列 n 行数据。
一个窗口可能存在多行数据，输入序列按时间对齐后，每个窗口作为的输入序列的一个数据点。
（每个窗口可能存在 i 行，每行数据可能存在某一列为`null`值，但不会全部都是`null`） | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | | SlidingSizeWindowAccessStrategy | 自定义时间序列生成函数，以固定行数的方式处理原始数据，即每个数据处理窗口都会包含固定行数的数据（最后一个窗口除外）。
框架会为每一个原始数据输入窗口调用一次`transform`方法，输入 k 列时间序列 m 行数据，输出 1 列时间序列 n 行数据。
一个窗口可能存在多行数据，输入序列按时间对齐后，每个窗口作为的输入序列的一个数据点。
（每个窗口可能存在 i 行，每行数据可能存在某一列为`null`值，但不会全部都是`null`） | void transform(RowWindow rowWindow, PointCollector collector) throws Exception | @@ -119,7 +121,7 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th #### 接口详情： -- `RowByRowAccessStrategy`的构造不需要任何参数。 +- `MappableRowByRowStrategy` 和 `RowByRowAccessStrategy`的构造不需要任何参数。 - `SlidingTimeWindowAccessStrategy` @@ -212,7 +214,98 @@ void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) th } ``` -3. **void transform(Row row, PointCollector collector) throws Exception** +3. **Object transform(Row row) throws Exception** + +当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `MappableRowByRowAccessStrategy`，您就需要该方法和下面的`void transform(Column[] columns, ColumnBuilder builder) throws Exception` 二选一来实现，在该方法中增加对原始数据处理的逻辑。 + +该方法每次处理原始数据的一行。原始数据由`Row`读入，由返回值输出。您必须在一次`transform`方法调用中，根据每个输入的数据点输出一个对应的数据点，即输入和输出依然是一对一的。需要注意的是，输出数据点的类型必须与您在`beforeStart`方法中设置的一致，而输出数据点的时间戳必须是严格单调递增的。 + +下面是一个实现了`Object transform(Row row) throws Exception`方法的完整 UDF 示例。它是一个加法器，接收两列时间序列输入，输出这两个数据点的代数和。 + +```java +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.access.Row; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type dataType; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + dataType = parameters.getDataType(0); + configurations + .setAccessStrategy(new MappableRowByRowAccessStrategy()) + .setOutputDataType(dataType); + } + + @Override + public Object transform(Row row) throws Exception { + return row.getLong(0) + row.getLong(1); + } +} +``` + +4. **void transform(Column[] columns, ColumnBuilder builder) throws Exception** + +当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `MappableRowByRowAccessStrategy`，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 + +该方法每次处理原始数据的多行，经过性能测试，我们发现一次性处理多行的 UDTF 比一次处理一行的 UDTF 性能更好。原始数据由`Column[]`读入，由`ColumnBuilder`输出。您必须在一次`transform`方法调用中，根据每个输入的数据点输出一个对应的数据点，即输入和输出依然是一对一的。需要注意的是，输出数据点的类型必须与您在`beforeStart`方法中设置的一致，而输出数据点的时间戳必须是严格单调递增的。 + +下面是一个实现了`void transform(Column[] columns, ColumnBuilder builder) throws Exceptionn`方法的完整 UDF 示例。它是一个加法器，接收两列时间序列输入，输出这两个数据点的代数和。 + +``` java +import org.apache.iotdb.tsfile.read.common.block.column.Column; +import org.apache.iotdb.tsfile.read.common.block.column.ColumnBuilder; +import org.apache.iotdb.udf.api.UDTF; +import org.apache.iotdb.udf.api.customizer.config.UDTFConfigurations; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameterValidator; +import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters; +import org.apache.iotdb.udf.api.customizer.strategy.MappableRowByRowAccessStrategy; +import org.apache.iotdb.udf.api.type.Type; + +public class Adder implements UDTF { + private Type type; + + @Override + public void validate(UDFParameterValidator validator) throws Exception { + validator + .validateInputSeriesNumber(2) + .validateInputSeriesDataType(0, Type.INT64) + .validateInputSeriesDataType(1, Type.INT64); + } + + @Override + public void beforeStart(UDFParameters parameters, UDTFConfigurations configurations) { + type = parameters.getDataType(0); + configurations.setAccessStrategy(new MappableRowByRowAccessStrategy()).setOutputDataType(type); + } + + @Override + public void transform(Column[] columns, ColumnBuilder builder) throws Exception { + long[] inputs1 = columns[0].getLongs(); + long[] inputs2 = columns[1].getLongs(); + + int count = columns[0].getPositionCount(); + for (int i = 0; i < count; i++) { + builder.writeLong(inputs1[i] + inputs2[i]); + } + } +} +``` + +5. **void transform(Row row, PointCollector collector) throws Exception** 当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `RowByRowAccessStrategy`，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 @@ -248,7 +341,7 @@ public class Adder implements UDTF { } ``` -4. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** +6. **void transform(RowWindow rowWindow, PointCollector collector) throws Exception** 当您在`beforeStart`方法中指定 UDF 读取原始数据的策略为 `SlidingTimeWindowAccessStrategy`或者`SlidingSizeWindowAccessStrategy`时，您就需要实现该方法，在该方法中增加对原始数据处理的逻辑。 @@ -288,7 +381,7 @@ public class Counter implements UDTF { } ``` -5. **void terminate(PointCollector collector) throws Exception** +7. **void terminate(PointCollector collector) throws Exception** 在一些场景下，UDF 需要遍历完所有的原始数据后才能得到最后的输出结果。`terminate`接口为这类 UDF 提供了支持。 @@ -341,13 +434,13 @@ public class Max implements UDTF { } ``` -6. **void beforeDestroy()** +8. **void beforeDestroy()** UDTF 的结束方法，您可以在此方法中进行一些资源释放等的操作。 此方法由框架调用。对于一个 UDF 类实例而言，生命周期中会且只会被调用一次，即在处理完最后一条记录之后被调用。 -### UDAF（User Defined Aggregation Function） +### 1.3 UDAF（User Defined Aggregation Function） 一个完整的 UDAF 定义涉及到 State 和 UDAF 两个类。 @@ -577,16 +670,16 @@ UDAF 的结束方法，您可以在此方法中进行一些资源释放等的操 此方法由框架调用。对于一个 UDF 类实例而言，生命周期中会且只会被调用一次，即在处理完最后一条记录之后被调用。 -### 完整 Maven 项目示例 +### 1.4 完整 Maven 项目示例 如果您使用 [Maven](http://search.maven.org/)，可以参考我们编写的示例项目**udf-example**。您可以在 [这里](https://github.com/apache/iotdb/tree/master/example/udf) 找到它。 -## 为iotdb贡献通用的内置UDF函数 +## 2. 为iotdb贡献通用的内置UDF函数 该部分主要讲述了外部用户如何将自己编写的 UDF 贡献给 IoTDB 社区。 -#### 前提条件 +## 2.1 前提条件 1. UDF 具有通用性。 @@ -596,30 +689,32 @@ UDAF 的结束方法，您可以在此方法中进行一些资源释放等的操 2. UDF 已经完成测试，且能够正常运行在用户的生产环境中。 -#### 贡献清单 +### 2.2 贡献清单 1. UDF 的源代码 2. UDF 的测试用例 3. UDF 的使用说明 -#### 源代码 +### 2.3 贡献内容 + +#### 2.3.1 源代码 1. 在`iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin`中创建 UDF 主类和相关的辅助类。 2. 在`iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/BuiltinTimeSeriesGeneratingFunction.java`中注册编写的 UDF。 -#### 测试用例 +#### 2.3.2 测试用例 至少需要为贡献的 UDF 编写集成测试。 可以在`integration-test/src/test/java/org/apache/iotdb/db/it/udf`中为贡献的 UDF 新增一个测试类进行测试。 -#### 使用说明 +#### 2.3.3 使用说明 使用说明需要包含：UDF 的名称、UDF 的作用、执行函数必须的属性参数、函数的适用的场景以及使用示例等。 使用说明需包含中英文两个版本。应分别在 `docs/zh/UserGuide/Operation Manual/DML Data Manipulation Language.md` 和 `docs/UserGuide/Operation Manual/DML Data Manipulation Language.md` 中新增使用说明。 -#### 提交 PR +#### 2.3.4 提交 PR 当准备好源代码、测试用例和使用说明后，就可以将 UDF 贡献到 IoTDB 社区了。在 [Github](https://github.com/apache/iotdb) 上面提交 Pull Request (PR) 即可。具体提交方式见：[贡献指南](https://iotdb.apache.org/zh/Community/Development-Guide.html)。 diff --git a/src/zh/UserGuide/latest/SQL-Manual/SQL-Manual.md b/src/zh/UserGuide/latest/SQL-Manual/SQL-Manual.md index 0f5bffeff..9779f8972 100644 --- a/src/zh/UserGuide/latest/SQL-Manual/SQL-Manual.md +++ b/src/zh/UserGuide/latest/SQL-Manual/SQL-Manual.md @@ -392,27 +392,22 @@ IoTDB > select * from root.sg1.d1 ``` ### 加载 TsFile 文件数据 -load '' [sglevel=int][verify=true/false][onSuccess=delete/none] +load '' [sglevel=int][onSuccess=delete/none] #### 通过指定文件路径(绝对路径)加载单 tsfile 文件 - `load '/Users/Desktop/data/1575028885956-101-0.tsfile'` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1` - `load '/Users/Desktop/data/1575028885956-101-0.tsfile' onSuccess=delete` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=true onSuccess=none` -- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data/1575028885956-101-0.tsfile' sglevel=1 onSuccess=delete` + #### 通过指定文件夹路径(绝对路径)批量加载文件 - `load '/Users/Desktop/data'` -- `load '/Users/Desktop/data' verify=false` -- `load '/Users/Desktop/data' verify=true` -- `load '/Users/Desktop/data' verify=true sglevel=1` -- `load '/Users/Desktop/data' verify=false sglevel=1 onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1` +- `load '/Users/Desktop/data' onSuccess=delete` +- `load '/Users/Desktop/data' sglevel=1 onSuccess=delete` ## 删除数据 diff --git a/src/zh/UserGuide/latest/User-Manual/Operator-and-Expression.md b/src/zh/UserGuide/latest/User-Manual/Operator-and-Expression.md index 0c704d4be..4c4be4392 100644 --- a/src/zh/UserGuide/latest/User-Manual/Operator-and-Expression.md +++ b/src/zh/UserGuide/latest/User-Manual/Operator-and-Expression.md @@ -94,6 +94,12 @@ OR, |, || | SUM | 求和。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | | COUNT | 计算数据点数。 | 所有类型 | INT | | AVG | 求平均值。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV | STDDEV_SAMP 的别名，求样本标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV_POP | 求总体标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| STDDEV_SAMP | 求样本标准差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VARIANCE | VAR_SAMP 的别名，求样本方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VAR_POP | 求总体方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | +| VAR_SAMP | 求样本方差。 | INT32 INT64 FLOAT DOUBLE | DOUBLE | | EXTREME | 求具有最大绝对值的值。如果正值和负值的最大绝对值相等，则返回正值。 | INT32 INT64 FLOAT DOUBLE | 与输入类型一致 | | MAX_VALUE | 求最大值。 | INT32 INT64 FLOAT DOUBLE | 与输入类型一致 | | MIN_VALUE | 求最小值。 | INT32 INT64 FLOAT DOUBLE | 与输入类型一致 |