Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/.vuepress/sidebar/V1.3.x/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ export const enSidebar = {
text: 'Syntax-Rule',
collapsible: true,
children: [
{ text: 'Reserved Words&Identifiers', link: 'Syntax-Rule' },
{ text: 'Identifiers', link: 'Syntax-Rule' },
{ text: 'Keywords', link: 'Keywords' },
],
},
Expand Down
4 changes: 2 additions & 2 deletions src/.vuepress/sidebar/V1.3.x/zh.ts
Original file line number Diff line number Diff line change
Expand Up @@ -205,14 +205,14 @@ export const zhSidebar = {
collapsible: true,
children: [
{ text: 'UDF函数库', link: 'UDF-Libraries_apache' },
{ text: '函数和操作符', link: 'Operator-and-Expression' },
{ text: '函数与运算符', link: 'Operator-and-Expression' },
],
},
{
text: '语法约定',
collapsible: true,
children: [
{ text: '保留字&标识符', link: 'Syntax-Rule' },
{ text: '标识符', link: 'Syntax-Rule' },
{ text: '关键字', link: 'Keywords' },
],
},
Expand Down
2 changes: 1 addition & 1 deletion src/.vuepress/sidebar_timecho/V1.3.x/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ export const enSidebar = {
text: 'Syntax-Rule',
collapsible: true,
children: [
{ text: 'Reserved Words&Identifiers', link: 'Syntax-Rule' },
{ text: 'Identifiers', link: 'Syntax-Rule' },
{ text: 'Keywords', link: 'Keywords' },
],
},
Expand Down
4 changes: 2 additions & 2 deletions src/.vuepress/sidebar_timecho/V1.3.x/zh.ts
Original file line number Diff line number Diff line change
Expand Up @@ -218,14 +218,14 @@ export const zhSidebar = {
collapsible: true,
children: [
{ text: 'UDF函数库', link: 'UDF-Libraries_timecho' },
{ text: '函数和操作符', link: 'Operator-and-Expression' },
{ text: '函数与运算符', link: 'Operator-and-Expression' },
],
},
{
text: '语法约定',
collapsible: true,
children: [
{ text: '保留字&标识符', link: 'Syntax-Rule' },
{ text: '标识符', link: 'Syntax-Rule' },
{ text: '关键字', link: 'Keywords' },
],
},
Expand Down
25 changes: 0 additions & 25 deletions src/UserGuide/Master/Tree/Reference/Operator-and-Expression.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,31 +252,6 @@ For details and examples, see the document [Sample Functions](../Reference/Funct

For details and examples, see the document [Time-Series](../Reference/Function-and-Expression.md#time-series-processing).

## DATA QUALITY FUNCTION LIBRARY

### About

For applications based on time series data, data quality is vital. **UDF Library** is IoTDB User Defined Functions (UDF) about data quality, including data profiling, data quality evalution and data repairing. It effectively meets the demand for data quality in the industrial field.

### Quick Start

The functions in this function library are not built-in functions, and must be loaded into the system before use.

1. [Download](https://archive.apache.org/dist/iotdb/1.0.1/apache-iotdb-1.0.1-library-udf-bin.zip) the JAR with all dependencies and the script of registering UDF.
2. Copy the JAR package to `ext\udf` under the directory of IoTDB system (Please put JAR to this directory of all DataNodes if you use Cluster).
3. Run `sbin\start-confignode.bat` and then `sbin\start-datanode.bat` (for Windows) or `sbin\start-confignode.sh` and `sbin\start-datanode.sh` (for Linux or MacOS) to start IoTDB server.
4. Copy the script to the directory of IoTDB system (under the root directory, at the same level as `sbin`), modify the parameters in the script if needed and run it to register UDF.

### Implemented Functions

1. Data Quality related functions, such as `Completeness`. For details and examples, see the document [Data-Quality](../Reference/UDF-Libraries.md#data-quality).
2. Data Profiling related functions, such as `ACF`. For details and examples, see the document [Data-Profiling](../Reference/UDF-Libraries.md#data-profiling).
3. Anomaly Detection related functions, such as `IQR`. For details and examples, see the document [Anomaly-Detection](../Reference/UDF-Libraries.md#anomaly-detection).
4. Frequency Domain Analysis related functions, such as `Conv`. For details and examples, see the document [Frequency-Domain](../Reference/UDF-Libraries.md#frequency-domain-analysis).
5. Data Matching related functions, such as `DTW`. For details and examples, see the document [Data-Matching](../Reference/UDF-Libraries.md#data-matching).
6. Data Repairing related functions, such as `TimestampRepair`. For details and examples, see the document [Data-Repairing](../Reference/UDF-Libraries.md#timestamprepair).
7. Series Discovery related functions, such as `ConsecutiveSequences`. For details and examples, see the document [Series-Discovery](../Reference/UDF-Libraries.md).
8. Machine Learning related functions, such as `AR`. For details and examples, see the document [Machine-Learning](../Reference/UDF-Libraries.md#machine-learning).

## LAMBDA EXPRESSION

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

-->

# Syntax Rule
# Identifiers

## Literal Values

Expand Down Expand Up @@ -278,16 +278,3 @@ Examples of case in which quoted identifier is used :

- Nodes except database in the path are allowed to contain the "*" symbol, when using this symbol it is required to enclose the node in backquotes, e.g., root.db.`*`, but this usage is only recommended when the path cannot avoid containing the "*" symbol.

## KeyWords Words

Keywords are words that have significance in SQL. Keywords can be used as an identifier. Certain keywords, such as TIME/TIMESTAMP and ROOT, are reserved and cannot use as identifiers.

[Keywords](../Reference/Keywords.md) shows the keywords in IoTDB.

## Detailed Definitions of Lexical and Grammar

Please read the lexical and grammar description files in our code repository:

Lexical file: `antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/IoTDBSqlLexer.g4`

Grammer file: `antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4`
53 changes: 26 additions & 27 deletions src/UserGuide/Master/Tree/User-Manual/Database-Programming.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,16 @@

-->

# Database Programming
# CONTINUOUS QUERY(CQ)

## CONTINUOUS QUERY (CQ)

### Introduction
## Introduction

Continuous queries(CQ) are queries that run automatically and periodically on realtime data and store query results in other specified time series.

Users can implement sliding window streaming computing through continuous query, such as calculating the hourly average temperature of a sequence and writing it into a new sequence. Users can customize the `RESAMPLE` clause to create different sliding windows, which can achieve a certain degree of tolerance for out-of-order data.

### Syntax
## Syntax

```sql
CREATE (CONTINUOUS QUERY | CQ) <cq_id>
Expand Down Expand Up @@ -58,7 +57,7 @@ END
> 2. GROUP BY TIME CLAUSE is different, it doesn't contain its original first display window parameter which is [start_time, end_time). It's still because IoTDB will automatically generate a time range for the query each time it's executed.
> 3. If there is no group by time clause in query, EVERY clause is required, otherwise IoTDB will throw an error.

#### Descriptions of parameters in CQ syntax
### Descriptions of parameters in CQ syntax

- `<cq_id>` specifies the globally unique id of CQ.
- `<every_interval>` specifies the query execution time interval. We currently support the units of ns, us, ms, s, m, h, d, w, and its value should not be lower than the minimum threshold configured by the user, which is `continuous_query_min_every_interval`. It's an optional parameter, default value is set to `group_by_interval` in group by clause.
Expand All @@ -80,19 +79,19 @@ END
> - If `<start_time_offset>` is less than `<every_interval>`, there may be uncovered data between each query window.
> - `start_time_offset` should be larger than `end_time_offset`, otherwise the system will throw an error.

##### `<start_time_offset>` == `<every_interval>`
#### `<start_time_offset>` == `<every_interval>`

![1](https://alioss.timecho.com/docs/img/UserGuide/Process-Data/Continuous-Query/pic1.png?raw=true)

##### `<start_time_offset>` > `<every_interval>`
#### `<start_time_offset>` > `<every_interval>`

![2](https://alioss.timecho.com/docs/img/UserGuide/Process-Data/Continuous-Query/pic2.png?raw=true)

##### `<start_time_offset>` < `<every_interval>`
#### `<start_time_offset>` < `<every_interval>`

![3](https://alioss.timecho.com/docs/img/UserGuide/Process-Data/Continuous-Query/pic3.png?raw=true)

##### `<every_interval>` is not zero
#### `<every_interval>` is not zero

![](https://alioss.timecho.com/docs/img/UserGuide/Process-Data/Continuous-Query/pic4.png?raw=true)

Expand All @@ -102,7 +101,7 @@ END
- `DISCARD` means that we just discard the current cq execution task and wait for the next execution time and do the next time interval cq task. If using `DISCARD` policy, some time intervals won't be executed when the execution time of one cq task is longer than the `<every_interval>`. However, once a cq task is executed, it will use the latest time interval, so it can catch up at the sacrifice of some time intervals being discarded.


### Examples of CQ
## Examples of CQ

The examples below use the following sample data. It's a real time data stream and we can assume that the data arrives on time.

Expand All @@ -122,7 +121,7 @@ The examples below use the following sample data. It's a real time data stream a
+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
````

#### Configuring execution intervals
### Configuring execution intervals

Use an `EVERY` interval in the `RESAMPLE` clause to specify the CQ’s execution interval, if not specific, default value is equal to `group_by_interval`.

Expand Down Expand Up @@ -180,7 +179,7 @@ At **2021-05-11T22:19:00.000+08:00**, `cq1` executes a query within the time ran
+-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
````

#### Configuring time range for resampling
### Configuring time range for resampling

Use `start_time_offset` in the `RANGE` clause to specify the start time of the CQ’s time range, if not specific, default value is equal to `EVERY` interval.

Expand Down Expand Up @@ -255,7 +254,7 @@ At **2021-05-11T22:19:00.000+08:00**, `cq2` executes a query within the time ran
+-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
````

#### Configuring execution intervals and CQ time ranges
### Configuring execution intervals and CQ time ranges

Use an `EVERY` interval and `RANGE` interval in the `RESAMPLE` clause to specify the CQ’s execution interval and the length of the CQ’s time range. And use `fill()` to change the value reported for time intervals with no data.

Expand Down Expand Up @@ -320,7 +319,7 @@ Notice that `cq3` will calculate the results for some time interval many times,
+-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
````

#### Configuring end_time_offset for CQ time range
### Configuring end_time_offset for CQ time range

Use an `EVERY` interval and `RANGE` interval in the RESAMPLE clause to specify the CQ’s execution interval and the length of the CQ’s time range. And use `fill()` to change the value reported for time intervals with no data.

Expand Down Expand Up @@ -379,7 +378,7 @@ Notice that `cq4` will calculate the results for all time intervals only once af
+-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
````

#### CQ without group by clause
### CQ without group by clause

Use an `EVERY` interval in the `RESAMPLE` clause to specify the CQ’s execution interval and the length of the CQ’s time range.

Expand Down Expand Up @@ -485,9 +484,9 @@ At **2021-05-11T22:19:00.000+08:00**, `cq5` executes a query within the time ran
+-----------------------------+-------------------------------+-----------+
````

### CQ Management
## CQ Management

#### Listing continuous queries
### Listing continuous queries

List every CQ on the IoTDB Cluster with:

Expand All @@ -497,7 +496,7 @@ SHOW (CONTINUOUS QUERIES | CQS)

`SHOW (CONTINUOUS QUERIES | CQS)` order results by `cq_id`.

##### Examples
#### Examples

```sql
SHOW CONTINUOUS QUERIES;
Expand All @@ -510,7 +509,7 @@ we will get:
| s1_count_cq | CREATE CQ s1_count_cq<br/>BEGIN<br/>SELECT count(s1)<br/>INTO root.sg_count.d.count_s1<br/>FROM root.sg.d<br/>GROUP BY(30m)<br/>END | active |


#### Dropping continuous queries
### Dropping continuous queries

Drop a CQ with a specific `cq_id`:

Expand All @@ -520,36 +519,36 @@ DROP (CONTINUOUS QUERY | CQ) <cq_id>

DROP CQ returns an empty result.

##### Examples
#### Examples

Drop the CQ named `s1_count_cq`:

```sql
DROP CONTINUOUS QUERY s1_count_cq;
```

#### Altering continuous queries
### Altering continuous queries

CQs can't be altered once they're created. To change a CQ, you must `DROP` and re`CREATE` it with the updated settings.


### CQ Use Cases
## CQ Use Cases

#### Downsampling and Data Retention
### Downsampling and Data Retention

Use CQs with `TTL` set on database in IoTDB to mitigate storage concerns. Combine CQs and `TTL` to automatically downsample high precision data to a lower precision and remove the dispensable, high precision data from the database.

#### Recalculating expensive queries
### Recalculating expensive queries

Shorten query runtimes by pre-calculating expensive queries with CQs. Use a CQ to automatically downsample commonly-queried, high precision data to a lower precision. Queries on lower precision data require fewer resources and return faster.

> Pre-calculate queries for your preferred graphing tool to accelerate the population of graphs and dashboards.

#### Substituting for sub-query
### Substituting for sub-query

IoTDB does not support sub queries. We can get the same functionality by creating a CQ as a sub query and store its result into other time series and then querying from those time series again will be like doing nested sub query.

##### Example
#### Example

IoTDB does not accept the following query with a nested sub query. The query calculates the average number of non-null values of `s1` at 30 minute intervals:

Expand Down Expand Up @@ -584,7 +583,7 @@ SELECT avg(count_s1) from root.sg_count.d;
```


### System Parameter Configuration
## System Parameter Configuration

| Name | Description | Data Type | Default Value |
| :------------------------------------------ | ------------------------------------------------------------ | --------- | ------------- |
Expand Down
25 changes: 0 additions & 25 deletions src/UserGuide/latest/Reference/Operator-and-Expression.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,31 +252,6 @@ For details and examples, see the document [Sample Functions](../Reference/Funct

For details and examples, see the document [Time-Series](../Reference/Function-and-Expression.md#time-series-processing).

## DATA QUALITY FUNCTION LIBRARY

### About

For applications based on time series data, data quality is vital. **UDF Library** is IoTDB User Defined Functions (UDF) about data quality, including data profiling, data quality evalution and data repairing. It effectively meets the demand for data quality in the industrial field.

### Quick Start

The functions in this function library are not built-in functions, and must be loaded into the system before use.

1. [Download](https://archive.apache.org/dist/iotdb/1.0.1/apache-iotdb-1.0.1-library-udf-bin.zip) the JAR with all dependencies and the script of registering UDF.
2. Copy the JAR package to `ext\udf` under the directory of IoTDB system (Please put JAR to this directory of all DataNodes if you use Cluster).
3. Run `sbin\start-confignode.bat` and then `sbin\start-datanode.bat` (for Windows) or `sbin\start-confignode.sh` and `sbin\start-datanode.sh` (for Linux or MacOS) to start IoTDB server.
4. Copy the script to the directory of IoTDB system (under the root directory, at the same level as `sbin`), modify the parameters in the script if needed and run it to register UDF.

### Implemented Functions

1. Data Quality related functions, such as `Completeness`. For details and examples, see the document [Data-Quality](../Reference/UDF-Libraries.md#data-quality).
2. Data Profiling related functions, such as `ACF`. For details and examples, see the document [Data-Profiling](../Reference/UDF-Libraries.md#data-profiling).
3. Anomaly Detection related functions, such as `IQR`. For details and examples, see the document [Anomaly-Detection](../Reference/UDF-Libraries.md#anomaly-detection).
4. Frequency Domain Analysis related functions, such as `Conv`. For details and examples, see the document [Frequency-Domain](../Reference/UDF-Libraries.md#frequency-domain-analysis).
5. Data Matching related functions, such as `DTW`. For details and examples, see the document [Data-Matching](../Reference/UDF-Libraries.md#data-matching).
6. Data Repairing related functions, such as `TimestampRepair`. For details and examples, see the document [Data-Repairing](../Reference/UDF-Libraries.md#timestamprepair).
7. Series Discovery related functions, such as `ConsecutiveSequences`. For details and examples, see the document [Series-Discovery](../Reference/UDF-Libraries.md).
8. Machine Learning related functions, such as `AR`. For details and examples, see the document [Machine-Learning](../Reference/UDF-Libraries.md#machine-learning).

## LAMBDA EXPRESSION

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

-->

# Syntax Rule
# Identifiers

## Literal Values

Expand Down Expand Up @@ -277,17 +277,3 @@ Examples of case in which quoted identifier is used :
- The key/value of an attribute can be String Literal and identifier, more details can be found at **key-value pair** part.

- Nodes except database in the path are allowed to contain the "*" symbol, when using this symbol it is required to enclose the node in backquotes, e.g., root.db.`*`, but this usage is only recommended when the path cannot avoid containing the "*" symbol.

## KeyWords Words

Keywords are words that have significance in SQL. Keywords can be used as an identifier. Certain keywords, such as TIME/TIMESTAMP and ROOT, are reserved and cannot use as identifiers.

[Keywords](../Reference/Keywords.md) shows the keywords in IoTDB.

## Detailed Definitions of Lexical and Grammar

Please read the lexical and grammar description files in our code repository:

Lexical file: `antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/IoTDBSqlLexer.g4`

Grammer file: `antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4`
Loading