Skip to content

Commit 1943b2c

Browse files
W1y1rW1y1r
authored
Operations statement - region migration and related modifications (#511)
1 parent 1d2f3a2 commit 1943b2c

File tree

23 files changed

+915
-431
lines changed

23 files changed

+915
-431
lines changed

‎src/.vuepress/sidebar/V1.3.3/en.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,15 @@ export const enSidebar = {
116116
{ text: 'Stream Processing', link: 'Streaming_apache' },
117117
],
118118
},
119-
{ text: 'Maintenance SQL', link: 'Maintennance' },
119+
{
120+
text: 'Maintenance SQL',
121+
collapsible: true,
122+
children: [
123+
{ text: 'Query Performance Analysis', link: 'Query-Performance-Analysis' },
124+
{ text: 'Load Balance', link: 'Load-Balance' },
125+
{ text: 'Data Recovery', link: 'Data-Recovery' },
126+
],
127+
},
120128
],
121129
},
122130
{

‎src/.vuepress/sidebar/V1.3.3/zh.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,15 @@ export const zhSidebar = {
103103
{ text: '流处理框架', link: 'Streaming_apache' },
104104
],
105105
},
106-
{ text: '运维语句', link: 'Maintennance' },
106+
{
107+
text: '运维语句',
108+
collapsible: true,
109+
children: [
110+
{ text: '查询性能分析', link: 'Query-Performance-Analysis' },
111+
{ text: '负载均衡', link: 'Load-Balance' },
112+
{ text: '数据修复', link: 'Data-Recovery' },
113+
],
114+
},
107115
],
108116
},
109117
{

‎src/.vuepress/sidebar_timecho/V1.3.3/en.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,15 @@ export const enSidebar = {
131131
{ text: 'Stream Processing', link: 'Streaming_timecho' },
132132
],
133133
},
134-
{ text: 'Maintenance SQL', link: 'Maintennance' },
134+
{
135+
text: 'Maintenance SQL',
136+
collapsible: true,
137+
children: [
138+
{ text: 'Query Performance Analysis', link: 'Query-Performance-Analysis' },
139+
{ text: 'Load Balance', link: 'Load-Balance' },
140+
{ text: 'Data Recovery', link: 'Data-Recovery' },
141+
],
142+
},
135143
],
136144
},
137145
{

‎src/.vuepress/sidebar_timecho/V1.3.3/zh.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,15 @@ export const zhSidebar = {
114114
{ text: '流处理框架', link: 'Streaming_timecho' },
115115
],
116116
},
117-
{ text: '运维语句', link: 'Maintennance' },
117+
{
118+
text: '运维语句',
119+
collapsible: true,
120+
children: [
121+
{ text: '查询性能分析', link: 'Query-Performance-Analysis' },
122+
{ text: '负载均衡', link: 'Load-Balance' },
123+
{ text: '数据修复', link: 'Data-Recovery' },
124+
],
125+
},
118126
],
119127
},
120128
{

‎src/.vuepress/sidebar_timecho/V2.0.1/zh-Tree.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,15 @@ export const zhSidebar = {
114114
{ text: '流处理框架', link: 'Streaming_timecho' },
115115
],
116116
},
117-
{ text: '运维语句', link: 'Maintennance' },
117+
{
118+
text: '运维语句',
119+
collapsible: true,
120+
children: [
121+
{ text: '查询性能分析', link: 'Query-Performance-Analysis' },
122+
{ text: '负载均衡', link: 'Load-Balance' },
123+
{ text: '数据修复', link: 'Data-Recovery' },
124+
],
125+
},
118126
],
119127
},
120128
{

‎src/UserGuide/Master/Tree/User-Manual/Data-Recovery.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
<!--
2+
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
20+
-->
21+
22+
## Data Recovery
23+
24+
Used to fix issues in data, such as data in sequential space not being arranged in chronological order.
25+
26+
### START REPAIR DATA
27+
28+
Start a repair task to scan all files created before current time.
29+
The repair task will scan all tsfiles and repair some bad files.
30+
31+
```sql
32+
IoTDB> START REPAIR DATA
33+
IoTDB> START REPAIR DATA ON LOCAL
34+
IoTDB> START REPAIR DATA ON CLUSTER
35+
```
36+
37+
### STOP REPAIR DATA
38+
39+
Stop the running repair task. To restart the stopped task.
40+
If there is a stopped repair task, it can be restart and recover the repair progress by executing SQL `START REPAIR DATA`.
41+
42+
```sql
43+
IoTDB> STOP REPAIR DATA
44+
IoTDB> STOP REPAIR DATA ON LOCAL
45+
IoTDB> STOP REPAIR DATA ON CLUSTER
46+
```
47+

‎src/UserGuide/Master/Tree/User-Manual/Load-Balance.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
<!--
2+
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
20+
-->
21+
22+
## Load Balance
23+
24+
Region migration belongs to advanced operations and maintenance functions, which have certain operational costs. It is recommended to read the entire document before using this function. If you have any questions about the solution design, please contact the IoTDB team for technical support.
25+
26+
27+
### Feature introduction
28+
29+
IoTDB is a distributed database, and the balanced distribution of data plays an important role in load balancing the disk space and write pressure of the cluster. Region is the basic unit for distributed storage of data in IoTDB cluster, and the specific concept can be seen in [region](../Background-knowledge/Cluster-Concept.md)。
30+
31+
Under normal operation of the cluster, IoTDB will automatically perform load balancing on data. However, in scenarios where a new DataNode node is added to the cluster or where the hard disk of the machine where the DataNode is located is damaged and data needs to be recovered, manual region migration can be used to finely adjust the cluster load and operations.
32+
33+
Here is a schematic diagram of the region migration process :
34+
35+
36+
![](https://alioss.timecho.com/docs/img/region%E8%BF%81%E7%A7%BB%E7%A4%BA%E6%84%8F%E5%9B%BE20241210.png)
37+
38+
### Notes
39+
40+
1. It is recommended to only use the Region Migration feature on IoTDB 1.3.3 and higher versions.
41+
2. Region migration is only supported when the consensus protocol is IoTConsus or Ratis (in iotdb system. properties, the `schema_region_consensus_protocol_class` and`data_region_consensus_protocol_class`).
42+
3. Region migration will occupy system resources such as hard drives and network bandwidth, and it is recommended to perform it during low business loads.
43+
4. The region migration process will occupy WAL files, blocking the deletion of WAL files in this consensus group. If the total number of WAL files reaches `wal_file_size_threshold_in_byte`, writing will be blocked.
44+
45+
### Instructions for use
46+
47+
- **Grammar definition** :
48+
49+
Submit an asynchronous task to migrate a region from one DataNode to another.
50+
51+
52+
```SQL
53+
migrateRegion
54+
: MIGRATE REGION regionId=INTEGER_LITERAL FROM fromId=INTEGER_LITERAL TO toId=INTEGER_LITERAL
55+
;
56+
```
57+
58+
- **Example** :
59+
60+
Migrating region 1 from DataNode 2 to DataNode 3:
61+
62+
```SQL
63+
IoTDB> migrate region 1 from 2 to 3
64+
Msg: The statement is executed successfully.
65+
```
66+
67+
"The statement is executed successfully" only represents the successful submission of the region migration task, not the completion of execution. The execution status of the task can be viewed through the CLI command `show regions `.
68+
- **Related configuration** :
69+
- Migration speed control : modify `iotdb-system.properties `parameters `region_migration_speed_limit_bytes_per_second `control region migration speed.
70+
- **Time cost estimation** :
71+
- If there are no concurrent writes during the migration process, the time consumption can be simply estimated by dividing the region data volume by the data transfer speed. For example, for a 1TB region, the hard disk internet bandwidth and speed limit parameters jointly determine that the data transfer speed is 100MB/s, so it takes about 3 hours to complete the migration.
72+
- If there are concurrent writes in the migration process, the time consumption will increase, and the specific time consumption depends on various factors such as write pressure and system resources. It can be simply estimated as `no concurrent write time × 1.5 `.
73+
- **Migration progress observation** : During the migration process, the state changes can be observed through the CLI command `show regions `. Taking the 2 replicas as an example, the state of the consensus group where the region is located will go through the following process:
74+
- Before migration starts: `Running `, `Running `.
75+
- Expansion phase: `Running `, `Running `, `Adding `. Due to the large number of file transfers involved, it may take a long time. If using IoTConsensus, the specific file transfer progress can be searched in the DataNode log `[SNAPSHOT TRANSMISSION] `.
76+
- Stages: `Removing `, `Running `, `Running `.
77+
- Migration complete: `Running `, `Running `.
78+
79+
Taking the expansion phase as an example, the result of `show regions` may be:
80+
81+
```Plain
82+
IoTDB> show regions
83+
+--------+------------+-------+--------+-------------+-----------+----------+----------+-------+---------------+--------+-----------------------+
84+
|RegionId| Type| Status|Database|SeriesSlotNum|TimeSlotNum|DataNodeId|RpcAddress|RpcPort|InternalAddress| Role| CreateTime|
85+
+--------+------------+-------+--------+-------------+-----------+----------+----------+-------+---------------+--------+-----------------------+
86+
| 0|SchemaRegion|Running| root.ln| 1| 0| 1| 0.0.0.0| 6668| 127.0.0.1| Leader|2024-04-15T18:55:17.691|
87+
| 0|SchemaRegion|Running| root.ln| 1| 0| 2| 0.0.0.0| 6668| 127.0.0.1| Leader|2024-04-15T18:55:17.691|
88+
| 0|SchemaRegion|Running| root.ln| 1| 0| 3| 0.0.0.0| 6668| 127.0.0.1| Leader|2024-04-15T18:55:17.691|
89+
| 1| DataRegion|Running| root.ln| 1| 1| 1| 0.0.0.0| 6667| 127.0.0.1| Leader|2024-04-15T18:55:19.457|
90+
| 1| DataRegion|Running| root.ln| 1| 1| 2| 0.0.0.0| 6668| 127.0.0.1|Follower|2024-04-15T18:55:19.457|
91+
| 1| DataRegion| Adding| root.ln| 1| 1| 3| 0.0.0.0| 6668| 127.0.0.1|Follower|2024-04-15T18:55:19.457|
92+
+--------+------------+-------+--------+-------------+-----------+----------+----------+-------+---------------+--------+-----------------------+
93+
Total line number = 3
94+
It costs 0.003s
95+
```

src/UserGuide/V2.0.1/Tree/User-Manual/Maintennance.md renamed to src/UserGuide/Master/Tree/User-Manual/Query-Performance-Analysis.md

Lines changed: 1 addition & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222

2323
# Maintennance
2424

25-
## Explain/Explain Analyze Statements
25+
## Query Performance Analysis
2626

2727
The purpose of query analysis is to assist users in understanding the execution mechanism and performance bottlenecks of queries, thereby facilitating query optimization and performance enhancement. This is crucial not only for the efficiency of query execution but also for the user experience of applications and the efficient utilization of resources. For effective query analysis, IoTDB versions V1.3.2 and above offer the query analysis statements: Explain and Explain Analyze.
2828

@@ -346,27 +346,3 @@ Observing the results, we found that it is because the query did not add a time
346346

347347
The final optimization plan is: Add a time filtering condition to avoid a full table scan.
348348

349-
350-
## Start/Stop Repair Data Statements
351-
Used to repair the unsorted data generate by system bug.
352-
### START REPAIR DATA
353-
354-
Start a repair task to scan all files created before current time.
355-
The repair task will scan all tsfiles and repair some bad files.
356-
357-
```sql
358-
IoTDB> START REPAIR DATA
359-
IoTDB> START REPAIR DATA ON LOCAL
360-
IoTDB> START REPAIR DATA ON CLUSTER
361-
```
362-
363-
### STOP REPAIR DATA
364-
365-
Stop the running repair task. To restart the stopped task.
366-
If there is a stopped repair task, it can be restart and recover the repair progress by executing SQL `START REPAIR DATA`.
367-
368-
```sql
369-
IoTDB> STOP REPAIR DATA
370-
IoTDB> STOP REPAIR DATA ON LOCAL
371-
IoTDB> STOP REPAIR DATA ON CLUSTER
372-
```

‎src/UserGuide/V2.0.1/Tree/User-Manual/Data-Recovery.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
<!--
2+
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
20+
-->
21+
22+
## Data Recovery
23+
24+
Used to fix issues in data, such as data in sequential space not being arranged in chronological order.
25+
26+
### START REPAIR DATA
27+
28+
Start a repair task to scan all files created before current time.
29+
The repair task will scan all tsfiles and repair some bad files.
30+
31+
```sql
32+
IoTDB> START REPAIR DATA
33+
IoTDB> START REPAIR DATA ON LOCAL
34+
IoTDB> START REPAIR DATA ON CLUSTER
35+
```
36+
37+
### STOP REPAIR DATA
38+
39+
Stop the running repair task. To restart the stopped task.
40+
If there is a stopped repair task, it can be restart and recover the repair progress by executing SQL `START REPAIR DATA`.
41+
42+
```sql
43+
IoTDB> STOP REPAIR DATA
44+
IoTDB> STOP REPAIR DATA ON LOCAL
45+
IoTDB> STOP REPAIR DATA ON CLUSTER
46+
```
47+

0 commit comments

Comments
 (0)