Skip to content

Commit bc07621

Browse files
W1y1rW1y1r
authored
Multi level storage configuration parameter modification (#413)
* Multi level storage configuration parameter modification * Add Tiered Storage configuration parameters * Modify Tiered Storage configuration parameters * Supplement of low-level configuration parameters for multi-level storage * fix 1.3.0/1/2 dn_default_space_usage_thresholds
1 parent c7315bd commit bc07621

24 files changed

+800
-84
lines changed

‎src/UserGuide/Master/Reference/Common-Config-Manual.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -676,14 +676,14 @@ Different configuration parameters take effect in the following three ways:
676676
| Default | ms |
677677
| Effective | Only allowed to be modified in first start up |
678678

679-
* default\_ttl\_in\_ms
680-
681-
| Name | default\_ttl\_in\_ms |
682-
| :---------: | :------------------------------------- |
683-
| Description | Default ttl when each database created |
684-
| Type | Long |
685-
| Default | Infinity |
686-
| Effective | After restarting system |
679+
* tier\_ttl\_in\_ms
680+
681+
|Name| tier\_ttl\_in\_ms |
682+
|:---:|:--------------|
683+
|Description| Define the maximum age of data for which each tier is responsible |
684+
|Type| long |
685+
|Default| -1 |
686+
|Effective| After restarting system |
687687

688688
* max\_waiting\_time\_when\_insert\_blocked
689689

‎src/UserGuide/Master/Reference/DataNode-Config-Manual.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -497,3 +497,88 @@ There will be at most 10 gc.log.* files and each one can reach to 10MB.
497497
|Type| int32 |
498498
|Default| 5000 |
499499
|Effective| After restarting system |
500+
501+
502+
503+
#### Storage engine configuration
504+
505+
* dn\_default\_space\_usage\_thresholds
506+
507+
|Name| dn\_default\_space\_usage\_thresholds |
508+
|:---:|:--------------|
509+
|Description| Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY |
510+
|Type| double |
511+
|Default| 0.85 |
512+
|Effective| hot-load |
513+
514+
* remote\_tsfile\_cache\_dirs
515+
516+
|Name| remote\_tsfile\_cache\_dirs |
517+
|:---:|:--------------|
518+
|Description| Cache directory stored locally in the cloud |
519+
|Type| string |
520+
|Default| data/datanode/data/cache |
521+
|Effective| After restarting system |
522+
523+
* remote\_tsfile\_cache\_page\_size\_in\_kb
524+
525+
|Name| remote\_tsfile\_cache\_page\_size\_in\_kb |
526+
|:---:|:--------------|
527+
|Description| Block size of locally cached files stored in the cloud |
528+
|Type| int |
529+
|Default| 20480 |
530+
|Effective| After restarting system |
531+
532+
* remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb
533+
534+
|Name| remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb |
535+
|:---:|:--------------|
536+
|Description| Maximum Disk Occupancy Size for Cloud Storage Local Cache |
537+
|Type| long |
538+
|Default| 51200 |
539+
|Effective| After restarting system |
540+
541+
* object\_storage\_type
542+
543+
|Name| object\_storage\_type |
544+
|:---:|:--------------|
545+
|Description| Cloud Storage Type |
546+
|Type| string |
547+
|Default| AWS_S3 |
548+
|Effective| After restarting system |
549+
550+
* object\_storage\_bucket
551+
552+
|Name| object\_storage\_bucket |
553+
|:---:|:--------------|
554+
|Description| Name of cloud storage bucket |
555+
|Type| string |
556+
|Default| iotdb_data |
557+
|Effective| After restarting system |
558+
559+
* object\_storage\_endpoiont
560+
561+
|Name| object\_storage\_endpoiont |
562+
|:---:|:--------------|
563+
|Description| endpoint of cloud storage |
564+
|Type| string |
565+
|Default| None |
566+
|Effective| After restarting system |
567+
568+
* object\_storage\_access\_key
569+
570+
|Name| object\_storage\_access\_key |
571+
|:---:|:--------------|
572+
|Description| Authentication information stored in the cloud: key |
573+
|Type| string |
574+
|Default| None |
575+
|Effective| After restarting system |
576+
577+
* object\_storage\_access\_secret
578+
579+
|Name| object\_storage\_access\_secret |
580+
|:---:|:--------------|
581+
|Description| Authentication information stored in the cloud: secret |
582+
|Type| string |
583+
|Default| None |
584+
|Effective| After restarting system |

‎src/UserGuide/Master/User-Manual/Tiered-Storage_timecho.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,11 @@ The specific parameter definitions and their descriptions are as follows.
3636

3737
| Configuration | Default | Description | Constraint |
3838
| ---------------------------------------- | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
39-
| dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
40-
| default_ttl_in_ms | None | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs；"-1" means "unlimited". |
41-
| dn_default_space_move_thresholds | 0.15 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
39+
| dn_data_dirs | data/datanode/data | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
40+
| tier_ttl_in_ms | -1 | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs；"-1" means "unlimited". |
41+
| dn_default_space_usage_thresholds | 0.85 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
4242
| object_storage_type | AWS_S3 | Cloud Storage Type | IoTDB currently only supports AWS S3 as a remote storage type, and this parameter can't be modified |
43-
| object_storage_bucket | None | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
43+
| object_storage_bucket | iotdb_data | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
4444
| object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required |
4545
| object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 credential key；If remote storage is not used, no configuration required |
4646
| object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 credential secret；If remote storage is not used, no configuration required |
@@ -55,8 +55,8 @@ The following is an example of a local two-level storage configuration.
5555
```JavaScript
5656
//Required configuration items
5757
dn_data_dirs=/data1/data;/data2/data,/data3/data;
58-
default_ttl_in_ms=86400000;-1
59-
dn_default_space_move_thresholds=0.2;0.1
58+
tier_ttl_in_ms=86400000;-1
59+
dn_default_space_usage_thresholds=0.2;0.1
6060
```
6161

6262
In this example, two levels of storage are configured, specifically:
@@ -73,8 +73,8 @@ The following takes three-level storage as an example:
7373
```JavaScript
7474
//Required configuration items
7575
dn_data_dirs=/data1/data;/data2/data,/data3/data;OBJECT_STORAGE
76-
default_ttl_in_ms=86400000;864000000;-1
77-
dn_default_space_move_thresholds=0.2;0.15;0.1
76+
tier_ttl_in_ms=86400000;864000000;-1
77+
dn_default_space_usage_thresholds=0.2;0.15;0.1
7878
object_storage_name=AWS_S3
7979
object_storage_bucket=iotdb
8080
object_storage_endpoiont=<your_endpoint>

‎src/UserGuide/V1.2.x/Reference/Common-Config-Manual.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -649,11 +649,11 @@ Different configuration parameters take effect in the following three ways:
649649
* default\_ttl\_in\_ms
650650

651651
| Name | default\_ttl\_in\_ms |
652-
| :---------: | :------------------------------------- |
653-
| Description | Default ttl when each database created |
654-
| Type | Long |
655-
| Default | Infinity |
656-
| Effective | After restarting system |
652+
|:---:|:--------------|
653+
|Description| Define the maximum age of data for which each tier is responsible |
654+
|Type| long |
655+
|Default| -1 |
656+
|Effective| After restarting system |
657657

658658
* max\_waiting\_time\_when\_insert\_blocked
659659

‎src/UserGuide/V1.2.x/Reference/DataNode-Config-Manual.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,3 +357,89 @@ sbin\start-datanode.bat printgc
357357

358358
GC log is stored at `IOTDB_HOME/logs/gc.log`.
359359
There will be at most 10 gc.log.* files and each one can reach to 10MB.
360+
361+
362+
#### Storage engine configuration
363+
364+
* dn\_default\_space\_move\_thresholds
365+
366+
|Name| dn\_default\_space\_move\_thresholds |
367+
|:---:|:--------------|
368+
|Description|Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY |
369+
|Type| double |
370+
|Default| 0.15 |
371+
|Effective| hot-load |
372+
373+
374+
375+
* remote\_tsfile\_cache\_dirs
376+
377+
|Name| remote\_tsfile\_cache\_dirs |
378+
|:---:|:--------------|
379+
|Description| Cache directory stored locally in the cloud |
380+
|Type| string |
381+
|Default| data/datanode/data/cache |
382+
|Effective| After restarting system |
383+
384+
* remote\_tsfile\_cache\_page\_size\_in\_kb
385+
386+
|Name| remote\_tsfile\_cache\_page\_size\_in\_kb |
387+
|:---:|:--------------|
388+
|Description| Block size of locally cached files stored in the cloud |
389+
|Type| int |
390+
|Default| 20480 |
391+
|Effective| After restarting system |
392+
393+
* remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb
394+
395+
|Name| remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb |
396+
|:---:|:--------------|
397+
|Description| Maximum Disk Occupancy Size for Cloud Storage Local Cache |
398+
|Type| long |
399+
|Default| 51200 |
400+
|Effective| After restarting system |
401+
402+
* object\_storage\_type
403+
404+
|Name| object\_storage\_type |
405+
|:---:|:--------------|
406+
|Description| Cloud Storage Type |
407+
|Type| string |
408+
|Default| AWS_S3 |
409+
|Effective| After restarting system |
410+
411+
* object\_storage\_bucket
412+
413+
|Name| object\_storage\_bucket |
414+
|:---:|:--------------|
415+
|Description| Name of cloud storage bucket |
416+
|Type| string |
417+
|Default| iotdb_data |
418+
|Effective| After restarting system |
419+
420+
* object\_storage\_endpoiont
421+
422+
|Name| object\_storage\_endpoiont |
423+
|:---:|:--------------|
424+
|Description| endpoint of cloud storage |
425+
|Type| string |
426+
|Default| None |
427+
|Effective| After restarting system |
428+
429+
* object\_storage\_access\_key
430+
431+
|Name| object\_storage\_access\_key |
432+
|:---:|:--------------|
433+
|Description| Authentication information stored in the cloud: key |
434+
|Type| string |
435+
|Default| None |
436+
|Effective| After restarting system |
437+
438+
* object\_storage\_access\_secret
439+
440+
|Name| object\_storage\_access\_secret |
441+
|:---:|:--------------|
442+
|Description| Authentication information stored in the cloud: secret |
443+
|Type| string |
444+
|Default| None |
445+
|Effective| After restarting system |

‎src/UserGuide/V1.2.x/User-Manual/Tiered-Storage_timecho.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,11 @@ The specific parameter definitions and their descriptions are as follows.
3636

3737
| Configuration | Default | Description | Constraint |
3838
| ---------------------------------------- | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
39-
| dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
40-
| default_ttl_in_ms | None | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
39+
| dn_data_dirs | data/datanode/data | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
40+
| default_ttl_in_ms | -1 | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
4141
| dn_default_space_move_thresholds | 0.15 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
4242
| object_storage_type | AWS_S3 | Cloud Storage Type | IoTDB currently only supports AWS S3 as a remote storage type, and this parameter can't be modified |
43-
| object_storage_bucket | None | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
43+
| object_storage_bucket | iotdb_data | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
4444
| object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required |
4545
| object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 credential key；If remote storage is not used, no configuration required |
4646
| object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 credential secret；If remote storage is not used, no configuration required |

‎src/UserGuide/V1.3.0-2/Reference/Common-Config-Manual.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -640,11 +640,11 @@ Different configuration parameters take effect in the following three ways:
640640
* default\_ttl\_in\_ms
641641

642642
| Name | default\_ttl\_in\_ms |
643-
| :---------: | :------------------------------------- |
644-
| Description | Default ttl when each database created |
645-
| Type | Long |
646-
| Default | Infinity |
647-
| Effective | After restarting system |
643+
|:---:|:--------------|
644+
|Description| Define the maximum age of data for which each tier is responsible |
645+
|Type| long |
646+
|Default| -1 |
647+
|Effective| After restarting system |
648648

649649
* max\_waiting\_time\_when\_insert\_blocked
650650

0 commit comments

Comments
 (0)