Skip to content

Commit 8b20432

Browse files
W1y1rW1y1r
committed
Add Tiered Storage configuration parameters
1 parent efe1a8e commit 8b20432

File tree

8 files changed

+566
-12
lines changed

8 files changed

+566
-12
lines changed

‎src/UserGuide/Master/Reference/DataNode-Config-Manual.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -497,3 +497,187 @@ There will be at most 10 gc.log.* files and each one can reach to 10MB.
497497
|Type| int32 |
498498
|Default| 5000 |
499499
|Effective| After restarting system |
500+
501+
502+
503+
#### Tiered Storage Configuration
504+
505+
* tier\_ttl\_in\_ms
506+
507+
|Name| tier\_ttl\_in\_ms |
508+
|:---:|:--------------|
509+
|Description| Define the maximum age of data for which each tier is responsible |
510+
|Type| long |
511+
|Default| None |
512+
|Effective| After restarting system |
513+
514+
* dn\_default\_space\_usage\_thresholds
515+
516+
|Name| dn\_default\_space\_usage\_thresholds |
517+
|:---:|:--------------|
518+
|Description| Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY |
519+
|Type| double |
520+
|Default| 0.15 |
521+
|Effective| hot-load |
522+
523+
* remote\_tsfile\_cache\_dirs
524+
525+
|Name| remote\_tsfile\_cache\_dirs |
526+
|:---:|:--------------|
527+
|Description| Cache directory stored locally in the cloud |
528+
|Type| string |
529+
|Default| data/datanode/data/cache |
530+
|Effective| After restarting system |
531+
532+
* remote\_tsfile\_cache\_page\_size\_in\_kb
533+
534+
|Name| remote\_tsfile\_cache\_page\_size\_in\_kb |
535+
|:---:|:--------------|
536+
|Description| Block size of locally cached files stored in the cloud |
537+
|Type| int |
538+
|Default| 20480 |
539+
|Effective| After restarting system |
540+
541+
* remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb
542+
543+
|Name| remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb |
544+
|:---:|:--------------|
545+
|Description| Maximum Disk Occupancy Size for Cloud Storage Local Cache |
546+
|Type| long |
547+
|Default| 51200 |
548+
|Effective| After restarting system |
549+
550+
* object\_storage\_type
551+
552+
|Name| object\_storage\_type |
553+
|:---:|:--------------|
554+
|Description| Cloud Storage Type |
555+
|Type| string |
556+
|Default| AWS_S3 |
557+
|Effective| After restarting system |
558+
559+
* object\_storage\_bucket
560+
561+
|Name| object\_storage\_bucket |
562+
|:---:|:--------------|
563+
|Description| Name of cloud storage bucket |
564+
|Type| string |
565+
|Default| None |
566+
|Effective| After restarting system |
567+
568+
* object\_storage\_endpoiont
569+
570+
|Name| object\_storage\_endpoiont |
571+
|:---:|:--------------|
572+
|Description| endpoint of cloud storage |
573+
|Type| string |
574+
|Default| None |
575+
|Effective| After restarting system |
576+
577+
* object\_storage\_access\_key
578+
579+
|Name| object\_storage\_access\_key |
580+
|:---:|:--------------|
581+
|Description| Authentication information stored in the cloud: key |
582+
|Type| string |
583+
|Default| None |
584+
|Effective| After restarting system |
585+
586+
* object\_storage\_access\_secret
587+
588+
|Name| object\_storage\_access\_secret |
589+
|:---:|:--------------|
590+
|Description| Authentication information stored in the cloud: secret |
591+
|Type| string |
592+
|Default| None |
593+
|Effective| After restarting system |
594+
595+
* tier\_ttl\_in\_ms
596+
597+
|Name| tier\_ttl\_in\_ms |
598+
|:---:|:--------------|
599+
|Description| Define the maximum age of data for which each tier is responsible |
600+
|Type| long |
601+
|Default| None |
602+
|Effective| After restarting system |
603+
604+
* dn\_default\_space\_usage\_thresholds
605+
606+
|Name| dn\_default\_space\_usage\_thresholds |
607+
|:---:|:--------------|
608+
|Description| Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY |
609+
|Type| double |
610+
|Default| 0.15 |
611+
|Effective| hot-load |
612+
613+
* remote\_tsfile\_cache\_dirs
614+
615+
|Name| remote\_tsfile\_cache\_dirs |
616+
|:---:|:--------------|
617+
|Description| Cache directory stored locally in the cloud |
618+
|Type| string |
619+
|Default| data/datanode/data/cache |
620+
|Effective| After restarting system |
621+
622+
* remote\_tsfile\_cache\_page\_size\_in\_kb
623+
624+
|Name| remote\_tsfile\_cache\_page\_size\_in\_kb |
625+
|:---:|:--------------|
626+
|Description| Block size of locally cached files stored in the cloud |
627+
|Type| int |
628+
|Default| 20480 |
629+
|Effective| After restarting system |
630+
631+
* remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb
632+
633+
|Name| remote\_tsfile\_cache\_max\_disk\_usage\_in\_mb |
634+
|:---:|:--------------|
635+
|Description| Maximum Disk Occupancy Size for Cloud Storage Local Cache |
636+
|Type| long |
637+
|Default| 51200 |
638+
|Effective| After restarting system |
639+
640+
* object\_storage\_type
641+
642+
|Name| object\_storage\_type |
643+
|:---:|:--------------|
644+
|Description| Cloud Storage Type |
645+
|Type| string |
646+
|Default| AWS_S3 |
647+
|Effective| After restarting system |
648+
649+
* object\_storage\_bucket
650+
651+
|Name| object\_storage\_bucket |
652+
|:---:|:--------------|
653+
|Description| Name of cloud storage bucket |
654+
|Type| string |
655+
|Default| None |
656+
|Effective| After restarting system |
657+
658+
* object\_storage\_endpoiont
659+
660+
|Name| object\_storage\_endpoiont |
661+
|:---:|:--------------|
662+
|Description| endpoint of cloud storage |
663+
|Type| string |
664+
|Default| None |
665+
|Effective| After restarting system |
666+
667+
* object\_storage\_access\_key
668+
669+
|Name| object\_storage\_access\_key |
670+
|:---:|:--------------|
671+
|Description| Authentication information stored in the cloud: key |
672+
|Type| string |
673+
|Default| None |
674+
|Effective| After restarting system |
675+
676+
* object\_storage\_access\_secret
677+
678+
|Name| object\_storage\_access\_secret |
679+
|:---:|:--------------|
680+
|Description| Authentication information stored in the cloud: secret |
681+
|Type| string |
682+
|Default| None |
683+
|Effective| After restarting system |

‎src/UserGuide/Master/User-Manual/Tiered-Storage_timecho.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ The specific parameter definitions and their descriptions are as follows.
3838
| ---------------------------------------- | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
3939
| dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
4040
| tier_ttl_in_ms | None | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs；"-1" means "unlimited". |
41-
| dn_default_space_move_thresholds | 0.15 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
41+
| dn_default_space_usage_thresholds | 0.15 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
4242
| object_storage_type | AWS_S3 | Cloud Storage Type | IoTDB currently only supports AWS S3 as a remote storage type, and this parameter can't be modified |
4343
| object_storage_bucket | None | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
4444
| object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required |
@@ -56,7 +56,7 @@ The following is an example of a local two-level storage configuration.
5656
//Required configuration items
5757
dn_data_dirs=/data1/data;/data2/data,/data3/data;
5858
tier_ttl_in_ms=86400000;-1
59-
dn_default_space_move_thresholds=0.2;0.1
59+
dn_default_space_usage_thresholds=0.2;0.1
6060
```
6161

6262
In this example, two levels of storage are configured, specifically:
@@ -74,7 +74,7 @@ The following takes three-level storage as an example:
7474
//Required configuration items
7575
dn_data_dirs=/data1/data;/data2/data,/data3/data;OBJECT_STORAGE
7676
tier_ttl_in_ms=86400000;864000000;-1
77-
dn_default_space_move_thresholds=0.2;0.15;0.1
77+
dn_default_space_usage_thresholds=0.2;0.15;0.1
7878
object_storage_name=AWS_S3
7979
object_storage_bucket=iotdb
8080
object_storage_endpoiont=<your_endpoint>

0 commit comments

Comments
 (0)