Merge exception is occured when locale TSFile to HDFS.

[chariswang188996]

I've tried to locale the TSFiles to HDFS, the config as below:

tsfile_storage_fs=HDFS
core_site_path=/home/hadoop-2.7.7/etc/hadoop/core-site.xml
hdfs_site_path=/home/hadoop-2.7.7/etc/hadoop/hdfs-site.xml
hdfs_ip=IP
hdfs_port=port

The exception was :
2021-08-10 14:23:28,262 [pool-4-IoTDB-Compaction-10] ERROR o.a.i.d.e.c.l.LevelCompactionTsFileManagement:740 - Error occurred in Compaction Merge thread java.lang.UnsupportedOperationException: Unsupported operation. at org.apache.iotdb.hadoop.fileSystem.HDFSFile.getParent(HDFSFile.java:293) at org.apache.iotdb.db.engine.storagegroup.TsFileResource.modifyTsFileNameMergeCnt(TsFileResource.java:822) at org.apache.iotdb.db.engine.compaction.level.LevelCompactionTsFileManagement.merge(LevelCompactionTsFileManagement.java:668) at org.apache.iotdb.db.engine.compaction.level.LevelCompactionTsFileManagement.merge(LevelCompactionTsFileManagement.java:596) at org.apache.iotdb.db.engine.compaction.TsFileManagement$CompactionMergeTask.call(TsFileManagement.java:172) at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.syncCompactOnePartition(StorageGroupProcessor.java:2003) at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.access$300(StorageGroupProcessor.java:137) at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor$CompactionAllPartitionTask.call(StorageGroupProcessor.java:527) at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor$CompactionAllPartitionTask.call(StorageGroupProcessor.java:514) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-08-10 14:23:28,262 [pool-4-IoTDB-Compaction-10] INFO o.a.i.d.e.c.l.LevelCompactionTsFileManagement:744 - root.vehicle [Compaction] merge end time isSeq = true, consumption: 928 ms

[HTHou]

Hi, the compaction of IoTDB doesn't adapt the HDFS yet. Please close the compaction by setting compaction_strategy=NO_COMPACTION and enable_unseq_compaction=false when using HDFS.

[chariswang188996]

I'll try this right away, thanks!

[cigarl]

If used these parameters in a production environment, are there any problems?
For example, queries will be slow, too many files and so on.

[HTHou]

If used these parameters in a production environment, are there any problems?
For example, queries will be slow, too many files and so on.

Yes, if turn off the compaction, queries will be slow as you say.
But on HDFS, the compaction cannot work correctly now. Closing the compaction is the better solution.
For other file system, we recommend to keep the compaction working for better query performance.

[cigarl]

If used these parameters in a production environment, are there any problems?
For example, queries will be slow, too many files and so on.

Yes, if turn off the compaction, queries will be slow as you say.
But on HDFS, the compaction cannot work correctly now. Closing the compaction is the better solution.
For other file system, we recommend to keep the compaction working for better query performance.

Thank you so much.

By the way,I suggest that when users need to modify these parameters, we alert them to the risks and avoid unexpected problems in the production environment.

After all, it's our suggestion for this change : )