INTERNAL Error: DuckLakeCompactor: Files have different hive partition path

[patrickdevivo]

What happens?

Encountering this error in production when programmatically running ducklake_merge_adjacent_files on a table partitioned by year(ts), month(ts), and day(ts). Using the Neo Node.js client. What's odd, is that running the same command from a duckdb CLI session I don't encounter the error.

CALL ducklake_merge_adjacent_files('my_ducklake', 'my_table');

INTERNAL Error: DuckLakeCompactor: Files have different hive partition path

Stack Trace:

/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0xb01606) [0x7fb3ce035606]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0xb016b4) [0x7fb3ce0356b4]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0xb03711) [0x7fb3ce037711]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0x1d4471) [0x7fb3cd708471]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(_ZN6duckdb17DuckLakeCompactor19GenerateCompactionsERNS_18DuckLakeTableEntryERNS_6vectorINS_10unique_ptrINS_15LogicalOperatorESt14default_deleteIS5_ELb1EEELb1EEE+0xc0c) [0x7fb3cd9f880c]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0x4c4f6a) [0x7fb3cd9f8f6a]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(_ZN6duckdb14BindCompactionERNS_13ClientContextERNS_22TableFunctionBindInputEmNS_14CompactionTypeE+0x543) [0x7fb3cd9f94c3]
/root/.duckdb/extensions/v1.4.1/linux_amd64/ducklake.duckdb_extension(+0x4c5a08) [0x7fb3cd9f9a08]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder25BindTableFunctionInternalERNS_13TableFunctionERKNS_16TableFunctionRefENS_6vectorINS_5ValueELb1EEESt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_NS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIKSF_S7_EEENS6_INS_11LogicalTypeELb1EEENS6_ISF_Lb1EEE+0x17e) [0x7fb3f166de1e]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_16TableFunctionRefE+0x85c) [0x7fb3f1677bac]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_8TableRefE+0x128) [0x7fb3f16cc4c8]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder8BindNodeERNS_10SelectNodeE+0x3b) [0x7fb3f1612e3b]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder8BindNodeERNS_9QueryNodeE+0x9e) [0x7fb3f16ccaee]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_9QueryNodeE+0x2f) [0x7fb3f16ccb4f]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_15SelectStatementE+0x3a) [0x7fb3f161dcfa]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_13CallStatementE+0x1c6) [0x7fb3f161ebb6]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb6Binder4BindERNS_12SQLStatementE+0xf5) [0x7fb3f16cd355]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb7Planner10CreatePlanERNS_12SQLStatementE+0xa5) [0x7fb3f16d9f35]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext31CreatePreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEE+0x27e) [0x7fb3f248f56e]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext23CreatePreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEENS_21PreparedStatementModeE+0x1a1) [0x7fb3f2490141]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext24PendingStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERKNS_22PendingQueryParametersE+0x9c) [0x7fb3f249617c]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext35PendingStatementOrPreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0xa7) [0x7fb3f249a787]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext43PendingStatementOrPreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0x7f) [0x7fb3f249a99f]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS4_ELb1EEERKNS_22PendingQueryParametersEb+0xa9) [0x7fb3f249b8c9]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb13ClientContext5QueryERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x468) [0x7fb3f249c0f8]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(_ZN6duckdb10Connection5QueryERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f) [0x7fb3f249c76f]
/app/node_modules/@duckdb/node-bindings-linux-x64/libduckdb.so(duckdb_query+0x7c) [0x7fb3f23af46c]
/app/node_modules/@duckdb/node-bindings-linux-x64/duckdb.node(_ZN11QueryWorker7ExecuteEv+0x5e) [0x7fb43a65cefe]
/app/node_modules/@duckdb/node-bindings-linux-x64/duckdb.node(_ZN4Napi11AsyncWorker18OnAsyncWorkExecuteEP10napi_env__Pv+0x24) [0x7fb43a65d444]
bun() [0x2f48cdb]

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors

To Reproduce

To be honest, not sure how to exactly produce. I'm running this via the Neo Node.js client SDK against a partitioned DuckLake table.

OS:

x86_64

DuckDB Version:

1.4

DuckLake Version:

1.4

DuckDB Client:

Nodejs (1.4.1-r.5)

Hardware:

No response

Full Name:

Patrick DeVivo

Affiliation:

Powerset

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[guillesd]

hi @patrickdevivo I remember we fixed a similar issue to this one. Could you check what commit hash of DuckLake are you actually running?

SELECT * FROM duckdb_extensions WHERE installed

[patrickdevivo]

hi @guillesd sure, looks like it's f134ad8

[guillesd]

#496 this got in right after! You should consider updating the DuckLake extensions (FORCE INSTALL ducklake) or more gerally UPDATE EXTENSIONS;

Let me know if this works!

[patrickdevivo]

Hi @guillesd - thanks for the help! I ended up upgrading the duckdb client version in my application (https://duckdb.org/docs/stable/clients/node_neo/overview) and it reported back the ducklake version it's using is now at 77f2512. After this upgrade, I haven't encountered the error, so I think that's resolved it (though I have not actually confirmed that 77f2512 is a ducklake version that includes the fix you mentioned).

Closing this out, thank you!