v1.0.0-Beta3

[Jolanrensen]

We're one step closer to DataFrame 1.0! We've been working hard to iron out bugs and improve documentation and examples throughout.

Try out this release by updating to 1.0.0-Beta3 in your project or use %use dataframe in your notebook!

See below for a complete list of changes in this release grouped by category.

1.0.0-Beta3 Highlights

• NEW: Reading data from Parquet files
• NEW: Reading data from DuckDB databases
• Our KDocs and website have received many, many additions, as you can see below, but some that stand out are:
  • Docs about the compiler plugin: https://kotlin.github.io/dataframe/compiler-plugin.html
  • A setup guide with instructions for each supported platform: https://kotlin.github.io/dataframe/setup.html
  • (Un)supported Data sources overview: https://kotlin.github.io/dataframe/data-sources.html
  • DataFrame Concepts and Principles: https://kotlin.github.io/dataframe/concepts.html
  • Frequently Asked Questions: https://kotlin.github.io/dataframe/faq.html
  • Migration away from Gradle/KSP Plugin: https://kotlin.github.io/dataframe/migration-from-plugins.html
• New examples:
  • Using the compiler plugin in a project: https://github.com/Kotlin/dataframe/tree/master/examples/kotlin-dataframe-plugin-example
  • DataFrame on Android: https://github.com/Kotlin/dataframe/tree/master/examples/android-example
  • "Unsupported data sources" examples, including Exposed, Spark, Hibernate, and Multik: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/examples
  • DataFrame with Spark and Parquet files: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/spark-parquet-dataframe
• format to HTML has had several improvements and bug fixes; though, the API might have breaking changes.
• We continue to improve the compiler plugin to better reflect the schemas of dataframes in compile time. You can see the compile time schema by hovering over a function call in the IDE — the schema will be displayed in the popup. If there are any unexpected results, feel free to open an issue or contact us!

Known Issues

• IN NOTEBOOKS use version 1.0.0-Beta3n instead of 1.0.0-Beta3. This uses the patch of DefinitelyNotNullable patch for notebooks only #1435 for issue Notebooks fail with T : Comparable<T & Any>? notation #1116 avoiding DefinitelyNotNullable errors. This is the version that will be provided when writing %use dataframe. Check dataFrameConfig.version in your notebook to check which version you are using.
• median and percentile require explicit type arguments for non-numeric columns
  • median and percentile require explicit type arguments for non-numeric columns #1189
• FormattedFrame does not render formatted in notebooks
  • This has since been fixed but might require a yet to be released version of IntelliJ.
  • See Adds is_formatted to json encoding for DF #1405

Features

• DataFrameSchema.generateCode() by @Jolanrensen in DataFrameSchema.generateCode() #1230
• add nulls parse option in readExcel by @AndreiKingsley in add nulls parse option in readExcel #1247
• Kotlin 2.2 by @Jolanrensen in Kotlin 2.2 #1286
• Parse String to UUID Parse String to UUID #1006 by @EmmanuelBerkowicz in Parse String to UUID #1006 #1287
• Makes prev()?.newValue() work in Add DSL by @Jolanrensen in Makes prev()?.newValue() work in Add DSL #1303
• Added readOnly mode by @zaleslaw in Added readOnly mode #1325
• Add support for validating SQL queries with the WITH clause by @zaleslaw in Add support for validating SQL queries with the WITH clause  #1327
• Add deprecated filterNotNull for discoverability of dropNulls by @koperagen in Add deprecated filterNotNull for discoverability of dropNulls #1334
• parseExperimentalUuid in ParserOptions by @Jolanrensen in parseExperimentalUuid in ParserOptions #1306
• Add support for reading parquet file thanks to arrow-dataset Add support for reading Parquet files #576 by @fb64 in Add support for reading parquet file thanks to arrow-dataset #576 #577
• duckdb support by @Jolanrensen in duckdb support #1366
• Format colgroups by @Jolanrensen in Format colgroups #1374
• Add support for read-only mode in DuckDB and SQLite by @zaleslaw in Add support for read-only mode in DuckDB and SQLite #1383
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in migration from kotlinx.datetime.Instant -> kotlin.time.Instant #1368
• Make AnyFrame.renderToString public by @bwjohnson92 in Make AnyFrame.renderToString public #1395

Docs, Examples, and KDocs

• Website docs landing by @AndrewKis in Website docs landing #1170
• Website docs examples dev by @AndreiKingsley in Website docs examples dev #1179
• fix snippets iframes by @AndreiKingsley in fix snippets iframes #1190
• Website docs kotlin notebook by @AndreiKingsley in Website docs kotlin notebook #1180
• Generate docs by @AndreiKingsley in Generate docs #1194
• Rename WS docs by @AndreiKingsley in Rename WS docs #1191
• Website docs dev by @AndreiKingsley in Website docs dev #1196
• fix kodex iframes by @AndreiKingsley in fix kodex iframes #1201
• Number unification docs by @Jolanrensen in Number unification docs #1200
• Add initial compiler plugin documentation by @koperagen in Add initial compiler plugin documentation #1202
• Add few more pages about compiler plugin by @koperagen in Add few more pages about compiler plugin #1204
• Remove some of the outdated info from README.md by @koperagen in Remove some of the outdated info from README.md #1205
• Minor fixes for the docs by @koperagen in Minor fixes for the docs #1208
• Update README.md by @koperagen in Update README.md #1209
• Update Gradle plugin docs and fix hardcoded versions by @koperagen in Update Gradle plugin docs and fix hardcoded versions #1211
• Update documentation for "create dataframe" by @koperagen in Update documentation for "create dataframe" #1212
• Remove examples for deprecated overloads from tests by @koperagen in Remove examples for deprecated overloads from tests #1213
• Notebook updates by @Jolanrensen in Notebook updates #1224
• Adding links to column selectors in docs by @Jolanrensen in Adding links to column selectors in docs #1226
• fix: fixed German companies notebook sample by @kikoso in fix: fixed German companies notebook sample #1234
• fix kodex docs iframes by @AndreiKingsley in fix kodex docs iframes #1233
• fix escaping in iframes by @AndreiKingsley in fix escaping in iframes #1237
• add pictures for guides by @AndreiKingsley in add pictures for guides #1239
• various docs fixes to lower the number of errors by @Jolanrensen in various docs fixes to lower the number of errors #1238
• Plugin example by @AndreiKingsley in Plugin example #1241
• KTNB-1041: Fix resource loading by @ileasile in KTNB-1041: Fix resource loading #1250
• Extension properties docs by @AndreiKingsley in Extension properties docs #1246
• Readme update by @AndreiKingsley in Readme update #1251
• IDE sample of "unsupported sources"->DataFrame by @Jolanrensen in IDE sample of "unsupported sources"->DataFrame #1231
• add kdocs by @AndreiKingsley in add kdocs #1261
• Gather kdocs by @AndreiKingsley in Gather kdocs #1272
• Corr kdocs by @AndreiKingsley in Corr kdocs #1275
• filter kdocs by @AndreiKingsley in filter kdocs #1288
• columns selector type by @AndreiKingsley in columns selector type #1274
• explode docs and tests by @AndreiKingsley in explode docs and tests #1291
• improve concept docs structure by @AndreiKingsley in improve concept docs structure #1305
• Simplify compiler plugin setup in documentation by @koperagen in Simplify compiler plugin setup in documentation #1317
• modules topic by @AndreiKingsley in modules topic #1314
• Flatten navigation of Modify operations by @koperagen in Flatten navigation of Modify operations #1312
• faq topic by @AndreiKingsley in faq topic #1316
• Docs small fixes by @AndreiKingsley in Docs small fixes #1323
• generateCode.kt refactor, KDocs, documentation fixes, and tests by @Jolanrensen in generateCode.kt refactor, KDocs, documentation fixes, and tests #1311
• Data schemas docs by @AndreiKingsley in Data schemas docs #1339
• Data sources docs by @AndreiKingsley in Data sources docs #1352
• Setup docs by @AndreiKingsley in Setup docs #1357
• format kdocs++ by @Jolanrensen in format kdocs++ #1346
• count and countDistinct kdocs by @AndreiKingsley in count and countDistinct kdocs #1301
• clarify how to create BaseColumn by @koperagen in clarify how to create BaseColumn #1371
• add spelling conventions and name fixes in a whole project by @AndreiKingsley in add spelling conventions and name fixes in a whole project #1364
• android example by @AndreiKingsley in android example #1349
• Android example: removing arrow from the examples by @Jolanrensen in Android example: removing arrow from the examples #1426
• Add Hibernate example with H2 in-memory database to idea-examples by @zaleslaw in Add Hibernate example with H2 in-memory database to idea-examples #1367
• Extension property name generation fix & docs & tests by @AndreiKingsley in Extension property name generation fix & docs & tests #1380
• Add SQL-to-Kotlin DataFrame transition guide for backend developers by @zaleslaw in Add SQL-to-Kotlin DataFrame transition guide for backend developers #1377
• [Junie]: docs: update ColumnGroup description with relevant links by @jetbrains-junie[bot] in [Junie]: docs: update ColumnGroup description with relevant links #1404
• Added Spark-Parquet-KDF example by @zaleslaw in Added Spark-Parquet-KDF example #1391
• Rename kdocs by @AndreiKingsley in Rename kdocs #1402
• Insert kdocs by @AndreiKingsley in Insert kdocs #1406
• Util functions docs by @AndreiKingsley in Util functions docs #1414

Fixes

• Bump apache poi to 5.4.1 by @Badya in Bump apache poi to 5.4.1 #1198
• fixed compilation on android by @Jolanrensen in fixed compilation on android #1203
• Android compilation fix by @Jolanrensen in Android compilation fix #1218
• IDEA goodness by @Jolanrensen in IDEA goodness #1235
• Use DayOfWeek.isoDayNumber instead of DayOfWeek.value in DateTests.kt. by @chaoren in Use DayOfWeek.isoDayNumber instead of DayOfWeek.value in DateTests.kt. #1232
• Run configs by @AndreiKingsley in Run configs #1236
• Reusage schemas fix by @Jolanrensen in Reusage schemas fix #1252
• Remove mandatory mariadb dependency from jdbc by @koperagen in Remove mandatory mariadb dependency from jdbc #1267
• GroupBy Jupyter codegen by @Jolanrensen in GroupBy Jupyter codegen #1263
• Add more specific explodeLists overload that helps to avoid unnecessary cast by @koperagen in Add more specific explodeLists overload that helps to avoid unnecessary cast #1269
• Bump gradle to 8.14.2 by @Jolanrensen in Bump gradle to 8.14.2 #1285
• Removing recursively() remnants from public api by @Jolanrensen in Removing recursively() remnants from public api #1302
• Replace static initialization of defaultTimeZone with getter by @koperagen in Replace static initialization of defaultTimeZone with getter  #1315
• Improve SQL validation logic and add related test by @zaleslaw in Improve SQL validation logic and add related test #1324
• Migrate publishing by @koperagen in Migrate publishing #1331
• Hide mapToColumn in CS DSL in favor of expr by @koperagen in Hide mapToColumn in CS DSL in favor of expr #1360
• Adds tableTypes to DbType by @Jolanrensen in Adds tableTypes to DbType #1283
• format kdocs++ by @Jolanrensen in format kdocs++ #1346
• skipKodex gradle parameter in dataframe-csv by @koperagen in skipKodex gradle parameter in dataframe-csv #1370
• Fix jupyter logger by @AndreiKingsley in Fix jupyter logger #1361
• Initialize JetBrains Junie 🚀 by @jetbrains-junie[bot] in Initialize JetBrains Junie 🚀 #1379
• Fixed reading Parquet file on Windows by @zaleslaw in Fixed reading Parquet file on Windows #1381
• Extension property name generation fix & docs & tests by @AndreiKingsley in Extension property name generation fix & docs & tests #1380
• Migrate usages of deprecated API in implementation & warning fixes by @koperagen in Migrate usages of deprecated API in implementation & warning fixes #1382
• [Junie]: fix: update deprecation message for DataRow.get method by @jetbrains-junie[bot] in [Junie]: fix: update deprecation message for DataRow.get method #1399
• Removed deprecatedSchemaGeneratorPlugin Gradle plugin publication by @Jolanrensen in Removed deprecatedSchemaGeneratorPlugin Gradle plugin publication #1400
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in migration from kotlinx.datetime.Instant -> kotlin.time.Instant #1368
• Columngroup map fix by @AndreiKingsley in Columngroup map fix #1396
• Adds is_formatted to json encoding for DF by @Jolanrensen in Adds is_formatted to json encoding for DF #1405
• Util functions fixes by @AndreiKingsley in Util functions fixes #1412
• Add API that debugger renderer can rely on to avoid breaking changes by @koperagen in Add API that debugger renderer can rely on to avoid breaking changes #1421
• Update Gradle to 9.0 by @koperagen in Update Gradle to 9.0 #1428
• What's the version 1727 and 1548? Fixed by @Jolanrensen in What's the version 1727 and 1548? #436

Compiler Plugin

Note that many changes to the compiler plugin are made in Kotlin, as this is where the compiler plugin now lives.

• Prepare Split for compiler plugin support by @koperagen in Prepare Split for compiler plugin support #1197
• Prepare update and implode operations for compiler plugin support by @koperagen in Prepare update and implode operations for compiler plugin support #1207
• Deprecate predicate argument in single, colsInGroups, colsAtAnyDepth by @koperagen in Deprecate predicate argument in single, colsInGroups, colsAtAnyDepth #1176
• Add missed compiler plugin annotations for Split by @koperagen in Add missed compiler plugin annotations for Split #1262
• add compiler annotations for dropNA by @koperagen in add compiler annotations for dropNA #1330
• add missed compiler annotations for update by @koperagen in add missed compiler annotations for update #1333
• Add plugin annotations for a new batch of functions by @koperagen in Add plugin annotations for a new batch of functions #1362
• Rollback parameter name to keep compatibility with the compiler plugin by @koperagen in Rollback parameter name to keep compatibility with the compiler plugin #1358
• cols with default argument deprecation by @AndreiKingsley in cols with default argument deprecation #1369
• Add basic tests that API exposed for compiler plugin works without excluded dependencies by @koperagen in Add basic tests that API exposed for compiler plugin works without excluded dependencies #1397
• Remove annotations dependency from compiler-plugin-core by @koperagen in Remove annotations dependency from compiler-plugin-core #1409
• Add annotations to support types extending DataRow in the compiler plugin by @koperagen in Add annotations to support types extending DataRow in the compiler plugin #1410
• [Compiler plugin] Support more functions in toDataFrame DSL and AddDsl by @koperagen in [Compiler plugin] Support more functions in toDataFrame DSL and AddDsl #1425
• Prepare utility to assert compile time column order by @koperagen in Prepare utility to assert compile time column order #1427

New Contributors

• @Badya made their first contribution in Bump apache poi to 5.4.1 #1198
• @kikoso made their first contribution in fix: fixed German companies notebook sample #1234
• @chaoren made their first contribution in Use DayOfWeek.isoDayNumber instead of DayOfWeek.value in DateTests.kt. #1232
• @EmmanuelBerkowicz made their first contribution in Parse String to UUID #1006 #1287
• @jetbrains-junie[bot] made their first contribution in Initialize JetBrains Junie 🚀 #1379
• @bwjohnson92 made their first contribution in Make AnyFrame.renderToString public #1395

Full Changelog: v1.0.0-Beta2...v1.0.0-Beta3

---

This discussion was created from the release v1.0.0-Beta3.