Explicitly null check the stdlib #23566

hamzaremmal · 2025-07-18T19:19:42Z

No description provided.

hamzaremmal · 2025-07-18T19:20:07Z

@noti0na1 As agreed to, please push the changes to explicitly null check the stdlib here.

library/src/scala/collection/View.scala

sjrd · 2025-08-30T11:37:05Z

library/src/scala/collection/immutable/ArraySeq.scala

@@ -313,7 +313,7 @@ object ArraySeq extends StrictOptimizedClassTagSeqFactory[ArraySeq] { self =>
   * `ArraySeq.unsafeWrapArray(a.asInstanceOf[Array[Int]])` does not work, it throws a
   * `ClassCastException` at runtime.
   */
-  def unsafeWrapArray[T](x: Array[T]): ArraySeq[T] = ((x: @unchecked) match {
+  def unsafeWrapArray[T](x: Array[T] | Null): ArraySeq[T] | Null = ((x: @unchecked) match {


I wonder if we could pull something off with a universal match type and its mapNull helper.

scala> type MapNull[X, Y] <: Y | Null = X match { case Null => Null; case _ => Y } scala> inline def mapNull[A, B](a: A | Null, inline f: A => B): MapNull[a.type, B] = (if a == null then null else f(a)).asInstanceOf[MapNull[a.type, B]] def mapNull[A, B](a: A | Null, f: A => B): MapNull[a.type, B] scala> def foo(x: String | Null): MapNull[x.type, Array[Char]] = mapNull(x, _.toCharArray()) def foo(x: String | Null): MapNull[x.type, Array[Char]] scala> foo("hello"): Array[Char] val res7: Array[Char] = Array(h, e, l, l, o) scala> foo(null): Null val res8: Null = null scala> var s: String | Null = "hello" var s: String | Null = hello scala> foo(s): Array[Char] -- [E007] Type Mismatch Error: ------------------------------------------------- 1 |foo(s): Array[Char] |^^^^^^ |Found: MapNull[(?1 : String | Null), Array[Char]] |Required: Array[Char] | |where: ?1 is an unknown value of type String | Null | | |Note: a match type could not be fully reduced: | | trying to reduce MapNull[(?1 : String | Null), Array[Char]] | failed since selector (?1 : String | Null) | does not match case Null => Null | and cannot be shown to be disjoint from it either. | Therefore, reduction cannot advance to the remaining case | | case _ => Array[Char] | | longer explanation available when compiling with `-explain` 1 error found scala> foo(s): Array[Char] | Null val res9: Array[Char] | Null = Array(h, e, l, l, o)

Note that the signature of foo is specially crafted so that its TASTy Signature and its binary descriptor are the same as if it were defined def foo(s: String | Null): Array[Char] | Null.

It's not bullet-proof. Sometimes, use site inference gets in the way, as in:

scala> foo(s): Array[Char] | Null val res9: Array[Char] | Null = Array(h, e, l, l, o) scala> Array(foo(s)) -- [E172] Type Error: ---------------------------------------------------------- 1 |Array(foo(s)) | ^ | No ClassTag available for MapNull[(?1 : String | Null), Array[Char]] | | where: ?1 is an unknown value of type String | Null | | | Note: a match type could not be fully reduced: | | trying to reduce MapNull[(?1 : String | Null), Array[Char]] | failed since selector (?1 : String | Null) | does not match case Null => Null | and cannot be shown to be disjoint from it either. | Therefore, reduction cannot advance to the remaining case | | case _ => Array[Char] 1 error found scala> Array(foo("hello")) val res10: Array[Array[Char]] = Array(Array(h, e, l, l, o))

However, that's only for the nullable argument case, and in some use site scenarios. It might be worth exploring.

This is an interesting idea. I will try this and see how it looks.

noti0na1 · 2025-09-01T14:58:26Z

@hamzaremmal Do you know how the stdlib is compiled at the first step? When I do scalac, it seems compiling my new code using the old library.

library/src/scala/collection/convert/AsJavaConverters.scala

When we compute `afterPatternContext`, the wrong `ctx` is passed to the function. ```scala def f(s: AnyRef | Null) = s match case null => 0 case s: String => s.length case _ => val a: AnyRef = s a.toString.length ``` Will be useful for #23566

olhotak · 2025-09-08T00:30:42Z

library/src/scala/Predef.scala

 }

 private[scala] abstract class LowPriorityImplicits2 {
  @deprecated("implicit conversions from Array to immutable.IndexedSeq are implemented by copying; use `toIndexedSeq` explicitly if you want to copy, or use the more efficient non-copying ArraySeq.unsafeWrapArray", since="2.13.0")
  implicit def copyArrayToImmutableIndexedSeq[T](xs: Array[T]): IndexedSeq[T] =
-    if (xs eq null) null
+    if (xs eq null) null.asInstanceOf[IndexedSeq[T]]


Why does this not use mapNull?

I can move the mapNull definition to Predef then I can use it.

olhotak · 2025-09-08T00:46:16Z

library/src/scala/collection/convert/AsScalaConverters.scala

@@ -41,10 +41,10 @@ trait AsScalaConverters {
   * @param i The Java `Iterator` to be converted.
   * @return  A Scala `Iterator` view of the argument.
   */
-  def asScala[A](i: ju.Iterator[A]): Iterator[A] = i match {
+  def asScala[A](i: ju.Iterator[A] | Null): Iterator[A] | Null = i match {


I wonder if we could do some kind of trick like mapNull, with match types, to give asScala a null-polymorphic signature, so that it returns a non-null when you pass it a non-null. That would reduce the need for many .nns not only within the library itself, but probably also in code that uses the library.

I would like to add a match type as shown in @sjrd 's comment. But this means the new type would show up at the type signature, and adding a new public definition to stdlib is impossible at this point...

Maybe I should just forbid null like the wrappers wrapRefArray

Forbidding nulls would break the current contract. That's a big no-no.

I also agree that we shouldn't introduce the match type I suggested at this time. We can reconsider in a future release, as we get closer to general adoption of explicit nulls.

I would argue for non-explicit-nulls users, forbidding nulls will not change any contract. It is just a more strict type for explicit-nulls.

It would make the body inconsistent with its typing. Constant-folding in the compiler could for example mis-"optimize" x == null as false because the type of x is not nullable. So even non-explicit-nulls users could get affected, indirectly.

In general I don't think the types under explicit-nulls should be any more restrictive than the existing (sometimes tacit 🤷‍♂️) contract. Typing should better describe the contract; not make the contract stricter.

I think we have to make a decision here. Of course we will ensure the behaviour of the body will not change, no matter passing null or not. Take asScala as an example:

Keep the original type: def asScala[A](i: ju.Iterator[A]): Iterator[A]. No effect for non-explicit-nulls in terms of typing. Not able to pass null in explicit nulls, more convenient to have a chain of collection operations.

Change the type to: def asScala[A](i: ju.Iterator[A] | Null): Iterator[A] | Null. Still no effect for non-explicit-nulls. More precise to the original behaviour and document. Have to add .nn at more places in explicit-nulls.

After experimenting with the wrapper, I personally think 1 is batter choice for the library.

Since we disagree, I added this question as an agenda item for tomorrow's core meeting.

library/src/scala/collection/immutable/RedBlackTree.scala

library/src/scala/collection/mutable/AnyRefMap.scala

library/src/scala/collection/mutable/ListBuffer.scala

olhotak · 2025-09-08T01:17:17Z

library/src/scala/math/BigDecimal.scala

@@ -305,7 +305,7 @@ object BigDecimal {
  implicit def double2bigDecimal(d: Double): BigDecimal = decimal(d)

  /** Implicit conversion from `java.math.BigDecimal` to `scala.BigDecimal`. */
-  implicit def javaBigDecimal2bigDecimal(x: BigDec): BigDecimal = if (x == null) null else apply(x)
+  implicit def javaBigDecimal2bigDecimal(x: BigDec | Null): BigDecimal | Null = if (x == null) null else apply(x)


Same comment as above for asScala.

library/src/scala/util/Properties.scala

library/src/scala/util/Using.scala

noti0na1 · 2025-09-15T18:17:52Z

Why does the MiMa check a java class?

Error:  scala-library-nonbootstrapped: Failed binary compatibility check against org.scala-lang:fat-stdlib:3.7.3! Found 2 potential problems (filtered 5132)
Error:   * abstract method get(java.lang.Object)java.lang.Object in class java.util.Dictionary does not have a correspondent in current version
Error:     filter with: ProblemFilters.exclude[DirectAbstractMethodProblem]("java.util.Dictionary.get")
Error:   * abstract method remove(java.lang.Object)java.lang.Object in class java.util.Dictionary does not have a correspondent in current version
Error:     filter with: ProblemFilters.exclude[DirectAbstractMethodProblem]("java.util.Dictionary.remove")
Error:  
Error:  Filters in MiMaFilters.Scala3Library are used in this check.

sjrd · 2025-09-15T18:40:02Z

Why should it not? Java classes are also part of the ABI of our artifacts.

Even tasty-mima checks Java classes!

noti0na1 · 2025-09-15T18:42:10Z

Why should it not? Java classes are also part of the ABI of our artifacts.

Oh, I thought we only check scala code.

noti0na1 · 2025-09-16T12:29:10Z

I added java.util.Dictionary to MiMaFilter temporarily, and the check passed.

So I guess at least the stdlib itself is good?

noti0na1 · 2025-09-18T12:44:11Z

Strange tailrac errors only during scala-library-sjs/compile:

[info] compiling 650 Scala sources and 55 Java sources to /Users/Work/dotty/library-js/target/scala-library/classes ...
[error] -- Error: /Users/Work/dotty/library/src/scala/collection/concurrent/TrieMap.scala:60:26 
[error] 60 |        else GCAS_Complete(/*READ*/mainnode, ct)
[error]    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |             Cannot rewrite recursive call: it is not in tail position
[error] -- Error: /Users/Work/dotty/library/src/scala/collection/concurrent/TrieMap.scala:73:28 
[error] 73 |          else GCAS_Complete(m, ct)
[error]    |               ^^^^^^^^^^^^^^^^^^^^
[error]    |               Cannot rewrite recursive call: it is not in tail position
[error] -- Error: /Users/Work/dotty/library/src/scala/collection/concurrent/TrieMap.scala:77:23 
[error] 77 |          GCAS_Complete(/*READ*/mainnode, ct)
[error]    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |          Cannot rewrite recursive call: it is not in tail position
[error] -- Error: /Users/Work/dotty/library/src/scala/collection/concurrent/TrieMap.scala:179:53 
[error] 179 |              if (startgen eq in.gen) in.rec_insertif(k, v, hc, cond, fullEquals, lev + 5, this, startgen, ct)
[error]     |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]     |             Cannot rewrite recursive call: it is not in tail position
[error] -- Error: /Users/Work/dotty/library/src/scala/collection/concurrent/TrieMap.scala:181:72 
[error] 181 |                if (GCAS(cn, cn.renewed(startgen, ct), ct)) rec_insertif(k, v, hc, cond, fullEquals, lev, parent, startgen, ct)
[error]     |                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]     |             Cannot rewrite recursive call: it is not in tail position
[error] -- Error: /Users/Work/dotty/library/src/scala/runtime/MethodCache.scala:73:33 
[error] 73 |      case x: PolyMethodCache => x findInternal forReceiver
[error]    |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |               Cannot rewrite recursive call: it is not in tail position

The same files was fine when only compiling the stdlib.

noti0na1 · 2025-09-18T13:57:42Z

Strange tailrac errors only during scala-library-sjs/compile:

It seems there are extra asInstanceOf inserted after the recursive calls, which prevents the tailrec rewriting.
Not sure if this is caused by interaction with union types, or some special handling in scalajs.

hamzaremmal assigned noti0na1 Jul 18, 2025

noti0na1 force-pushed the explicit-nulls-stdlib branch from 2fe2627 to 2d178a7 Compare August 26, 2025 14:17

sjrd reviewed Aug 30, 2025

View reviewed changes

noti0na1 mentioned this pull request Aug 31, 2025

[DO NOT MERGE] Try to enable explicit nulls for Scala 3 stdlib #23319

Closed

noti0na1 marked this pull request as ready for review September 1, 2025 12:20

noti0na1 requested a review from a team as a code owner September 1, 2025 12:20

noti0na1 force-pushed the explicit-nulls-stdlib branch from cc174ce to 99df51b Compare September 1, 2025 14:49

noti0na1 requested a review from Copilot September 2, 2025 12:04

This comment was marked as outdated.

Sign in to view

noti0na1 mentioned this pull request Sep 2, 2025

Fix NotNullInfo in multi-case match #23864

Merged

noti0na1 reviewed Sep 2, 2025

View reviewed changes

library/src/scala/collection/convert/AsJavaConverters.scala Outdated Show resolved Hide resolved

olhotak reviewed Sep 8, 2025

View reviewed changes

noti0na1 force-pushed the explicit-nulls-stdlib branch from 216877c to 2a854a1 Compare September 8, 2025 12:47

hamzaremmal force-pushed the explicit-nulls-stdlib branch from 5b251c0 to 747eb65 Compare September 15, 2025 16:30

noti0na1 mentioned this pull request Sep 17, 2025

-Yexplicit-nulls breaks obtaining quoted.Type #23937

Open

hamzaremmal and others added 8 commits September 18, 2025 13:26

chore: enable explicit-nulls in the stdlib

4022588

Port stdlib to explicit nulls

7a18397

Adjust some type parameters

0390729

More changes

fe71a71

Migrate RBT

bb723d3

Another RBT

e6bf63a

Update TrieMap

f45a002

Fix some cc errors

6abcdcc

noti0na1 added 14 commits September 18, 2025 13:26

Fix remaining cc errors

0117bfe

Clean some unnecessary .nn

737e814

Removing remaining unnecessary .nn

603e310

Fix crush

3119223

Always apply asInstanceOf with arguments

1a90f87

Refine .nn usages

1fbb084

Address some review comments

29c06ad

Make wrappers and convertors non-nullable

dea3d92

Fix NPEs revealed by tests

da2d0c5

Fix a missing comment

d8f6684

Remove some unnecessary .nn

f2ef977

Try to migrate sjs library

a4e9f5f

Update tests about Regex

7a4f257

Update required type signatures in sjs

582c5c5

noti0na1 force-pushed the explicit-nulls-stdlib branch from a22fa25 to 582c5c5 Compare September 18, 2025 12:24

noti0na1 requested review from Linyxus and natsukagami September 19, 2025 11:35

Explicitly null check the stdlib #23566

Are you sure you want to change the base?

Explicitly null check the stdlib #23566

Conversation

hamzaremmal commented Jul 18, 2025 • edited by noti0na1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hamzaremmal commented Jul 18, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noti0na1 commented Sep 1, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

noti0na1 commented Sep 15, 2025

Uh oh!

sjrd commented Sep 15, 2025

Uh oh!

noti0na1 commented Sep 15, 2025

Uh oh!

noti0na1 commented Sep 16, 2025

Uh oh!

noti0na1 commented Sep 18, 2025

Uh oh!

noti0na1 commented Sep 18, 2025

Uh oh!

Uh oh!

hamzaremmal commented Jul 18, 2025 •

edited by noti0na1

Loading