Initialize python containers with fixed size

[LecrisUT]

I would like to pre-allocate the sizes of the python native objects. With ndarray this is partially possible by specifying nb::shape<>, but if you have a dynamic size that you want to initialize, what then? And what about nb::list? Am I missing something in the interface?

[wjakob]

ndarray does not allocate new memory, it only wraps memory by you already. This is by design -- nanobind, e.g., does not want to interface with CUDA to be able to do allocations for the GPU.

The tricky thing with initialization of nb::list and nb::tuple with a fixed size is that those are invalid objects that will crash the Python interpreter when used there unless you have initialized all objects. So therefore this is not exposed. But of course you can still drop down the CPython API to make an array and then wrap the result with nb::steal<nb::list>(...).

[LecrisUT]

ndarray does not allocate new memory, it only wraps memory by you already. This is by design -- nanobind, e.g., does not want to interface with CUDA to be able to do allocations for the GPU.

Yeah, that's another issue, on how would we do the initialization. I am still trying to wrap my head around the documentation. I believe I could just add a new double[a][b] to the constructor, but I haven't checked if that would be ok with the ownership. Then there are several nuances of how to create the array for a specific backend, do operations like reshape, CPU/GPU differences etc.

The tricky thing with initialization of nb::list and nb::tuple with a fixed size is that those are invalid objects that will crash the Python interpreter when used there unless you have initialized all objects.

What if it was pre-populated with None or any value that we specify during the construction?

But of course you can still drop down the CPython API

I would very much want to avoid that. The main motivation for the initialization is to to avoid any resizing overhead, and have a cleaner transition from pybind11 to nanobind. Admittedly the overhead of nb::list would be small, but still it's something that could be more easily parallelized.