Not clear how to expose existing C++ vector as numpy array

[yesint]

This is a question of documentation rather than an issue. I can't find any example of the following very common scenario:

std::vector<int> some_func();
...
// We want to expose returned std::vector as a numpy array without copying
m.def("some_func", []() -> py::array {
   auto data = some_func();
   // What to do with data?? Map it with Eigen (then return what?), wrap somehow with py::buffer (how?)
})

I don't know the answer. It would be very nice to have this explained in docs since this scenario if rather common.

[YannickJadoul]

py::array will automatically copy your data if you don't give it a base argument in the constructor (though maybe that's indeed not very well documented).

If you don't want to copy, one solution would be to move the std::vector into a py::capsule and use that capsule as the base for a new py::array, then just continue using the v.data() of that moved vector to construct. If I'm not mistaken, the returned py::array will then keep that capsule alive and delete the vector once capsule can be and is garbage collected.

[YannickJadoul]

Untested code, but this should be the implementation of the non-copy approach:

auto v = new std::vector<int>(some_func());
auto capsule = py::capsule(v, [](void *v) { delete reinterpret_cast<std::vector<int>*>(v); });
return py::array(v->size(), v->data(), capsule);

Yes, probably more black magic than you might expect. But then again, you're not doing something simple either. You are keeping a C++ object alive to make sure you can access its internal data safely, without leaking the memory.

But if you don't mind the copy, just go:

auto v = some_func();
return py::array(v.size(), v.data());

[yesint]

@YannickJadoul, thank you very it really works! I don't mind doing black magic (and the magic is in fact quite logical) but currently the user is not even aware that this kind of magic exists.
Are there any plans to document the usage of py::array and py::capsule? The constructors of these types are non-trivial and usage of base argument is, well, a bit arcane.

[yesint]

Another suggestion. Probably it makes sense to provide an easy non-copying conversion from any contiguous buffer to py::arrray? Something like:

auto v = new std::vector<int>(some_func());
py::array array_from_buffer<int>(v, int ndim, shape, strides);

which will create corresponding py::buffer_info and capsule internally?
Could be a great addition in the cases when numerical data have to be returned, especially if one needs to wrap the function like:

void some_func(vector<int>& val1, vector<vector<float>>& val2);

Manual wrapping of each argument with py::buffer, py::capsule into the py::array becomes tedious in such cases.

[YannickJadoul]

but currently the user is not even aware that this kind of magic exists.

Agreed, I had to look into the actual headers to check the exact constructors, etc. But I don't know about planned documentation updates. If you feel like it, I'm sure a PR with more documentation on this would be gladly accepted ;-) Then again, I'm not always sure what's stable API and what're implementation details.

[YannickJadoul]

Probably it makes sense to provide an easy non-copying conversion from any contiguous buffer to py::arrray?

Not sure how easy that is to do (and how much more confusing this will make the whole situation). Maybe some kind of a static function as 'named constructor' could make sense, though?

By the way, std::vector<std::vector<int>> is not a contiguous structure. And I don't think this technique works when (un)wrapping the arguments of a function. What I just described was a way of not copying a return std::vector.

[yesint]

Sure, it won't work with "input" function parameters but works for "output" when one transforms c++ signature to python function returning tuple of numpy arrays instead of bunch of ref parameters (tht's exactly my case). In any case such thing should not be automatic - the user have to make it explicit in lambda.

Vector is indeed not contigous, sorry. But for example vectorEigen::Vector3f is contigous and could be returned efficiently as 2d array. Such funny structures are common when dealing with variable number of space points when Eigen::MatrixXf is not usable due to unknown dimension.

[xkunglu]

@YannickJadoul your code works, thanks for the reference, I just wanted to point out that you are missing a parenthesis at the end of the line
auto cap = py::capsule(v, [](void *v) { delete reinterpret_cast<std::vector<int>*>(v); });

[arquolo]

By the way,
anybody knows, how to get py::array_t from std::shared_ptr<std::vector<T>> without copy (and using new/delete)?
I tried this:

std::shared_ptr<std::vector<float>> ptr = get_data();
return py::array_t<float>{
    ptr->size(),
    ptr->data(),
    py::capsule(ptr.get(), [](void* p){ reinterpret_cast<decltype(ptr)*>(p)->reset(); }),
};

Obviously, this will never work, because when return happens, ptr will be deallocated from stack.
Using capture also does not help, because py::capsule can't accept them:

std::shared_ptr<std::vector<float>> ptr = get_data();
return py::array_t<float>{
    ptr->size(),
    ptr->data(),
    py::capsule([ptr](){ }),  // using lambda-capture to increase lifetime of ptr
};

Worked this solution (which seems very dirty):

std::shared_ptr<std::vector<float>> ptr = get_data();
return py::array_t<float>{
    ptr->size(),
    ptr->data(),
    py::capsule(
        new auto(ptr),  // <- can leak
        [](void* p){ delete reinterpret_cast<decltype(ptr)*>(p); }
    )
};

[YannickJadoul]

@arquolo Indeed, the only data that can be stored in a py::capsule is a single void * and a simple function pointer (this is a Python C API thing, by the way; pybind11 just made a C++ wrapper around it). So if you want the capsule to be a (co-)owner of the shared_ptr, I would think that the last solution is the only one that works and stores the actual shared_ptr object.

Is it that dirty, though? In the end, a capsule taking a std::function (or any kind of lambda/functor object) would incur this same allocation (inside of the std::function) because of the variable size of the capture.

The one thing to note, though, is that the object doesn't need to be a capsule. I can just as well be any other object (though hopefully one that keeps the data alive), so if your shared_ptr would be stored as member in a C++ class that is exposed to Python, you could also take use that py::object.

[ferdonline]

We define the following utility functions, which have proven to be live savers :)

template <typename Sequence>
inline py::array_t<typename Sequence::value_type> as_pyarray(Sequence&& seq) {
    // Move entire object to heap (Ensure is moveable!). Memory handled via Python capsule
    Sequence* seq_ptr = new Sequence(std::move(seq));
    auto capsule = py::capsule(seq_ptr, [](void* p) { delete reinterpret_cast<Sequence*>(p); });
    return py::array(seq_ptr->size(),  // shape of array
                     seq_ptr->data(),  // c-style contiguous strides for Sequence
                     capsule           // numpy array references this parent
    );
}

and the copy version

template <typename Sequence>
inline py::array_t<typename Sequence::value_type> to_pyarray(const Sequence& seq) {
    return py::array(seq.size(), seq.data());
}

[arquolo]

Thanks @ferdonline
However, the move-helper needs to change signature to:

template <typename Sequence,
          typename = std::enable_if_t<std::is_rvalue_reference_v<Sequence&&>>>
inline py::array_t<typename Sequence::value_type> as_pyarray(Sequence&& seq)

With such fix, the compiler will warn you if you calls with without std::move