mapobs problems with composite datasets and vector indexes

[CarloLucibello]

With tuples:

julia> t = (zeros(3), ones(3))
([0.0, 0.0, 0.0], [1.0, 1.0, 1.0])

julia> m = mapobs(x -> (x[1], 2*x[2]), t)
mapobs(#15, Tuple{Vector{Float64}, Vector{Float64}})

julia> m[1]
(0.0, 2.0)

julia> m[2]
(0.0, 2.0)

julia> m[1:2]
((0.0, 0.0), (1.0, 2.0))  # expected ([0.0, 0.0], [2.0, 2.0]) 

With named tuples:

julia> t = (a=zeros(3),b=ones(3))
(a = [0.0, 0.0, 0.0], b = [1.0, 1.0, 1.0])

julia> m = mapobs(x -> (x[1], 2*x[2]), t)
mapobs(#17, NamedTuple{(:a, :b), Tuple{Vector{Float64}, Vector{Float64}}})

julia> m[1]
(0.0, 2.0)

julia> m[2]
(0.0, 2.0)

julia> m[1:2]
ERROR: ArgumentError: broadcasting over dictionaries and `NamedTuple`s is reserved
Stacktrace:
 [1] broadcastable(#unused#::NamedTuple{(:a, :b), Tuple{Vector{Float64}, Vector{Float64}}})
   @ Base.Broadcast ./broadcast.jl:705
 [2] broadcasted
   @ ./broadcast.jl:1295 [inlined]
 [3] getindex(data::MLUtils.MappedData{var"#17#18", NamedTuple{(:a, :b), Tuple{Vector{Float64}, Vector{Float64}}}}, idxs::UnitRange{Int64})
   @ MLUtils ~/.julia/packages/MLUtils/8OXl7/src/obstransform.jl:14
 [4] top-level scope
   @ REPL[35]:1

The fix could be turning

MLUtils.jl/src/obstransform.jl

Line 14 in 1da3c53

Base.getindex(data::MappedData, idxs::AbstractVector) = data.f.(getobs(data.data, idxs))

into

Base.getindex(data::MappedData, idxs::AbstractVector) = Flux.batch([data.f(getobs(data.data, i)) for i in idxs]) 

[CarloLucibello]

This is no longer an error

julia> t = (a=zeros(3),b=ones(3))
(a = [0.0, 0.0, 0.0], b = [1.0, 1.0, 1.0])

julia> m = mapobs(x -> (x[1], 2*x[2]), t)
mapobs(#17, NamedTuple{(:a, :b), Tuple{Vector{Float64}, Vector{Float64}}})

julia> m[1]
(0.0, 2.0)

julia> m[2]
(0.0, 2.0)

julia> m[1:2]
([0.0, 0.0], [2.0, 2.0])

Mapobs now does the following:

  mapobs(f, data; batched=:auto)

  Lazily map f over the observations in a data container data. Returns a new data container mdata that can be indexed and has a length. Indexing triggers the transformation f.

  The batched keyword argument controls the behavior of mdata[idx] and mdata[idxs] where idx is an integer and idxs is a vector of integers:

    •  batched=:auto (default). Let f handle the two cases. Calls f(getobs(data, idx)) and f(getobs(data, idxs)).

    •  batched=:never. The function f is always called on a single observation. Calls f(getobs(data, idx)) and [f(getobs(data, idx)) for idx in idxs].

    •  batched=:always. The function f is always called on a batch of observations. Calls getobs(f(getobs(data, [idx])), 1) and f(getobs(data, idxs)).