`NestedFrame.__setitem__("new_nested.x", flat_values)` repeats values for non-unique index

[hombit]

Bug report

from nested_pandas.datasets import generate_data

nf = generate_data(5, 3)
nf['new.flux'] = nf['nested.flux']
print(nf)

          a  ...                               new
0  0.768677  ...  [{flux: -85.021725}; …] (6 rows)
0  0.618566  ...  [{flux: -85.021725}; …] (6 rows)
1  0.308658  ...  [{flux: -16.160369}; …] (9 rows)
1  0.612433  ...  [{flux: -16.160369}; …] (9 rows)
1  0.332888  ...  [{flux: -16.160369}; …] (9 rows)

Note 6 and 9 nested rows instead of 3

Before submitting
Please check the following:

• I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
• I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
• If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

[dougbrn]

I agree that this behavior is weird. It's very understandable from the way the package works itself, as we adhere very strongly to the notion that the index has the most authority on how data is nested. I would bet there's a lot of things like this in our API when dealing with non-unique indices. That being said, this seems like really bad behavior in how much we can explode memory usage!

My thought on how this "should" work, would be to have "new" mirror the structure of "nested", so having 3 per iloc as you note above. Without looking into the code, I'm not sure how difficult it would be to make that change.

[hombit]

@dougbrn thanks! Yes, I agree that it would be hard to make it work properly, at least with the current data structures.

Two really large changes which could help us solving it could be:

1. Adding secondary index to flatten series,
2. Or introduce FlattenSeries class to handle relationship between (not implemented) NestedSeries and its flatten version.