Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: loosing required type when using to_pyarrow #10896

Closed
1 task done
christophediprima opened this issue Feb 25, 2025 · 1 comment · Fixed by #10897
Closed
1 task done

bug: loosing required type when using to_pyarrow #10896

christophediprima opened this issue Feb 25, 2025 · 1 comment · Fixed by #10897
Labels
bug Incorrect behavior inside of ibis

Comments

@christophediprima
Copy link

What happened?

When using to_pyarrow on a Ibis expression the required types are lost. Here is an example to reproduce it:

import ibis

ibis.options.interactive = True

schema = ibis.schema([
  ('id', '!string')
])

new_ids = ibis.memtable([{
  'id': 'my_id',
  }]).cast(schema)

print('--- ibis exp ---')
print(new_ids)
pyarrow_df = new_ids.to_pyarrow()
print('--- pyarrow_df ---')
print(pyarrow_df)
print('--- pyarrow_df after recasting ---')
print(pyarrow_df.cast(schema.to_pyarrow()))

Using things like ibis.literal('my_id', type=ibis.dtype("string", nullable=False)) to set the value won't fix the issue.

What version of ibis are you using?

10.0.0

What backend(s) are you using, if any?

No response

Relevant log output

--- ibis exp ---
┏━━━━━━━━━┓
┃ id      ┃
┡━━━━━━━━━┩
│ !string │
├─────────┤
│ my_id   │
└─────────┘
--- pyarrow_df ---
pyarrow.Table
id: string
----
id: [["my_id"]]
--- pyarrow_df after recasting ---
pyarrow.Table
id: string not null
----
id: [["my_id"]]

Code of Conduct

  • I agree to follow this project's Code of Conduct
@cpcloud
Copy link
Member

cpcloud commented Feb 25, 2025

Thanks for the issue, PR to fix it is up at #10897.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis
Projects
Status: done
Development

Successfully merging a pull request may close this issue.

2 participants