Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improvement: proper data types mapping #141

Open
github-actions bot opened this issue Mar 12, 2025 · 0 comments
Open

improvement: proper data types mapping #141

github-actions bot opened this issue Mar 12, 2025 · 0 comments
Assignees

Comments

@github-actions
Copy link

github-actions bot commented Mar 12, 2025

Problem

Currently, Olake maps all integer and floating-point types to generalized types like Int64 or Float64, without considering more specific types such as Int32, UInt32, Float32, etc.

This results in:

  • Loss of type precision
  • Potential compatibility issues for downstream systems
  • Increased memory usage for smaller types

Goals

  • Implement accurate and granular type mappings based on the source column types
  • Support all major numeric variants:
    • Signed and unsigned integers: tinyint, smallint, int, bigint, etc.
    • Floating-point numbers: float, real, double precision
  • Ensure consistent type preservation across different sources (e.g., MySQL, PostgreSQL)

Tasks

  • Audit current type mapping logic
  • Add proper mappings for Int32, UInt32, Int16, UInt16, Float32, etc.
  • Ensure compatibility across all supported source systems
  • Document all supported type mappings in the developer guide

Outcome

  • Enhanced data accuracy and fidelity
  • Better downstream integration and usability

Example: below types not properly mapped


// TODO: add proper types (not only int64)
"bigint": types.Int64,
"tinyint": types.Int64,
"integer": types.Int64,
"smallint": types.Int64,
"smallserial": types.Int64,


This issue was generated by todo-issue based on a TODO comment in ed705d2. It's been assigned to @hash-data because they committed the code.
@hash-data hash-data changed the title add proper types (not only int64) improvement: proper data types Apr 9, 2025
@hash-data hash-data changed the title improvement: proper data types improvement: proper data types mapping Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant