Feat : Source mysql #118

vikash390 · 2025-02-28T06:26:21Z

Description

Implemented MySQL Connector for Full Load and CDC: Developed a connector that supports both full load and Change Data Capture (CDC) functionality for seamless data migration and synchronization .
Used Row Chunking for Faster Backfill Performance: Implemented row chunking to optimize backfill operations, significantly improving data load speed and reducing system load during large data transfers.

Fixes # (issue)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Full Load

Test inserting all data from the source table to the target table
Ensure data is correctly copied, with no missing or extra records.

CDC

Test handling of new records added to the source table after the initial full load.
Test updates in the source table and verify they are reflected correctly in the target.
Test deletion of records in the source table and ensure they are removed from the target (if applicable).
Handle cases where no changes occur during CDC, ensuring no unnecessary updates.
Test adding new columns to the source table and ensure the target table schema is updated automatically.
Test removing columns from the source table and ensure the target table is updated accordingly.
Test renaming columns in the source table and verify the target schema reflects these changes

Verify data types and column mapping between source and target.

Scenario A
Scenario B

Screenshots or Recordings

Related PR's (If Any):

…at/source_mysql

go.mod

drivers/postgres/internal/backfill.go

pkg/jdbc/jdbc.go

drivers/mysql/internal/cdc.go

drivers/mysql/internal/backfill.go

drivers/mysql/internal/cdc.go

drivers/mysql/internal/backfill.go

…at/source_mysql

drivers/mysql/internal/backfill.go

drivers/mysql/internal/config.go

pkg/binlog/connection.go

pkg/binlog/filter.go

into feat/source_mysql

drivers/mysql/internal/backfill.go

drivers/mysql/internal/cdc.go

hash-data · 2025-03-24T12:46:02Z

drivers/mysql/internal/mysql.go

+			logger.Warnf("Unsupported MySQL type '%s'for column '%s.%s', defaulting to String", dataType, streamName, columnName)
+			datatype = types.String
+		}


@shubham19may decision required...

drivers/mysql/internal/mysql.go

pkg/binlog/connection.go

pkg/binlog/filter.go

pkg/jdbc/jdbc.go

hash-data · 2025-03-24T19:37:51Z

@vikash390 add readme in order to merge Mysql

into feat/source_mysql

hash-data

LGTM

Co-authored-by: Priyansh Khodiyar <[email protected]> Co-authored-by: hash-data <[email protected]> Co-authored-by: Ankit Sharma <[email protected]>

vikash390 added 7 commits February 12, 2025 12:10

WIP:mysql as source

660981e

Add discovery and catalog generation for mysql source

287f46c

Merge branch 'master' of https://github.com/datazip-inc/olake into fe…

21c3839

…at/source_mysql

backfill in mysql

8de362a

feat: cdc in mysql

25a536a

Merge branch 'master' of https://github.com/datazip-inc/olake into fe…

9d05b3c

…at/source_mysql

fix:chunking query

47bb198

vikash390 changed the title ~~Feat/source mysql~~ WIP: Source mysql Feb 28, 2025

vikash390 added 14 commits February 28, 2025 18:26

fix : CDC process to detect completion much faster

56f6d84

fix: handling of non numeric PK and missing record

a04d9ef

feat:chunksize based on numbers of rows and threads

b2777c2

feat : implement Repeatable read isolation level

e6ab224

refactor:unnecessary checks

2606928

fix: handling of misssing record

0ef4f1f

chore: backoff in backfill

db30487

fix : handle mysql datatypes conversion

56901a5

chore : implement jdbc reader

6d4c88c

fix : datatype conversion and mysql text issue

b1a52e8

fix : schema evolution in cdc

72163dc

chore: refactor based on self review

75ba1a9

Merge branch 'master' of https://github.com/datazip-inc/olake into fe…

8ad149b

…at/source_mysql

refcator code and merge with master

4b4a7f3

vikash390 changed the title ~~WIP: Source mysql~~ Feat : Source mysql Mar 13, 2025

vikash390 marked this pull request as ready for review March 13, 2025 05:38

hash-data linked an issue Mar 13, 2025 that may be closed by this pull request

[Feat] MySql as a Source #77

Closed

hash-data reviewed Mar 19, 2025

View reviewed changes

vikash390 added 4 commits March 19, 2025 18:26

refactor backfill based on review

5364230

chore : refactor mysql cdc code

e3b4a4a

chore : refactor mysql code file

7e9303b

chore : refactor config file code

630ab44

vikash390 added 6 commits March 20, 2025 18:51

fix : lint issue

72b7411

fix :lint issue

0740bd9

Merge branch 'master' of https://github.com/datazip-inc/olake into fe…

3f3a6bd

…at/source_mysql

fix : lint issue

54091a6

fix : lint issue

8e18da8

lint issue

49ba03f

vikash390 force-pushed the feat/source_mysql branch from 938ef29 to 49ba03f Compare March 21, 2025 07:21

hash-data reviewed Mar 23, 2025

View reviewed changes

hash-data and others added 5 commits March 23, 2025 23:39

Merge branch 'master' into feat/source_mysql

605b19e

chore : refactor code based on review

76fee5d

Merge branch 'feat/source_mysql' of https://github.com/datazip-inc/olake

3f1e672

into feat/source_mysql

fix : lint issue

49c8893

fix:lint issue

fd0a3a6

hash-data reviewed Mar 24, 2025

View reviewed changes

vikash390 and others added 3 commits March 24, 2025 18:56

chore : refactor bassed on review

ca3a311

chore: some improvments in mysql

54cb2f1

fix: golangci

e0cba45

hash-data had a problem deploying to Publish Driver March 24, 2025 19:19 — with GitHub Actions Failure

hash-data temporarily deployed to Publish Driver March 24, 2025 19:27 — with GitHub Actions Inactive

vikash390 and others added 4 commits March 25, 2025 13:17

add readme.md

11d18ce

Merge branch 'feat/source_mysql' of https://github.com/datazip-inc/olake

e489b83

into feat/source_mysql

chore: fix for null chunks

5698b34

fix : varchar to byte issue

b406d87

hash-data changed the base branch from master to development March 25, 2025 11:25

hash-data approved these changes Mar 25, 2025

View reviewed changes

hash-data merged commit 5a1e949 into development Mar 25, 2025
5 checks passed

hash-data deleted the feat/source_mysql branch March 25, 2025 11:26

hash-data added a commit that referenced this pull request Mar 25, 2025

Feat : Source mysql (#118) (#175)

48dfd59

Co-authored-by: Priyansh Khodiyar <[email protected]> Co-authored-by: hash-data <[email protected]> Co-authored-by: Ankit Sharma <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat : Source mysql #118

Feat : Source mysql #118

vikash390 commented Feb 28, 2025 •

edited

Loading

hash-data Mar 24, 2025

hash-data commented Mar 24, 2025

hash-data left a comment

Feat : Source mysql #118

Feat : Source mysql #118

Conversation

vikash390 commented Feb 28, 2025 • edited Loading

Description

Type of change

How Has This Been Tested?

Full Load

CDC

Screenshots or Recordings

Related PR's (If Any):

hash-data Mar 24, 2025

Choose a reason for hiding this comment

hash-data commented Mar 24, 2025

hash-data left a comment

Choose a reason for hiding this comment

vikash390 commented Feb 28, 2025 •

edited

Loading