[fix] Fix skip empty string proto representation mc #92
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* chore: modify error message on stopped error bq sink * chore: fix typo --------- Co-authored-by: jesry.pandawa <[email protected]>
bump up version to 0.4.1 Co-authored-by: jesry.pandawa <[email protected]>
* feat: migrating to gotocompany * feat: add bigquery write storage client * feat: adding unit tests * feat: add converter * test: add convertor tests * test: float to double and int to long in payload * tets: add nested tests * test: checkstyle * chore: fix tests * chore: add more asserts * feat: timestamp values in INT64 * fix: timestamp column * feat: add metadata in biqguery * test: invalid records * feat: Schema update * feat: error parsing * refactor: minor refactors * tests: unit tests for error parsing * test: add tests for schema update * feat: bq operation metrics * feat: error metrics * test: add tests for stream writer metrics * refactor: use instrumentation instead of log in ProtoWriter * chore: convert the metric to count * chore: version bump * chore: only send payload with data * fix: add status.getCode() as a tag --------- Co-authored-by: Prakhar Mathur <[email protected]> Co-authored-by: mayur.gubrele <[email protected]>
* fix: recreate writer if closed * tests: add unit test
* chore: getFieldByName with lowercase * chore: version bump
* fix: check timestamp only for partition column * fix: add timestamp check
Update jedis client from 3.0.1 to 3.10.0 In redis config, acl can be configured with * Both username and password. eg: user <user_name> ..<acls>.. on >somepassword * Only username with no password. eg: user <user_name> ..<acls>.. on nopass * Only password via requirepass config. eg: requirepass foobar
* bug: recreate connection when idle * chore: cleanup * chore: checkstyle * chore: logging * chore: typo in method name
* bug: recreate connection when idle * chore: cleanup * chore: checkstyle * chore: logging * chore: typo in method name * chore: version bump
* bug: recreate connection when idle * chore: cleanup * chore: checkstyle * chore: logging * chore: typo in method name * chore: version bump * fixes: call fetch schema before conversion * fix: check streamwriter new schema before convert * refactor: only keep the lock while refreshing * test: fix test
#31) * bug: recreate connection when idle * chore: cleanup * chore: checkstyle * chore: logging * chore: typo in method name * chore: version bump * fixes: call fetch schema before conversion * fix: check streamwriter new schema before convert * refactor: only keep the lock while refreshing * test: fix test * fix: adding 60 second refresh even if no incoming traffic * fix: use stencil ttl for refresh
* bug: recreate connection when idle * chore: cleanup * chore: checkstyle * chore: logging * chore: typo in method name * chore: version bump * fixes: call fetch schema before conversion * fix: check streamwriter new schema before convert * refactor: only keep the lock while refreshing * test: fix test * fix: adding 60 second refresh even if no incoming traffic * chore: fix name
* fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * chore: version bump * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: fix checkstyle * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: temp fix for redis broken connection issue * fix: fix checkstyle * fix: temp fix for redis broken connection issue * tests: add tests for redis sink temp fix
* fix: add constant backoff for redis connection issue * fix: add constant backoff for redis connection issue * fix: add constant backoff for redis connection issue
* updating protobuf modules
* Enables building code on M1 macs
* Updated protobuf used in project
* updating version
---------
Co-authored-by: lavkesh <[email protected]>
* feat: add socket and connection timeout configs in redis sink * chore: version bump to 0.8.1 * fix: remove retry mechanism of temp fix * tests: fix tests * feat: make retry mechanism configurable * docs: add docs for redis client retry configs * tests: fix tests * fix: redis client to handle retry mechanism * fix: redis client to handle retry mechanism * fix: redis client to handle retry mechanism * fix: redis client to handle retry mechanism * fix: redis client to handle retry mechanism * fix: remove redundant RedisStandaloneClient constructor * fix: add break after successfull push to redis * fix: change default timeout values
* feat: add metrics for Redis sink * feat: add metrics for Redis sink * feat: add metrics for Redis sink * feat: add metrics for Redis sink * feat: add metrics for Redis sink * tests: add tests for redis sink metrics * tests: add tests for redis sink metrics * tests: add tests for redis sink metrics * tests: add tests for redis sink metrics * docs: add docs for Redis sink metrics
* feat: create common http client (#53) * feat: create bare minimum http sink (#56) * feat: create bare minimum http sink * feat: add unit test * test: fix test * refactor: refactor request creation into factory class * refactor: remove comments * chore: minor fixes * chore: disable parameters in configs * chore: minor fixes * fix: fix metadata to also support generic value * chore: abstract metadata util * chore: http sink config extend to http client config * chore: refactor config class definition * chore: query param as map * chore: remove null check * chore: refactor * chore: minor refactoring * chore: create converter for header config Co-authored-by: lavkesh <[email protected]> Co-authored-by: jesry.pandawa <[email protected]> * fix: fix single request (#64) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable error parsing (#55) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: minor fixes (#59) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: add bq error logs (#57) * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests * chore: small fixes Co-authored-by: Mayur Gubrele <[email protected]> Co-authored-by: mayur.gubrele <[email protected]> * docs: adds bigtable sink documentation (#58) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * docs: add bigtable sink documentation Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * chore: rc version * fix: fix uri and exception handler * feat: exclude io.grpc from bigtable library and bump the version to 0.3.4-beta.4 * chore: add iterator in http record and toString in Template * chore: add more exception handler * Revert "chore: add more exception handler" This reverts commit 0edb3b7. * Revert "chore: add iterator in http record and toString in Template" This reverts commit a18bad6. * fix: fix template, uri and http exception handling * fix: fix exception handling on response and request body * fix: fix template exception * Revert "fix: fix template exception" This reverts commit 746754a. * chore: move IOException handling for request and response body Co-authored-by: Mayur Gubrele <[email protected]> Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: lavkesh <[email protected]> * feat: add batch request (#68) Co-authored-by: lavkesh <[email protected]> * Http batch request (#70) * feat: add batch request * chore: minor refactor Co-authored-by: jesrypandawa <[email protected]> * feat: header builder generates headers from template (#63) * feat: header builder generates headers from template * refactor: fix for the bug where baseheaders were getting updated with template ones * add a maven repo in build.gradle * feat: add messageContainer class and handle exceptions for headerBuilder flow * refactor: add tests for headers in singleRequestTest * refactor: resolve review comments * refactor: minor refactoring as per review comments * feat: add http query parameter builder (#71) * feat: add query param builder * chore: add minor changes * refactor: refactor builder class to cache schema, header,queryparam,and url * feat: get sink_retryable_error for configured retry http response code (#75) * feat: get sink_retryable_error for configured retry http response code ranges * refactor: changes based on review comments * refactor: getMapping on ParsedMessage * feat: add ProtoJsonProvider to work with JSONPath * feat: use ProtoJsonProvider for getFieldByName on ProtoParsedMessage * chore: remove unused imports * feat: add config to log http request based on response code (#74) * feat: add config to log http request based on response code * refactor: remove unused import * refactor: fix error after merge * refactor: refactor log request method name * refactor: refactor http request for parsedMessage design changes * refactor: refactor request body (#77) * feat: add http templatized json body (#79) * refactor: refactor request body * feat: add templatized json body * chore: explicit the exception in http sink factory, add more test for templatized json * feat: moving to gotocompany * chore: cleanup schema class references * feat: implement getMapping using ParsedMessage interface * wip: add proto implementation * feat: add json implementation * chore: add tests for schema implementation * chore: add tests for generic json schema implementation * chore: remoeve getMapping from parsedMessage and clean up test * chore: add tests for byte and enum types in RecordConverter * chore: clean up unused protoField and jsonfield class * chore: add tests for getFields ProtoParsedMessage * feat: include default fields in Proto message * feat: add float point check for Double * refactor: inline proto logical type parse * refactor: ProtoLogicalValue constructor --------- Co-authored-by: jesry.pandawa <[email protected]> * feat: add toJson in parsedMessage and implementation (raystack#6) * feat: add toJson in parsedMessage and implementation * chore: fix config and unit test * chore: fix checkstyle * chore: fix test * chore: remove preserve proto field names config * feat: add http sink message payload type (#8) * feat: add http sink json payload type (#7) * feat: add http sink json payload type * fix: http sink payload key cases --------- Co-authored-by: Harikrishna <[email protected]> * feat: add http delete method (#9) * feat: add http delete method * refactor: remove comments --------- Co-authored-by: Harikrishna <[email protected]> * chore: requestutils refactor * refactor: http to httpv2 prefix (#20) * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * fix: add SINK prefix in HttpClientConfigs (#17) * fix: add SINK prefix in HttpClientConfigs * fix add SINK prefix in HttpClientConfigs * fix: add httpv2 prefix in http configs * tests: fix tests * tests: fix tests * tests: fix tests * fix: fix configs * fix: fix checkstyle * feat: http sink metrics (#14) * feat: http sink metrics * test: add http metrics tests * chore: beta release version * refactor: update exception message * chore: version bump * chore: version bump * fix: remove simple date format (#23) * fix: throw exception when template is used in batch mode for url , query and header * fix: log request payload body in debug mode * chore: fix build * test: log request payload body in debug mode * test: log request payload body in debug mode * fix: add config for default value for empty fields (#25) * fix: add config for default value for empty fields * fix: add config for default value for empty fields * refactor: rename variables which are not final anymore * fix: default value behaviour for empty fields * test: fix tests * test: fix tests * test: fix tests * test: fix tests * test: fix tests * feat: add support for JSON array and primitives in Json body template (#40) * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * tests: add unit tests for json parser * refactor: removed extra checks * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * tests: fix unit tests * tests: fix unit tests * refactor: change implementation to use jackson library instead of gson * refactor: change implementation to use jackson library instead of gso * refactor: move json parsing logic to JsonParserUtils * tests: add unit tests for JsonParserUtil * fix: fix check style * fix: fix check style * fix: fix bugs and unit tests * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix PR review comments * docs: add java doc for utility methods * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils --------- Co-authored-by: lavkesh <[email protected]> * docs: add docs for Httpv2 sink (#41) * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * fix: remove messageschema get schema method * fix: remove default values of oauth configs * refactor: revert bq storage sink changes (#42) * fix: fix http connection pool error (#43) * fix: fix http connection pool error * fix: fix build error * fix: fix response code logic and consuming response body --------- Co-authored-by: gagandhand <[email protected]> Co-authored-by: gdgagan696 <[email protected]> * docs: add docs for httpv2 sink metrics * docs: add docs for httpv2 sink metrics * fix: change default field value enable to true --------- Co-authored-by: jesrypandawa <[email protected]> Co-authored-by: lavkesh <[email protected]> Co-authored-by: jesry.pandawa <[email protected]> Co-authored-by: Mayur Gubrele <[email protected]> Co-authored-by: mayur.gubrele <[email protected]> Co-authored-by: jesrypandawa <[email protected]> Co-authored-by: Harikrishna <[email protected]> Co-authored-by: gagandhand <[email protected]> Co-authored-by: gdgagan696 <[email protected]>
* fix: allow commas in the template string in json body template * fix: allow commas in the template string in json body template * docs: add docs to allow commas in the template string in json body template * docs: add docs to allow commas in the template string in json body template * test: add tests for fixing comma invalidation in json body template * refactor: change template pattern to final instance variable * chore: bump up version to 0.9.1 * docs: fix httpv2 payload examples in docs * refactor: change comma parsing unit tests to parameterized tests * refactor: change comma escaping pattern to forward slash * docs: change comma escaping pattern to forward slash
* chore: upgrade jfrog version to 4.33.1 * chore: upgrade jfrog version to 4.33.1
…#56) Enable publish to maven local - ./gradlew clean publishToMavenLocal Co-authored-by: rajuGT <[email protected]>
* feat: Add Type Info Converter Implementation * feat: Add Test * feat: Complete test for MessageTypeInfoConverterTest * feat: Complete test for TimestampTypeInfoConverter * feat: Complete test for StructTypeInfoConverter.java * feat: Complete test for DurationTypeInfoConverter.java * feat: Complete test for BaseTypeInfoConverterTest.java * feat: get implementation diff from feat branch * test: update ConverterOrchestratorTest * test: Add InsertManagerFactoryTest * chore: remove unused constructor * test: - add DefaultPartitioningStrategyTest - add TimestampPartitioningStrategyTest - add PartitioningStrategyFactoryTest * fix: PartitioningStrategyFactoryTest * fix: remove unused dependency injection * test: add non null injected decorator * test: add test for MaxComputeSchemaCache * chore: remove unused lombok annotations * test: Add MaxComputeSinkTest * test: add test for close * test: fix typeinfo * fix: add error handler in MaxComputeSink * add ProtoUnknownFieldValidationType * test: Add test for ProtoUnknownFieldValidationType * fix: use correct converter * fix: rename method * chore: rename * chore: exclude conflicting deps * chore: fix checkstyle main * chore: fix checkstyle test * chore: fix dependencies issue * chore: remove main * fix: fix test * chore: exclude model package and bump version * feat: add max compute metrics * feat: add compression option * fix: fix ordering on instantiation * feat: Add utils and implement schema update * feat: Add logging when executing SQL * chore: checkstyle main * chore: fix wrong params * feat: add converter for MaxComputeCompressionAlgorithmConverter * chore: fix checkstyle * fix: byte array conversion * test: fix test * fix: remove record reordering, put metadata at the beginning * fix: use timestamp_ntz for proto timestamp and partition * fix: adjust unit test to use timestamp_ntz instead of timestamp * chore: remove unused lombok annotations * chore: exclude maxcompute client and factory from coverage * chore: exclude maxcompute client from coverage check * chore: delete unused proto * test: add test case for log key * chore: remove unused getters * test: add compression option test * chore: remove redundant column field * fix: duration converter cast to message * feat: add partition precondition on MaxComputeClient * feat: add allowSchemaMismatch to false * chore: remove unused config * chore: adjust config name * chore: add sink and config docs * chore: docs layout * chore: docs layout + adjust default value of config * fix: revert mistakenly pushed build gradle change * chore: fix magic constant * fix: set null instead of default value to nonexistent field * chore: fix partitioning strategy * test: complete unit test * feat: add configurable date format for timestamp partition key * chore: change default value and docs * Maxcompute instrumentation (#55) * added instrumentation for MC sink * chore: refactor InsertManager to abstract class * chore: revert spacing change * chore: refactor --------- Co-authored-by: Eka Winata <[email protected]> * feat: add zone offset for timestamp * feat: use zone offset config for metadata util * feat: Use ZoneId instead of ZoneOffset * feat: use ali auto partitioning strategy [todo fix ProtoDataColumnRecordDecoratorTest] * test: fix ProtoDataColumnRecordDecoratorTest * chore: remove unused config * fix: set MaxComputeCache when partitioningStrategy is not null * chore: checkstyle * feat: add checking on sql result * fix: metadata util * chore: checkstyle * fix: add validation for enum based config and make it fail fast on initialization * chore: fix exception type * chore: checkstyle * fix: class naming and add validation for enum value * chore: Fix MetadataUtil to accept List of TupleString * chore: switch metadata ordering * - refactor schema cache to factory method - fix schema cache to fetch from metadata server * Fix MaxComputeSchemaCache and its test * chore: if not exists true table creation parameter * - checkstyle SinkConfigUtils - add test for SinkConfigUtils * - add retry utils - separate DDL operation to DdlManager * test: Add test for RetryUtils.java * feat: add global configs as externalized parameter * feat: refactor streaming management to StreamingSessionManager class * feat: externalize partition time unit configuration * fix: refactor TimestampPartitioningStrategy * test: add case when passed object is not Record * chore: remove null checking since message couldn't be null * chore: add synchronization on schema update method * chore: checkstyle main * chore: checkstyle main * chore: update docs * chore: update schema docs * chore: reorder annotation in config * chore: change version to 0.10.0 * chore: bump aliyun version * chore: remove redundant enum converter class * feat: use guava cache for streaming session * chore: use sessionCache.getUnchecked * test: update test * chore: checkstyle * chore: use const instead of literal string * chore: wrap update statement with backtick * test: add IOException use case * chore: refactor RetryUtils to receive exception predicate * chore: remove validateConfig() * fix: make RecordWrapper immutable * chore: optimize variable declaration on MaxComputeOdpsGlobalSettingsConverter * fix: remove unused TableTunnel dependencies from InsertManager.java and its implementation * fix: setup FlushOption on the constructor for InsertManager * fix: rename static factory method of streaming session manager * chore: add javadoc for PayloadConverter * chore: refactor StreamingSessionManager to receive LoadingCache in its constructor * chore: Refactor PrimitiveTypeInfoConverter to use ImmutableMap * chore: Rename ConverterOrchestrator.java to ProtobufConverterOrchestrator.java * chore: refactor name and type info to static var * chore: move MaxComputeSchemaHelper to root MaxCompute package * chore: rename getDdlDeclaration to getDDL * chore: remove builder on MaxComputeSchema * chore: fix indent * refactor: MaxComputeSchemaCache to separate handling between non-null and null descriptor * refactor: use google Sets on PartitioningStrategyFactory * refactor: make TableValidator limit configurable * refactor: use immutable map on MetadataUtil * refactor: remove SinkConfigUtils.java and its test. Transfer the implementation of util method directly in class using it * chore: add static imports for assertions lib * chore: rename PayloadConverter.java to ProtobufPayloadConverter * feat: add validation for timestamp type * chore: update docs and default value for valid min and max timestamp * refactor: wrap payload converter to work with DTO * chore: set default maximum session count to 2 for SINK_MAXCOMPUTE_STREAMING_INSERT_MAXIMUM_SESSION_COUNT * test: fix assertion * feat: add timestamp difference validation * feat: Add NaN and Infinite check float and double * chore: rename upsertTable to createOrUpdateTable * chore: refactor inline lambda to static method * chore: update docs * fix: use server side schema for building metadata * chore: add docs for SINK_CONNECTOR_SCHEMA_PROTO_UNKNOWN_FIELDS_VALIDATION * chore: rename MaxComputeSchemaHelper.java to MaxComputeSchemaBuilder * refactor: Add method to add uniform errors in SinkResponse and apply it to MaxComputeSink * feat: parameterized SINK_MAXCOMPUTE_STREAMING_INSERT_TUNNEL_SLOT_COUNT_PER_SESSION * chore: update generic.md to use more explicit context * chore: update maxcompute.md to show the correct type mapping * chore: rename object to parsedObject in ProtoPayload.java * chore: refactor `convert` method under ProtobufConverterOrchestrator to toMaxComputeTypeInfo and toMaxComputeValue * chore: add javadoc for ProtobufTypeInfoConverter.java * chore: checkstyle * chore: add static import * chore: remove extraneous space --------- Co-authored-by: Vaishnavi190900 <[email protected]>
) * feat: Add Type Info Converter Implementation * feat: Add Test * feat: Complete test for MessageTypeInfoConverterTest * feat: Complete test for TimestampTypeInfoConverter * feat: Complete test for StructTypeInfoConverter.java * feat: Complete test for DurationTypeInfoConverter.java * feat: Complete test for BaseTypeInfoConverterTest.java * feat: get implementation diff from feat branch * test: update ConverterOrchestratorTest * test: Add InsertManagerFactoryTest * chore: remove unused constructor * test: - add DefaultPartitioningStrategyTest - add TimestampPartitioningStrategyTest - add PartitioningStrategyFactoryTest * fix: PartitioningStrategyFactoryTest * fix: remove unused dependency injection * test: add non null injected decorator * test: add test for MaxComputeSchemaCache * chore: remove unused lombok annotations * test: Add MaxComputeSinkTest * test: add test for close * test: fix typeinfo * fix: add error handler in MaxComputeSink * add ProtoUnknownFieldValidationType * test: Add test for ProtoUnknownFieldValidationType * fix: use correct converter * fix: rename method * chore: rename * chore: exclude conflicting deps * chore: fix checkstyle main * chore: fix checkstyle test * chore: fix dependencies issue * chore: remove main * fix: fix test * chore: exclude model package and bump version * feat: add max compute metrics * feat: add compression option * fix: fix ordering on instantiation * feat: Add utils and implement schema update * feat: Add logging when executing SQL * chore: checkstyle main * chore: fix wrong params * feat: add converter for MaxComputeCompressionAlgorithmConverter * chore: fix checkstyle * fix: byte array conversion * test: fix test * fix: remove record reordering, put metadata at the beginning * fix: use timestamp_ntz for proto timestamp and partition * fix: adjust unit test to use timestamp_ntz instead of timestamp * chore: remove unused lombok annotations * chore: exclude maxcompute client and factory from coverage * chore: exclude maxcompute client from coverage check * chore: delete unused proto * test: add test case for log key * chore: remove unused getters * test: add compression option test * chore: remove redundant column field * fix: duration converter cast to message * feat: add partition precondition on MaxComputeClient * feat: add allowSchemaMismatch to false * chore: remove unused config * chore: adjust config name * chore: add sink and config docs * chore: docs layout * chore: docs layout + adjust default value of config * fix: revert mistakenly pushed build gradle change * chore: fix magic constant * fix: set null instead of default value to nonexistent field * chore: fix partitioning strategy * test: complete unit test * feat: add configurable date format for timestamp partition key * chore: change default value and docs * Maxcompute instrumentation (#55) * added instrumentation for MC sink * chore: refactor InsertManager to abstract class * chore: revert spacing change * chore: refactor --------- Co-authored-by: Eka Winata <[email protected]> * feat: add zone offset for timestamp * feat: use zone offset config for metadata util * feat: Use ZoneId instead of ZoneOffset * feat: use ali auto partitioning strategy [todo fix ProtoDataColumnRecordDecoratorTest] * test: fix ProtoDataColumnRecordDecoratorTest * chore: remove unused config * fix: set MaxComputeCache when partitioningStrategy is not null * chore: checkstyle * feat: add checking on sql result * fix: metadata util * chore: checkstyle * fix: add validation for enum based config and make it fail fast on initialization * chore: fix exception type * chore: checkstyle * fix: class naming and add validation for enum value * chore: Fix MetadataUtil to accept List of TupleString * chore: switch metadata ordering * - refactor schema cache to factory method - fix schema cache to fetch from metadata server * Fix MaxComputeSchemaCache and its test * chore: if not exists true table creation parameter * - checkstyle SinkConfigUtils - add test for SinkConfigUtils * - add retry utils - separate DDL operation to DdlManager * test: Add test for RetryUtils.java * feat: add global configs as externalized parameter * feat: refactor streaming management to StreamingSessionManager class * feat: externalize partition time unit configuration * fix: refactor TimestampPartitioningStrategy * test: add case when passed object is not Record * chore: remove null checking since message couldn't be null * chore: add synchronization on schema update method * chore: checkstyle main * chore: checkstyle main * chore: update docs * chore: update schema docs * chore: reorder annotation in config * chore: change version to 0.10.0 * chore: bump aliyun version * chore: remove redundant enum converter class * feat: use guava cache for streaming session * chore: use sessionCache.getUnchecked * test: update test * chore: checkstyle * chore: use const instead of literal string * chore: wrap update statement with backtick * test: add IOException use case * chore: refactor RetryUtils to receive exception predicate * chore: remove validateConfig() * fix: make RecordWrapper immutable * chore: optimize variable declaration on MaxComputeOdpsGlobalSettingsConverter * fix: remove unused TableTunnel dependencies from InsertManager.java and its implementation * fix: setup FlushOption on the constructor for InsertManager * fix: rename static factory method of streaming session manager * chore: add javadoc for PayloadConverter * chore: refactor StreamingSessionManager to receive LoadingCache in its constructor * chore: Refactor PrimitiveTypeInfoConverter to use ImmutableMap * chore: Rename ConverterOrchestrator.java to ProtobufConverterOrchestrator.java * chore: refactor name and type info to static var * chore: move MaxComputeSchemaHelper to root MaxCompute package * chore: rename getDdlDeclaration to getDDL * chore: remove builder on MaxComputeSchema * chore: fix indent * refactor: MaxComputeSchemaCache to separate handling between non-null and null descriptor * refactor: use google Sets on PartitioningStrategyFactory * refactor: make TableValidator limit configurable * refactor: use immutable map on MetadataUtil * refactor: remove SinkConfigUtils.java and its test. Transfer the implementation of util method directly in class using it * chore: add static imports for assertions lib * chore: rename PayloadConverter.java to ProtobufPayloadConverter * feat: add validation for timestamp type * chore: update docs and default value for valid min and max timestamp * refactor: wrap payload converter to work with DTO * chore: set default maximum session count to 2 for SINK_MAXCOMPUTE_STREAMING_INSERT_MAXIMUM_SESSION_COUNT * test: fix assertion * feat: add timestamp difference validation * feat: Add NaN and Infinite check float and double * chore: rename upsertTable to createOrUpdateTable * chore: refactor inline lambda to static method * chore: update docs * fix: use server side schema for building metadata * chore: add docs for SINK_CONNECTOR_SCHEMA_PROTO_UNKNOWN_FIELDS_VALIDATION * chore: rename MaxComputeSchemaHelper.java to MaxComputeSchemaBuilder * refactor: Add method to add uniform errors in SinkResponse and apply it to MaxComputeSink * feat: parameterized SINK_MAXCOMPUTE_STREAMING_INSERT_TUNNEL_SLOT_COUNT_PER_SESSION * chore: update generic.md to use more explicit context * chore: update maxcompute.md to show the correct type mapping * chore: rename object to parsedObject in ProtoPayload.java * chore: refactor `convert` method under ProtobufConverterOrchestrator to toMaxComputeTypeInfo and toMaxComputeValue * chore: add javadoc for ProtobufTypeInfoConverter.java * chore: checkstyle * chore: add static import * chore: remove extraneous space * refactor: Consolidate TypeInfoConverter and PayloadConverter to single interface * chore: remove consolidated interface and implementation * chore: checkstyle * chore: checkstyle * refactor: Add typeInfo caching * fix: checkstyle * refactor: remove computeIfAbsent * fix: Cache in top level * chore: use cached convertSingularPayload * refactor: remove proxy object reference and bootstrap * refactor: remove proxy MaxComputeSinkConfig to TimestampProtobufMaxComputeConverter * refactor: unknown field validation to not use from config directly * chore: checkstyle * chore: invert the condition to validate ParsedMessage * feat: add instrumentation for unknown field validation latency * feat: add session count metrics * chore: fix checkstyle test + use captureValue instead of captureCount for session manager * feat: add streaming session initialization metrics * feat: add feature flag to instrument metrics * refactor: change instrumentation metric naming for session created * refactor: refactor class where unknown field validation happened to refer to bootstrapped flag field * chore: bump depot version * chore: Update docs for generic config * chore: Use concurrent hashmap * chore: use junit4 assertions * feat: use thread name as part of the session key * docs: Add Javadocs for maxcompute * docs: Add Javadocs for ProtoUnknownFieldValidationType * chore: revert to use partition spec key * refactor: use partition spec as sole key, remove MaxComputeClient dependency from MaxComputeSink, fix checkstyle * feat: add configuration to enable default proto value * fix: add null object directly for default value enabled message field * chore: change odps version to 0.51.0-public * fix: revert default unset field * refactor: abstract base session builder * refactor: simplify interface method structure * refactor: refactor access modifier for const * refactor: checkstyle * refactor: simplify getStreamingSessionManager * test: add static import on mockito methods * refactor: register primitives as predefined set * refactor: move MaxComputeProtobufConverterCache.java to converter package * test: add unit test for MaxComputeProtobufConverterCache * refactor: simplify InsertManager creation * refactor: remove unused interface method and its test * refactor: use get to check the existence of entry * refactor: pass StatsDReporter instead of instrumentation * chore: Add TableTunnel docs on StreamingSessionManager + refactor params * refactor: remove unused constant * refactor: use switch case * refactor: explicitly mention unknown field only checked on the first element --------- Co-authored-by: Vaishnavi190900 <[email protected]>
…#60) * fix: add non-retryable and retryable error handling in insert manager * fix: fix assertion * test: add unit test for non retryable exception in MaxComputeSinkTest * chore: bump version * fix assertion * refresh all session instead of clearing it out * feat: track individual schema mismatch count * chore: bump odps SDK to 0.51.3-public * chore: exclude slf4j from ODPS lib * chore: change log keyword from schema to sessions * fix: remove direct reference from sinkConfig * fix: fix test by removing backtick since the API definition changed by ali * refactor: make StreamingSessionManager protected and refer it directly from implementor * revert
* feat: support switching between timestamp vs timestampntz * chore: bump version to 0.10.2 * chore: checkstyle * chore: add docs * chore: wrap factory params * Rename config * refactor: rename MaxComputeTimeUnitType.java to MaxComputeTimestampDataType.java * refactor: change ternary operation to if else * refactor: make Ntz to uppercase * refactor: extract common logic to validator class * refactor: extract magic word * fix: checkstyle * chore: bump MC SDK to 0.51.3-public * fix: remove direct reference from SinkConfig in conversion method call * chore: exclude slf4j from odps deps * chore: bump version to 0.10.3
* feat: Add different casting strategy for numeric types * chore: checkstyle * fix: throw exception when attempting to modify non struct type * test: Change IllegalArgumentException to UnsupportedOperationException and add more test for SchemaDifferenceUtils * refactor: rename initializer package to strategy * test: Add BasePrimitiveProtobufMappingStrategyTest, IntegerPrimitiveProtobufMappingStrategyTest, UpcastedIntegerPrimitiveProtobufMappingStrategyTest * test: [WIP] add DecimalCastedFloatPrimitiveProtobufMappingStrategyTest and FloatPrimitiveProtobufMappingStrategyTest * refactor: check whether differences are primitive or not * test: add test for Double stategy * refactor: rename maxcomputesinkconfig numeric related * refactor: naming * checkstyle * add test ProtoPrimitiveDataTypeMappingStrategyFactoryTest * add docs * chore: bump version to 0.10.4 * chore: checkstyle * use String method * Toggle INT or BIGINT in metadata * feat: pass table properties when creating tables * chore: bump odps version to 0.51.6-public * chore: add props * refactor: rename to KeyValuePairsToMapConverter * refactor: rename to ProtoPrimitiveDataTypeMapper * refactor: rename method to toBinaryFrom * refactor: rename to isValid * refactor: separate package * test: add non numeric test * refactor: rename config * refactor: ProtoPrimitiveDataTypeMapper remove merge, build complete map on the factory method * refactor: append TYPE suffix in the config
* feat: add float to double converter * chore: bump version
* bugfix: set nanos in Duration to BIGINT MC type to comply with MMS * bump version * checkstyle
* fix: negative nano handling * test: adding unit tests and refactor code * test case changed * refactoring and docs added
…he Improvement (#69) * [chore] Bump odps lib version & conditionally apply tunnel endpoint * [refactor] Use stingUtils nonempty * [refactor] use dynamicMessage.getField(...) when getting the field values * [refactor] clear cache after schema creation to avoid concurrent update by other threads * [refactor] bump odps to 0.51.9-public * [fix] revert gradle
…age Discrepancy (#66) * change the signature type * Add config for max message depth * checkstyle * bump depot * [fix] Add level as cache key * [refactor] remove isRootLevel field and infer it from level * [refactor] add docs and more checking * [refactor] add docs and more checking * [fix] print config name * [fix] remove upper limit checking * [refactor] remove unused config * checkstyle test * extract filter logic * simplify constructor * [refactor] Refactor ProtobufConverterOrchestrator * [refactor] Refactor method * [fix] Use default value for primitive types and skip message type for unset field * [chore] bump to 0.10.9 * [fix] checkstyle * [refactor] add logging when filtering * [fix] Treat unset string as null instead of empty string * [fix] fix test
[chore] use reoderable struct
* [fix] use lowercase when building tree path in schema difference utils * [chore] bump version to 0.10.11
* [fix] Fix empty message parsed causing all messages within single poll fails * Update build.gradle
* [fix] Fix empty message parsed causing all messages within single poll fails * Update build.gradle * [refactor] Externalize allowSchemaMismatch to configuration props * [chore] update docs * [chore] update docs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.