Destination MSSQL: standard inserts uses new typing interface #55849
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| } | ||
| } else { | ||
| if (field.name == COLUMN_NAME_AB_META) { | ||
| // don't populate _airbyte_meta yet - we might run into errors in the other fields |
There was a problem hiding this comment.
this isn't new, it matches the previous behavior (COLUMN_NAME_AB_META -> airbyteMetaStatementIndex = statementIndex). Just adding a comment b/c it's nonobvious.
...tion-mssql/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/MSSQLQueryBuilder.kt
Outdated
Show resolved
Hide resolved
| setAdditionalProperty("sync_id", stream.syncId) | ||
| } | ||
| val enrichedRecord = | ||
| plainRecord.asDestinationRecordRaw().asEnrichedDestinationRecordAirbyteValue() |
There was a problem hiding this comment.
Hm, the two steps feels wrong. If the point is to get everyone on the same interface (a single type that can be marshaled into a bunch of stuff) then oughtn't we be giving everyone the "RecordRaw"
There was a problem hiding this comment.
yeah, I misread what we were doing in s3datalake. how does 6525221 look?
I do think passing RecordRaw is correct though? B/c it's useful for destinations that just want the plain JSON blob - blob destination non-flattening csv/jsonl, warehouses in raw tables mode, etc
There was a problem hiding this comment.
Yes, pass the thing that can be marshaled to lots of stuff, and have a layer between whatever the framework uses and that.
frifriSF59
force-pushed
the
frifri/as_enriched_values
branch
from
01f2d5c to
5b44513
Compare
...rc/integrationTest/kotlin/io/airbyte/cdk/load/mock_integration_test/MockDestinationWriter.kt
Show resolved
Hide resolved
frifriSF59
force-pushed
the
frifri/as_enriched_values
branch
from
5b44513 to
6a23592
Compare
edgao
force-pushed
the
frifri/as_enriched_values
branch
from
2cf1eb1 to
726abb8
Compare
edgao
changed the title
| @@ -302,7 +302,7 @@ abstract class IntegrationTest( | |||
| fun updateConfig(config: String): String = configUpdater.update(config) | |||
|
|
|||
| companion object { | |||
| val randomizedNamespaceRegex = Regex("test(\\d{8})[A-Za-z]{4}") | |||
| val randomizedNamespaceRegex = Regex("test(\\d{8})[A-Za-z]{4}.*") | |||
There was a problem hiding this comment.
Sorry, can you explain this change please?
There was a problem hiding this comment.
yeah, this was what I mean by
fix the DataCleaner regex - now it correctly matches test20241212abcd_1
in the pr description. I.e. previously the regex only recognized test20241212abcd, so we weren't deleting any namespaces that ended with a _1 / _2 suffix 🤦
(... in related news, I deleted about 1K databases from AWS glue today)
airbyteMetaField: Meta.AirbyteMetaFields?to EnrichedAirbyteValue (we thought we didn't need it b/c we separated airbyteMetaFields / declaredFields; turns out it's still useful)airbyteMetaspecifically from EnrichedDestinationRecordAirbyteValue, instead of hiding it inside airbyteMetaFieldstest20241212abcd_1