Skip to content

Commit 8150737

Browse files
vischaaronsteers
visch
and
aaronsteers
authored
feat: Activate Version (#89)
* Activat Version example test * Activate version working * Lint and Schema fix for config * Activate Version test to show data being deleted * Lint Fix * Activate version is doing things correctly, existing implementations of ACTIVATE_VERSION sometimes send an ACTIVATE_MESSAGE before records are sent which I think is just a bad tap implemention but it is valid * Consistent spacing for docs * Update README.md Co-authored-by: Aaron ("AJ") Steers <[email protected]> --------- Co-authored-by: Aaron ("AJ") Steers <[email protected]>
1 parent c66b0e2 commit 8150737

8 files changed

+249
-18
lines changed

‎README.md

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -14,20 +14,22 @@ Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Target
1414
* `schema-flattening`
1515

1616
## Settings
17-
18-
| Setting | Required | Default | Description |
19-
| :------------------- | :------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
20-
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
21-
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
22-
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
23-
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
24-
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
25-
| sqlalchemy_url | False | None | SQLAlchemy connection string. This will override using host, user, password, port, dialect. Note that you must esacpe password special characters properly see https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords |
26-
| dialect+driver | False | postgresql+psycopg2 | Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. Note if sqlalchemy_url is set this will be ignored. |
27-
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
28-
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
29-
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
30-
| flattening_max_depth | False | None | The max depth to flatten schemas. |
17+
| Setting | Required | Default | Description |
18+
| :-------------------- | :------: | :-----------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
19+
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
20+
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
21+
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
22+
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
23+
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
24+
| sqlalchemy_url | False | None | SQLAlchemy connection string. This will override using host, user, password, port,dialect. Note that you must escape password special characters properly. See https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords |
25+
| dialect+driver | False | postgresql+psycopg2 | Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. Note if sqlalchemy_url is set this will be ignored. |
26+
| default_target_schema | False | None | Postgres schema to send data to, example: tap-clickup |
27+
| hard_delete | False | 0 | When activate version is sent from a tap this specefies if we should delete the records that don't match, or mark them with a date in the `_sdc_deleted_at` column. |
28+
| add_record_metadata | False | 1 | Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information. |
29+
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
30+
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
31+
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
32+
| flattening_max_depth | False | None | The max depth to flatten schemas. |
3133

3234
A full list of supported settings and capabilities is available by running: `target-postgres --about`
3335

‎target_postgres/sinks.py

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,12 @@
22
import uuid
33
from typing import Any, Dict, Iterable, List, Optional, Union
44

5+
import sqlalchemy
6+
from pendulum import now
57
from singer_sdk.sinks import SQLSink
68
from sqlalchemy import Column, MetaData, Table, insert
79
from sqlalchemy.sql import Executable
10+
from sqlalchemy.sql.expression import bindparam
811

912
from target_postgres.connector import PostgresConnector
1013

@@ -252,3 +255,67 @@ def schema_name(self) -> Optional[str]:
252255

253256
# Schema name not detected.
254257
return None
258+
259+
def activate_version(self, new_version: int) -> None:
260+
"""Bump the active version of the target table.
261+
262+
Args:
263+
new_version: The version number to activate.
264+
"""
265+
# There's nothing to do if the table doesn't exist yet
266+
# (which it won't the first time the stream is processed)
267+
if not self.connector.table_exists(self.full_table_name):
268+
return
269+
270+
deleted_at = now()
271+
# Different from SingerSDK as we need to handle types the
272+
# same as SCHEMA messsages
273+
datetime_type = self.connector.to_sql_type(
274+
{"type": "string", "format": "date-time"}
275+
)
276+
277+
# Different from SingerSDK as we need to handle types the
278+
# same as SCHEMA messsages
279+
integer_type = self.connector.to_sql_type({"type": "integer"})
280+
281+
if not self.connector.column_exists(
282+
full_table_name=self.full_table_name,
283+
column_name=self.version_column_name,
284+
):
285+
self.connector.prepare_column(
286+
self.full_table_name,
287+
self.version_column_name,
288+
sql_type=integer_type,
289+
)
290+
291+
self.logger.info("Hard delete: %s", self.config.get("hard_delete"))
292+
if self.config["hard_delete"] is True:
293+
self.connection.execute(
294+
f"DELETE FROM {self.full_table_name} "
295+
f"WHERE {self.version_column_name} <= {new_version} "
296+
f"OR {self.version_column_name} IS NULL"
297+
)
298+
return
299+
300+
if not self.connector.column_exists(
301+
full_table_name=self.full_table_name,
302+
column_name=self.soft_delete_column_name,
303+
):
304+
self.connector.prepare_column(
305+
self.full_table_name,
306+
self.soft_delete_column_name,
307+
sql_type=datetime_type,
308+
)
309+
# Need to deal with the case where data doesn't exist for the version column
310+
query = sqlalchemy.text(
311+
f"UPDATE {self.full_table_name}\n"
312+
f"SET {self.soft_delete_column_name} = :deletedate \n"
313+
f"WHERE {self.version_column_name} < :version "
314+
f"OR {self.version_column_name} IS NULL \n"
315+
f" AND {self.soft_delete_column_name} IS NULL\n"
316+
)
317+
query = query.bindparams(
318+
bindparam("deletedate", value=deleted_at, type_=datetime_type),
319+
bindparam("version", value=new_version, type_=integer_type),
320+
)
321+
self.connector.connection.execute(query)

‎target_postgres/target.py

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ def __init__(
7171
th.StringType,
7272
description=(
7373
"User name used to authenticate. "
74-
+ "Note if sqlalchemy_url is set this will be ignored.",
74+
+ "Note if sqlalchemy_url is set this will be ignored."
7575
),
7676
),
7777
th.Property(
@@ -95,9 +95,9 @@ def __init__(
9595
th.StringType,
9696
description=(
9797
"SQLAlchemy connection string. "
98-
+ "This will override using host, user, password, port,"
99-
+ "dialect. Note that you must esacpe password special"
100-
+ "characters properly see"
98+
+ "This will override using host, user, password, port, "
99+
+ "dialect. Note that you must esacpe password special "
100+
+ "characters properly see "
101101
+ "https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords" # noqa: E501
102102
),
103103
),
@@ -117,6 +117,27 @@ def __init__(
117117
th.StringType,
118118
description="Postgres schema to send data to, example: tap-clickup",
119119
),
120+
th.Property(
121+
"hard_delete",
122+
th.BooleanType,
123+
default=False,
124+
description=(
125+
"When activate version is sent from a tap this specefies "
126+
+ "if we should delete the records that don't match, or mark "
127+
+ "them with a date in the `_sdc_deleted_at` column."
128+
),
129+
),
130+
th.Property(
131+
"add_record_metadata",
132+
th.BooleanType,
133+
default=True,
134+
description=(
135+
"Note that this must be enabled for activate_version to work!"
136+
+ "This adds _sdc_extracted_at, _sdc_batched_at, and more to every "
137+
+ "table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html " # noqa: E501
138+
+ "for more information."
139+
),
140+
),
120141
).to_dict()
121142
default_sink_class = PostgresSink
122143

‎target_postgres/tests/data_files/activate_version_hard.singer

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{"type": "SCHEMA", "stream": "test_activate_version_hard", "schema": {"type": "object", "properties": {"code": {"type": ["string"]}, "name": {"type": ["null", "string"]}}}, "key_properties": ["code"], "bookmark_properties": []}
2+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_hard", "version": 1674486431563}
3+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "AF", "name": "Africa"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
4+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "AN", "name": "Antarctica"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
5+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "AS", "name": "Asia"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
6+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "EU", "name": "Europe"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
7+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "NA", "name": "North America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
8+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "OC", "name": "Oceania"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
9+
{"type": "RECORD", "stream": "test_activate_version_hard", "record": {"code": "SA", "name": "South America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
10+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_hard", "version": 1674486431563}

‎target_postgres/tests/data_files/activate_version_soft.singer

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{"type": "SCHEMA", "stream": "test_activate_version_soft", "schema": {"type": "object", "properties": {"code": {"type": ["string"]}, "name": {"type": ["null", "string"]}}}, "key_properties": ["code"], "bookmark_properties": []}
2+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_soft", "version": 1674486431563}
3+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "AF", "name": "Africa"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
4+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "AN", "name": "Antarctica"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
5+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "AS", "name": "Asia"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
6+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "EU", "name": "Europe"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
7+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "NA", "name": "North America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
8+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "OC", "name": "Oceania"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
9+
{"type": "RECORD", "stream": "test_activate_version_soft", "record": {"code": "SA", "name": "South America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
10+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_soft", "version": 1674486431563}

‎target_postgres/tests/data_files/test_activate_version_deletes_data_properly.singer

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{"type": "SCHEMA", "stream": "test_activate_version_deletes_data_properly", "schema": {"type": "object", "properties": {"code": {"type": ["string"]}, "name": {"type": ["null", "string"]}}}, "key_properties": ["code"], "bookmark_properties": []}
2+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_deletes_data_properly", "version": 1674486431563}
3+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "AF", "name": "Africa"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
4+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "AN", "name": "Antarctica"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
5+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "AS", "name": "Asia"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
6+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "EU", "name": "Europe"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
7+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "NA", "name": "North America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
8+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "OC", "name": "Oceania"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
9+
{"type": "RECORD", "stream": "test_activate_version_deletes_data_properly", "record": {"code": "SA", "name": "South America"}, "version": 1674486431563, "time_extracted": "2023-01-23T15:07:11.563063Z"}
10+
{"type": "ACTIVATE_VERSION", "stream": "test_activate_version_deletes_data_properly", "version": 1674486431563}

0 commit comments

Comments
 (0)