Skip to content

Commit 0b39232

Browse files
psainicssgarg-CS
psainics
authored and
sgarg-CS
committed
Set TRANSACTION_ISOLATION_LEVEL config in MySQL, PostgreSQL & MSSQL plugins
Add transaction isolation level to PostgreSQL, MYSQL and MSSQL Plugins.
1 parent 4359d59 commit 0b39232

34 files changed

+366
-56
lines changed

‎.github/workflows/build-report.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@ on:
2121
types:
2222
- completed
2323

24+
permissions:
25+
actions: read # Allows reading workflow run information
26+
statuses: write # Required if the action updates commit statuses
27+
checks: write # Required if it updates GitHub Checks API
28+
2429
jobs:
2530
build:
2631
runs-on: ubuntu-latest

‎.github/workflows/build.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,12 @@ jobs:
3737
&& (github.event.action != 'labeled' || github.event.label.name == 'build')
3838
)
3939
steps:
40-
- uses: actions/checkout@v3
40+
- uses: actions/checkout@v4
4141
with:
4242
ref: ${{ github.event.workflow_run.head_sha }}
4343

4444
- name: Cache
45-
uses: actions/cache@v3
45+
uses: actions/cache@v4
4646
with:
4747
path: ~/.m2/repository
4848
key: ${{ runner.os }}-maven-${{ github.workflow }}-${{ hashFiles('**/pom.xml') }}

‎.github/workflows/e2e.yml

Lines changed: 29 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@ name: Build e2e tests
1616

1717
on:
1818
push:
19-
branches: [ develop, release/* ]
19+
branches: [ develop, release/** ]
2020
pull_request:
21-
branches: [ develop, release/* ]
21+
branches: [ develop, release/** ]
2222
types: [ opened, synchronize, reopened, labeled ]
2323
workflow_dispatch:
2424

@@ -45,7 +45,7 @@ jobs:
4545

4646
steps:
4747
# Pinned 1.0.0 version
48-
- uses: actions/checkout@v3
48+
- uses: actions/checkout@v4
4949
with:
5050
path: plugin
5151
submodules: 'recursive'
@@ -61,14 +61,14 @@ jobs:
6161
- '${{ matrix.module }}/**/e2e-test/**'
6262
6363
- name: Checkout e2e test repo
64-
uses: actions/checkout@v3
64+
uses: actions/checkout@v4
6565
with:
6666
repository: cdapio/cdap-e2e-tests
6767
path: e2e
6868
ref: release/6.10
6969

7070
- name: Cache
71-
uses: actions/cache@v3
71+
uses: actions/cache@v4
7272
with:
7373
path: ~/.m2/repository
7474
key: ${{ runner.os }}-maven-${{ github.workflow }}-${{ hashFiles('**/pom.xml') }}
@@ -96,6 +96,12 @@ jobs:
9696
POSTGRESQL_USERNAME:cdapio-github-builds/POSTGRESQL_USERNAME
9797
POSTGRESQL_PASSWORD:cdapio-github-builds/POSTGRESQL_PASSWORD
9898
POSTGRESQL_PORT:cdapio-github-builds/POSTGRESQL_PORT
99+
CLOUDSQL_POSTGRESQL_USERNAME:cdapio-github-builds/CLOUDSQL_POSTGRESQL_USERNAME
100+
CLOUDSQL_POSTGRESQL_PASSWORD:cdapio-github-builds/CLOUDSQL_POSTGRESQL_PASSWORD
101+
CLOUDSQL_POSTGRESQL_CONNECTION_NAME:cdapio-github-builds/CLOUDSQL_POSTGRESQL_CONNECTION_NAME
102+
CLOUDSQL_MYSQL_USERNAME:cdapio-github-builds/CLOUDSQL_MYSQL_USERNAME
103+
CLOUDSQL_MYSQL_PASSWORD:cdapio-github-builds/CLOUDSQL_MYSQL_PASSWORD
104+
CLOUDSQL_MYSQL_CONNECTION_NAME:cdapio-github-builds/CLOUDSQL_MYSQL_CONNECTION_NAME
99105
100106
- name: Run required e2e tests
101107
if: github.event_name != 'workflow_dispatch' && github.event_name != 'push' && steps.filter.outputs.e2e-test == 'false'
@@ -117,6 +123,12 @@ jobs:
117123
POSTGRESQL_USERNAME: ${{ steps.secrets.outputs.POSTGRESQL_USERNAME }}
118124
POSTGRESQL_PASSWORD: ${{ steps.secrets.outputs.POSTGRESQL_PASSWORD }}
119125
POSTGRESQL_PORT: ${{ steps.secrets.outputs.POSTGRESQL_PORT }}
126+
CLOUDSQL_POSTGRESQL_USERNAME: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_USERNAME }}
127+
CLOUDSQL_POSTGRESQL_PASSWORD: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_PASSWORD }}
128+
CLOUDSQL_POSTGRESQL_CONNECTION_NAME: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_CONNECTION_NAME }}
129+
CLOUDSQL_MYSQL_USERNAME: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_USERNAME }}
130+
CLOUDSQL_MYSQL_PASSWORD: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_PASSWORD }}
131+
CLOUDSQL_MYSQL_CONNECTION_NAME: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_CONNECTION_NAME }}
120132

121133
- name: Run all e2e tests
122134
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || steps.filter.outputs.e2e-test == 'true'
@@ -138,25 +150,28 @@ jobs:
138150
POSTGRESQL_USERNAME: ${{ steps.secrets.outputs.POSTGRESQL_USERNAME }}
139151
POSTGRESQL_PASSWORD: ${{ steps.secrets.outputs.POSTGRESQL_PASSWORD }}
140152
POSTGRESQL_PORT: ${{ steps.secrets.outputs.POSTGRESQL_PORT }}
141-
142-
- name: Upload report
143-
uses: actions/upload-artifact@v3
144-
if: always()
145-
with:
146-
name: Cucumber report - ${{ matrix.module }}
147-
path: ./**/target/cucumber-reports
153+
CLOUDSQL_POSTGRESQL_USERNAME: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_USERNAME }}
154+
CLOUDSQL_POSTGRESQL_PASSWORD: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_PASSWORD }}
155+
CLOUDSQL_POSTGRESQL_CONNECTION_NAME: ${{ steps.secrets.outputs.CLOUDSQL_POSTGRESQL_CONNECTION_NAME }}
156+
CLOUDSQL_MYSQL_USERNAME: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_USERNAME }}
157+
CLOUDSQL_MYSQL_PASSWORD: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_PASSWORD }}
158+
CLOUDSQL_MYSQL_CONNECTION_NAME: ${{ steps.secrets.outputs.CLOUDSQL_MYSQL_CONNECTION_NAME }}
148159

149160
- name: Upload debug files
150-
uses: actions/upload-artifact@v3
161+
uses: actions/upload-artifact@v4
151162
if: always()
152163
with:
153164
name: Debug files - ${{ matrix.module }}
154165
path: ./**/target/e2e-debug
155166

156167
- name: Upload files to GCS
157-
uses: google-github-actions/upload-cloud-storage@v0
168+
uses: google-github-actions/upload-cloud-storage@v2
158169
if: always()
159170
with:
160171
path: ./plugin
161172
destination: e2e-tests-cucumber-reports/${{ github.event.repository.name }}/${{ github.ref }}
162173
glob: '**/target/cucumber-reports/**'
174+
175+
- name: Cucumber Report URL
176+
if: always()
177+
run: echo "https://storage.googleapis.com/e2e-tests-cucumber-reports/${{ github.event.repository.name }}/${{ github.ref }}/plugin/${{ matrix.module }}/target/cucumber-reports/advanced-reports/cucumber-html-reports/overview-features.html"

‎database-commons/src/main/java/io/cdap/plugin/db/ConnectionConfig.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ public abstract class ConnectionConfig extends PluginConfig implements DatabaseC
4545
public static final String CONNECTION_ARGUMENTS = "connectionArguments";
4646
public static final String JDBC_PLUGIN_NAME = "jdbcPluginName";
4747
public static final String JDBC_PLUGIN_TYPE = "jdbc";
48+
public static final String TRANSACTION_ISOLATION_LEVEL = "transactionIsolationLevel";
4849

4950
@Name(JDBC_PLUGIN_NAME)
5051
@Description("Name of the JDBC driver to use. This is the value of the 'jdbcPluginName' key defined in the JSON " +

‎database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBSpecificConnectorConfig.java

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,9 @@
2020
import io.cdap.cdap.api.annotation.Macro;
2121
import io.cdap.cdap.api.annotation.Name;
2222
import io.cdap.plugin.db.ConnectionConfig;
23+
import io.cdap.plugin.db.TransactionIsolationLevel;
2324

24-
import java.util.Collections;
25+
import java.util.HashMap;
2526
import java.util.Map;
2627
import javax.annotation.Nullable;
2728

@@ -42,6 +43,12 @@ public abstract class AbstractDBSpecificConnectorConfig extends AbstractDBConnec
4243
@Nullable
4344
protected Integer port;
4445

46+
@Name(ConnectionConfig.TRANSACTION_ISOLATION_LEVEL)
47+
@Description("The transaction isolation level for the database session.")
48+
@Macro
49+
@Nullable
50+
protected String transactionIsolationLevel;
51+
4552
public String getHost() {
4653
return host;
4754
}
@@ -55,4 +62,21 @@ public int getPort() {
5562
public boolean canConnect() {
5663
return super.canConnect() && !containsMacro(ConnectionConfig.HOST) && !containsMacro(ConnectionConfig.PORT);
5764
}
65+
66+
@Override
67+
public Map<String, String> getAdditionalArguments() {
68+
Map<String, String> additonalArguments = new HashMap<>();
69+
if (getTransactionIsolationLevel() != null) {
70+
additonalArguments.put(TransactionIsolationLevel.CONF_KEY, getTransactionIsolationLevel());
71+
}
72+
return additonalArguments;
73+
}
74+
75+
public String getTransactionIsolationLevel() {
76+
if (transactionIsolationLevel == null) {
77+
return null;
78+
}
79+
return TransactionIsolationLevel.Level.valueOf(transactionIsolationLevel).name();
80+
}
5881
}
82+

‎mssql-plugin/docs/SQL Server-connector.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ authentication. Optional for databases that do not require authentication.
2222

2323
**Password:** Password to use to connect to the specified database.
2424

25+
**Transaction Isolation Level** The transaction isolation level of the database connection
26+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
27+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
28+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
29+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
30+
31+
For more details on the Transaction Isolation Levels supported in SQL Server, refer to the [SQL Server documentation](https://learn.microsoft.com/en-us/sql/t-sql/statements/set-transaction-isolation-level-transact-sql?view=sql-server-ver16)
32+
2533
**Authentication Type:** Indicates which authentication method will be used for the connection. Use 'SQL Login'. to
2634
connect to a SQL Server using username and password properties. Use 'Active Directory Password' to connect to an Azure
2735
SQL Database/Data Warehouse using an Azure AD principal name and password.

‎mssql-plugin/docs/SqlServer-batchsink.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,14 @@ an Azure SQL Database/Data Warehouse using an Azure AD principal name and passwo
4646

4747
**Password:** Password to use to connect to the specified database.
4848

49+
**Transaction Isolation Level** The transaction isolation level of the database connection
50+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
51+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
52+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
53+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
54+
55+
For more details on the Transaction Isolation Levels supported in SQL Server, refer to the [SQL Server documentation](https://learn.microsoft.com/en-us/sql/t-sql/statements/set-transaction-isolation-level-transact-sql?view=sql-server-ver16)
56+
4957
**Instance Name:** SQL Server instance name to connect to. When it is not specified, a
5058
connection is made to the default instance. For the case where both the instanceName and port are specified,
5159
see the notes for port. If you specify a Virtual Network Name in the Server connection property, you cannot

‎mssql-plugin/docs/SqlServer-batchsource.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,14 @@ an Azure SQL Database/Data Warehouse using an Azure AD principal name and passwo
5656

5757
**Password:** Password to use to connect to the specified database.
5858

59+
**Transaction Isolation Level** The transaction isolation level of the database connection
60+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
61+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
62+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
63+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
64+
65+
For more details on the Transaction Isolation Levels supported in SQL Server, refer to the [SQL Server documentation](https://learn.microsoft.com/en-us/sql/t-sql/statements/set-transaction-isolation-level-transact-sql?view=sql-server-ver16)
66+
5967
**Instance Name:** SQL Server instance name to connect to. When it is not specified, a
6068
connection is made to the default instance. For the case where both the instanceName and port are specified,
6169
see the notes for port. If you specify a Virtual Network Name in the Server connection property, you cannot

‎mssql-plugin/src/main/java/io/cdap/plugin/mssql/SqlServerSink.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,11 @@ public Map<String, String> getDBSpecificArguments() {
167167
packetSize, queryTimeout);
168168
}
169169

170+
@Override
171+
public String getTransactionIsolationLevel() {
172+
return connection.getTransactionIsolationLevel();
173+
}
174+
170175
@Override
171176
public String getConnectionString() {
172177
return String.format(SqlServerConstants.SQL_SERVER_CONNECTION_STRING_FORMAT,

‎mssql-plugin/src/main/java/io/cdap/plugin/mssql/SqlServerSource.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,11 @@ public List<String> getInitQueries() {
188188
return Collections.emptyList();
189189
}
190190

191+
@Override
192+
public String getTransactionIsolationLevel() {
193+
return connection.getTransactionIsolationLevel();
194+
}
195+
191196
@Override
192197
public void validate(FailureCollector collector) {
193198
ConfigUtil.validateConnection(this, useConnection, connection, collector);

‎mssql-plugin/widgets/SQL Server-connector.json

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,20 @@
6464
"widget-type": "password",
6565
"label": "Password",
6666
"name": "password"
67+
},
68+
{
69+
"widget-type": "select",
70+
"label": "Transaction Isolation Level",
71+
"name": "transactionIsolationLevel",
72+
"widget-attributes": {
73+
"values": [
74+
"TRANSACTION_READ_UNCOMMITTED",
75+
"TRANSACTION_READ_COMMITTED",
76+
"TRANSACTION_REPEATABLE_READ",
77+
"TRANSACTION_SERIALIZABLE"
78+
],
79+
"default": "TRANSACTION_SERIALIZABLE"
80+
}
6781
}
6882
]
6983
},

‎mssql-plugin/widgets/SqlServer-batchsink.json

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,20 @@
8484
"label": "Password",
8585
"name": "password"
8686
},
87+
{
88+
"widget-type": "select",
89+
"label": "Transaction Isolation Level",
90+
"name": "transactionIsolationLevel",
91+
"widget-attributes": {
92+
"values": [
93+
"TRANSACTION_READ_UNCOMMITTED",
94+
"TRANSACTION_READ_COMMITTED",
95+
"TRANSACTION_REPEATABLE_READ",
96+
"TRANSACTION_SERIALIZABLE"
97+
],
98+
"default": "TRANSACTION_SERIALIZABLE"
99+
}
100+
},
87101
{
88102
"widget-type": "keyvalue",
89103
"label": "Connection Arguments",
@@ -267,6 +281,10 @@
267281
{
268282
"type": "property",
269283
"name": "connectionArguments"
284+
},
285+
{
286+
"type": "property",
287+
"name": "transactionIsolationLevel"
270288
}
271289
]
272290
},

‎mssql-plugin/widgets/SqlServer-batchsource.json

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,20 @@
8484
"label": "Password",
8585
"name": "password"
8686
},
87+
{
88+
"widget-type": "select",
89+
"label": "Transaction Isolation Level",
90+
"name": "transactionIsolationLevel",
91+
"widget-attributes": {
92+
"values": [
93+
"TRANSACTION_READ_UNCOMMITTED",
94+
"TRANSACTION_READ_COMMITTED",
95+
"TRANSACTION_REPEATABLE_READ",
96+
"TRANSACTION_SERIALIZABLE"
97+
],
98+
"default": "TRANSACTION_SERIALIZABLE"
99+
}
100+
},
87101
{
88102
"widget-type": "keyvalue",
89103
"label": "Connection Arguments",
@@ -316,6 +330,10 @@
316330
{
317331
"type": "property",
318332
"name": "connectionArguments"
333+
},
334+
{
335+
"type": "property",
336+
"name": "transactionIsolationLevel"
319337
}
320338
]
321339
},

‎mysql-plugin/docs/MySQL-connector.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ authentication. Optional for databases that do not require authentication.
2222

2323
**Password:** Password to use to connect to the specified database.
2424

25+
**Transaction Isolation Level** The transaction isolation level of the databse connection
26+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
27+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
28+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
29+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
30+
31+
For more details on the Transaction Isolation Levels supported in MySQL, refer to the [MySQL documentation](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html)
32+
2533
**Connection Arguments:** A list of arbitrary string tag/value pairs as connection arguments. These arguments
2634
will be passed to the JDBC driver, as connection arguments, for JDBC drivers that may need additional configurations.
2735
This is a semicolon-separated list of key-value pairs, where each pair is separated by a equals '=' and specifies

‎mysql-plugin/docs/Mysql-batchsink.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,14 @@ You also can use the macro function ${conn(connection-name)}.
3939

4040
**Password:** Password to use to connect to the specified database.
4141

42+
**Transaction Isolation Level** The transaction isolation level of the databse connection
43+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
44+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
45+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
46+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
47+
48+
For more details on the Transaction Isolation Levels supported in MySQL, refer to the [MySQL documentation](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html)
49+
4250
**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
4351
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.
4452

‎mysql-plugin/docs/Mysql-batchsource.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,14 @@ For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is s
4949

5050
**Password:** Password to use to connect to the specified database.
5151

52+
**Transaction Isolation Level** The transaction isolation level of the database connection
53+
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
54+
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
55+
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
56+
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.
57+
58+
For more details on the Transaction Isolation Levels supported in MySQL, refer to the [MySQL documentation](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html)
59+
5260
**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
5361
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.
5462

0 commit comments

Comments
 (0)