You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If both `active-region.lookup-source` and `active-region.lookup-key` are set, then the workflow will be associated with the entity specified and the active cluster will be determined based on that entity's region.
159
+
160
+
Otherwise, the default behavior is to pick the cluster that received the start workflow request.
161
+
162
+
### Workflow activeness metadata in execution table
163
+
A new row will be added to executions table as part of StartWorkflow LWT for active-active-domains. It will grab external entity source/key info from workflow’s headers if provided. If no such header is provided then current cluster’s region is stored as origin. These records are immutable and are replicated as part of regular per-shard replication tasks.
158
164
159
-
If none of the new parameters are set, the default behavior is to pick the cluster that received the start workflow request.
160
-
It's not allowed to have both active-region.lookup-key and active-region.origin set.
-`wf1` is created on us-west. No external entity associations exist. Its tasks will have failover version of the active cluster in us-west specified in domain's `RegionToClusterMap` which is 0.
173
+
-`wf2` is created with an external entity association. Its tasks will have failover version of following row in `EntityActiveRegion` table:
174
+
<type=user-location, key=london> which is 5. Its tasks will be processed by cluster3 because version 5 maps to us-east region and cluster3 is active in us-east.
175
+
-`wf3` is a long running workflow which was created when the domain was active-passive (migrated to active-active later on). It doesn't have an activeness metadata row in the executions table. Its existing tasks already have failover version of the cluster that it was active in so it will continue to be active in that cluster.
176
+
177
+
TODO: How to come up with such version when loading mutable state of `wf3`?
178
+
179
+
180
+
### Domain records
181
+
182
+
Domain records will have a new field called ActiveClusters which contains the information about region to cluster mapping and failover versions of the clusters. Active-active domains will have this field set and ActiveClusterName field will be empty.
183
+
Active-passive domains will not have this field set and ActiveClusterName field will be set.
184
+
185
+
```
186
+
{
187
+
// Clusters can be a subset of clusters in the group.
188
+
Clusters: [us-west, us-east, eu-west, eu-central]
189
+
190
+
// Active clusters can have at most one from each region. Remaining clusters will be considered passive.
191
+
ActiveClusters: {
192
+
RegionToClusterMap: {
193
+
us-west: {
194
+
ActiveClusterName: cluster0,
195
+
FailoverVersion: 0, # failover version of cluster0.
196
+
},
197
+
us-east: {
198
+
ActiveClusterName: cluster3,
199
+
FailoverVersion: 6, # failover version of cluster3.
200
+
},
201
+
},
202
+
}
203
+
}
204
+
```
205
+
206
+
**RegionToClusterMap:**
207
+
208
+
A map of regions to clusters that the domain can have active workflows in. It's initially formed based on cluster group configuration and given active clusters of the domain. Each region that the domain is active in will have an entry in this map.
209
+
210
+
Workflow task versions are assigned based on the failover version of the region that their associated entity belongs to.
211
+
If there's no entity associated with a workflow, then the task version is assigned based on the failover version of the region that the workflow is created on.
212
+
213
+
Active-passive domains can be turned into active-active domains by setting the ActiveClusters field. There can be open workflows with task versions mapping to a cluster failover version instead of a region failover version. Mapping logic will support this fallback to support migration from active-passive to active-active domains.
214
+
215
+
e.g.
216
+
A long running workflow of an active-passive domain has task in the queue with version 0. This is supposed to be active in cluster0.
217
+
If we make this domain active-active, the task version should resolve to:
218
+
- cluster0 if domain is active in cluster0 (direct mapping).
219
+
- cluster1 if domain is active in cluster1 (fallback to active cluster of the region).
220
+
- if domain has no active cluster in the region, then it's a resolution error and the task will not be processed as active anywhere.
161
221
162
222
163
223
### EntityActiveRegion Lookup Table
224
+
This table is used to lookup the active cluster of a workflow based on the entity that the workflow is associated with.
225
+
164
226
- Primary key of the table is <EntityType, EntityKey>
165
227
- A new plugin system will be introduced to watch supported entity types and populate the table
166
228
- Watchers are per cluster group
@@ -178,43 +240,131 @@ It's not allowed to have both active-region.lookup-key and active-region.origin
### Workflow activeness metadata in execution table
189
-
A new row will be added to executions table as part of StartWorkflow LWT for active-active-domains. It will grab external entity lookup info from workflow’s headers if provided. If no such header is provided then current cluster’s region is stored as origin. These records are immutable and are replicated as part of regular per-shard replication tasks.
245
+
This table is not domain specific so failover versions are not the failover versions of a cluster but instead the failover versions of a region. Region to cluster mapping is specific to domain.
Workflows associated with an entity will have task versions as indicated in the table above. Since these workflows are bound to a region, we cannot use the cluster failover versions.
195
248
196
249
197
-
In the above example:
198
-
-`wf1` is created on us-west. No external entity associations exist. Its tasks will have failover version of the
199
-
<type=domain, key=test-domain.us-west> which is 1.
200
-
-`wf2` is created with an external entity. Its tasks will have failover version of following row
201
-
<type=user-location, key=london> which is 5.
202
250
251
+
### Failover scenarios
203
252
204
-
### Domain configuration
253
+
**1. Regional failover (disabling a region):**
254
+
This is a typical operation when the operator updates `ActiveClusterName` field in the domain record to failover the active-passive domain to one of the passive clusters.
255
+
It's called cluster failover because of active-passive domains where only one cluster can be active.
205
256
206
-
Domain configuration will have a new field called ActiveClusters which is a list of clusters that workflows of the domain can be active in. Active-active domains will have this field set. Active-passive domains will not have this field set.
257
+
In the active-active domain context, this operation will be done by updating `ActiveClusters` field in the domain record. Arbitrary changes can be made to the `ActiveClusters` field as long as following criterias are satisfied
258
+
- it's a subset of clusters in the cluster group.
259
+
- it has at most one active cluster per region.
207
260
261
+
For example, let's say we want to failover cluster0 to cluster3. (us-west -> us-east).
262
+
263
+
Before failover:
264
+
```
265
+
ActiveClusters: {
266
+
RegionToClusterMap: {
267
+
us-west: {
268
+
ActiveClusterName: cluster0,
269
+
FailoverVersion: 1,
270
+
},
271
+
us-east: {
272
+
ActiveClusterName: cluster3,
273
+
FailoverVersion: 3,
274
+
},
275
+
},
276
+
}
208
277
```
209
-
{
210
-
// Clusters can be a subset of clusters in the group.
211
-
Clusters: [us-west, us-east, eu-west, eu-central]
212
278
213
-
// Active clusters can have at most one from each region. Remaining clusters will be considered passive.
214
-
ActiveClusters: [us-west, eu-west]
279
+
After failover:
280
+
```
281
+
ActiveClusters: {
282
+
RegionToClusterMap: {
283
+
us-west: { // failover version of us-west region is changed to 3 so that it maps to us-east region.
284
+
ActiveClusterName: cluster3,
285
+
FailoverVersion: 3,
286
+
},
287
+
us-east: {
288
+
ActiveClusterName: cluster3,
289
+
FailoverVersion: 3,
290
+
},
291
+
},
215
292
}
216
293
```
217
294
295
+
Failover version to cluster mapping:
296
+
- Workflows whose task version resolves to region us-west will start using version 3 from now on.
297
+
- Workflows whose task version resolves to region us-east will continue using version 3.
298
+
299
+
300
+
**2. Within region failover (changing active cluster of a region):**
301
+
302
+
In the example active-active domain above, the cluster0 & cluster1 are available in us-west region and cluster0 is active.
303
+
304
+
If we want to failover from cluster0 to cluster1, then we need to update the `RegionToClusterMap` in the domain record and replace cluster0 with cluster1.
305
+
306
+
Before failover:
307
+
```
308
+
ActiveClusters: {
309
+
RegionToClusterMap: {
310
+
us-west: {
311
+
ActiveClusterName: cluster0,
312
+
FailoverVersion: 1,
313
+
},
314
+
us-east: {
315
+
ActiveClusterName: cluster3,
316
+
FailoverVersion: 3,
317
+
},
318
+
},
319
+
}
320
+
```
321
+
322
+
After failover:
323
+
```
324
+
ActiveClusters: {
325
+
RegionToClusterMap: {
326
+
us-west: {
327
+
ActiveClusterName: cluster1,
328
+
FailoverVersion: 2,
329
+
},
330
+
us-east: {
331
+
ActiveClusterName: cluster3,
332
+
FailoverVersion: 3,
333
+
},
334
+
},
335
+
}
336
+
```
337
+
338
+
Failover version to cluster mapping:
339
+
- Workflows whose task version resolves to region us-west will start using version 2 from now on.
340
+
- Workflows whose task version resolves to region us-east will continue using version 3.
341
+
342
+
343
+
**3. Entity failover:**
344
+
345
+
The external entity support is optional. If no entity is provided, the workflow will be associated with the region that the workflow is created on and they will be failed over as explained in the previous section.
346
+
347
+
If an external entity is provided for a workflow, then it can be failed over by updating the corresponding record in `EntityActiveRegion` lookup table.
348
+
349
+
Let's say we want to failover the entity with key "seattle" to region us-east so it gets processed by cluster3.
350
+
351
+
Before failover:
352
+
| EntityType | EntityKey | Region | Failover Version | LastUpdated |
Notice that the failover version is updated to 3. This means that all workflows associated with this entity will be considered active in us-east region from now on. Since there can only be one active cluster per region, the coresponding workflows will be active in cluster3.
362
+
363
+
Failover version to cluster mapping:
364
+
There's no change in the failover version to cluster mapping in entity failovers. Same versions map to same clusters as before.
365
+
However, new task versions of workflows belonging to "seattle" entity will be 3 from now on and therefore they will be active in cluster3.
366
+
367
+
218
368
### API Call Forwarding
219
369
220
370
Cadence frontend proxies some API calls to the active cluster of the domain. Currently, this request to cluster lookup is done by checking the failover version of the active-passive domain.
0 commit comments