Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
96cf7d3
5793: Create CDKDefinedVariants of properties
rtjd6554 Nov 14, 2025
8e6c763
5793: Application of Accout and Region
rtjd6554 Nov 14, 2025
3be913d
5793: Remove Account from CommonProperty and repoint imports
rtjd6554 Nov 14, 2025
52fb097
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 14, 2025
de6e28c
5793: Update missing account reference
rtjd6554 Nov 14, 2025
3c4a135
5793: Adjust common property Region
rtjd6554 Nov 14, 2025
e8aab6d
5793: Update documentation
rtjd6554 Nov 14, 2025
2df4545
5793: Test updates for property changes
rtjd6554 Nov 17, 2025
f1896ce
5793: Update to SleeperPropertiesValidationTest
rtjd6554 Nov 17, 2025
144b671
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 17, 2025
a47612a
5793: Update mandatory details for Account and Region
rtjd6554 Nov 17, 2025
52fb4db
5793: Remove properties not UserDefined
rtjd6554 Nov 17, 2025
aae952b
5793: Update change for ID tag property
rtjd6554 Nov 18, 2025
838ef0d
5793: Update to remove populate for production code
rtjd6554 Nov 19, 2025
cff7089
5793: Add missing configuration file
rtjd6554 Nov 19, 2025
262238c
5793: Refactor DeployInstanceConfigurationIT
rtjd6554 Nov 19, 2025
f7ea82d
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 19, 2025
27983ba
5793: Correct class usage
rtjd6554 Nov 19, 2025
d48d53e
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 20, 2025
56bf9f4
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 20, 2025
319baad
5793: Remove remaining populate usage in DeployInstance
rtjd6554 Nov 21, 2025
e24ab9b
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 21, 2025
ac65c53
5793: Update property definitions
rtjd6554 Nov 21, 2025
30dfc5c
5793: Update to replace populate within tests
rtjd6554 Nov 21, 2025
d909491
5793: Update more testhelper usages
rtjd6554 Nov 21, 2025
d1c8285
5793: Delete test class
rtjd6554 Nov 21, 2025
1aba20c
5793: Refactor location of setting of test instance properties
rtjd6554 Nov 24, 2025
b599681
5793: Break unintentional loop
rtjd6554 Nov 24, 2025
e8294d9
5793: Fix method signature
rtjd6554 Nov 24, 2025
c8e1fa1
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 24, 2025
9c59978
5793: Update method signature
rtjd6554 Nov 24, 2025
d08a5b4
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 24, 2025
8b2c371
5793: Correctt static method usage
rtjd6554 Nov 24, 2025
e4b91e7
5793: Refactor remaining populate usages out of testing
rtjd6554 Nov 24, 2025
81b92bb
5793: Refactor out the populateInstaceProperties class and last remai…
rtjd6554 Nov 24, 2025
0bba933
5793: Typo fix
rtjd6554 Nov 24, 2025
405d36a
5793: Refactor location of loadTags
rtjd6554 Nov 24, 2025
475ed84
5793: Update local variable definition
rtjd6554 Nov 24, 2025
435c227
5793: Update wording
rtjd6554 Nov 24, 2025
2ddd95b
5793: Remove properties not required
rtjd6554 Nov 24, 2025
0b995c0
5793: Update templates definiton
rtjd6554 Nov 24, 2025
159bc0c
5793: Update for a common property usage
rtjd6554 Nov 24, 2025
9c332cf
5793: Update property.txt
rtjd6554 Nov 24, 2025
364690a
5793: Update region and account setting
rtjd6554 Nov 24, 2025
56497c5
5793: Removal of unused Region from SyncJarsRequest
rtjd6554 Nov 24, 2025
029b5ae
5793: Update wording
rtjd6554 Nov 24, 2025
b334c60
5793: Refactor TablePropertiesTestHelper to within relevant tests
rtjd6554 Nov 24, 2025
8887d57
5793: Update for a different CommonProperty
rtjd6554 Nov 24, 2025
87e6d9b
5793: Update for determination of Account and Region
rtjd6554 Nov 24, 2025
d4905df
5793: Simply InstanePropertiesTestHelper
rtjd6554 Nov 24, 2025
9b40887
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 25, 2025
1e909b0
5793: Temp change to pass tests
rtjd6554 Nov 25, 2025
542b4b4
5793: Update UploadDockerImagesToEcr to use Account and Region
rtjd6554 Nov 25, 2025
1d8685f
5793: Update admin Client
rtjd6554 Nov 25, 2025
c4d7c3f
5793: Refactor for StsClients
rtjd6554 Nov 25, 2025
f14148e
5793: Remove TODO's
rtjd6554 Nov 25, 2025
65cf608
5793: Correct missing method usage
rtjd6554 Nov 26, 2025
12ff099
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 26, 2025
ec77e82
5793: Update missing tags validation
rtjd6554 Nov 26, 2025
cf71cdc
5793: Update javadoc
rtjd6554 Nov 26, 2025
ebdc298
5793: Update Region and Account descriptions
rtjd6554 Nov 26, 2025
19f0098
5793: Update SystemTestApp
rtjd6554 Nov 26, 2025
e9845e5
5793: Remove redundant TODO
rtjd6554 Nov 26, 2025
ef1393e
5793: Rename method arguments
rtjd6554 Nov 26, 2025
6924829
5793: Update argument verification
rtjd6554 Nov 26, 2025
aea3b69
5793: Update captilisation on JavaDoc
rtjd6554 Nov 26, 2025
eee31b7
5793: New test for tagProperties
rtjd6554 Nov 26, 2025
331eecd
5793: Include value as part of validation
rtjd6554 Nov 26, 2025
73ea104
5793: Change RegionProvider to field over local variable
rtjd6554 Nov 26, 2025
70b2867
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 26, 2025
facdef1
5793: Update approval for default-instance
rtjd6554 Nov 26, 2025
2427c5f
5793: Fix for spot bugs issue
rtjd6554 Nov 26, 2025
7280d21
Merge branch 'develop' into 5793-Remove-prepopulate-properties
rtjd6554 Nov 27, 2025
9c99c37
5793: System test adjust for spot bugs
rtjd6554 Nov 27, 2025
fed550f
5793: Remove invalid modifier
rtjd6554 Nov 27, 2025
cd297bd
5793: Update from review comments
rtjd6554 Nov 27, 2025
77ef963
5793: Update logic surrounding unit tests
rtjd6554 Nov 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/usage/properties/instance/cdk/common.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ The following instance properties are commonly used throughout Sleeper.
| sleeper.version | The version of Sleeper that is being used. This property is used to identify the correct jars in the S3 jars bucket and to select the correct tag in the ECR repositories. |
| sleeper.admin.role.arn | The ARN of a role that has permissions to administer the instance. |
| sleeper.config.bucket | The S3 bucket name used to store configuration files. |
| sleeper.account | The AWS account number. This is the AWS account that the instance is deployed in. |
| sleeper.region | The AWS region the instance is deployed in. |
| sleeper.data.bucket | The S3 bucket name used to store table data. |
| sleeper.tables.name.index.dynamo.table | The name of the DynamoDB table indexing tables by their name. |
| sleeper.tables.id.index.dynamo.table | The name of the DynamoDB table indexing tables by their ID. |
Expand Down
2 changes: 0 additions & 2 deletions docs/usage/properties/instance/user/common.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ The following instance properties are commonly used throughout Sleeper.
| sleeper.retain.logs.after.destroy | Whether to keep the sleeper log groups when the instance is destroyed. | true | true |
| sleeper.optional.stacks | The optional stacks to deploy. Not case sensitive.<br>Valid values: [IngestStack, IngestBatcherStack, EmrServerlessBulkImportStack, EmrBulkImportStack, PersistentEmrBulkImportStack, EksBulkImportStack, EmrStudioStack, BulkExportStack, QueryStack, WebSocketQueryStack, AthenaStack, KeepLambdaWarmStack, CompactionStack, GarbageCollectorStack, PartitionSplittingStack, DashboardStack, TableMetricsStack] | IngestStack,IngestBatcherStack,EmrServerlessBulkImportStack,EmrStudioStack,QueryStack,CompactionStack,GarbageCollectorStack,PartitionSplittingStack,DashboardStack,TableMetricsStack | true |
| sleeper.lambda.deploy.type | The deployment type for AWS Lambda. Not case sensitive.<br>There are two types of Lambda deployments, jar and container.<br>If the size of the jar file is too large, it will always be deployed as a container.<br>Valid values: [jar, container] | jar | true |
| sleeper.account | The AWS account number. This is the AWS account that the instance will be deployed to. | | false |
| sleeper.region | The AWS region to deploy to. | | false |
| sleeper.endpoint.url | The AWS endpoint URL. This should only be set for a non-standard service endpoint. Usually this is used to set the URL to LocalStack for a locally deployed instance. | | false |
| sleeper.vpc | The id of the VPC to deploy to. | | false |
| sleeper.vpc.endpoint.check | Whether to check that the VPC that the instance is deployed to has an S3 endpoint. If there is no S3 endpoint then the NAT costs can be very significant. | true | false |
Expand Down
6 changes: 0 additions & 6 deletions example/basic/instance.properties
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,6 @@ sleeper.retain.logs.after.destroy=true
# PartitionSplittingStack, DashboardStack, TableMetricsStack]
sleeper.optional.stacks=IngestStack,IngestBatcherStack,EmrServerlessBulkImportStack,EmrStudioStack,QueryStack,CompactionStack,GarbageCollectorStack,PartitionSplittingStack,DashboardStack,TableMetricsStack

# The AWS account number. This is the AWS account that the instance will be deployed to.
sleeper.account=1234567890

# The AWS region to deploy to.
sleeper.region=eu-west-2

# The id of the VPC to deploy to.
sleeper.vpc=1234567890

Expand Down
6 changes: 0 additions & 6 deletions example/full/instance.properties
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,6 @@ sleeper.optional.stacks=IngestStack,IngestBatcherStack,EmrServerlessBulkImportSt
# Valid values: [jar, container]
sleeper.lambda.deploy.type=jar

# The AWS account number. This is the AWS account that the instance will be deployed to.
sleeper.account=1234567890

# The AWS region to deploy to.
sleeper.region=eu-west-2

# The AWS endpoint URL. This should only be set for a non-standard service endpoint. Usually this is
# used to set the URL to LocalStack for a locally deployed instance.
# sleeper.endpoint.url=
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,14 @@

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.tuple;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.CONFIG_BUCKET;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.DATA_BUCKET;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.VERSION;
import static sleeper.core.properties.instance.CommonProperty.ACCOUNT;
import static sleeper.core.properties.instance.CommonProperty.FILE_SYSTEM;
import static sleeper.core.properties.instance.CommonProperty.ID;
import static sleeper.core.properties.instance.CommonProperty.JARS_BUCKET;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.SUBNETS;
import static sleeper.core.properties.instance.CommonProperty.VPC_ID;
import static sleeper.core.properties.table.TableProperty.TABLE_NAME;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.CONFIG_BUCKET;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.VERSION;
import static sleeper.core.properties.instance.CommonProperty.ACCOUNT;
import static sleeper.core.properties.instance.CommonProperty.ID;
import static sleeper.core.properties.instance.CommonProperty.JARS_BUCKET;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.SUBNETS;
import static sleeper.core.properties.instance.CommonProperty.VPC_ID;

Expand Down
10 changes: 4 additions & 6 deletions java/cdk/src/main/java/sleeper/cdk/SleeperCdkApp.java
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@

import sleeper.core.properties.instance.InstanceProperties;

import static sleeper.core.properties.instance.CommonProperty.ACCOUNT;
import static sleeper.core.properties.instance.CommonProperty.ID;
import static sleeper.core.properties.instance.CommonProperty.REGION;

/**
* Deploys an instance of Sleeper, including any configured optional stacks.
Expand All @@ -46,18 +44,18 @@ public static void main(String[] args) {
DynamoDbClient dynamoClient = DynamoDbClient.create()) {
SleeperInstanceProps props = SleeperInstanceProps.fromContext(app, s3Client, dynamoClient);
InstanceProperties instanceProperties = props.getInstanceProperties();

String id = instanceProperties.get(ID);

Environment environment = Environment.builder()
.account(instanceProperties.get(ACCOUNT))
.region(instanceProperties.get(REGION))
.account(System.getenv("CDK_DEFAULT_ACCOUNT"))
.region(System.getenv("CDK_DEFAULT_REGION"))
.build();
SleeperInstance.createAsRootStack(app, id,
StackProps.builder()
.stackName(id)
.env(environment)
.build(),
SleeperInstanceProps.fromContext(app, s3Client, dynamoClient));
props);
instanceProperties.getTags()
.forEach((key, value) -> Tags.of(app).add(key, value));

Expand Down
7 changes: 7 additions & 0 deletions java/cdk/src/main/java/sleeper/cdk/SleeperInstance.java
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
import sleeper.cdk.stack.core.SleeperInstanceRoles;
import sleeper.core.properties.instance.InstanceProperties;

import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;

/**
* Deploys an instance of Sleeper, including any configured optional stacks.
* <p>
Expand Down Expand Up @@ -62,6 +65,8 @@ public SleeperInstanceRoles getRoles() {
*/
public static SleeperInstance createAsNestedStack(Construct scope, String id, NestedStackProps stackProps, SleeperInstanceProps sleeperProps) {
NestedStack stack = new NestedStack(scope, id, stackProps);
sleeperProps.getInstanceProperties().set(ACCOUNT, stack.getAccount());
sleeperProps.getInstanceProperties().set(REGION, stack.getRegion());
return addNestedStacks(stack, sleeperProps);
}

Expand All @@ -76,6 +81,8 @@ public static SleeperInstance createAsNestedStack(Construct scope, String id, Ne
*/
public static Stack createAsRootStack(Construct scope, String id, StackProps stackProps, SleeperInstanceProps sleeperProps) {
Stack stack = new Stack(scope, id, stackProps);
sleeperProps.getInstanceProperties().set(ACCOUNT, stack.getAccount());
sleeperProps.getInstanceProperties().set(REGION, stack.getRegion());
addNestedStacks(stack, sleeperProps);
return stack;
}
Expand Down
5 changes: 5 additions & 0 deletions java/cdk/src/main/java/sleeper/cdk/SleeperInstanceProps.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import java.nio.file.Path;
import java.util.List;
import java.util.Objects;
import java.util.Properties;

import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.VERSION;
import static sleeper.core.properties.instance.CommonProperty.ID;
Expand Down Expand Up @@ -260,6 +261,10 @@ public SleeperInstanceProps build() {
Objects.requireNonNull(tableProperties, "tableProperties must not be null");
Objects.requireNonNull(jars, "jars must not be null");

Properties tagsProperties = instanceProperties.getTagsProperties();
tagsProperties.setProperty("InstanceID", instanceProperties.get(ID));
instanceProperties.loadTags(tagsProperties);

if (validateProperties) {
instanceProperties.validate();
try {
Expand Down
4 changes: 2 additions & 2 deletions java/cdk/src/main/java/sleeper/cdk/stack/AthenaStack.java
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@
import static sleeper.core.properties.instance.AthenaProperty.ATHENA_COMPOSITE_HANDLER_TIMEOUT_IN_SECONDS;
import static sleeper.core.properties.instance.AthenaProperty.ATHENA_SPILL_MASTER_KEY_ARN;
import static sleeper.core.properties.instance.AthenaProperty.SPILL_BUCKET_AGE_OFF_IN_DAYS;
import static sleeper.core.properties.instance.CommonProperty.ACCOUNT;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;

@SuppressFBWarnings("NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE")
public class AthenaStack extends NestedStack {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@
import java.util.Map;
import java.util.stream.IntStream;

import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.ID;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.MetricsProperty.DASHBOARD_TIME_WINDOW_MINUTES;
import static sleeper.core.properties.instance.MetricsProperty.METRICS_NAMESPACE;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_EXPORT_CLUSTER;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_EXPORT_TASK_CREATION_CLOUDWATCH_RULE;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_EXPORT_TASK_CREATION_LAMBDA_FUNCTION;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.VERSION;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.TASK_RUNNER_LAMBDA_MEMORY_IN_MB;
import static sleeper.core.properties.instance.CommonProperty.TASK_RUNNER_LAMBDA_TIMEOUT_IN_SECONDS;
import static sleeper.core.properties.instance.CommonProperty.VPC_ID;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@
import java.util.Map;
import java.util.stream.Collectors;

import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_IMPORT_EMR_CLUSTER_ROLE_NAME;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_IMPORT_EMR_EC2_ROLE_NAME;
import static sleeper.core.properties.instance.CommonProperty.ACCOUNT;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.JARS_BUCKET;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.SUBNETS;
import static sleeper.core.properties.instance.CommonProperty.VPC_ID;
import static sleeper.core.properties.instance.EMRProperty.BULK_IMPORT_EMR_EBS_ENCRYPTION_KEY_ARN;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@
import static sleeper.core.properties.instance.BulkImportProperty.BULK_IMPORT_STARTER_LAMBDA_MEMORY;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_IMPORT_EKS_JOB_QUEUE_ARN;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.BULK_IMPORT_EKS_JOB_QUEUE_URL;
import static sleeper.core.properties.instance.CdkDefinedInstanceProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.JARS_BUCKET;
import static sleeper.core.properties.instance.CommonProperty.REGION;
import static sleeper.core.properties.instance.CommonProperty.SUBNETS;
import static sleeper.core.properties.instance.CommonProperty.VPC_ID;
import static sleeper.core.properties.instance.EKSProperty.EKS_CLUSTER_ADMIN_ROLES;
Expand Down
Loading
Loading