Skip to content

Commit b30d5e6

Browse files
jbeemsterpaulboocock
jbeemster
authored and
paulboocock
committed
Add GCP Pipeline for 21.08 release (closes #15)
1 parent cf9f85d commit b30d5e6

File tree

12 files changed

+943
-0
lines changed

12 files changed

+943
-0
lines changed

‎terraform/gcp/pipeline/default/README.md

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
## Requirements
2+
3+
| Name | Version |
4+
|------|---------|
5+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | ~> 1 |
6+
| <a name="requirement_google"></a> [google](#requirement\_google) | ~> 3.44.0 |
7+
| <a name="requirement_random"></a> [random](#requirement\_random) | ~> 3.1.0 |
8+
9+
## Providers
10+
11+
No providers.
12+
13+
## Modules
14+
15+
| Name | Source | Version |
16+
|------|--------|---------|
17+
| <a name="module_bad_1_topic"></a> [bad\_1\_topic](#module\_bad\_1\_topic) | snowplow-devops/pubsub-topic/google | 0.1.0 |
18+
| <a name="module_collector_lb"></a> [collector\_lb](#module\_collector\_lb) | snowplow-devops/lb/google | 0.1.0 |
19+
| <a name="module_collector_pubsub"></a> [collector\_pubsub](#module\_collector\_pubsub) | snowplow-devops/collector-pubsub-ce/google | 0.2.0 |
20+
| <a name="module_enrich_pubsub"></a> [enrich\_pubsub](#module\_enrich\_pubsub) | snowplow-devops/enrich-pubsub-ce/google | 0.1.0 |
21+
| <a name="module_enriched_topic"></a> [enriched\_topic](#module\_enriched\_topic) | snowplow-devops/pubsub-topic/google | 0.1.0 |
22+
| <a name="module_pipeline_db"></a> [pipeline\_db](#module\_pipeline\_db) | snowplow-devops/cloud-sql/google | 0.1.0 |
23+
| <a name="module_postgres_loader_bad"></a> [postgres\_loader\_bad](#module\_postgres\_loader\_bad) | snowplow-devops/postgres-loader-pubsub-ce/google | 0.2.0 |
24+
| <a name="module_postgres_loader_enriched"></a> [postgres\_loader\_enriched](#module\_postgres\_loader\_enriched) | snowplow-devops/postgres-loader-pubsub-ce/google | 0.2.0 |
25+
| <a name="module_raw_topic"></a> [raw\_topic](#module\_raw\_topic) | snowplow-devops/pubsub-topic/google | 0.1.0 |
26+
27+
## Resources
28+
29+
No resources.
30+
31+
## Inputs
32+
33+
| Name | Description | Type | Default | Required |
34+
|------|-------------|------|---------|:--------:|
35+
| <a name="input_iglu_server_dns_name"></a> [iglu\_server\_dns\_name](#input\_iglu\_server\_dns\_name) | The DNS name of your Iglu Server | `string` | n/a | yes |
36+
| <a name="input_iglu_super_api_key"></a> [iglu\_super\_api\_key](#input\_iglu\_super\_api\_key) | A UUIDv4 string to use as the master API key for Iglu Server management | `string` | n/a | yes |
37+
| <a name="input_network"></a> [network](#input\_network) | The name of the network to deploy within | `string` | n/a | yes |
38+
| <a name="input_pipeline_db_name"></a> [pipeline\_db\_name](#input\_pipeline\_db\_name) | The name of the database to connect to | `string` | n/a | yes |
39+
| <a name="input_pipeline_db_password"></a> [pipeline\_db\_password](#input\_pipeline\_db\_password) | The password to use to connect to the database | `string` | n/a | yes |
40+
| <a name="input_pipeline_db_username"></a> [pipeline\_db\_username](#input\_pipeline\_db\_username) | The username to use to connect to the database | `string` | n/a | yes |
41+
| <a name="input_prefix"></a> [prefix](#input\_prefix) | Will be prefixed to all resource names. Use to easily identify the resources created | `string` | n/a | yes |
42+
| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | The project ID in which the stack is being deployed | `string` | n/a | yes |
43+
| <a name="input_region"></a> [region](#input\_region) | The name of the region to deploy within | `string` | n/a | yes |
44+
| <a name="input_ssh_ip_allowlist"></a> [ssh\_ip\_allowlist](#input\_ssh\_ip\_allowlist) | The list of CIDR ranges to allow SSH traffic from | `list(any)` | n/a | yes |
45+
| <a name="input_subnetwork"></a> [subnetwork](#input\_subnetwork) | The name of the sub-network to deploy within | `string` | n/a | yes |
46+
| <a name="input_labels"></a> [labels](#input\_labels) | The labels to append to the resources in this module | `map(string)` | `{}` | no |
47+
| <a name="input_pipeline_db_authorized_networks"></a> [pipeline\_db\_authorized\_networks](#input\_pipeline\_db\_authorized\_networks) | The list of CIDR ranges to allow access to the Pipeline Database over | <pre>list(object({<br> name = string<br> value = string<br> }))</pre> | `[]` | no |
48+
| <a name="input_pipeline_db_tier"></a> [pipeline\_db\_tier](#input\_pipeline\_db\_tier) | The instance type to assign to the deployed Cloud SQL instance | `string` | `"db-g1-small"` | no |
49+
| <a name="input_ssh_key_pairs"></a> [ssh\_key\_pairs](#input\_ssh\_key\_pairs) | The list of SSH key-pairs to add to the servers | <pre>list(object({<br> user_name = string<br> public_key = string<br> }))</pre> | `[]` | no |
50+
| <a name="input_ssl_information"></a> [ssl\_information](#input\_ssl\_information) | The ID of an Google Managed certificate to bind to the load balancer | <pre>object({<br> enabled = bool<br> certificate_id = string<br> })</pre> | <pre>{<br> "certificate_id": "",<br> "enabled": false<br>}</pre> | no |
51+
| <a name="input_telemetry_enabled"></a> [telemetry\_enabled](#input\_telemetry\_enabled) | Whether or not to send telemetry information back to Snowplow Analytics Ltd | `bool` | `true` | no |
52+
| <a name="input_user_provided_id"></a> [user\_provided\_id](#input\_user\_provided\_id) | An optional unique identifier to identify the telemetry events emitted by this stack | `string` | `""` | no |
53+
54+
## Outputs
55+
56+
| Name | Description |
57+
|------|-------------|
58+
| <a name="output_collector_ip_address"></a> [collector\_ip\_address](#output\_collector\_ip\_address) | The IP address for the Pipeline Collector |
59+
| <a name="output_db_ip_address"></a> [db\_ip\_address](#output\_db\_ip\_address) | The IP address of the database where your data is being streamed |
60+
| <a name="output_db_port"></a> [db\_port](#output\_db\_port) | The port of the database where your data is being streamed |

‎terraform/gcp/pipeline/default/main.tf

+189
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
locals {
2+
custom_iglu_resolvers = [
3+
{
4+
name = "Iglu Server"
5+
priority = 0
6+
uri = "${var.iglu_server_dns_name}/api"
7+
api_key = var.iglu_super_api_key
8+
vendor_prefixes = []
9+
}
10+
]
11+
}
12+
13+
# 1. Deploy PubSub Topics
14+
module "raw_topic" {
15+
source = "snowplow-devops/pubsub-topic/google"
16+
version = "0.1.0"
17+
18+
name = "${var.prefix}-raw-topic"
19+
20+
labels = var.labels
21+
}
22+
23+
module "bad_1_topic" {
24+
source = "snowplow-devops/pubsub-topic/google"
25+
version = "0.1.0"
26+
27+
name = "${var.prefix}-bad-1-topic"
28+
29+
labels = var.labels
30+
}
31+
32+
module "enriched_topic" {
33+
source = "snowplow-devops/pubsub-topic/google"
34+
version = "0.1.0"
35+
36+
name = "${var.prefix}-enriched-topic"
37+
38+
labels = var.labels
39+
}
40+
41+
# 2. Deploy Collector stack
42+
module "collector_pubsub" {
43+
source = "snowplow-devops/collector-pubsub-ce/google"
44+
version = "0.2.0"
45+
46+
name = "${var.prefix}-collector-server"
47+
48+
network = var.network
49+
subnetwork = var.subnetwork
50+
region = var.region
51+
52+
ssh_ip_allowlist = var.ssh_ip_allowlist
53+
ssh_key_pairs = var.ssh_key_pairs
54+
55+
topic_project_id = var.project_id
56+
good_topic_name = module.raw_topic.name
57+
bad_topic_name = module.bad_1_topic.name
58+
59+
telemetry_enabled = var.telemetry_enabled
60+
user_provided_id = var.user_provided_id
61+
62+
labels = var.labels
63+
}
64+
65+
module "collector_lb" {
66+
source = "snowplow-devops/lb/google"
67+
version = "0.1.0"
68+
69+
name = "${var.prefix}-collector-lb"
70+
71+
instance_group_named_port_http = module.collector_pubsub.named_port_http
72+
instance_group_url = module.collector_pubsub.instance_group_url
73+
health_check_self_link = module.collector_pubsub.health_check_self_link
74+
75+
ssl_certificate_enabled = var.ssl_information.enabled
76+
ssl_certificate_id = var.ssl_information.certificate_id
77+
}
78+
79+
# 3. Deploy Enrichment
80+
module "enrich_pubsub" {
81+
source = "snowplow-devops/enrich-pubsub-ce/google"
82+
version = "0.1.0"
83+
84+
name = "${var.prefix}-enrich-server"
85+
86+
network = var.network
87+
subnetwork = var.subnetwork
88+
region = var.region
89+
90+
ssh_ip_allowlist = var.ssh_ip_allowlist
91+
ssh_key_pairs = var.ssh_key_pairs
92+
93+
raw_topic_name = module.raw_topic.name
94+
good_topic_id = module.enriched_topic.id
95+
bad_topic_id = module.bad_1_topic.id
96+
97+
# Linking in the custom Iglu Server here
98+
custom_iglu_resolvers = local.custom_iglu_resolvers
99+
100+
telemetry_enabled = var.telemetry_enabled
101+
user_provided_id = var.user_provided_id
102+
103+
labels = var.labels
104+
}
105+
106+
# 4. Deploy Postgres Loader
107+
module "pipeline_db" {
108+
source = "snowplow-devops/cloud-sql/google"
109+
version = "0.1.0"
110+
111+
name = "${var.prefix}-pipeline-db"
112+
113+
region = var.region
114+
db_name = var.pipeline_db_name
115+
db_username = var.pipeline_db_username
116+
db_password = var.pipeline_db_password
117+
118+
authorized_networks = var.pipeline_db_authorized_networks
119+
120+
tier = var.pipeline_db_tier
121+
122+
labels = var.labels
123+
}
124+
125+
module "postgres_loader_enriched" {
126+
source = "snowplow-devops/postgres-loader-pubsub-ce/google"
127+
version = "0.2.0"
128+
129+
name = "${var.prefix}-pg-loader-enriched-server"
130+
131+
network = var.network
132+
subnetwork = var.subnetwork
133+
region = var.region
134+
project_id = var.project_id
135+
136+
ssh_ip_allowlist = var.ssh_ip_allowlist
137+
ssh_key_pairs = var.ssh_key_pairs
138+
139+
in_topic_name = module.enriched_topic.name
140+
purpose = "ENRICHED_EVENTS"
141+
schema_name = "atomic"
142+
143+
db_instance_name = module.pipeline_db.connection_name
144+
db_port = module.pipeline_db.port
145+
db_name = var.pipeline_db_name
146+
db_username = var.pipeline_db_username
147+
db_password = var.pipeline_db_password
148+
149+
# Linking in the custom Iglu Server here
150+
custom_iglu_resolvers = local.custom_iglu_resolvers
151+
152+
telemetry_enabled = var.telemetry_enabled
153+
user_provided_id = var.user_provided_id
154+
155+
labels = var.labels
156+
}
157+
158+
module "postgres_loader_bad" {
159+
source = "snowplow-devops/postgres-loader-pubsub-ce/google"
160+
version = "0.2.0"
161+
162+
name = "${var.prefix}-pg-loader-bad-server"
163+
164+
network = var.network
165+
subnetwork = var.subnetwork
166+
region = var.region
167+
project_id = var.project_id
168+
169+
ssh_ip_allowlist = var.ssh_ip_allowlist
170+
ssh_key_pairs = var.ssh_key_pairs
171+
172+
in_topic_name = module.bad_1_topic.name
173+
purpose = "JSON"
174+
schema_name = "atomic_bad"
175+
176+
db_instance_name = module.pipeline_db.connection_name
177+
db_port = module.pipeline_db.port
178+
db_name = var.pipeline_db_name
179+
db_username = var.pipeline_db_username
180+
db_password = var.pipeline_db_password
181+
182+
# Linking in the custom Iglu Server here
183+
custom_iglu_resolvers = local.custom_iglu_resolvers
184+
185+
telemetry_enabled = var.telemetry_enabled
186+
user_provided_id = var.user_provided_id
187+
188+
labels = var.labels
189+
}

‎terraform/gcp/pipeline/default/outputs.tf

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
output "collector_ip_address" {
2+
description = "The IP address for the Pipeline Collector"
3+
value = module.collector_lb.ip_address
4+
}
5+
6+
output "db_ip_address" {
7+
description = "The IP address of the database where your data is being streamed"
8+
value = module.pipeline_db.first_ip_address
9+
}
10+
11+
output "db_port" {
12+
description = "The port of the database where your data is being streamed"
13+
value = module.pipeline_db.port
14+
}

‎terraform/gcp/pipeline/default/terraform.tfvars

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Will be prefixed to all resource names
2+
# Use this to easily identify the resources created and provide entropy for subsequent environments
3+
prefix = "sp"
4+
5+
# The project to deploy the infrastructure into
6+
project_id = "PROJECT_ID_TO_DEPLOY_INTO"
7+
8+
# Where to deploy the infrastructure
9+
region = "REGION_TO_DEPLOY_INTO"
10+
11+
# --- Default Network
12+
# Update to the network you would like to deploy into
13+
#
14+
# Note: If you opt to use your own network then you will need to define a subnetwork to deploy into as well
15+
network = "default"
16+
subnetwork = ""
17+
18+
# --- SSH
19+
# Update this to your IP Address
20+
ssh_ip_allowlist = ["999.999.999.999/32"]
21+
# Generate a new SSH key locally with `ssh-keygen`
22+
# ssh-keygen -t rsa -b 4096
23+
ssh_key_pairs = [
24+
{
25+
user_name = "snowplow"
26+
public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQA0jSi9//bRsHW4M6czodTs6smCXsxZ0gijzth0aBmycE= [email protected]"
27+
}
28+
]
29+
30+
# --- Iglu Server Configuration
31+
# Iglu Server DNS output from the Iglu Server stack
32+
iglu_server_dns_name = "http://CHANGE-TO-MY-IGLU-IP"
33+
# Used for API actions on the Iglu Server
34+
# Change this to the same UUID from when you created the Iglu Server
35+
iglu_super_api_key = "00000000-0000-0000-0000-000000000000"
36+
37+
# NOTE: Needed for Postgres Loader to work
38+
# igluctl: https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/igluctl
39+
# Ensure you have seeded Iglu Server:
40+
# git clone https://github.com/snowplow/iglu-central
41+
# cd iglu-central
42+
# igluctl static push --public schemas/ http://CHANGE-TO-MY-IGLU-URL.elb.amazonaws.com 00000000-0000-0000-0000-000000000000
43+
44+
# --- Snowplow Postgres Loader
45+
pipeline_db_name = "snowplow"
46+
pipeline_db_username = "snowplow"
47+
# Change and keep this secret!
48+
pipeline_db_password = "Hell0W0rld!2"
49+
# IP ranges that you want to query the Pipeline Postgres Cloud SQL instance from directly over the internet. An alternative access method is to leverage
50+
# the Cloud SQL Proxy service which creates an IAM authenticated tunnel to the instance
51+
#
52+
# Details: https://cloud.google.com/sql/docs/postgres/sql-proxy
53+
#
54+
# Note: this exposes your data to the internet - take care to ensure your allowlist is strict enough
55+
pipeline_db_authorized_networks = [
56+
{
57+
name = "foo"
58+
value = "999.999.999.999/32"
59+
},
60+
{
61+
name = "bar"
62+
value = "888.888.888.888/32"
63+
}
64+
]
65+
# Note: the size of the database instance determines the number of concurrent connections - each Postgres Loader instance creates 10 open connections so having
66+
# a sufficiently powerful database tier is important to not running out of connection slots
67+
pipeline_db_tier = "db-g1-small"
68+
69+
# See for more information: https://registry.terraform.io/modules/snowplow-devops/collector-pubsub-ce/google/latest#telemetry
70+
# Telemetry principles: https://docs.snowplowanalytics.com/docs/open-source-quick-start/what-is-the-quick-start-for-open-source/telemetry-principles/
71+
user_provided_id = ""
72+
telemetry_enabled = true
73+
74+
# --- SSL Configuration (optional)
75+
ssl_information = {
76+
certificate_id = ""
77+
enabled = false
78+
}
79+
80+
# --- Extra Labels to append to created resources (optional)
81+
labels = {}

0 commit comments

Comments
 (0)