From 44b1ba70d11db5f1c4f009b72d403a0d1122c08f Mon Sep 17 00:00:00 2001 From: nelsonic Date: Tue, 26 Nov 2024 10:09:27 +0000 Subject: [PATCH 01/17] init hetzner #97 --- analytics/README.md | 5 ++++- postgres/backup-fly-postgres.md | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/analytics/README.md b/analytics/README.md index 86607cf..df53a12 100644 --- a/analytics/README.md +++ b/analytics/README.md @@ -1,2 +1,5 @@ Placeholder for: -https://github.com/dwyl/learn-devops/issues/91 \ No newline at end of file +https://github.com/dwyl/learn-devops/issues/91 + +Meanwhile, see: +https://github.com/dwyl/learn-analytics/tree/main/plausible \ No newline at end of file diff --git a/postgres/backup-fly-postgres.md b/postgres/backup-fly-postgres.md index e1eea0c..3b421c8 100644 --- a/postgres/backup-fly-postgres.md +++ b/postgres/backup-fly-postgres.md @@ -16,7 +16,7 @@ from a Fly.io `Postgres` instance. ## How? 👩‍💻 ### 0. Before You Start -n + Before you attempt to access the `Postgres` database on `Fly.io`, ensure you are authenticated with your `Fly.io` account; run the command: From 0be89214e40594f80833c99375956e6064883fae Mon Sep 17 00:00:00 2001 From: nelsonic Date: Tue, 26 Nov 2024 22:53:09 +0000 Subject: [PATCH 02/17] init hetzner #97 --- hetzner/README.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 hetzner/README.md diff --git a/hetzner/README.md b/hetzner/README.md new file mode 100644 index 0000000..a9a3d7d --- /dev/null +++ b/hetzner/README.md @@ -0,0 +1,45 @@ +
+ +![hetzner-logo-banner](https://github.com/user-attachments/assets/5b3b5a63-33ae-41d6-bcff-6a25744db465 "Hetzner Logo") + +We've made the switch to `Hetzner`, +we think you should too. + +
+ +# Why? + +After more than a decade using various "Cloud" providers +including all the _Big_ Tech (`AWS`, `Azure`, `GCP`, `DigitalOcean`, `Linode`) +and many PaaS such as Lambda, Fly.io, Heroku, Vercel, etc. +we've finally bitten the bullet and gone _back_ to our roots; Servers! + +## Brief Aside on Motivation + +There two types of motivation: towards and away. +**Toward** is the **_positive_** motivation such as getting fit/healthy. +**Away** motivation is when we want to _avoid_ something, like being unfit. +Both types of motivation have their place. +Some people are exclusively motivated by loss/pain/risk aversion, +while others are driven by gain/reward/returns and downplay the downside. +I go through phases of being super risk averse +and others when I'm happy to take calculated risks +that others who haven't done the math think are _crazy_. + +In the case of self-hosting our web apps on barebones servers, +we have _plenty_ of experience from the pre-AWS days. +Yes, this ages us, but the experience was formative. +And means I'm not afraid to dive in. + +I'm motivated _away_ from the +[data loss]() +we experienced on `Fly.io` +and _toward_ the high availability/affordability of `Hetzner`. +I know this will require some setup/config work, +but I am not deterred + + +## Recommended Reading + ++ `Hetzner` sustainability report: +https://www.hetzner.com/unternehmen/nachhaltigkeit \ No newline at end of file From 70b13b5db6acbe3531c23f5880f9bd5c4f003a30 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Fri, 29 Nov 2024 04:54:13 +0000 Subject: [PATCH 03/17] add context on motivation #97 --- hetzner/README.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/hetzner/README.md b/hetzner/README.md index a9a3d7d..27bd7be 100644 --- a/hetzner/README.md +++ b/hetzner/README.md @@ -26,20 +26,31 @@ I go through phases of being super risk averse and others when I'm happy to take calculated risks that others who haven't done the math think are _crazy_. -In the case of self-hosting our web apps on barebones servers, -we have _plenty_ of experience from the pre-AWS days. +In the case of self-hosting our web apps on **barebones servers**, +we have _plenty_ of experience from the _pre-AWS_ days. Yes, this ages us, but the experience was formative. And means I'm not afraid to dive in. I'm motivated _away_ from the -[data loss]() +[data loss](https://github.com/dwyl/auth/issues/325#issuecomment-1792297886) we experienced on `Fly.io` and _toward_ the high availability/affordability of `Hetzner`. I know this will require some setup/config work, -but I am not deterred +but am undeterred; +that's why we write systematic & meticulous notes! + +> “_Notes aren’t a **record** of my thinking process. +> They **are** my thinking process_.” +~ Richard Feynman ## Recommended Reading -+ `Hetzner` sustainability report: -https://www.hetzner.com/unternehmen/nachhaltigkeit \ No newline at end of file ++ Good summary including "incidents": +[wikipedia.org/wiki/Hetzner](https://en.wikipedia.org/wiki/Hetzner) ++ `Hetzner` "about" page: +[hetzner.com/unternehmen/ueber-uns](https://www.hetzner.com/unternehmen/ueber-uns/) ++ Sustainability report: +[hetzner.com/unternehmen/nachhaltigkeit](https://www.hetzner.com/unternehmen/nachhaltigkeit) ++ Finances: +[northdata.com/Hetzner](https://www.northdata.com/Hetzner%20Online%20GmbH,%20Gunzenhausen/Amtsgericht%20Ansbach%20HRB%206089) \ No newline at end of file From f15d72b7e63bdddefcf8aff0bef85e2371006382 Mon Sep 17 00:00:00 2001 From: ATC Date: Sat, 7 Dec 2024 15:20:50 +0000 Subject: [PATCH 04/17] add ouline of autobase deployment on hetzner #97 --- postgres/autobase-ha-cluster.md | 364 ++++++++++++++++++++++++++++++++ 1 file changed, 364 insertions(+) create mode 100644 postgres/autobase-ha-cluster.md diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md new file mode 100644 index 0000000..3d4f308 --- /dev/null +++ b/postgres/autobase-ha-cluster.md @@ -0,0 +1,364 @@ +Hi friends, my name is Nelson and this is dwyl.
+Today we're going to deploy a Postgres Database Cluster +on +[`Hetzner`] +using [**autobase**](https://autobase.tech). + +As always, detailed instructions +for how we do _everything_ are available on `GitHub`; +link in the description. 🔗 + +Along the way we will clarify the steps as possible. +But keep in mind it's _not possible_ to cover everything in a **7 minute video**. + +If you have questions, suggestions or just want to say hi, +**please comment on YouTube**; +thanks. + +With all that out of the way, lets dive in! + +## 1. Login to `Hetzner` Cloud + + +When you _first_ login to `Hetzner`, +you will see the message: + +"You don't have any servers yet." + +hetzner-no-servers + +Click the "**Add Server**" button to begin your quest! + +## 2. Create a New Server (VPS) + +Select all the default options, +add your `ssh` `public` key +and create your server. + +new-server-created + +## 3. `SSH` into the `Hetzner` Server + +Use your `Terminal` to login to the newly created `Hetzner`server, e.g: + +```sh +ssh root@116.202.31.52 +``` + +Once you have successfully connected via `ssh`, +a best practice we recommend is to run a quick update. + +### Update The Server + +Run the following command chain: + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y +``` + +> Updates and installs usually take a couple of minutes. +> We speed installs up for brevity. + +With everything up-to-date, install the necessary dependencies. + + +## 4. Install Dependencies + +As per the `autobase` getting started guide: +[autobase.tech/docs#getting-started](https://autobase.tech/docs#getting-started) +run the following command to get the necessary dependencies: + +```sh +sudo apt update && sudo apt install -y python3-pip sshpass git +pip3 install ansible +``` + +### Install `Docker` + +The `Ubuntu` Server + +Follow the installation instructions in the **official `Docker` docs**: +https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository + +Run: + +```sh +# Add Docker's official GPG key: +sudo apt-get update +sudo apt-get install ca-certificates curl +sudo install -m 0755 -d /etc/apt/keyrings +sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc +sudo chmod a+r /etc/apt/keyrings/docker.asc + +# Add the repository to Apt sources: +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update +``` + +Followed by: +```sh +sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +Verify that the installation is successful by running the `hello-world` image: + +```sh +$ sudo docker run hello-world +``` + +With that confirmed working, +go back to the previous step and run the `autobase`command. + + + +## 5. Run `autobase` Console Boot Script + +Sample: + +```sh +docker run -d --name autobase-console \ + --publish 80:80 \ + --publish 8080:8080 \ + --env PG_CONSOLE_API_URL=http://localhost:8080/api/v1 \ + --env PG_CONSOLE_AUTHORIZATION_TOKEN=secret_token \ + --env PG_CONSOLE_DOCKER_IMAGE=autobase/automation:latest \ + --volume console_postgres:/var/lib/postgresql \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume /tmp/ansible:/tmp/ansible \ + --restart=unless-stopped \ + autobase/console:latest +``` + +You will nee to replace the `localhost` in the `PG_CONSOLE_API_URL` +with the IP (v4) address of your server +and `secret_token`in the `PG_CONSOLE_AUTHORIZATION_TOKEN` + ++ IP: 116.202.31.52 (yours will be different!) ++ Token: 5b0b6259-a7d4-4435-947d-0dff528912ba (create your own!) + +Actual: + +```sh +docker run -d --name autobase-console \ + --publish 80:80 \ + --publish 8080:8080 \ + --env PG_CONSOLE_API_URL=http://116.202.31.52:8080/api/v1 \ + --env PG_CONSOLE_AUTHORIZATION_TOKEN=5b0b6259-a7d4-4435-947d-0dff528912ba \ + --env PG_CONSOLE_DOCKER_IMAGE=autobase/automation:latest \ + --volume console_postgres:/var/lib/postgresql \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume /tmp/ansible:/tmp/ansible \ + --restart=unless-stopped \ + autobase/console:latest +``` + + +Confirm it worked with the `docker ps` command. You should see something similar to the following: + +```sh +CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                                                                    NAMES + +9740dfd66c42   autobase/console:latest   "/usr/bin/supervisor…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 5432/tcp   autobase-console +``` + + +## 6. Login To `autobase`Console Web UI + +Visit the IP Address of your server in you web browser e.g: +http://116.202.31.52/ + +You should see a login screen: + +autobase-console-login + +Copy-paste the Token you defined in step 5 above. + +When you first login you should see that there are **No Postgres Clusters**: + +autobase-no-clusters + +## 7. Create Postgres Cluster + +Click the "**CREATE CLUSTER**" button: + +create-cluster-button + +Select `hetzner`and the datacenter region you prefer, in our case Europe: + +create-cluster-europe + +The default disk storage is **`100Gb`**; + +cluster-disk-storage + +this is _way_ too much for most simple projects. +lower it to **`10Gb`** for each instance to instantly save **50%** of the cost! +(you're welcome!) + +disk-storage-10gb + +> **Note**: all values for `DISK`storage, `RAM`, and `CPU`can easily be scaled later. + +Finally, you'll need to add your `public` SSH key. + +### Copy Your SSH Public Key + +```sh +cat ~/.ssh/id_ed25519.pub | pbcopy +``` + +Paste it into the `SSH public key*`field: + +add-ssh-key + +Then scroll down and click the "**CREATE CLUSTER**" button. + +You will see a modal window appear prompting you to input a `Hetzner`API Key: + +hetzner-api-key-modal + +### 8. Generate an API token + +Follow the instructions in the official `Hetzner` docs: +https://docs.hetzner.com/cloud/api/getting-started/generating-api-token/ + +In the `Hetzner`console, navigate to **Security** > **API tokens**. +You should see the message: +"**You haven't generated an API token yet.**" + +hetzner-api-tokens + +Click on the "**Generate API token**" button: + +generate-api-token + +That will open _another_ modal window, input the description for your key, +e.g: +"postgres-cluster-api-key" +and select "**Read and Write**": + +generate-api-token-modal + +Finally, click on the "**Generate API token**" button. +You should see a confirmation message: + +token-created + +Click to reveal the token you created: + +copy-token + +Copy the token to your clipboard, e.g: + +```sh +zH2qdgCeogrKjVKgV7sngMRxCfewgSdDARUBr8yqcjuHhGzlNdY72H13Sjh1il2D +``` + +Paste it into the Cluster creation window: + +paste-token-in-auto-window + +_Optionally_ save the API Key to the console and then +click "**CREATE CLUSTER**": + +create-cluster + +created: + +cluster-created + +Cluster details: + +cluster-details + +The `Postgres` cluster _appears_ to be deployed, +but how do we _know_ that it worked? + +## 9. Test The Cluster! 👩‍🔬 + +First: _connect_ to the **primary** `Postgres` instance. +In our case this is: `10.0.1.4` + +postgres-primary + +Sample: + +```sh +export PGPASSWORD='password'; +psql -h 127.0.0.1 -p 5432 -U postgres -d postgres +``` + +Get the **Password** and **Port** from the **Connection info** panel: + +postgres-connection-info + +Actual: + +```sh +export PGPASSWORD='9Djw2LNRMWwaDS1F9TlxeXiGj4dV3zNk'; +psql -h 88.99.81.115 -p 5432 -U postgres -d postgres +``` + +```sh +psql -h 10.0.1.4 -p 6432 -U postgres -d postgres +``` + +```sh +psql -h 10.0.1.4 -p 6432 -U postgres -d postgres -c "select version()" +``` + +Got the following error: + +```sh +Command 'psql' not found, but can be installed with: + +apt install postgresql-client-common +``` + +This is a barebones `Ubuntu` instance, remember, so it's not surprising that it doesn't have `psql` installed. So follow the instruction and install it: + +```sh +sudo apt install postgresql-client-common +``` + +The output is: + +But when trying to run `psql` again, we still get an error: + +```sh +Error: You must install at least one postgresql-client- package +``` + + + +## Outro: + +Given that this is a technical guide for an evolving system, +it may need to be enhanced/extended or updated in future, +that will be done on GitHub; +_everyone_ is welcome to and _encouraged_ to contribute! +Again, link in the description. + +Thanks for watching/listening. +If you found it useful and want to see more, +please subscribe. + + +## Privacy Disclaimer + +By the time you read/watch this,  +all of the sensitive data such as passwords, IP addresses,  +public keys and auth tokens will have been updated. +This avoids anyone getting ideas about accessing backend systems. + +We publish our notes and videos on how we do things +so that we can be as transparent as possible. +We have a strong security & privacy focus for all our systems +so all private backend systems like databases are always locked down. + +As always, if you have a security question or concern,  +Please contact us responsibly. + From 0eacffb2a616fc57c4c633eba30aac69205cb515 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Fri, 21 Mar 2025 10:53:08 +0000 Subject: [PATCH 05/17] init nginx hetzner setup https://github.com/dwyl/learn-devops/issues/105 --- nginx/README.md | 41 +++++++++++++++++++++++++++++++++ postgres/README.md | 19 +++++++++------ postgres/autobase-ha-cluster.md | 11 +++++---- 3 files changed, 60 insertions(+), 11 deletions(-) create mode 100644 nginx/README.md diff --git a/nginx/README.md b/nginx/README.md new file mode 100644 index 0000000..9e6c3b2 --- /dev/null +++ b/nginx/README.md @@ -0,0 +1,41 @@ +# `nginx` Fast Setup + +This is a speed run of using `nginx` +to proxy an app on a `Hetzner` server. + +## 1. Install `nginx` on `Ubuntu` + +```sh +sudo apt install nginx +``` + +That installs and automatically starts the `nginx` server. + +Check the status: + +```sh +service nginx status +``` + +Output: + +```sh +● nginx.service - A high performance web server and a reverse proxy server + Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled) + Active: active (running) since Fri 2025-03-21 10:49:46 UTC; 56s ago + Docs: man:nginx(8) + Process: 754 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) + Process: 778 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) + Main PID: 803 (nginx) + Tasks: 3 (limit: 4540) + Memory: 3.7M (peak: 3.8M) + CPU: 34ms + CGroup: /system.slice/nginx.service + ├─803 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;" + ├─804 "nginx: worker process" + └─805 "nginx: worker process" +``` + +Official instructions: +https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview + diff --git a/postgres/README.md b/postgres/README.md index cc360fa..de9290e 100644 --- a/postgres/README.md +++ b/postgres/README.md @@ -1,23 +1,26 @@
-Postgres logo wide # Deployment
-`Postgres` deployment is divided into two options: +`Postgres` deployment is split into two options: 1. Managed - the infrastructure provider manages the instances for you including data backups and failover in the event of a crash/corruption. + 2. Unmanaged - we the engineers or operations team need to manage it including data integrity and availability. Both have their place. But if you are not an _experienced_ -Database Administrator (DBA) -or System Administrator (SysAdmin), +Database Administrator +([DBA](https://en.wikipedia.org/wiki/Database_administration)) +or System Administrator +([SysAdmin](https://en.wikipedia.org/wiki/System_administrator)), you should seriously consider a _managed_ service. The risk of data loss greatly outweighs the cost of a _managed_ service. @@ -35,10 +38,12 @@ There are _many_ other options for "cloud" providers for managed `Postgres` and other `SQL` databases. We looked at all the major ones including `AWS`, `GCP`, `Azure`. -Sadly, `AWS Aurara` while appealing, -has _deliberately_ confusing pricing: -https://aws.amazon.com/rds/aurora/pricing +Sadly, `AWS Aurara` while appealing, +has **_deliberately_ confusing pricing**: +[aws.amazon.com/rds/aurora/pricing](https://aws.amazon.com/rds/aurora/pricing) They have costs for I/O requests, storage, backtrack (backup) and data transfer. By contrast `DigitalOcean` has _transparent_ pricing based on the VPS (Memory, CPU and SSD) used. +But `DigitalOcean` also gets _very_ expensive ... +So we decided to invest the time to _self-manage_. diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md index 3d4f308..61208e2 100644 --- a/postgres/autobase-ha-cluster.md +++ b/postgres/autobase-ha-cluster.md @@ -1,17 +1,20 @@ + +Deploy a Postgres Database Cluster on [`Hetzner`] using [**autobase**](https://autobase.tech). -As always, detailed instructions +As always, detailed instructions for how we do _everything_ are available on `GitHub`; link in the description. 🔗 Along the way we will clarify the steps as possible. But keep in mind it's _not possible_ to cover everything in a **7 minute video**. -If you have questions, suggestions or just want to say hi, +If you have questions, suggestions or just want to say hi, **please comment on YouTube**; thanks. @@ -20,7 +23,7 @@ With all that out of the way, lets dive in! ## 1. Login to `Hetzner` Cloud -When you _first_ login to `Hetzner`, +When you _first_ login to `Hetzner`, you will see the message: "You don't have any servers yet." From 627f8e91b99830c48dd94ae4e8befd478bc15ea8 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Fri, 21 Mar 2025 18:53:03 +0000 Subject: [PATCH 06/17] add snapshot of nginx.conf for #105 --- nginx/README.md | 86 ++++++++++++++++++++++++++++++++++++++++++++++-- nginx/nginx.conf | 84 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+), 2 deletions(-) create mode 100644 nginx/nginx.conf diff --git a/nginx/README.md b/nginx/README.md index 9e6c3b2..a5464ee 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -5,6 +5,9 @@ to proxy an app on a `Hetzner` server. ## 1. Install `nginx` on `Ubuntu` +Official instructions: +https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview + ```sh sudo apt install nginx ``` @@ -36,6 +39,85 @@ Output: └─805 "nginx: worker process" ``` -Official instructions: -https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview +Visit: +http://88.99.81.115 + +![nginx-running](https://github.com/user-attachments/assets/f8754c78-7243-4844-9ab6-eb642d4ab2e7) + + + +## 2. Certbot + +Instructions: +https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal + +```sh +sudo snap install --classic certbot +``` + +Output: + +```sh +2025-03-21T11:48:11Z INFO Waiting for automatic snapd restart... +certbot 3.3.0 from Certbot Project (certbot-eff✓) installed +``` + +Link the command: + +```sh +sudo ln -s /snap/bin/certbot /usr/bin/certbot +``` +```sh +sudo certbot --nginx +``` + +Output: + +```sh +Requesting a certificate for dwy.is + +Successfully received certificate. +Certificate is saved at: /etc/letsencrypt/live/dwy.is/fullchain.pem +Key is saved at: /etc/letsencrypt/live/dwy.is/privkey.pem +This certificate expires on 2025-06-19. +These files will be updated when the certificate renews. +Certbot has set up a scheduled task to automatically renew this certificate in the background. + +Deploying certificate +Successfully deployed certificate for dwy.is to /etc/nginx/sites-enabled/default +Congratulations! You have successfully enabled HTTPS on https://dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +If you like Certbot, please consider supporting our work by: + * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + * Donating to EFF: https://eff.org/donate-le +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +Dry run renewal: + +```sh +sudo certbot renew --dry-run +``` + +```sh +Saving debug log to /var/log/letsencrypt/letsencrypt.log + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Processing /etc/letsencrypt/renewal/dwy.is.conf +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Account registered. +Simulating renewal of an existing certificate for dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Congratulations, all simulated renewals succeeded: + /etc/letsencrypt/live/dwy.is/fullchain.pem (success) +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +## 3. Configure `nginx` Proxy + +```sh +cd /etc/nginx/ +``` \ No newline at end of file diff --git a/nginx/nginx.conf b/nginx/nginx.conf new file mode 100644 index 0000000..0388fd0 --- /dev/null +++ b/nginx/nginx.conf @@ -0,0 +1,84 @@ +# /etc/nginx/nginx.conf +user www-data; +worker_processes auto; +pid /run/nginx.pid; +error_log /var/log/nginx/error.log; +include /etc/nginx/modules-enabled/*.conf; + +events { + worker_connections 768; + # multi_accept on; +} + +http { + + ## + # Basic Settings + ## + + sendfile on; + tcp_nopush on; + types_hash_max_size 2048; + # server_tokens off; + + # server_names_hash_bucket_size 64; + # server_name_in_redirect off; + + include /etc/nginx/mime.types; + default_type application/octet-stream; + + ## + # SSL Settings + ## + + ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE + ssl_prefer_server_ciphers on; + + ## + # Logging Settings + ## + + access_log /var/log/nginx/access.log; + + ## + # Gzip Settings + ## + + gzip on; + + # gzip_vary on; + # gzip_proxied any; + # gzip_comp_level 6; + # gzip_buffers 16 8k; + # gzip_http_version 1.1; + # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; + + ## + # Virtual Host Configs + ## + + include /etc/nginx/conf.d/*.conf; + include /etc/nginx/sites-enabled/*; +} + + +#mail { +# # See sample authentication script at: +# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript +# +# # auth_http localhost/auth.php; +# # pop3_capabilities "TOP" "USER"; +# # imap_capabilities "IMAP4rev1" "UIDPLUS"; +# +# server { +# listen localhost:110; +# protocol pop3; +# proxy on; +# } +# +# server { +# listen localhost:143; +# protocol imap; +# proxy on; +# } +#} \ No newline at end of file From 7300fa2b323a00c446c42330d81359a2eaeff0c3 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Fri, 21 Mar 2025 19:26:40 +0000 Subject: [PATCH 07/17] snapshot nginx/sites-available/default for #105 --- nginx/sites-available/default | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 nginx/sites-available/default diff --git a/nginx/sites-available/default b/nginx/sites-available/default new file mode 100644 index 0000000..e69de29 From 6945486123d4006546f16a8f904d5a7e52728350 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Fri, 21 Mar 2025 19:26:51 +0000 Subject: [PATCH 08/17] snapshot nginx/sites-available/default for #105 --- nginx/sites-available/default | 163 ++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) diff --git a/nginx/sites-available/default b/nginx/sites-available/default index e69de29..c23bb5f 100644 --- a/nginx/sites-available/default +++ b/nginx/sites-available/default @@ -0,0 +1,163 @@ +## +# You should look at the following URL's in order to grasp a solid understanding +# of Nginx configuration files in order to fully unleash the power of Nginx. +# https://www.nginx.com/resources/wiki/start/ +# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/ +# https://wiki.debian.org/Nginx/DirectoryStructure +# +# In most cases, administrators will remove this file from sites-enabled/ and +# leave it as reference inside of sites-available where it will continue to be +# updated by the nginx packaging team. +# +# This file will automatically load configuration files provided by other +# applications, such as Drupal or Wordpress. These applications will be made +# available underneath a path with that package name, such as /drupal8. +# +# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples. +## + +# Default server configuration +# +server { + listen 80 default_server; + listen [::]:80 default_server; + + # SSL configuration + # + # listen 443 ssl default_server; + # listen [::]:443 ssl default_server; + # + # Note: You should disable gzip for SSL traffic. + # See: https://bugs.debian.org/773332 + # + # Read up on ssl_ciphers to ensure a secure configuration. + # See: https://bugs.debian.org/765782 + # + # Self signed certs generated by the ssl-cert package + # Don't use them in a production server! + # + # include snippets/snakeoil.conf; + + root /var/www/html; + + # Add index.php to the list if you are using PHP + index index.html index.htm index.nginx-debian.html; + + server_name _; + + location / { + # First attempt to serve request as file, then + # as directory, then fall back to displaying a 404. + try_files $uri $uri/ =404; + } + + # pass PHP scripts to FastCGI server + # + #location ~ \.php$ { + # include snippets/fastcgi-php.conf; + # + # # With php-fpm (or other unix sockets): + # fastcgi_pass unix:/run/php/php7.4-fpm.sock; + # # With php-cgi (or other tcp sockets): + # fastcgi_pass 127.0.0.1:9000; + #} + + # deny access to .htaccess files, if Apache's document root + # concurs with nginx's one + # + #location ~ /\.ht { + # deny all; + #} +} + + +# Virtual Host configuration for example.com +# +# You can move that to a different file under sites-available/ and symlink that +# to sites-enabled/ to enable it. +# +#server { +# listen 80; +# listen [::]:80; +# +# server_name example.com; +# +# root /var/www/example.com; +# index index.html; +# +# location / { +# try_files $uri $uri/ =404; +# } +#} + +server { + + # SSL configuration + # + # listen 443 ssl default_server; + # listen [::]:443 ssl default_server; + # + # Note: You should disable gzip for SSL traffic. + # See: https://bugs.debian.org/773332 + # + # Read up on ssl_ciphers to ensure a secure configuration. + # See: https://bugs.debian.org/765782 + # + # Self signed certs generated by the ssl-cert package + # Don't use them in a production server! + # + # include snippets/snakeoil.conf; + + root /var/www/html; + + # Add index.php to the list if you are using PHP + index index.html index.htm index.nginx-debian.html; + server_name dwy.is; # managed by Certbot + + + location / { + # First attempt to serve request as file, then + # as directory, then fall back to displaying a 404. + try_files $uri $uri/ =404; + } + + # pass PHP scripts to FastCGI server + # + #location ~ \.php$ { + # include snippets/fastcgi-php.conf; + # + # # With php-fpm (or other unix sockets): + # fastcgi_pass unix:/run/php/php7.4-fpm.sock; + # # With php-cgi (or other tcp sockets): + # fastcgi_pass 127.0.0.1:9000; + #} + + # deny access to .htaccess files, if Apache's document root + # concurs with nginx's one + # + #location ~ /\.ht { + # deny all; + #} + + + listen [::]:443 ssl ipv6only=on; # managed by Certbot + listen 443 ssl; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot + +} +server { + if ($host = dwy.is) { + return 301 https://$host$request_uri; + } # managed by Certbot + + + listen 80 ; + listen [::]:80 ; + server_name dwy.is; + return 404; # managed by Certbot + + +} \ No newline at end of file From 80c32347db0d15429b8da60a9b6b82747f91fa6d Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sat, 22 Mar 2025 22:09:55 +0000 Subject: [PATCH 09/17] add nginx config for autobase subdomain #105 --- nginx/README.md | 106 +++++++++++++++++++++++++++++++-- nginx/sites-available/autobase | 24 ++++++++ nginx/sites-available/default | 2 - 3 files changed, 126 insertions(+), 6 deletions(-) create mode 100644 nginx/sites-available/autobase diff --git a/nginx/README.md b/nginx/README.md index a5464ee..a127f9c 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -45,7 +45,6 @@ http://88.99.81.115 ![nginx-running](https://github.com/user-attachments/assets/f8754c78-7243-4844-9ab6-eb642d4ab2e7) - ## 2. Certbot Instructions: @@ -116,8 +115,107 @@ Congratulations, all simulated renewals succeeded: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ``` -## 3. Configure `nginx` Proxy +The full config including the TLS is in +`/etc/nginx/sites-available/default` + +## 3. Configure `nginx` Subdomain + +```sh +cd /etc/nginx/sites-enabled/autobase.dwy.is +``` + +Test `nginx` config: + +```sh +nginx -t +``` + +You should see output similar to the following: + +```sh +nginx: the configuration file /etc/nginx/nginx.conf syntax is ok +nginx: configuration file /etc/nginx/nginx.conf test is successful +``` + +Test a specific configuration file: ```sh -cd /etc/nginx/ -``` \ No newline at end of file +nginx -t -c /path/to/conf +``` + +In our case: + +```sh +nginx -t -c /etc/nginx/sites-enabled/autobase.dwy.is +``` + +If your config fails the test for any reason, +try checking it online: +[google.com/search?q=nginx+syntax+check+online](https://www.google.com/search?q=nginx+syntax+check+online) +e.g: +[getpagespeed.com/check-nginx-config](https://www.getpagespeed.com/check-nginx-config) + +Restart `nginx`: + +```sh +sudo service nginx restart +``` + +Wildcard Certificate for Domain: +https://www.baeldung.com/linux/letsencrypt-certbot-add-subdomains + +```sh +sudo certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com +``` + +In our case: + +```sh +sudo certbot certonly --manual --preferred-challenges=dns -d dwy.is -d *.dwy.is +``` + +Output: + +```sh +Saving debug log to /var/log/letsencrypt/letsencrypt.log +Plugins selected: Authenticator manual, Installer None + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +You have an existing certificate that contains a portion of the domains you +requested (ref: /etc/letsencrypt/renewal/dwy.is.conf) + +It contains these names: dwy.is + +You requested these names for the new certificate: dwy.is, *.dwy.is. + +Do you want to expand and replace this existing certificate with the new +certificate? +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +(E)xpand/(C)ancel: E +Renewing an existing certificate for dwy.is and *.dwy.is +Performing the following challenges: +dns-01 challenge for dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Please deploy a DNS TXT record under the name: + +_acme-challenge.dwy.is. + +with the following value: + +8GC-85xs1BGQDlU7YKpxA5fyHBV20PqBU8aMA9lAN10 + +Before continuing, verify the TXT record has been deployed. Depending on the DNS +provider, this may take some time, from a few seconds to multiple minutes. You can +check if it has finished deploying with aid of online tools, such as the Google +Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.dwy.is. +Look for one or more bolded line(s) below the line ';ANSWER'. It should show the +value(s) you've just added. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Press Enter to Continue +``` + +```sh +dig -t txt _acme-challenge.dwy.is +``` diff --git a/nginx/sites-available/autobase b/nginx/sites-available/autobase new file mode 100644 index 0000000..1b140ad --- /dev/null +++ b/nginx/sites-available/autobase @@ -0,0 +1,24 @@ +server { + listen 80; + server_name autobase.dwy.is; + + location / { + proxy_pass http://172.17.0.1:82; + } +} + +server { + server_name autobase.dwy.is; + + location / { + proxy_pass http://172.17.0.1:82; + } + + # listen [::]:443 ssl ipv6only=on; # managed by Certbot + listen 443 ssl; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot +} + diff --git a/nginx/sites-available/default b/nginx/sites-available/default index c23bb5f..ebc6122 100644 --- a/nginx/sites-available/default +++ b/nginx/sites-available/default @@ -158,6 +158,4 @@ server { listen [::]:80 ; server_name dwy.is; return 404; # managed by Certbot - - } \ No newline at end of file From 32147a57da1b9add7e4d10b269db8271f9709a22 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sun, 23 Mar 2025 19:42:39 +0000 Subject: [PATCH 10/17] https://autobase.dwy.is secured with letsencrypt tls cert closes #105 --- nginx/README.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 3 deletions(-) diff --git a/nginx/README.md b/nginx/README.md index a127f9c..67c545c 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -146,7 +146,7 @@ nginx -t -c /path/to/conf In our case: ```sh -nginx -t -c /etc/nginx/sites-enabled/autobase.dwy.is +nginx -t -c /etc/nginx/sites-enabled/autobase ``` If your config fails the test for any reason, @@ -161,17 +161,19 @@ Restart `nginx`: sudo service nginx restart ``` +## 4. Wildcard Certificate (Failed) + Wildcard Certificate for Domain: https://www.baeldung.com/linux/letsencrypt-certbot-add-subdomains ```sh -sudo certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com +sudo certbot certonly -i nginx -d example.com -d *.example.com ``` In our case: ```sh -sudo certbot certonly --manual --preferred-challenges=dns -d dwy.is -d *.dwy.is +sudo certbot certonly -i nginx -d dwy.is -d *.dwy.is -v ``` Output: @@ -216,6 +218,50 @@ value(s) you've just added. Press Enter to Continue ``` +I created the `TXT` record immediatley: + +https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns + +![dwyis-txt-record](https://github.com/user-attachments/assets/80ddea19-d06c-4d71-8c8e-f86ee2acd9dc) + +But it never propagated ... ⏳ + +https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.dwy.is + +![google-dig-txt](https://github.com/user-attachments/assets/6ccbb156-6c34-4d67-9c13-f69db9b47a76) + +I refreshed this like a million times over `48h` +but it never updated. + ```sh dig -t txt _acme-challenge.dwy.is ``` + +Sadly, adding the wildcard TLS cert was a dead-end +because the `TXT` record never updates on `NameCheap` ... +I guess it's _Cheap_ for a _reason_ ... 😢 + +... + +I decided to contact `NameCheap` support via live chat. + + +Final output: + +```sh +Renewing an existing certificate for dwy.is and *.dwy.is +Reloading nginx server after certificate issuance + +Successfully received certificate. +Certificate is saved at: /etc/letsencrypt/live/dwy.is/fullchain.pem +Key is saved at: /etc/letsencrypt/live/dwy.is/privkey.pem +This certificate expires on 2025-06-21. +These files will be updated when the certificate renews. +Certbot has set up a scheduled task to automatically renew this certificate in the background. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +If you like Certbot, please consider supporting our work by: + * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + * Donating to EFF: https://eff.org/donate-le +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` \ No newline at end of file From 38bee89db9a9c7be023dc97263e0bd4164a1a8f3 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sun, 23 Mar 2025 20:13:56 +0000 Subject: [PATCH 11/17] add screenshot of https://autobase.dwy.is working and full config for #105 --- nginx/README.md | 21 ++++++++++++++++++--- nginx/sites-available/autobase | 16 ++++++---------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/nginx/README.md b/nginx/README.md index 67c545c..4ca5f2f 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -1,7 +1,7 @@ # `nginx` Fast Setup This is a speed run of using `nginx` -to proxy an app on a `Hetzner` server. +to proxy an app running on a `Hetzner` server. ## 1. Install `nginx` on `Ubuntu` @@ -243,7 +243,18 @@ I guess it's _Cheap_ for a _reason_ ... 😢 ... -I decided to contact `NameCheap` support via live chat. +I decided to contact `NameCheap` support via live chat: +https://www.namecheap.com/help-center/live-chat + +They were helpful and together we determined that _I_ had misconfigured the `TXT` record ... 🤦 + +Updated config: + +https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns + +![dwy.is-dns-txt-record](https://github.com/user-attachments/assets/c19e6cae-132c-4f70-ba99-6bd8829f0d13) + +Full transcript: [Chat_Transcript_23_Mar_2025.pdf](https://github.com/user-attachments/files/19410954/Chat_Transcript_23_Mar_2025.pdf) Final output: @@ -264,4 +275,8 @@ If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` \ No newline at end of file +``` + +Working! + +![autobase.dwy.is-with-ssl](https://github.com/user-attachments/assets/15411040-860f-4a56-9c2d-91fc8702c318) \ No newline at end of file diff --git a/nginx/sites-available/autobase b/nginx/sites-available/autobase index 1b140ad..54c400b 100644 --- a/nginx/sites-available/autobase +++ b/nginx/sites-available/autobase @@ -1,24 +1,20 @@ server { listen 80; server_name autobase.dwy.is; - - location / { - proxy_pass http://172.17.0.1:82; - } + + return 301 https://$host$request_uri; } server { server_name autobase.dwy.is; - location / { - proxy_pass http://172.17.0.1:82; - } - - # listen [::]:443 ssl ipv6only=on; # managed by Certbot listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot -} + location / { + proxy_pass http://172.17.0.1:82; + } +} \ No newline at end of file From 69ed6e7ac9c6c105a3acf33792ffc38944c1066b Mon Sep 17 00:00:00 2001 From: nelsonic Date: Mon, 24 Mar 2025 05:50:39 +0000 Subject: [PATCH 12/17] proof read `nginx/README.md for #105 --- nginx/README.md | 61 ++++++++++++++----- .../autobase | 0 .../default | 0 postgres/autobase-ha-cluster.md | 4 +- 4 files changed, 49 insertions(+), 16 deletions(-) rename nginx/{sites-available => sites-enabled}/autobase (100%) rename nginx/{sites-available => sites-enabled}/default (100%) diff --git a/nginx/README.md b/nginx/README.md index 4ca5f2f..92514ec 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -1,10 +1,30 @@ -# `nginx` Fast Setup +# `nginx` _Speedy_ Setup This is a speed run of using `nginx` to proxy an app running on a `Hetzner` server. ## 1. Install `nginx` on `Ubuntu` +`SSH` into the virtual machine, e.g: + +```sh +ssh root@88.99.81.115 +``` + +Ensure that everything is up-to-date on the VM: + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y +``` + +Followed by: + +```sh +sudo reboot +``` + +Now we can proceed with installing `nginx`. + Official instructions: https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview @@ -67,6 +87,8 @@ Link the command: sudo ln -s /snap/bin/certbot /usr/bin/certbot ``` +Basic `certbot` setup for an `nginx` server: + ```sh sudo certbot --nginx ``` @@ -118,12 +140,18 @@ Congratulations, all simulated renewals succeeded: The full config including the TLS is in `/etc/nginx/sites-available/default` +Now create a new config file +_just_ for the subdomain. + ## 3. Configure `nginx` Subdomain ```sh -cd /etc/nginx/sites-enabled/autobase.dwy.is +vi /etc/nginx/sites-enabled/autobase ``` +Paste the contents from this file: +`nginx/sites-enabled/autobase` + Test `nginx` config: ```sh @@ -161,11 +189,16 @@ Restart `nginx`: sudo service nginx restart ``` -## 4. Wildcard Certificate (Failed) +## 4. Wildcard Certificate + +In our case, I actually wanted a wildcard certificate +so that I can add any subdomain I want later. -Wildcard Certificate for Domain: +Wildcard Certificate instructions: https://www.baeldung.com/linux/letsencrypt-certbot-add-subdomains +Sample command: + ```sh sudo certbot certonly -i nginx -d example.com -d *.example.com ``` @@ -218,13 +251,17 @@ value(s) you've just added. Press Enter to Continue ``` -I created the `TXT` record immediatley: +I created the `TXT` record: https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns ![dwyis-txt-record](https://github.com/user-attachments/assets/80ddea19-d06c-4d71-8c8e-f86ee2acd9dc) -But it never propagated ... ⏳ +But this was incorrect! +The host needed to be `_acme-challenge` +***NOT*** `_acme-challenge.dwy.is` +as was implied by `certbot`. +i.e. the domain `dwy.is` should not be in the host! https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.dwy.is @@ -237,12 +274,6 @@ but it never updated. dig -t txt _acme-challenge.dwy.is ``` -Sadly, adding the wildcard TLS cert was a dead-end -because the `TXT` record never updates on `NameCheap` ... -I guess it's _Cheap_ for a _reason_ ... 😢 - -... - I decided to contact `NameCheap` support via live chat: https://www.namecheap.com/help-center/live-chat @@ -256,7 +287,6 @@ https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns Full transcript: [Chat_Transcript_23_Mar_2025.pdf](https://github.com/user-attachments/files/19410954/Chat_Transcript_23_Mar_2025.pdf) - Final output: ```sh @@ -279,4 +309,7 @@ If you like Certbot, please consider supporting our work by: Working! -![autobase.dwy.is-with-ssl](https://github.com/user-attachments/assets/15411040-860f-4a56-9c2d-91fc8702c318) \ No newline at end of file +![autobase.dwy.is-with-ssl](https://github.com/user-attachments/assets/15411040-860f-4a56-9c2d-91fc8702c318) + +Also used: +https://dnschecker.org/#TXT/_acme-challenge.dwy.is \ No newline at end of file diff --git a/nginx/sites-available/autobase b/nginx/sites-enabled/autobase similarity index 100% rename from nginx/sites-available/autobase rename to nginx/sites-enabled/autobase diff --git a/nginx/sites-available/default b/nginx/sites-enabled/default similarity index 100% rename from nginx/sites-available/default rename to nginx/sites-enabled/default diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md index 61208e2..e2775f6 100644 --- a/postgres/autobase-ha-cluster.md +++ b/postgres/autobase-ha-cluster.md @@ -3,8 +3,8 @@ Hi friends, my name is Nelson and this is dwyl.
Today we're going to --> Deploy a Postgres Database Cluster -on -[`Hetzner`] +(on +[`Hetzner`](../hetzner)) using [**autobase**](https://autobase.tech). As always, detailed instructions From b1cb5c59da2868f71a4bccf343eedac3208d3605 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Tue, 25 Mar 2025 12:15:17 +0000 Subject: [PATCH 13/17] add final nginx config for autobase.dwy.is subdomain with https #105 --- nginx/sites-enabled/autobase | 29 ++++++++++++++++++++--- postgres/autobase-ha-cluster.md | 41 +++++++++++++++------------------ 2 files changed, 45 insertions(+), 25 deletions(-) diff --git a/nginx/sites-enabled/autobase b/nginx/sites-enabled/autobase index 54c400b..057c1cd 100644 --- a/nginx/sites-enabled/autobase +++ b/nginx/sites-enabled/autobase @@ -1,20 +1,43 @@ server { listen 80; server_name autobase.dwy.is; - + return 301 https://$host$request_uri; } server { server_name autobase.dwy.is; + location / { + # stackoverflow.com/questions/14501047/add-response-header-nginx-proxy-pass + # 1. hide the Access-Control-Allow-Origin from the server response + proxy_hide_header Access-Control-Allow-Origin; + # 2. add a new custom header that allows all * origins instead + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Credentials' 'true'; + add_header 'Access-Control-Allow-Methods' '*'; + add_header 'Access-Control-Allow-Headers' '*'; + + proxy_pass http://172.17.0.1:82; + } + listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot +} + +server { + listen 8080 ssl; + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot + + server_name autobase.dwy.is; location / { - proxy_pass http://172.17.0.1:82; - } + proxy_pass http://172.17.0.1:8082/; + } } \ No newline at end of file diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md index e2775f6..bac49f2 100644 --- a/postgres/autobase-ha-cluster.md +++ b/postgres/autobase-ha-cluster.md @@ -140,7 +140,7 @@ with the IP (v4) address of your server and `secret_token`in the `PG_CONSOLE_AUTHORIZATION_TOKEN` + IP: 116.202.31.52 (yours will be different!) -+ Token: 5b0b6259-a7d4-4435-947d-0dff528912ba (create your own!) ++ Token: 5b0b6259-a7d4-4435-947dba (create your own!) Actual: @@ -149,7 +149,7 @@ docker run -d --name autobase-console \ --publish 80:80 \ --publish 8080:8080 \ --env PG_CONSOLE_API_URL=http://116.202.31.52:8080/api/v1 \ - --env PG_CONSOLE_AUTHORIZATION_TOKEN=5b0b6259-a7d4-4435-947d-0dff528912ba \ + --env PG_CONSOLE_AUTHORIZATION_TOKEN=5b0b6259-a7d4-4435-947dba \ --env PG_CONSOLE_DOCKER_IMAGE=autobase/automation:latest \ --volume console_postgres:/var/lib/postgresql \ --volume /var/run/docker.sock:/var/run/docker.sock \ @@ -158,7 +158,6 @@ docker run -d --name autobase-console \ autobase/console:latest ``` - Confirm it worked with the `docker ps` command. You should see something similar to the following: ```sh @@ -167,11 +166,10 @@ CONTAINER ID   IMAGE                     COMMAND              9740dfd66c42   autobase/console:latest   "/usr/bin/supervisor…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 5432/tcp   autobase-console ``` - ## 6. Login To `autobase`Console Web UI -Visit the IP Address of your server in you web browser e.g: -http://116.202.31.52/ +Visit the IP Address of your server in you web browser e.g: +http://116.202.31.52 You should see a login screen: @@ -193,17 +191,17 @@ Select `hetzner`and the datacenter region you prefer, in our case Europe: create-cluster-europe -The default disk storage is **`100Gb`**; +The default disk storage is **`100Gb`**; cluster-disk-storage this is _way_ too much for most simple projects. -lower it to **`10Gb`** for each instance to instantly save **50%** of the cost! +lower it to **`10Gb`** for each instance to instantly save **50%** of the cost! (you're welcome!) disk-storage-10gb -> **Note**: all values for `DISK`storage, `RAM`, and `CPU`can easily be scaled later. +> **Note**: all values for `DISK`storage, `RAM`, and `CPU`can easily be scaled later. Finally, you'll need to add your `public` SSH key. @@ -238,9 +236,10 @@ Click on the "**Generate API token**" button: generate-api-token -That will open _another_ modal window, input the description for your key, +That will open _another_ modal window, +input the description for your key, e.g: -"postgres-cluster-api-key" +"postgres-cluster-api-key" and select "**Read and Write**": generate-api-token-modal @@ -260,11 +259,13 @@ Copy the token to your clipboard, e.g: zH2qdgCeogrKjVKgV7sngMRxCfewgSdDARUBr8yqcjuHhGzlNdY72H13Sjh1il2D ``` +> **Note**: for security reasons, this API key is no longer valid. + Paste it into the Cluster creation window: paste-token-in-auto-window -_Optionally_ save the API Key to the console and then +_Optionally_ save the API Key to the console and then click "**CREATE CLUSTER**": create-cluster @@ -335,25 +336,22 @@ But when trying to run `psql` again, we still get an error: Error: You must install at least one postgresql-client- package ``` - - ## Outro: Given that this is a technical guide for an evolving system, it may need to be enhanced/extended or updated in future, -that will be done on GitHub; +that will be done on GitHub; _everyone_ is welcome to and _encouraged_ to contribute! Again, link in the description. Thanks for watching/listening. If you found it useful and want to see more, -please subscribe. - +please subscribe. ## Privacy Disclaimer -By the time you read/watch this,  -all of the sensitive data such as passwords, IP addresses,  +By the time you read/watch this, +all of the sensitive data such as passwords, IP addresses, public keys and auth tokens will have been updated. This avoids anyone getting ideas about accessing backend systems. @@ -362,6 +360,5 @@ so that we can be as transparent as possible. We have a strong security & privacy focus for all our systems so all private backend systems like databases are always locked down. -As always, if you have a security question or concern,  -Please contact us responsibly. - +As always, if you have a security question or concern, +Please contact us responsibly. \ No newline at end of file From 3a7f4fe5eaf7d0b1eb53a3c08e1c94d6a546df8c Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sat, 5 Apr 2025 09:14:09 +0100 Subject: [PATCH 14/17] tidy autobase deployment on hezner #97 --- nginx/README.md | 4 ++ postgres/autobase-ha-cluster.md | 67 ++++++++++++++++++++++----------- postgres/backup-fly-postgres.md | 27 +++++++++++-- 3 files changed, 72 insertions(+), 26 deletions(-) diff --git a/nginx/README.md b/nginx/README.md index 92514ec..1b3f026 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -1,8 +1,12 @@ +
+ # `nginx` _Speedy_ Setup This is a speed run of using `nginx` to proxy an app running on a `Hetzner` server. +
+ ## 1. Install `nginx` on `Ubuntu` `SSH` into the virtual machine, e.g: diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md index bac49f2..57ede7c 100644 --- a/postgres/autobase-ha-cluster.md +++ b/postgres/autobase-ha-cluster.md @@ -1,18 +1,31 @@ +
+ +# High Availability `Postgres` Cluster With `Autobase` + +![autobase-hero-image](https://github.com/user-attachments/assets/51ff8785-2f07-4a23-8892-41ff6a6d2aaa) + +
+ -Deploy a Postgres Database Cluster +Deploy a +[high availability](https://en.wikipedia.org/wiki/High_availability) +`Postgres` database cluster (on [`Hetzner`](../hetzner)) -using [**autobase**](https://autobase.tech). +using [**`autobase`**](https://autobase.tech). + -Along the way we will clarify the steps as possible. -But keep in mind it's _not possible_ to cover everything in a **7 minute video**. +Along the way we will clarify as many of the steps as possible.
+But please keep in mind it's _not possible_ +to cover everything in a **7 minute video**. If you have questions, suggestions or just want to say hi, **please comment on YouTube**; @@ -22,7 +35,6 @@ With all that out of the way, lets dive in! ## 1. Login to `Hetzner` Cloud - When you _first_ login to `Hetzner`, you will see the message: @@ -32,10 +44,16 @@ you will see the message: Click the "**Add Server**" button to begin your quest! -## 2. Create a New Server (VPS) +> If you don't yet have a `Hezner` account, +> please consider using our **referral link**: +> [hetzner.cloud/?ref=ahpZuUB3t0XI](https://hetzner.cloud/?ref=ahpZuUB3t0XI) 🔗 +> to get **`$20`** in credit. 💵
+> Helps us do what we love too. Thanks. ❤ + +## 2. Create a New "Cloud" Virtual Private Server (VPS) -Select all the default options, -add your `ssh` `public` key +Select all the default options, +add your `ssh` `public` key and create your server. new-server-created @@ -78,9 +96,9 @@ pip3 install ansible ### Install `Docker` -The `Ubuntu` Server - -Follow the installation instructions in the **official `Docker` docs**: +Install `Docker` on the `Ubuntu` server +following the installation instructions +in the **official `Docker` docs**: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository Run: @@ -102,6 +120,7 @@ sudo apt-get update ``` Followed by: + ```sh sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` @@ -112,13 +131,13 @@ Verify that the installation is successful by running the `hello-world` image: $ sudo docker run hello-world ``` -With that confirmed working, +With that confirmed working, go back to the previous step and run the `autobase`command. - - ## 5. Run `autobase` Console Boot Script +Install and run the `latest` version of `Autobase` console. + Sample: ```sh @@ -135,14 +154,14 @@ docker run -d --name autobase-console \ autobase/console:latest ``` -You will nee to replace the `localhost` in the `PG_CONSOLE_API_URL` +You will nee to replace the `localhost` in the `PG_CONSOLE_API_URL` with the IP (v4) address of your server -and `secret_token`in the `PG_CONSOLE_AUTHORIZATION_TOKEN` +and `secret_token` for the `PG_CONSOLE_AUTHORIZATION_TOKEN` + IP: 116.202.31.52 (yours will be different!) + Token: 5b0b6259-a7d4-4435-947dba (create your own!) -Actual: +Actual: ```sh docker run -d --name autobase-console \ @@ -158,7 +177,8 @@ docker run -d --name autobase-console \ autobase/console:latest ``` -Confirm it worked with the `docker ps` command. You should see something similar to the following: +Confirm it worked with the `docker ps` command. +You should see something similar to the following: ```sh CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                                                                    NAMES @@ -166,7 +186,7 @@ CONTAINER ID   IMAGE                     COMMAND              9740dfd66c42   autobase/console:latest   "/usr/bin/supervisor…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 5432/tcp   autobase-console ``` -## 6. Login To `autobase`Console Web UI +## 6. Login To `autobase` Console Web UI Visit the IP Address of your server in you web browser e.g: http://116.202.31.52 @@ -211,7 +231,7 @@ Finally, you'll need to add your `public` SSH key. cat ~/.ssh/id_ed25519.pub | pbcopy ``` -Paste it into the `SSH public key*`field: +Paste it into the **`SSH public key`** field: add-ssh-key @@ -338,7 +358,7 @@ Error: You must install at least one postgresql-client- package ## Outro: -Given that this is a technical guide for an evolving system, +Given that this is a technical guide for an evolving system, it may need to be enhanced/extended or updated in future, that will be done on GitHub; _everyone_ is welcome to and _encouraged_ to contribute! @@ -361,4 +381,7 @@ We have a strong security & privacy focus for all our systems so all private backend systems like databases are always locked down. As always, if you have a security question or concern, -Please contact us responsibly. \ No newline at end of file +please contact us responsibly. + diff --git a/postgres/backup-fly-postgres.md b/postgres/backup-fly-postgres.md index 3b421c8..4624cd4 100644 --- a/postgres/backup-fly-postgres.md +++ b/postgres/backup-fly-postgres.md @@ -1,9 +1,9 @@
-# How to Backup Fly.io Postgres Database +# How to Backup `Fly.io` `Postgres` Database A comprehensive step-by-step guide -to backing-up your Fly.io `Postgres` database +to backing-up your `Fly.io` `Postgres` database on your `localhost`.
@@ -12,6 +12,15 @@ on your `localhost`. You need to get the data from a Fly.io `Postgres` instance. +_Your_ reason may be different, +see our [context](#context-) below. + +## What? 🤔 + +Backup your `Postgres` DB running on `Fly.io` +and use the data somewhere `else`; +in our case we are migrating our DBs to `Hetzner` +where we have a High Availability Cluster. ## How? 👩‍💻 @@ -79,29 +88,39 @@ pg_dump -h localhost -U hits_e2k5m6j4k46d0v7p -d hits --verbose > backup.sql > **Note**: If you need to get the password for the `Postgres` instance, use the following command: + ```sh flyctl ssh console -a hits -C "printenv DATABASE_URL" ``` + > in our case it was: + ```sh flyctl ssh console -a hits -C "printenv DATABASE_URL" ``` + We saw: + ```sh postgres://hits_e2k5m6j4k46d0v7p:baf3d9f0bdf155bfetc@hits-db.internal:5432/hits?sslmode=disable ``` + Where the first section `postgres://` is the protocol, the `hits_e2k5m6j4k46d0v7p` is the DB username, `baf3d9f0bdf155bfetc` is the password and `hits` is the name of the database. Ref: https://community.fly.io/t/how-to-view-environment-variables-in-a-fly-machine/10830/2 + Once you have the password, export it as an environtment variable: + ```sh export PGPASSWORD="$put_here_the_password" ``` -> in our case it was: + +in our case it was: + ```sh export PGPASSWORD="baf3d9f0bdf155bfetc" ``` @@ -158,7 +177,7 @@ e.g: http://localhost:8081/# image -i.e. there have been 3 page views on https://github.com/dwyl/start-here since we did the SQL dump a few mins ago. +i.e. there have been 3 page views on https://github.com/dwyl/start-here since we did the SQL dump a few mins ago. We can check the "live" count at: https://hits.dwyl.com/dwyl/start-here.svg e.g: ![hits-start-here-svg](https://hits.dwyl.com/dwyl/start-here.svg) From 8c029c8fd7bcf7ebcf8b278d1675a1a35e0b5237 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sun, 6 Apr 2025 08:26:04 +0100 Subject: [PATCH 15/17] upload backup.sql to hetzner vps for https://github.com/dwyl/hits/issues/320 --- postgres/backup-fly-postgres.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/postgres/backup-fly-postgres.md b/postgres/backup-fly-postgres.md index 4624cd4..d9eb4f8 100644 --- a/postgres/backup-fly-postgres.md +++ b/postgres/backup-fly-postgres.md @@ -20,7 +20,9 @@ see our [context](#context-) below. Backup your `Postgres` DB running on `Fly.io` and use the data somewhere `else`; in our case we are migrating our DBs to `Hetzner` -where we have a High Availability Cluster. +where we have a +[high availability](https://en.wikipedia.org/wiki/High_availability) +cluster. ## How? 👩‍💻 From 5708472a38533ee5394c39c7ab20bc99486a9530 Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sun, 6 Apr 2025 08:26:37 +0100 Subject: [PATCH 16/17] upload backup.sql to hetzner vps for https://github.com/dwyl/hits/issues/320 --- postgres/backup-fly-postgres.md | 3 +- postgres/migrate-db.md | 56 +++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 postgres/migrate-db.md diff --git a/postgres/backup-fly-postgres.md b/postgres/backup-fly-postgres.md index d9eb4f8..ca59f33 100644 --- a/postgres/backup-fly-postgres.md +++ b/postgres/backup-fly-postgres.md @@ -135,7 +135,7 @@ Once the `pg_dump` command finishes, proceed to the next step. ### 3. Close your port forwarding -Kill the connection to the `Fly.io` instance +Kill the connection to the `Fly.io` instance using keyboard shortcut: `Ctrl` + `C` (twice). ### 4. Restore your local database @@ -144,7 +144,6 @@ To restore the database you just backed up to `Postgres` running on your `localhost`, you _first_ need to ensure that `Postgres` is indeed running! - With the `backup.sql` on your `localhost`, run the following command in the working directory: diff --git a/postgres/migrate-db.md b/postgres/migrate-db.md new file mode 100644 index 0000000..5301991 --- /dev/null +++ b/postgres/migrate-db.md @@ -0,0 +1,56 @@ +
+ +# Migrate `Postgres` DB to `Hetzner` Cluster + +
+ +Migrate/restore a snapshot of `Postgres` Database +from `Fly.io` (unreliable) to a +[high availability](https://en.wikipedia.org/wiki/High_availability) +cluster +running on `Hetzner`. + +## 0. Before You Start: Get the Snapshot + +We wrote _detailed_ instructions for backing up +a `Postgres` DB running on `Fly.io`, +see: +[postgres/backup-fly-postgres.md] + +backup.sql + +With the `backup.sql` on your `localhost`, +you can start. + +## 1. Connect To `Hezner` VPS Using `Cyberduck` + +There are several ways to upload large files to a remote server, +we've been using +[`Cyberduck`](https://en.wikipedia.org/wiki/Cyberduck) +for the past few decades and it works very well. +It uses +[`SFTP`](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) +to securely transfer files +and is Open Source: +[github.com/iterate-ch/cyberduck](https://github.com/iterate-ch/cyberduck) + +> The Official Docs are great: +[docs.cyberduck.io](https://docs.cyberduck.io/cyberduck/) +and if you get stuck, +just Google: +[google.com/search?q=cyberduck+tutorial](https://www.google.com/search?q=cyberduck+tutorial) + +Open `Cyberduck` +and navigate to the `/tmp` directory of the VPS: + +hits-upload-backup-to-hetzner + +Drag the `backup.sql` file from the `finder` window on `localhost` +to the `Cyberduc` window to start the upload. + +hits-upload-backup-in-progress + +Take a screen break and refill your water bottle +while you wait for upload to complete. + +hits-upload-complete \ No newline at end of file From b91805fd243d503dbea0914669aa77c2b6fbcbeb Mon Sep 17 00:00:00 2001 From: nelsonic Date: Sun, 11 May 2025 00:36:41 +0100 Subject: [PATCH 17/17] create cheatsheet.md with first useful ubuntu command #102 --- cheatsheet.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 cheatsheet.md diff --git a/cheatsheet.md b/cheatsheet.md new file mode 100644 index 0000000..7b54364 --- /dev/null +++ b/cheatsheet.md @@ -0,0 +1,12 @@ +# Cheat Sheet + +This [cheat sheet](https://en.wikipedia.org/wiki/Cheat_sheet) +helps us manage our `DevOps` _fast_. + +## Update & Reboot Ubuntu Server + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y && sudo reboot +``` + +The `alias` for this command on our servers is `upr`. \ No newline at end of file