What it does

Terraform code to deploy Ollama and Open WebUI in a lower-cost pre-emptible spot VM with an NVIDIA L4 GPU on Google Cloud. Ready to go in five minutes.

Background

I've been playing around with various local LLMs and wanted to run them on some better hardware. But I'm not interested in paying hundreds or thousands of dollars a month.

Spot VMs offer significant price savings, even with GPUs. The downside to them is that you may be pre-empted during busy times. So this isn't a solution for prod.

You'll have to see how it works for you for development. But spot VMs are a great lower-cost alternative if you can tolerate occasionally being pre-empted.

As for the name: spot VMs + Ollama = Spotted Llama.

GCP Considerations

Machine type

The L4 has 24GB of GPU RAM.

Using an L4 GPU requires a G2-series compute engine instance. The base g2-standard-4 only has 16GB of RAM, which isn't enough to load more than small models. I bumped up to g2-standard-8 ($0.34/hour) which has 32GB.

Persistent Disk

Spot VMs can be pre-empted at any time, and moving them to another zone means rebuilding them. I was hoping to use regional persistent disks to store data that should be retained when moving zones. Unfortunately, that's not possible. G2 machine types don't support regional disks.

Balanced PDs cost $0.10/GB/month, so I increased the base PD size from 50GB to 100GB to account for storage space for models and data. You may be able to decrease this. Current disk cost is $10/month.

This means if you are pre-empted and have to move to a different zone but want to retain your data, you'll need to create a snapshot of the current disk and use it to create the new disk in the new zone. Or just start from scratch again.

Another option is to modify the startup script for your needs, so it always installs whatever models you want. This is what I do.

Snapshots

If you want to create a snapshot:

gcloud compute disks create-snapshot \
    --source-disk=SOURCE_DISK_NAME \
    --name=SNAPSHOT_NAME \
    --source-disk-zone=SOURCE_DISK_ZONE

If you want to use it to spin up a new instance, create a google_compute_disk block in main.tf and reference the new snapshot by name:

resource "google_compute_disk" "boot" {
  name = "${var.base_name}-boot"
  size = 50
  source_instant_snapshot = "SNAPSHOT_NAME"
  type = "pd-balanced"
  zone = var.zone
}

You can then comment out the entire initialize_params block of the boot_disk block in google_compute_instance and add this line within the boot_disk block:

source = google_compute_disk.boot.name

Change the zone in your terraform.tfvars file to the new value, and run a terraform apply.

Once the new instance comes up and you've verified your data, you can delete the snapshot and remove the source_instance_snapshot entry from google_compute_disk

Private IP with Cloud NAT

For security purposes, I didn't want any public IP addresses. But if you want to be able to download and install models, you'll needed internet access. I chose to use Cloud NAT instead of a public IP.

Cloud NAT can be disabled by setting the variable enable_nat = false. Note that this will destroy the Cloud NAT and router instances in the project since they are only needed for outbound internet access. You will still be able to connect via an IAP tunnel to use Ollama, but you won't be able to update software or download new models.

Image Choice

Uses the common-cu124 image:

Google, Deep Learning VM with CUDA 12.4, M129, Debian 11, Python 3.10.
  With CUDA 12.4 preinstalled.
diskSizeGb: '50'
enableConfidentialCompute: false
family: common-cu124

Prerequisites

1. You need an adequate GPU quota allowance. I did not include this in Terraform, because:

   • it can take 24-48 hours or longer for Google to review these

   • it could get very messy if the code just assumed you were at zero and changed your quota to one, but you weren't actually at zero and had a higher quota.

      

     DO THIS BEFORE YOU DO ANYTHING ELSE

      

     This may take some time. Factors that control approval time depend on your history with GCP, billing, and other factors.

     When I submitted the quota increase request in March 2025, it was approved in about five minutes. But I've heard of it taking days for some people.

      

     To request GPU quota increase:

     • Go to Google Cloud Console
     • Navigate to IAM & Admin > Quotas
     • Filter for "GPUs (all regions)"
     • Select the quota
     • Click "EDIT QUOTAS"
     • Submit a request for the number of GPUs you need (1 if this is your first)

2. Make sure the account you are running this under has explicitly been granted roles/resourcemanager.projectCreator. Even if you are an Owner, you still need this granted. You need to grant this if you receive the error:

   googleapi: Error 403: Permission 'resourcemanager.projects.create' denied on resource (or it may not exist)., forbidden. If you received a 403 error, make sure you have the `roles/resourcemanager.projectCreator` permission
   

    

   First make sure you are logged in with the correct account gcloud auth application-default login. Try again after logging in again.

    

   If you continue to get this error:

   • Go to the Google Cloud Console

   • Select your top-level organization

   • Navigate to IAM & Admin

   • Find your account

   • Add the "Project Creator" role (roles/resourcemanager.projectCreator)

      

     You can also do this via gcloud:

     gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
         --member="user:your-email@domain.com" \
         --role="roles/resourcemanager.projectCreator"

     If it still fails after doing this, also make sure you have roles/billing.user (to associate billing accounts with new projects).

Variables

You will need to specify the region and zone you wish to use. Make sure that g2-standard-4 machine type and nvidia-l4 accelerator types are available in the region and zone you want to use.

You can check them both with gcloud:

gcloud compute machine-types list --filter="name=g2-standard-4"
gcloud compute accelerator-types list --filter="name=nvidia-l4"

Project Creation

You can specify a value for organization_id or folder_id, but not both. If you specify an organization_id, the project will be created under that organization. Likewise, if you specify a folder_id, the project will be created in that folder.

If you specify both organization_id and folder_id, Terraform will report "Error: Conflicting configuration arguments"

Installation

1. Create a terraform.tfvars file. At minimum, it must have the following:

   • billing_account_id
   • folder_id (or organization_id - but not both)
   • project_id
   • project_name
   • region
   • subnet_cidr
   • zone

   See variables.tf for the list of all possible variables.

2. Login to GCP

gcloud auth application-default login

3. Deploy it

terraform apply

4. Connect and tunnel over SSH

gcloud compute ssh spotted-llama-l4 -- -L 3000:localhost:3000 -L 11434:localhost:11434

5. Browse to http://localhost:3000

This also forwards localhost:11434 for direct API access to Ollama, e.g. for use in IDEs, like Zed

Using it

There are two ways to access it via a private IP:

1. SSH to the VM and tunnel the port from localhost
2. Use an IAP tunnel

Access via SSH

This is my preferred method for two reasons:

1. No open ports on the box beyond ssh
2. It leaves an open terminal to the box, which reminds me to shut down the instance when I'm done. This keeps my costs down.

Establish the tunnel:

gcloud compute ssh spotted-llama-l4 -- -L 3000:localhost:3000

Then just browse to http://localhost:3000

Using an IAP tunnel

Using an IAP tunnel will require:

One-time setup

• change startup.sh so that the docker run binds port 3000 to all interfaces, not just loopback

docker run -d -p 3000:8080 \
    --gpus=all \
    -v /opt/ollama:/root/.ollama \
    -v /opt/open-webui:/app/backend/data \
    --name open-webui \
    --pull always \
    --restart always \
    ghcr.io/open-webui/open-webui:ollama

• add a firewall rule allowing ingress to port 3000 from IAP. Do this by editing networking.tf and changing the allow{} block to this:

allow {
  protocol = "tcp"
  ports    = [22, 3000]
}

• terraform apply

Connecting each time

Bring up the tunnel:

 gcloud compute start-iap-tunnel VM_NAME 3000 --local-host-port=localhost:3000 --zone=ZONE --project=PROJECT_ID

Then just browse to http://localhost:3000

Next Steps

Need to install the Google Ops Agent.

I want to see how performance would be using GCS. It would be a significant cost savings, about 80% cheaper than balanced PD, but I'm not sure if performance would be a bottleneck. 500GB of regional storage is only $9.90/month.

There would also be the nice advantage of being able to store models and other data in a bucket that can be mounted anywhere in the region with no transfer costs. Which would make spinning up in a new zone much faster and easier.

There are ways to speed up reads of large files https://cloud.google.com/storage/docs/cloud-storage-fuse/performance#increase-read-ahead-size