GitHub - AdamShannag/volare: Kubernetes volume populator

About Volare

Volare is a Kubernetes volume populator that populates Persistent Volume Claims (PVCs) from multiple external sources concurrently.

Supported Source Types

HTTP – Direct file URLs (e.g., raw files from GitHub, CDN links).
Git – Clones the entire repository using Git over HTTPS, then extracts the requested files. Works with any Git provider (GitHub, GitLab, Bitbucket, etc.).
GitHub – Downloads only the requested files directly using the GitHub API. Supports public and private repositories.
GitLab – Downloads only the requested files directly using the GitLab API. Supports both public and private projects.
S3 – Downloads files from any S3-compatible storage provider (e.g., AWS S3, MinIO, DigitalOcean Spaces).
GCS – Downloads files from Google Cloud Storage (GCS) buckets. Supports private buckets via service account credentials.

Note: The git source type performs a full git clone, which can be slower for large repositories. In contrast, github and gitlab use provider-specific APIs to fetch only the requested files, making them faster.

Getting Started

1. Install VolumeDataSource Validator (Upstream Requirement)

Volare relies on the upstream CSI Volume Populator. Install the required CRDs and RBAC by running:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/client/config/crd/populator.storage.k8s.io_volumepopulators.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/rbac-data-source-validator.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/setup-data-source-validator.yaml

2. Deploy Volare CRDs and Populators

Apply the custom resource definitions and Volare volume populators:

kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/1.volarepopulators-crd.yaml
kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/2.volumepopulator.yaml

3. Deploy Volare Controller & Example PVC

Start the Volare controller and create a sample PVC using a VolarePopulator:

kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/3.volarepopulator.yaml
kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/4.volare-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/5.pvc.yaml

4. Verify the Mounted Data

Deploy a checker pod and inspect the mounted data:

kubectl apply -f https://raw.githubusercontent.com/AdamShannag/volare/master/manifests/6.checker.yaml

Then open a shell inside the pod:

kubectl exec -it nginx -- bash
cd /mnt/checker
ls

Sources Configuration Reference

A detailed overview of all supported source types (http, gitlab, github, s3, git, gcs), their available configuration options, and practical usage examples.

Note on Environment Variable For sensitive fields like tokens and credentials, values can be loaded from environment variables. If the value matches the name of an environment variable, it will be substituted; otherwise, the literal value will be used as-is.

Fields that support environment variable:

Source Type	Field
`http`	Headers (values only)
`gitlab`	`token`
`github`	`token`
`s3`	`accessKeyId`
`s3`	`secretAccessKey`
`s3`	`sessionToken`
`git`	`username`
`git`	`password`

Example: If you set token: GITLAB_TOKEN in your config and your environment has GITLAB_TOKEN=abcd1234, it will use abcd1234.

These environment variables can be injected directly into the controller deployment like this:

env:
  - name: GITLAB_TOKEN
    value: secret-token

Full example snippet from a Kubernetes controller deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller
  namespace: controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: controller
  template:
    metadata:
      labels:
        app: controller
    spec:
      serviceAccountName: vp-account
      containers:
        - name: volarepopulator
          image: ghcr.io/adamshannag/volare:v1.0.1
          args:
            - "--mode=controller"
            - "--image=ghcr.io/adamshannag/volare:v1.0.1"
            - "--namespace=controller"
            - "--mountpath=/mnt/checker"
          env:
            - name: GITLAB_TOKEN
              value: secret-token

Resources

The controller supports mounting a shared directory of static resources (e.g., credentials, policies, templates) via the --resources flag. This allows passing metadata to the populator — currently used only by the gcs source type.

Field	Type	Required	Description
`--resources`	string	❌	Filesystem path where resource files are mounted. Optional.
`volumeMounts`	-	❌	Mount the volume at the same `--resources` path. Typically mapped from a Kubernetes Secret or ConfigMap.

Example (with GCP credentials mounted under resources path)

containers:
  ...
  args:
    - "--mode=controller"
    - "--image=ghcr.io/adamshannag/volare:v1.0.1"
    - "--namespace=controller"
    - "--mountpath=/mnt/checker"
    - "--resources=/tmp/resources"

  volumeMounts:
    - name: controller-resources
      mountPath: /tmp/resources/gcs # relative to --resources
      #     mountPath: /tmp/resources/gcs/us
      #     mountPath: /tmp/resources
      readOnly: true

volumes:
  - name: controller-resources
    secret:
      secretName: gcs-secret

Volare scans all files under --resources, mapping each file's relative path to its contents for use by the Populator.

Common Required Fields (All Types)

Field	Type	Required	Description
`type`	string	✅	One of: `http`, `gitlab`, `github`, `s3`, `git`, `gcs`
`targetPath`	string	✅	Relative path under `mountPath` to store the file(s)

HTTP Source

Field	Type	Required	Description
`http.uri`	string	✅	URI of the file to download (must be full URL)
`http.headers`	object	❌	Optional HTTP headers (e.g., auth)

Example

- type: http
  targetPath: docs
  http:
    uri: https://example.com/readme.md
    headers:
      Authorization: BEARER_TOKEN
      Content-Type: application/json

GitLab Source

Field	Type	Required	Description
`gitlab.host`	string	✅	GitLab host (e.g., `https://gitlab.com`)
`gitlab.project`	string	✅	Full project path (e.g., `group/my-project`)
`gitlab.ref`	string	✅	Git reference (branch/tag/commit)
`gitlab.paths`	string[]	✅	List of file or directory keys to download. Keys ending with / will create the corresponding directory; otherwise only contents are extracted.
`gitlab.token`	string	❌	Required if private repo
`gitlab.workers`	integer	❌	Optional, default is 2

Example

- type: gitlab
  targetPath: integrations
  gitlab:
    host: https://gitlab.com
    project: group/my-project
    ref: master
    paths:
      - integration/testutils/ # will create testutils directory and its content
      - integration/testutils # won't create testutils directory, just its content
      - integration/testutils/db.go
    token: secret-token
    workers: 2

GitHub Source

Field	Type	Required	Description
`github.owner`	string	✅	GitHub repository owner
`github.repo`	string	✅	Repository name
`github.ref`	string	✅	Git reference (branch/tag/commit)
`github.paths`	string[]	✅	List of file or directory keys to download. Keys ending with / will create the corresponding directory; otherwise only contents are extracted.
`github.token`	string	❌	Required if private repo
`github.workers`	integer	❌	Optional, default is 2

Example

- type: github
  targetPath: /github
  github:
    owner: kubernetes-csi
    repo: lib-volume-populator
    ref: master
    paths:
      - example/ # will create example directory and its content
      - example # won't create example directory, just its content
    workers: 2

S3 Source

Field	Type	Required	Description
`s3.endpoint`	string	✅	S3-compatible service endpoint
`s3.secure`	boolean	❌	Use HTTPS (default: false)
`s3.bucket`	string	✅	Name of the bucket
`s3.paths`	string[]	✅	List of file or directory keys to download. Keys ending with / will create the corresponding directory; otherwise only contents are extracted.
`s3.region`	string	❌	Region (optional for some services)
`s3.accessKeyId`	string	✅	Access key ID
`s3.secretAccessKey`	string	✅	Secret access key
`s3.sessionToken`	string	❌	Temporary token (if using session auth)
`s3.workers`	integer	❌	Optional, default is 2

Example

- type: s3
  targetPath: /s3
  s3:
    endpoint: fra1.digitaloceanspaces.com
    secure: true
    bucket: bucket-name
    region: us-east-1
    accessKeyId: access-key-id
    secretAccessKey: secret
    paths:
      - backups/ # will create backup directory and its content
      - backups # won't create backup directory, just its content
      - images/image1.png
    workers: 3

Git Source (Generic)

Field	Type	Required	Description
`git.url`	string	✅	URL of the Git repository to clone
`git.paths`	string[]	✅	List of file or directory keys to download. Keys ending with / will create the corresponding directory; otherwise only contents are extracted.
`git.ref`	string	❌	Git reference (branch/tag/commit). Defaults to the repository’s default branch if omitted
`git.username`	string	❌	Username for basic HTTP authentication (if required)
`git.password`	string	❌	Password or token for basic HTTP authentication (if required)
`git.remote`	string	❌	Remote name to use (default is `origin`)
`git.workers`	integer	❌	Number of concurrent workers for file copying. Defaults to 2

Example

- type: git
  targetPath: /gitrepo
  git:
    # HTTPS URL to the repository — works with Bitbucket, GitHub, GitLab, etc.
    url: https://bitbucket.org/project/repo.git

    # List of file paths to extract from the repository
    paths:
      - README.md  # Only this file will be copied from the repo

    # Authentication (required for private repositories)
    username: BITBUCKET_USERNAME        # Your Bitbucket username
    password: BITBUCKET_APP_PASSWORD    # Use a Bitbucket app password or API access token

    # Number of parallel workers used to copy files
    workers: 2

GCS Source

Field	Type	Required	Description
`gcs.bucket`	string	✅	Name of the GCS bucket to fetch files from.
`gcs.paths`	string[]	✅	List of object paths or directories to download. Paths ending with `/` are treated as directories; otherwise, only the file is downloaded.
`gcs.credentialsFile`	string	❌	Relative path (within `--resources`) to a GCP service account JSON file for accessing private buckets.
`gcs.workers`	integer	❌	Number of concurrent download workers. Defaults to 2.

Authenticating with a Private GCS Bucket

To access a private GCS bucket, mount a service account key file using a Kubernetes Secret:

Step 1: Create the Secret

kubectl create secret generic gcs-secret \
  --from-file=credentials.json=/path/to/serviceAccountKeyfile.json -n controller

The key name (credentials.json) must match the value used in credentialsFile.

Refer to the official GCP guide for generating service account keys.

Required IAM Role:

Storage Object Viewer (roles/storage.objectViewer): Grants read access to bucket contents.

Step 2: Mount the Secret in Your Controller Pod

containers:
  ...
  args:
    - "--mode=controller"
    - "--image=ghcr.io/adamshannag/volare:v1.0.1"
    - "--namespace=controller"
    - "--mountpath=/mnt/checker"
    - "--resources=/tmp/resources"

  volumeMounts:
    - name: controller-resources
      mountPath: /tmp/resources/gcs
      readOnly: true

volumes:
  - name: controller-resources
    secret:
      secretName: gcs-secret

Nested paths are supported If your secret is mounted to /tmp/resources/gcs/credentials.json, set credentialsFile: gcs/credentials.json.

This enables clean resource structuring (e.g., grouping under gcs/, us-west/, secrets/, etc.).

Example Configuration

- type: gcs
  targetPath: /gcs-data
  gcs:
    bucket: volare-bucket
    paths:
      - assets/logo.png
      - data/config/
    credentialsFile: gcs/credentials.json
    workers: 3

Global Options

Field	Type	Required	Description
`workers`	integer	❌	Global number of concurrent source workers (default: 2)

Example `VolarePopulator`

apiVersion: k8s.volare.dev/v1alpha1
kind: VolarePopulator
metadata:
  name: volare-populator
spec:
  sources:
    - type: github
      targetPath: /github
      github:
        owner: kubernetes-csi
        repo: lib-volume-populator
        ref: master
        paths:
          - example
        # public repo       
        # token: GITHUB_TOKEN # Read from env if set, otherwise used as a raw string
    - type: s3
      targetPath: /s3
      s3:
        endpoint: s3.example.com
        bucket: data-bucket
        region: us-east-1
        accessKeyId: AWS_ACCESS_KEY # Read from env if set, otherwise used as a raw string
        secretAccessKey: AWS_SECRET_KEY # Read from env if set, otherwise used as a raw string
        secure: true
        paths:
          - data/
  workers: 4

Note: You can define as many sources as needed. For example, it's possible to configure multiple sources of the same or different types—such as 3 GitHub sources, 2 GitLab sources, 3 S3 sources, and even 20 HTTP sources—within the same configuration.

License

MIT

Contribution

If you have any questions or want to contribute, feel free to open an issue or PR.

Name		Name	Last commit message	Last commit date
Latest commit History 29 Commits
.github/workflows		.github/workflows
assets		assets
cmd/volare		cmd/volare
internal/populator		internal/populator
manifests		manifests
pkg		pkg
.gitignore		.gitignore
.goreleaser.yml		.goreleaser.yml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.MD		README.MD
go.mod		go.mod
go.sum		go.sum

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

About Volare

Supported Source Types

Getting Started

1. Install VolumeDataSource Validator (Upstream Requirement)

2. Deploy Volare CRDs and Populators

3. Deploy Volare Controller & Example PVC

4. Verify the Mounted Data

Sources Configuration Reference

Fields that support environment variable:

Resources

Example (with GCP credentials mounted under resources path)

Common Required Fields (All Types)

HTTP Source

Example

GitLab Source

Example

GitHub Source

Example

S3 Source

Example

Git Source (Generic)

Example

GCS Source

Authenticating with a Private GCS Bucket

Global Options

Example `VolarePopulator`

License

Contribution

About

Uh oh!

Releases 9

Packages

Uh oh!

Languages

License

AdamShannag/volare

Folders and files

Latest commit

History

Repository files navigation

About Volare

Supported Source Types

Getting Started

1. Install VolumeDataSource Validator (Upstream Requirement)

2. Deploy Volare CRDs and Populators

3. Deploy Volare Controller & Example PVC

4. Verify the Mounted Data

Sources Configuration Reference

Fields that support environment variable:

Resources

Example (with GCP credentials mounted under resources path)

Common Required Fields (All Types)

HTTP Source

Example

GitLab Source

Example

GitHub Source

Example

S3 Source

Example

Git Source (Generic)

Example

GCS Source

Authenticating with a Private GCS Bucket

Global Options

Example VolarePopulator

License

Contribution

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 9

Packages 0

Uh oh!

Languages

Example `VolarePopulator`

Packages