Skip to content

Dynamic provisioning

Secrets

After acquiring service account keys, create 2 secrets (we'll call them csi-gcs-secret-mounter and csi-gcs-secret-creator in the following example):

kubectl create secret generic csi-gcs-secret-mounter --from-file=key=<PATH_TO_SERVICE_ACCOUNT_KEY_1>
kubectl create secret generic csi-gcs-secret-creator --from-file=key=<PATH_TO_SERVICE_ACCOUNT_KEY_2> --from-literal=projectId=csi-gcs

Usage

Let's run another example application!

kubectl apply -k "github.com/ofek/csi-gcs/examples/dynamic?ref=v0.9.0"

Confirm it's working by running

kubectl get pods,sc,pv,pvc

You should see something like

NAME                                READY   STATUS    RESTARTS   AGE
pod/csi-gcs-test-5f677df9f9-qd8nt   2/2     Running   0          100s

NAME                                             PROVISIONER          AGE
storageclass.storage.k8s.io/csi-gcs              gcs.csi.ofek.dev     100s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
persistentvolume/pvc-906ed812-2c06-4eaa-a80e-7115e8ffd653   5Gi        RWO            Delete           Bound    default/csi-gcs-pvc   csi-gcs                 98s

NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/csi-gcs-pvc   Bound    pvc-906ed812-2c06-4eaa-a80e-7115e8ffd653   5Gi        RWO            csi-gcs        100s

Note the pod name, in this case csi-gcs-test-5f677df9f9-qd8nt. The pod in the example deployment has 2 containers: a writer and a reader.

Now create some data!

kubectl exec csi-gcs-test-5f677df9f9-qd8nt -c writer -- /bin/sh -c "echo Hello from Google Cloud Storage! > /data/test.txt"

Let's read what we just put in the bucket

$ kubectl exec csi-gcs-test-5f677df9f9-qd8nt -c reader -it -- /bin/sh
/ # ls -lh /data
total 1K
-rw-r--r--    1 root     root          33 May 26 21:23 test.txt
/ # cat /data/test.txt
Hello from Google Cloud Storage!

Notice that while the writer container's permission is completely governed by the mounter's service account key, the reader container is further restricted to read-only access

/ # touch /data/forbidden.txt
touch: /data/forbidden.txt: Read-only file system

To clean up everything, run the following commands

kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/deployment.yaml"
kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/pvc.yaml"
kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/sc.yaml"
kubectl delete -k "github.com/ofek/csi-gcs/deploy/overlays/stable?ref=v0.9.0"
kubectl delete secret csi-gcs-secret-creator
kubectl delete secret csi-gcs-secret-mounter
Note

Cleanup is necessarily verbose until this is resolved.

Driver options

StorageClass is the resource type that enables dynamic provisioning.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <STORAGE_CLASS_NAME>
provisioner: gcs.csi.ofek.dev
reclaimPolicy: Delete
parameters: ...

Storage Class Parameters

Annotation Description
csi.storage.k8s.io/node-publish-secret-name The name of the secret allowed to mount created buckets
csi.storage.k8s.io/node-publish-secret-namespace The namespace of the secret allowed to mount created buckets
csi.storage.k8s.io/provisioner-secret-name The name of the secret allowed to create buckets
csi.storage.k8s.io/provisioner-secret-namespace The namespace of the secret allowed to create buckets
csi.storage.k8s.io/controller-expand-secret-name The name of the secret allowed to expand bucket capacity
csi.storage.k8s.io/controller-expand-secret-namespace The namespace of the secret allowed to expand bucket capacity
gcs.csi.ofek.dev/project-id The project to create the buckets in. If not specified, projectId will be looked up in the provisioner's secret
gcs.csi.ofek.dev/location The location to create buckets at (default US multi-region)
gcs.csi.ofek.dev/kms-key-id (optional) KMS encryption key ID. (projects/my-pet-project/locations/us-east1/keyRings/my-key-ring/cryptoKeys/my-key)
gcs.csi.ofek.dev/max-retry-sleep The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

Tip

You may omit the secret definition and let the code automatically detect the service account key using standard heuristics.

Persistent Volume Claim Parameters

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations: ...
Annotation Description
gcs.csi.ofek.dev/project-id The project to create the buckets in. If not specified, projectId will be looked up in the provisioner's secret
gcs.csi.ofek.dev/location The location to create buckets at (default US multi-region)
gcs.csi.ofek.dev/bucket The name for the new bucket
gcs.csi.ofek.dev/kms-key-id (optional) KMS encryption key ID. (projects/my-pet-project/locations/us-east1/keyRings/my-key-ring/cryptoKeys/my-key)
gcs.csi.ofek.dev/max-retry-sleep The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

Persistent buckets

In our example, the dynamically created buckets are deleted during cleanup. If you want the buckets to not be ephemeral, you can set reclaimPolicy to Retain.

Extra flags

You can pass flags to gcsfuse. They will be forwarded to PersistentVolumeClaim.spec.csi.volumeAttributes.

The following flags are supported (ordered by precedence):

  1. PersistentVolumeClaim.metadata.annotations 
    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    gcs.csi.ofek.dev/gid: "63147"
    gcs.csi.ofek.dev/dir-mode: "0775"
    gcs.csi.ofek.dev/file-mode: "0664"
    Option Type Description
    gcs.csi.ofek.dev/dir-mode Octal Integer Permission bits for directories. (default: 0775)
    gcs.csi.ofek.dev/file-mode Octal Integer Permission bits for files. (default: 0664)
    gcs.csi.ofek.dev/gid Integer GID owner of all inodes. (default: 63147)
    gcs.csi.ofek.dev/uid Integer UID owner of all inodes. (default: -1)
    gcs.csi.ofek.dev/implicit-dirs Flag Implicitly define directories based on content.
    gcs.csi.ofek.dev/billing-project Text Project to use for billing when accessing requester pays buckets.
    gcs.csi.ofek.dev/limit-bytes-per-sec Integer Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
    gcs.csi.ofek.dev/limit-ops-per-sec Integer Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
    gcs.csi.ofek.dev/stat-cache-ttl Text How long to cache StatObject results and inode attributes e.g. 1h.
    gcs.csi.ofek.dev/type-cache-ttl Text How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
    gcs.csi.ofek.dev/fuse-mount-options Text[] Additional comma-separated system-specific mount options. Be careful!
    gcs.csi.ofek.dev/max-retry-sleep Integer The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.
  2. StorageClass.parameters 
    apiVersion: storage.k8s.io/v1
kind: StorageClass
parameters:
  gid: "63147"
  dirMode: "0775"
  fileMode: "0664"
    Option Type Description
    dirMode Octal Integer Permission bits for directories. (default: 0775)
    fileMode Octal Integer Permission bits for files. (default: 0664)
    gid Integer GID owner of all inodes. (default: 63147)
    uid Integer UID owner of all inodes. (default: -1)
    implicitDirs Flag Implicitly define directories based on content.
    billingProject Text Project to use for billing when accessing requester pays buckets.
    limitBytesPerSec Integer Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
    limitOpsPerSec Integer Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
    statCacheTTL Text How long to cache StatObject results and inode attributes e.g. 1h.
    typeCacheTTL Text How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
    fuseMountOptions Text[] Additional comma-separated system-specific mount options. Be careful!
    maxRetrySleep Integer The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.
  3. StorageClass.mountOptions 
    apiVersion: storage.k8s.io/v1
kind: StorageClass
mountOptions:
  - --gid=63147
  - --dir-mode=0775
  - --file-mode=0664
    Option Type Description
    dir-mode Octal Integer Permission bits for directories. (default: 0775)
    file-mode Octal Integer Permission bits for files. (default: 0664)
    gid Integer GID owner of all inodes. (default: 63147)
    uid Integer UID owner of all inodes. (default: -1)
    implicit-dirs Flag Implicitly define directories based on content.
    billing-project Text Project to use for billing when accessing requester pays buckets.
    limit-bytes-per-sec Integer Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
    limit-ops-per-sec Integer Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
    stat-cache-ttl Text How long to cache StatObject results and inode attributes e.g. 1h.
    type-cache-ttl Text How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
    fuse-mount-option Text Additional system-specific mount option. Be careful!
    max-retry-sleep Integer The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.
  4. StorageClass.parameters."csi.storage.k8s.io/provisioner-secret-name"
    Option Type Description
    dirMode Octal Integer Permission bits for directories, in octal. (default: 0775)
    fileMode Octal Integer Permission bits for files, in octal. (default: 0664)
    gid Integer GID owner of all inodes. (default: 63147)
    uid Integer UID owner of all inodes. (default: -1)
    implicitDirs Flag Implicitly define directories based on content.
    billingProject Text Project to use for billing when accessing requester pays buckets.
    limitBytesPerSec Integer Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
    limitOpsPerSec Integer Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
    statCacheTTL Text How long to cache StatObject results and inode attributes e.g. 1h.
    typeCacheTTL Text How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
    fuseMountOptions Text[] Additional comma-separated system-specific mount options. Be careful!
    maxRetrySleep Integer The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

Permission

In order to access anything stored in GCS, you will need service accounts with appropriate IAM roles. You will usually assign the role roles/storage.admin.

The easiest way to create service account keys, if you don't yet have any, is to run:

gcloud iam service-accounts list

to find the email of a desired service account, then run:

gcloud iam service-accounts keys create <FILE_NAME>.json --iam-account <EMAIL>

to create a key file.

Mounter

The Node Plugin is the component that is actually mounting and serving buckets to pods. If writes are needed, you will usually select roles/storage.objectAdmin scoped to the desired buckets.

Creator

The Controller Plugin is the component that is in charge of creating buckets. The service account will need the storage.buckets.create Cloud IAM permission.

Last update: May 14, 2022