Dynamic provisioning¶

Secrets¶

After acquiring service account keys, create 2 secrets (we'll call them csi-gcs-secret-mounter and csi-gcs-secret-creator in the following example):

kubectl create secret generic csi-gcs-secret-mounter --from-file=key=<PATH_TO_SERVICE_ACCOUNT_KEY_1>
kubectl create secret generic csi-gcs-secret-creator --from-file=key=<PATH_TO_SERVICE_ACCOUNT_KEY_2> --from-literal=projectId=csi-gcs

Usage¶

Let's run another example application!

kubectl apply -k "github.com/ofek/csi-gcs/examples/dynamic?ref=v0.9.0"

Confirm it's working by running

kubectl get pods,sc,pv,pvc

You should see something like

NAME                                READY   STATUS    RESTARTS   AGE
pod/csi-gcs-test-5f677df9f9-qd8nt   2/2     Running   0          100s

NAME                                             PROVISIONER          AGE
storageclass.storage.k8s.io/csi-gcs              gcs.csi.ofek.dev     100s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
persistentvolume/pvc-906ed812-2c06-4eaa-a80e-7115e8ffd653   5Gi        RWO            Delete           Bound    default/csi-gcs-pvc   csi-gcs                 98s

NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/csi-gcs-pvc   Bound    pvc-906ed812-2c06-4eaa-a80e-7115e8ffd653   5Gi        RWO            csi-gcs        100s

Note the pod name, in this case csi-gcs-test-5f677df9f9-qd8nt. The pod in the example deployment has 2 containers: a writer and a reader.

Now create some data!

kubectl exec csi-gcs-test-5f677df9f9-qd8nt -c writer -- /bin/sh -c "echo Hello from Google Cloud Storage! > /data/test.txt"

Let's read what we just put in the bucket

$ kubectl exec csi-gcs-test-5f677df9f9-qd8nt -c reader -it -- /bin/sh
/ # ls -lh /data
total 1K
-rw-r--r--    1 root     root          33 May 26 21:23 test.txt
/ # cat /data/test.txt
Hello from Google Cloud Storage!

Notice that while the writer container's permission is completely governed by the mounter's service account key, the reader container is further restricted to read-only access

/ # touch /data/forbidden.txt
touch: /data/forbidden.txt: Read-only file system

To clean up everything, run the following commands

kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/deployment.yaml"
kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/pvc.yaml"
kubectl delete -f "https://github.com/ofek/csi-gcs/blob/v0.9.0/examples/dynamic/sc.yaml"
kubectl delete -k "github.com/ofek/csi-gcs/deploy/overlays/stable?ref=v0.9.0"
kubectl delete secret csi-gcs-secret-creator
kubectl delete secret csi-gcs-secret-mounter

Cleanup is necessarily verbose until this is resolved.

Driver options¶

StorageClass is the resource type that enables dynamic provisioning.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <STORAGE_CLASS_NAME>
provisioner: gcs.csi.ofek.dev
reclaimPolicy: Delete
parameters: ...

Storage Class Parameters¶

Persistent Volume Claim Parameters¶

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations: ...

Persistent buckets¶

In our example, the dynamically created buckets are deleted during cleanup. If you want the buckets to not be ephemeral, you can set reclaimPolicy to Retain.

Extra flags¶

You can pass flags to gcsfuse. They will be forwarded to PersistentVolumeClaim.spec.csi.volumeAttributes.

The following flags are supported (ordered by precedence):

1. PersistentVolumeClaim.metadata.annotations

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     annotations:
       gcs.csi.ofek.dev/gid: "63147"
       gcs.csi.ofek.dev/dir-mode: "0775"
       gcs.csi.ofek.dev/file-mode: "0664"
   

   Option	Type	Description
   gcs.csi.ofek.dev/dir-mode	Octal Integer	Permission bits for directories. (default: 0775)
   gcs.csi.ofek.dev/file-mode	Octal Integer	Permission bits for files. (default: 0664)
   gcs.csi.ofek.dev/gid	Integer	GID owner of all inodes. (default: 63147)
   gcs.csi.ofek.dev/uid	Integer	UID owner of all inodes. (default: -1)
   gcs.csi.ofek.dev/implicit-dirs	Flag	Implicitly define directories based on content.
   gcs.csi.ofek.dev/billing-project	Text	Project to use for billing when accessing requester pays buckets.
   gcs.csi.ofek.dev/limit-bytes-per-sec	Integer	Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
   gcs.csi.ofek.dev/limit-ops-per-sec	Integer	Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
   gcs.csi.ofek.dev/stat-cache-ttl	Text	How long to cache StatObject results and inode attributes e.g. 1h.
   gcs.csi.ofek.dev/type-cache-ttl	Text	How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
   gcs.csi.ofek.dev/fuse-mount-options	Text[]	Additional comma-separated system-specific mount options. Be careful!
   gcs.csi.ofek.dev/max-retry-sleep	Integer	The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

2. StorageClass.parameters

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   parameters:
     gid: "63147"
     dirMode: "0775"
     fileMode: "0664"
   

   Option	Type	Description
   dirMode	Octal Integer	Permission bits for directories. (default: 0775)
   fileMode	Octal Integer	Permission bits for files. (default: 0664)
   gid	Integer	GID owner of all inodes. (default: 63147)
   uid	Integer	UID owner of all inodes. (default: -1)
   implicitDirs	Flag	Implicitly define directories based on content.
   billingProject	Text	Project to use for billing when accessing requester pays buckets.
   limitBytesPerSec	Integer	Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
   limitOpsPerSec	Integer	Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
   statCacheTTL	Text	How long to cache StatObject results and inode attributes e.g. 1h.
   typeCacheTTL	Text	How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
   fuseMountOptions	Text[]	Additional comma-separated system-specific mount options. Be careful!
   maxRetrySleep	Integer	The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

3. StorageClass.mountOptions

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   mountOptions:
     - --gid=63147
     - --dir-mode=0775
     - --file-mode=0664
   

   Option	Type	Description
   dir-mode	Octal Integer	Permission bits for directories. (default: 0775)
   file-mode	Octal Integer	Permission bits for files. (default: 0664)
   gid	Integer	GID owner of all inodes. (default: 63147)
   uid	Integer	UID owner of all inodes. (default: -1)
   implicit-dirs	Flag	Implicitly define directories based on content.
   billing-project	Text	Project to use for billing when accessing requester pays buckets.
   limit-bytes-per-sec	Integer	Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
   limit-ops-per-sec	Integer	Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
   stat-cache-ttl	Text	How long to cache StatObject results and inode attributes e.g. 1h.
   type-cache-ttl	Text	How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
   fuse-mount-option	Text	Additional system-specific mount option. Be careful!
   max-retry-sleep	Integer	The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

4. StorageClass.parameters."csi.storage.k8s.io/provisioner-secret-name"

   Option	Type	Description
   dirMode	Octal Integer	Permission bits for directories, in octal. (default: 0775)
   fileMode	Octal Integer	Permission bits for files, in octal. (default: 0664)
   gid	Integer	GID owner of all inodes. (default: 63147)
   uid	Integer	UID owner of all inodes. (default: -1)
   implicitDirs	Flag	Implicitly define directories based on content.
   billingProject	Text	Project to use for billing when accessing requester pays buckets.
   limitBytesPerSec	Integer	Bandwidth limit for reading data, measured over a 30-second window. The default is -1 (no limit).
   limitOpsPerSec	Integer	Operations per second limit, measured over a 30-second window. The default is 5. Use -1 for no limit.
   statCacheTTL	Text	How long to cache StatObject results and inode attributes e.g. 1h.
   typeCacheTTL	Text	How long to cache name -> file/dir mappings in directory inodes e.g. 1h.
   fuseMountOptions	Text[]	Additional comma-separated system-specific mount options. Be careful!
   maxRetrySleep	Integer	The maximum duration allowed to sleep in a retry loop with exponential backoff for failed requests to GCS backend. Once the backoff duration exceeds this limit, the retry stops. The default is 1 minute. A value of 0 disables retries.

Permission¶

In order to access anything stored in GCS, you will need service accounts with appropriate IAM roles. You will usually assign the role roles/storage.admin.

The easiest way to create service account keys, if you don't yet have any, is to run:

gcloud iam service-accounts list

to find the email of a desired service account, then run:

gcloud iam service-accounts keys create <FILE_NAME>.json --iam-account <EMAIL>

to create a key file.

Mounter¶

The Node Plugin is the component that is actually mounting and serving buckets to pods. If writes are needed, you will usually select roles/storage.objectAdmin scoped to the desired buckets.

Creator¶

The Controller Plugin is the component that is in charge of creating buckets. The service account will need the storage.buckets.create Cloud IAM permission.