CSI for Cloud Storage

MiniCluster YAML

This basic tutorial will walk through creating a MiniCluster to run a Snakemake workflow! We will use a “Container Storage Interface” (CSI) to connect to Snakemake assets in Google Cloud Storage. You should have already setup your workspace, including preparing the Snakemake data in Google Storage.

Create Cluster

Let’s use gcloud to create a cluster, and we are purposefully going to choose a very small node type to test. Note that I choose us-central1-a because it tends to be cheaper (and closer to me). First, here is our project for easy access:


Then create your cluster as follows:

$ gcloud container clusters create flux-cluster --project $GOOGLE_PROJECT \
    --zone us-central1-a --machine-type n1-standard-1 \
    --num-nodes=4 --enable-network-policy --tags=flux-cluster --enable-intra-node-visibility

As an example to use core-dns:

$ gcloud container clusters create flux-operator --cluster-dns=clouddns \
    --cluster-dns-scope=cluster --region=us-central1-a --project $GOOGLE_PROJECT \
    --machine-type n1-standard-2 --num-nodes=4 --enable-network-policy --tags=flux-cluster --enable-intra-node-visibility

If you need a particular Kubernetes version:

$ gcloud container clusters create flux-cluster --project $GOOGLE_PROJECT \
    --zone us-central1-a --cluster-version 1.23 --machine-type n1-standard-1 \
    --num-nodes=4 --enable-network-policy --tags=flux-cluster --enable-intra-node-visibility

Note that not all of the flags above might be necessary - I did a lot of testing to get this working and didn’t go back and try removing things after the fact! If you want to use cloud dns instead (after enabling it)

$ gcloud beta container clusters create flux-cluster --project $GOOGLE_PROJECT \
    --zone us-central1-a --cluster-version 1.23 --machine-type n1-standard-1 \
    --num-nodes=4 --enable-network-policy --tags=flux-cluster --enable-intra-node-visibility \
    --cluster-dns=clouddns \

In your Google cloud interface, you should be able to see the cluster! Note this might take a few minutes.


I also chose a tiny size (nodes and instances) anticipating having it up longer to figure things out.

Get Credentials

Next we need to ensure that we can issue commands to our cluster with kubectl. To get credentials, in the view shown above, select the cluster and click “connect.” Doing so will show you the correct statement to run to configure command-line access, which probably looks something like this:

$ gcloud container clusters get-credentials flux-cluster --zone us-central1-a --project $GOOGLE_PROJECT
Fetching cluster endpoint and auth data.
kubeconfig entry generated for flux-cluster.

Finally, use cloud IAM to ensure you can create roles, etc.

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account)
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created

At this point you should be able to get your nodes:

$ kubectl get nodes
NAME                                            STATUS   ROLES    AGE     VERSION
gke-flux-cluster-default-pool-f103d9d8-379m   Ready    <none>   3m41s   v1.23.14-gke.1800
gke-flux-cluster-default-pool-f103d9d8-3wf9   Ready    <none>   3m42s   v1.23.14-gke.1800
gke-flux-cluster-default-pool-f103d9d8-c174   Ready    <none>   3m42s   v1.23.14-gke.1800
gke-flux-cluster-default-pool-f103d9d8-zz1q   Ready    <none>   3m42s   v1.23.14-gke.1800

Deploy Operator

To deploy the Flux Operator, choose one of the options here to deploy the operator. Whether you apply a yaml file, use flux-cloud or clone the repository and make deploy you will see the operator install to the operator-system namespace.

For a quick “production deploy” from development, the Makefile has a directive that will build and push a test tag (you’ll need to edit DEVIMG to be one you can push to) and then generate a yaml file targeting that image, e.g.,

$ make test-deploy
$ kubectl apply -f examples/dist/flux-operator-dev.yaml

or the production version:

$ kubectl apply -f examples/dist/flux-operator.yaml
clusterrole.rbac.authorization.k8s.io/operator-manager-role created
clusterrole.rbac.authorization.k8s.io/operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/operator-proxy-rolebinding created
configmap/operator-manager-config created
service/operator-controller-manager-metrics-service created
deployment.apps/operator-controller-manager created

Ensure the operator-system namespace was created:

$ kubectl get namespace
NAME              STATUS   AGE
default           Active   6m39s
kube-node-lease   Active   6m42s
kube-public       Active   6m42s
kube-system       Active   6m42s
operator-system   Active   39s
$ kubectl describe namespace operator-system
Name:         operator-system
Labels:       control-plane=controller-manager
Annotations:  <none>
Status:       Active

Resource Quotas
  Name:                              gke-resource-quotas
  Resource                           Used  Hard
  --------                           ---   ---
  count/ingresses.extensions         0     100
  count/ingresses.networking.k8s.io  0     100
  count/jobs.batch                   0     5k
  pods                               1     1500
  services                           1     500

No LimitRange resource.

And you can find the name of the operator pod as follows:

$ kubectl get pod --all-namespaces
operator-system   operator-controller-manager-56b5bcf9fd-m8wg4               2/2     Running   0          73s

Create Flux Operator namespace

Make your namespace for the flux-operator custom resource definition (CRD), which is the yaml file above that generates the MiniCluster:

$ kubectl create namespace flux-operator

Akin to how we created a local volume, we can do something similar, but instead of pointing the Flux Operator to a volume on the host (e.g., in MiniKube) we are going to point it to a storage bucket with our data. For this tutorial, we will use the csi-gcs driver to connect a cloud storage bucket to our cluster.

Install the CSI

There are many drivers for kubernetes, and we will use this one that requires a stateful set and daemon set to work. Let’s install those first.

$ kubectl apply -k "github.com/ofek/csi-gcs/deploy/overlays/stable?ref=v0.9.0"
$ kubectl get CSIDriver,daemonsets,pods -n kube-system | grep csi

And to debug:

$ kubectl logs -l app=csi-gcs -c csi-gcs -n kube-system

As you are working, if the mounts seem to work but you don’t see files, keep in mind you need to be aware of implicit directories. The operator will do a mkdir -p on the working directory (and this will show content there) but if you don’t see content and expect to, you either need to interact in this way or set this flag as an annotation in your minicluster.yaml.

Storage Class

We can then create our storage class, this file is provided in examples/storage/google/storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
  name: csi-gcs
provisioner: gcs.csi.ofek.dev
$ kubectl apply -f examples/storage/google/gcs-csi/storageclass.yaml

Snakemake MiniCluster

The operator can now be run, telling it to use this storage class (named csi-gcs) - we provide an example Minicluster to run this snakemake tutorial to test this out, and note that if you want to debug you can change the command first to be an ls to see what data is there! Before you apply this file, make sure that the annotations for storage match your Google project, zone, etc.

  # Make this kind of persistent volume and claim available to pods
  # This is a type of storage that will use Google Storage
      class: csi-gcs
      path: /tmp/data
      secret: csi-gcs-secret
        gcs.csi.ofek.dev/location: us-central1
        gcs.csi.ofek.dev/project-id: my-project
        gcs.csi.ofek.dev/bucket: flux-operator-storage

Also note that we are setting the commands: -> runFluxAsRoot to true. This isn’t ideal, but it was the only way I could get the storage to both be seen and have permission to write there. Let’s create the job!

$ kubectl apply -f examples/storage/google/gcs-csi/minicluster.yaml

Wait to see the certificate generator pod come up, complete, and the worker pods (that depend on it) will finish creation and then come up:

$ kubectl get pods -n flux-operator

And I like to get the main pod and stream the output so I don’t miss it:

# Stream to your terminal
$ kubectl logs -n flux-operator flux-sample-0-fj6td -f

# Stream to file
$ kubectl logs -n flux-operator flux-sample-0-fj6td -f > output.txt
Snakemake output from Log
flux user identifiers:
uid=1000(flux) gid=1000(flux) groups=1000(flux)

As Flux prefix for flux commands: sudo -E PYTHONPATH=/usr/lib/flux/python3.8:/code -E PATH=/opt/micromamba/envs/snakemake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -E HOME=/home/flux

👋 Hello, I'm flux-sample-0
The main host is flux-sample-0
The working directory is /workflow/snakemake-workflow, contents include:
Dockerfile  README.md  Snakefile  data  environment.yaml  scripts
End of file listing, if you see nothing above there are no files.
flux R encode --hosts=flux-sample-[0-1]

📦 Resources
{"version": 1, "execution": {"R_lite": [{"rank": "0-1", "children": {"core": "0"}}], "starttime": 0.0, "expiration": 0.0, "nodelist": ["flux-sample-[0-1]"]}}

🐸 Diagnostics: false

🦊 Independent Minister of Privilege
allowed-users = [ "flux", "root" ]
allowed-shells = [ "/usr/libexec/flux/flux-shell" ]

🐸 Broker Configuration
# Flux needs to know the path to the IMP executable
imp = "/usr/libexec/flux/flux-imp"

allow-guest-user = true
allow-root-owner = true

# Point to resource definition generated with flux-R(1).
path = "/etc/flux/system/R"

curve_cert = "/etc/curve/curve.cert"
default_port = 8050
default_bind = "tcp://eth0:%p"
default_connect = "tcp://%h.flux-service.flux-operator.svc.cluster.local:%p"
hosts = [
        { host="flux-sample-[0-1]"},
#   ****  Generated on 2023-02-17 21:05:10 by CZMQ  ****
#   ZeroMQ CURVE **Secret** Certificate
#   DO NOT PROVIDE THIS FILE TO OTHER USERS nor change its permissions.

    name = "flux-sample-cert-generator"
    hostname = "flux-sample-0"
    public-key = ".!?zfo10Ew)m=+J:j^zehs&{Ayy#BGSV0Eets5Ne"
    secret-key = "vmk%8&dl7ICTfgx?*+0wgPb=@kFA>djvZU-Sl[T6"

🔒️ Working directory permissions:
total 3
-rw-rw-r-- 1 root 63147  233 Feb 10 22:57 Dockerfile
-rw-rw-r-- 1 root 63147  347 Feb 10 22:57 README.md
-rw-rw-r-- 1 root 63147 1144 Feb 10 22:57 Snakefile
drwxrwxr-x 1 root 63147    0 Feb 17 21:05 data
-rw-rw-r-- 1 root 63147  203 Feb 10 22:57 environment.yaml
drwxrwxr-x 1 root 63147    0 Feb 17 21:05 scripts

✨ Curve certificate generated by helper pod
#   ****  Generated on 2023-02-17 21:05:10 by CZMQ  ****
#   ZeroMQ CURVE **Secret** Certificate
#   DO NOT PROVIDE THIS FILE TO OTHER USERS nor change its permissions.

    name = "flux-sample-cert-generator"
    hostname = "flux-sample-0"
    public-key = ".!?zfo10Ew)m=+J:j^zehs&{Ayy#BGSV0Eets5Ne"
    secret-key = "vmk%8&dl7ICTfgx?*+0wgPb=@kFA>djvZU-Sl[T6"
Extra arguments are: snakemake --cores 1 --flux

🌀 flux start -o --config /etc/flux/config -Scron.directory=/etc/flux/system/cron.d   -Stbon.fanout=256   -Srundir=/run/flux   -Sstatedir=/var/lib/flux   -Slocal-uri=local:///run/flux/local   -Slog-stderr-level=6    -Slog-stderr-mode=local flux submit  -n 1 --quiet  --watch snakemake --cores 1 --flux
broker.info[0]: start: none->join 10.6371ms
broker.info[0]: parent-none: join->init 0.051043ms
cron.info[0]: synchronizing cron tasks to event heartbeat.pulse
job-manager.info[0]: restart: 0 jobs
job-manager.info[0]: restart: 0 running jobs
job-manager.info[0]: restart: checkpoint.job-manager not found
broker.info[0]: rc1.0: running /etc/flux/rc1.d/01-sched-fluxion
sched-fluxion-resource.warning[0]: create_reader: allowlist unsupported
sched-fluxion-resource.info[0]: populate_resource_db: loaded resources from core's resource.acquire
broker.info[0]: rc1.0: running /etc/flux/rc1.d/02-cron
broker.info[0]: rc1.0: /etc/flux/rc1 Exited (rc=0) 0.9s
broker.info[0]: rc1-success: init->quorum 0.922753s
broker.info[0]: online: flux-sample-0 (ranks 0)
broker.info[0]: online: flux-sample-[0-1] (ranks 0-1)
broker.info[0]: quorum-full: quorum->run 0.427513s
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job               count    min threads    max threads
--------------  -------  -------------  -------------
all                   1              1              1
bcftools_call         1              1              1
bwa_map               2              1              1
plot_quals            1              1              1
samtools_index        2              1              1
samtools_sort         2              1              1
total                 9              1              1

Select jobs to execute...

[Fri Feb 17 21:05:22 2023]
rule bwa_map:
    input: data/genome.fa, data/samples/A.fastq
    output: mapped_reads/A.bam
    jobid: 4
    reason: Missing output files: mapped_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp

[Fri Feb 17 21:05:42 2023]
Finished job 4.
1 of 9 steps (11%) done
Select jobs to execute...

[Fri Feb 17 21:05:43 2023]
rule samtools_sort:
    input: mapped_reads/A.bam
    output: sorted_reads/A.bam
    jobid: 3
    reason: Missing output files: sorted_reads/A.bam; Input files updated by another job: mapped_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp

[Fri Feb 17 21:05:52 2023]
Finished job 3.
2 of 9 steps (22%) done
Select jobs to execute...

[Fri Feb 17 21:05:52 2023]
rule bwa_map:
    input: data/genome.fa, data/samples/B.fastq
    output: mapped_reads/B.bam
    jobid: 6
    reason: Missing output files: mapped_reads/B.bam
    wildcards: sample=B
    resources: tmpdir=/tmp

[Fri Feb 17 21:06:02 2023]
Finished job 6.
3 of 9 steps (33%) done
Select jobs to execute...

[Fri Feb 17 21:06:02 2023]
rule samtools_sort:
    input: mapped_reads/B.bam
    output: sorted_reads/B.bam
    jobid: 5
    reason: Missing output files: sorted_reads/B.bam; Input files updated by another job: mapped_reads/B.bam
    wildcards: sample=B
    resources: tmpdir=/tmp

[Fri Feb 17 21:06:12 2023]
Finished job 5.
4 of 9 steps (44%) done
Select jobs to execute...

[Fri Feb 17 21:06:13 2023]
rule samtools_index:
    input: sorted_reads/A.bam
    output: sorted_reads/A.bam.bai
    jobid: 7
    reason: Missing output files: sorted_reads/A.bam.bai; Input files updated by another job: sorted_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp

[Fri Feb 17 21:06:22 2023]
Finished job 7.
5 of 9 steps (56%) done
Select jobs to execute...

[Fri Feb 17 21:06:22 2023]
rule samtools_index:
    input: sorted_reads/B.bam
    output: sorted_reads/B.bam.bai
    jobid: 8
    reason: Missing output files: sorted_reads/B.bam.bai; Input files updated by another job: sorted_reads/B.bam
    wildcards: sample=B
    resources: tmpdir=/tmp

[Fri Feb 17 21:06:32 2023]
Finished job 8.
6 of 9 steps (67%) done
Select jobs to execute...

[Fri Feb 17 21:06:32 2023]
rule bcftools_call:
    input: data/genome.fa, sorted_reads/A.bam, sorted_reads/B.bam, sorted_reads/A.bam.bai, sorted_reads/B.bam.bai
    output: calls/all.vcf
    jobid: 2
    reason: Missing output files: calls/all.vcf; Input files updated by another job: sorted_reads/A.bam, sorted_reads/B.bam, sorted_reads/A.bam.bai, sorted_reads/B.bam.bai
    resources: tmpdir=/tmp

[Fri Feb 17 21:06:43 2023]
Finished job 2.
7 of 9 steps (78%) done
Select jobs to execute...

[Fri Feb 17 21:06:43 2023]
rule plot_quals:
    input: calls/all.vcf
    output: plots/quals.svg
    jobid: 1
    reason: Missing output files: plots/quals.svg; Input files updated by another job: calls/all.vcf
    resources: tmpdir=/tmp

[Fri Feb 17 21:07:02 2023]
Finished job 1.
8 of 9 steps (89%) done
Select jobs to execute...

[Fri Feb 17 21:07:02 2023]
localrule all:
    input: plots/quals.svg
    jobid: 0
    reason: Input files updated by another job: plots/quals.svg
    resources: tmpdir=/tmp

[Fri Feb 17 21:07:02 2023]
Finished job 0.
9 of 9 steps (100%) done
Complete log: .snakemake/log/2023-02-17T210519.114509.snakemake.log
broker.info[0]: rc2.0: flux submit -n 1 --quiet --watch snakemake --cores 1 --flux Exited (rc=0) 115.1s
broker.info[0]: rc2-success: run->cleanup 1.91796m
broker.info[0]: cleanup.0: flux queue stop --quiet --all --nocheckpoint Exited (rc=0) 0.0s
broker.info[0]: cleanup.1: flux job cancelall --user=all --quiet -f --states RUN Exited (rc=0) 0.0s
broker.info[0]: cleanup.2: flux queue idle --quiet Exited (rc=0) 0.0s
broker.info[0]: cleanup-success: cleanup->shutdown 49.7479ms
broker.info[0]: children-complete: shutdown->finalize 93.2169ms
broker.info[0]: rc3.0: running /etc/flux/rc3.d/01-sched-fluxion
broker.info[0]: online: flux-sample-0 (ranks 0)
broker.info[0]: rc3.0: /etc/flux/rc3 Exited (rc=0) 0.2s
broker.info[0]: rc3-success: finalize->goodbye 0.246792s
broker.info[0]: goodbye: goodbye->exit 0.080512ms

After it finishes the job will cleanup (unless you’ve set cleanup: false) in your minicluster.yaml. If you check Google storage, you’ll see the output of the run (e.g., data in mapped_reads / mapped_samples /plots) and a “.snakemake” hidden directory with logs.


Note that when this is deployed in a larger scale or a more productions sense, you’ll want to account for the details (e.g., resource limits) of the storage. This is basically done by (instead of applying the GitHub provided files) editing them via kustomize. Note that I haven’t tried this yet.

Note: we would like to get this working without requiring running the workflow as root, but it hasn’t been figured out yet! If you have insight, please comment on this issue.

Clean up

Whatever tutorial you choose, don’t forget to clean up at the end! You can optionally undeploy the operator (this is again at the root of the operator repository clone)

$ make undeploy

Or the file you used to deploy it:

$ kubectl delete -f examples/dist/flux-operator.yaml
$ kubectl delete -f examples/dist/flux-operator-dev.yaml

And then to delete the cluster with gcloud:

$ gcloud container clusters delete --zone us-central1-a flux-cluster

I like to check in the cloud console to ensure that it was actually deleted.

Last update: Sep 10, 2023