MiniKube

This small tutorial wall walk through how to run the Flux Operator on MiniKube, deploying a workflow that requires storage.

External Examples

The following (external) example tutorials are available:

Workflow with Storage

The example here will use the Snakemake workflow that comes alongside the examples! Instead of using data that is already in the base container, we will mount it as a shared volume to demonstrate how that works.

Cluster

Bring up your MiniKube cluster!

$ minikube start

See the developer guide for installing the operator. or the main install guide.

Prepare Data

Let’s first clone the required data to our local machine. We are going to bind this to the pods.

$ git clone --depth 1 https://github.com/snakemake/snakemake-tutorial-data /tmp/workflow

You’ll want to add the Snakefile for your workflow along with a plotting script:

$ wget -O /tmp/workflow/Snakefile https://raw.githubusercontent.com/rse-ops/flux-hpc/main/snakemake/atacseq/Snakefile
$ mkdir -p /tmp/workflow/scripts
$ wget -O /tmp/workflow/scripts/plot-quals.py https://raw.githubusercontent.com/rse-ops/flux-hpc/main/snakemake/atacseq/scripts/plot-quals.py

You should have this structure:

$ tree /tmp/workflow
$ tree /tmp/workflow/
/tmp/workflow/
├── data
│   ├── genome.fa
│   ├── genome.fa.amb
│   ├── genome.fa.ann
│   ├── genome.fa.bwt
│   ├── genome.fa.fai
│   ├── genome.fa.pac
│   ├── genome.fa.sa
│   └── samples
│       ├── A.fastq
│       ├── B.fastq
│       └── C.fastq
├── Dockerfile
├── environment.yaml
├── README.md
├── scripts
│   └── plot-quals.py
└── Snakefile

In a real production setting, you would likely have this data in cloud storage somewhere. We are emulating that for MiniKube with a clone to a temporary directory! Note that to make it easier

Containers

MiniKube can burp sometimes on pulling large images, so let’s issue a command to it to pull the container we need via ssh.

$ minikube ssh docker pull ghcr.io/rse-ops/atacseq:app-latest

Mount Data

The Flux Operator is going to expect to find volumes on the host of a particular storage type. Since we are early in development, we currently (as the default) define a “hostpath” storage type, meaning the operator will expect the path to be present on the node where you are running the job. This means that we need to mount the data on our host into MiniKube (where the cluster is running) with minikube mount.

Note that in our minicluster.yaml we are defining the volume on the host (inside the MiniKube VM) to be at /tmp/data so let’s tell MiniKube to mount our local path there:

minikube ssh -- mkdir -p /tmp/data

minikube mount /tmp/workflow:/tmp/data

Leave that process running in a window and then open another terminal to interact with the cluster. If you want to double check the data is in the MiniKube vm:

$ minikube ssh -- ls /tmp/data
Dockerfile  README.md  Snakefile  data	environment.yaml  scripts

Run Workflow

At this point we want to run our Snakemake workflow, which means applying the custom resource definition for the MiniCluster. Conceptually, this is going to bring up the pods on your MiniCluster nodes, start a flux instance, run your command of interest, and then clean up. Here we can work from the examples/tests/snakemake directory with the minicluster.yaml file:

$ kubectl apply -f minicluster.yaml
minicluster.flux-framework.org/flux-sample created

You can see your pods:

$ kubectl get -n flux-operator pods
NAME                         READY   STATUS      RESTARTS   AGE
flux-sample-0-sk72j          1/1     Running     0          35s
flux-sample-1-6tw4z          1/1     Running     0          35s

Keep watching for when the job is finished! You can see logs (and snakemake output) as follows:

$ kubectl logs -n flux-operator flux-sample-0-pssf6 -f
Snakemake Output in Console
broker.info[0]: quorum-full: quorum->run 0.420945s
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job               count    min threads    max threads
--------------  -------  -------------  -------------
all                   1              1              1
bcftools_call         1              1              1
bwa_map               2              1              1
plot_quals            1              1              1
samtools_index        2              1              1
samtools_sort         2              1              1
total                 9              1              1

Select jobs to execute...

[Fri Feb 10 00:59:17 2023]
rule bwa_map:
    input: data/genome.fa, data/samples/A.fastq
    output: mapped_reads/A.bam
    jobid: 4
    reason: Missing output files: mapped_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp

[Fri Feb 10 00:59:27 2023]
Finished job 4.
1 of 9 steps (11%) done
Select jobs to execute...

[Fri Feb 10 00:59:27 2023]
rule samtools_sort:
    input: mapped_reads/A.bam
    output: sorted_reads/A.bam
    jobid: 3
    reason: Missing output files: sorted_reads/A.bam; Input files updated by another job: mapped_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp

[Fri Feb 10 00:59:37 2023]
Finished job 3.
2 of 9 steps (22%) done
Select jobs to execute...

[Fri Feb 10 00:59:37 2023]
rule bwa_map:
    input: data/genome.fa, data/samples/B.fastq
    output: mapped_reads/B.bam
    jobid: 6
    reason: Missing output files: mapped_reads/B.bam
    wildcards: sample=B
    resources: tmpdir=/tmp

[Fri Feb 10 00:59:47 2023]
Finished job 6.
3 of 9 steps (33%) done
Select jobs to execute...

[Fri Feb 10 00:59:47 2023]
rule samtools_sort:
    input: mapped_reads/B.bam
    output: sorted_reads/B.bam
    jobid: 5
    reason: Missing output files: sorted_reads/B.bam; Input files updated by another job: mapped_reads/B.bam
    wildcards: sample=B
    resources: tmpdir=/tmp

[Fri Feb 10 00:59:57 2023]
Finished job 5.
4 of 9 steps (44%) done
Select jobs to execute...

[Fri Feb 10 00:59:57 2023]
rule samtools_index:
    input: sorted_reads/A.bam
    output: sorted_reads/A.bam.bai
    jobid: 7
    reason: Missing output files: sorted_reads/A.bam.bai; Input files updated by another job: sorted_reads/A.bam
    wildcards: sample=A
    resources: tmpdir=/tmp
...

Keep in mind that when cleanup: true is set, the pods will be cleaned up (with the storage) when everything is finished, so you won’t be able to access the output log. To access it you can either stream it as it comes (e.g, flux-cloud supports this) or disable cleanup, and clean up your volumes yourself after things are finished! When the job is finished, everything should register as completed, and you can clean up like:

$ kubectl delete -f minicluster.yaml
minicluster.flux-framework.org "flux-sample" deleted

And then stop minikube and the bound directory and you are good!

$ minikube stop

But we might not actually need to get the logs from the operator! Since our data directory was bound to our host, we have all the output locally too:

/tmp/workflow/
├── calls
│   └── all.vcf
├── data
│   ├── genome.fa
│   ├── genome.fa.amb
│   ├── genome.fa.ann
│   ├── genome.fa.bwt
│   ├── genome.fa.fai
│   ├── genome.fa.pac
│   ├── genome.fa.sa
│   └── samples
│       ├── A.fastq
│       ├── B.fastq
│       └── C.fastq
├── Dockerfile
├── environment.yaml
├── mapped_reads
│   ├── A.bam
│   └── B.bam
├── plots
│   └── quals.svg
├── README.md
├── scripts
│   └── plot-quals.py
├── Snakefile
└── sorted_reads
    ├── A.bam
    ├── A.bam.bai
    ├── B.bam
    └── B.bam.bai

And snakemake keeps logs in a hidden directory:

$ tree /tmp/workflow/.snakemake/
/tmp/workflow/.snakemake/
├── auxiliary
├── conda
├── conda-archive
├── incomplete
├── locks
├── log
│   └── 2023-02-10T005917.395176.snakemake.log
├── metadata
│   ├── bWFwcGVkX3JlYWRzL0EuYmFt
│   ├── bWFwcGVkX3JlYWRzL0IuYmFt
│   ├── c29ydGVkX3JlYWRzL0EuYmFt
│   ├── c29ydGVkX3JlYWRzL0EuYmFtLmJhaQ==   ├── c29ydGVkX3JlYWRzL0IuYmFt
│   ├── c29ydGVkX3JlYWRzL0IuYmFtLmJhaQ==   ├── cGxvdHMvcXVhbHMuc3Zn
│   └── Y2FsbHMvYWxsLnZjZg==
├── scripts
│   └── tmp5ppvd29r.plot-quals.py
├── shadow
└── singularity

10 directories, 10 files

Wicked! This is actually the first “realish” workflow we’ve run with the Flux Operator and local data (that is public) so this is hugely cool to see!

Development Notes

Debugging Containers

If you load a container in your MiniKube cluster and need to debug it, it’s useful to shell into the container already pulled there:

$ minikube ssh -- docker run -it --entrypoint bash ghcr.io/rse-ops/atacseq:app-latest

This will avoid the redundant pull to your host!

Cleaning up

If you are testing and need to cleanup between runs, if you don’t finish via the operator (cleanup: true) and need to cleanup volumes:

kubectl delete -n flux-operator pods --all --grace-period=0 --force
kubectl delete -n flux-operator pvc --all --grace-period=0 --force
kubectl delete -n flux-operator pv --all --grace-period=0 --force
kubectl delete -n flux-operator jobs --all --grace-period=0 --force
kubectl delete -n flux-operator MiniCluster --all --grace-period=0 --force

That’s technically the set of commands to do all-the-things! However, setting the cleanup: true boolean in the Operator should handle this for you.


Last update: Mar 28, 2024