State

We are experimenting with saving state for the Flux Operator, which can have several different levels of difficulty:

  • Saving state of the jobs queue and metadata, after runs are complete (between two MiniClusters)

  • Saving state of the jobs queue and metadata, pausing a queue in the middle and resuming.

  • Saving state of the jobs queue and metadata, and including filesystem (storage) assets.

These small tutorials will walk through examples of each. The most likely use cases for doing this will be using the Flux Operator Python SDK (since we need to create multiple clusters) in a reasonable way) but for the purposes of explanation, minicluster.yaml files are provided as well. One important note is that since we cannot control the timing of a pod termination, while we can have Flux automatically load a saved archive, for the process to wait for jobs to finish and then dump the archive anew, we rely on issuing a command to the MiniCluster (done by a script or workflow tool). This can likely be improved upon.

Create hostpath volume

Create the hostpath volume and pvc first:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /data
---
apiVersion: v1 
kind: PersistentVolumeClaim
metadata: 
  name: data
spec: 
  accessModes: 
    - ReadWriteMany
  resources: 
    requests: 
      storage: 1Gi

If the above is volumes.yaml:

kubectl apply -f volumes.yaml

Saving Pending Jobs

Pausing scheduling and the queue in a populated queue

This example shows (via the Python SDK) how we can pause and stop a running queue and move the jobs to a new MiniCluster to continue.

Tutorial File

To run this example:

$ python sdk/python/v1alpha2/examples/state-pending-jobs-minicluster.py 

Using this example, we are able to (with slight modification) test:

  • Starting jobs on one cluster and running on another

  • Changing the size of the cluster to be larger

  • Changing the size of the cluster to be smaller

For the different cases, you can adjust the original size (and updated size) in the script by changing the minicluster.size. All cases are successful to pause and resume on the new cluster (regardless of size). Make sure (between runs) that you delete the previous archive so you aren’t loading jobs across all the clusters!

$ minikube ssh -- rm /tmp/data/archive.tar.gz

The commands we are issuing to flux are:

# Stop the queue
flux queue stop

# This should wait for running jobs to finish
flux queue idle

# And then do the dump!
flux dump /state/archive.tar.gz

And this means we will stop and wait for jobs to finish, and then this state is loaded into the next cluster. If you run the example you might want to insert an IPython.embed() before the delete command at the end, and then interactively shell into the new MiniCluster (when the node are running) and then look at jobs:

$ kubectl exec -it flux-sample-0-mbv54 -- source /mnt/flux/flux-view.sh && flux proxy ${fluxsocket} flux jobs -a

And always make sure to clean up your archive at the end!

$ minikube ssh -- rm /data/archive.tar.gz

The next (basic) example goes through the same ideas, but manually for each step so you can learn about what the script is doing.

Basic Saving Jobs and Metadata

Saving state of the jobs queue and metadata, after runs are complete (between two MiniClusters)

This example will walk through creating two MiniClusters - the first running a set of jobs (and finishing) and the second cluster then loading those states. The assets for these files are in examples/state/basic-job-completion. Note that in order for this to work, a shared storage location is required. Since it’s easier to submit multiple jobs interactively, we will do it that way. Here is the first minicluster.yaml to create:

apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
  name: flux-sample
spec:
  # Number of pods to create for MiniCluster
  size: 2

  # Make this interactive so we can launch a bunch of jobs!
  interactive: true

  # Define the archive load/save path here (in our volume mount that persists)
  archive:
    path: /state/archive.tar.gz

  # This is a list because a pod can support multiple containers
  containers:
    - image: rockylinux:9
      volumes:
        data:
          path: /state

Note that interactive mode is set to true - this will start a broker to keep running until we decide we are done. Since we are defining the archive path to /state/archive.tar.gz, this means that before Flux is started, we will load an archive from that path given that it exists with flux resource reload. This is done directly in the entrypoint. To have better control of the reverse sequence - saving the final state to that same location, we will run flux dump to that same archive as an interactive command. Note this is a simple approach that assumes we are OK replacing a previous state with a new one - for more complex workflows (where possibly we need to maintain an original state) we likely will need to do something differently. For the time being, let’s create this first MiniCluster to submit jobs to, and the plan will be that the second MiniCluster can load previous job history. Now let’s create it! You can either walk through this tutorial and learn about each step (continue) below with kubectl apply) or you can run a demo script that runs the commands on your behalf:

$ /bin/bash ./examples/state/basic-job-completion/example.sh
View the Interactive Example Output
$ bash examples/state/basic-job-completion/example.sh  
🌀️ Creating first MiniCluster...
minicluster.flux-framework.org/flux-sample created

🥱️ Sleeping 20 seconds to wait for cluster...Broker pod is flux-sample-0-qwsqw

🤓️ Contents of /tmp/data in MiniKube

✨️ Submitting jobs
ƒQK5i1V
ƒmXbRuh
ƒw92msM
ƒ27R1o9y
ƒ2JTUStw
ƒ2UhyUuD
ƒ2eVnjqH
ƒ2prDixw
ƒ2zV94Cw

🥱️ Waiting for jobs...
Jobs finished...
       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
    ƒ2zV94Cw flux     whoami      S      1      -        - 
    ƒ2prDixw flux     sleep       R      1      1   3.740s flux-sample-0
    ƒ2eVnjqH flux     whoami     CD      1      1   0.042s flux-sample-0
    ƒ2UhyUuD flux     sleep      CD      1      1   4.023s flux-sample-0
    ƒ2JTUStw flux     whoami     CD      1      1   0.045s flux-sample-0
    ƒ27R1o9y flux     sleep      CD      1      1   3.022s flux-sample-0
     ƒw92msM flux     whoami     CD      1      1   0.015s flux-sample-0
     ƒmXbRuh flux     sleep      CD      1      1   2.019s flux-sample-0

🥱️ Wait a minute to be sure we have saved...

🧊️ Current state directory at /var/lib/flux...
total 4332
-rw-r--r-- 1 flux flux  151552 Mar 12 16:36 content.sqlite
-rw-r--r-- 1 flux flux 4120032 Mar 12 16:37 content.sqlite-wal
-rw-r--r-- 1 flux flux    4096 Mar 12 16:36 job-archive.sqlite
-rw-r--r-- 1 flux flux   32768 Mar 12 16:37 job-archive.sqlite-shm
-rw-r--r-- 1 flux flux  123632 Mar 12 16:37 job-archive.sqlite-wal

🧊️ Current archive directory at /state... should be empty, not saved yet
total 0
Cleaning up...
minicluster.flux-framework.org "flux-sample" deleted
total 7
-rw-rw-r-- 1 docker docker 6165 Mar 12 16:38 archive.tar.gz

🌀️ Creating second MiniCluster
minicluster.flux-framework.org/flux-sample created

🥱️ Sleeping a minute to wait for cluster...
Broker pod is flux-sample-0-jpx76

🤓️ Contents of /tmp/data in MiniKube - should be populated with archive from first
total 7
-rw-rw-r-- 1 docker docker 6165 Mar 12 16:38 archive.tar.gz

🤓️ Inspecting state directory in new cluster...
total 1308
-rw-r--r-- 1 flux flux    4096 Mar 12 16:38 content.sqlite
-rw-r--r-- 1 flux flux 1281352 Mar 12 16:38 content.sqlite-wal
-rw-r--r-- 1 flux flux    4096 Mar 12 16:38 job-archive.sqlite
-rw-r--r-- 1 flux flux   32768 Mar 12 16:38 job-archive.sqlite-shm
-rw-r--r-- 1 flux flux   12392 Mar 12 16:38 job-archive.sqlite-wal

😎️ Looking to see if old job history exists...
       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
    ƒ2zV94Cw flux     whoami     CD      1      1   0.037s flux-sample-0
    ƒ2prDixw flux     sleep      CD      1      1   5.023s flux-sample-0
    ƒ2eVnjqH flux     whoami     CD      1      1   0.042s flux-sample-0
    ƒ2UhyUuD flux     sleep      CD      1      1   4.023s flux-sample-0
    ƒ2JTUStw flux     whoami     CD      1      1   0.045s flux-sample-0
    ƒ27R1o9y flux     sleep      CD      1      1   3.022s flux-sample-0
     ƒw92msM flux     whoami     CD      1      1   0.015s flux-sample-0
     ƒmXbRuh flux     sleep      CD      1      1   2.019s flux-sample-0
Cleaning up..
minicluster.flux-framework.org "flux-sample" deleted

You can look at that script to follow the logic and steps.


Last update: Mar 28, 2024