Submit Jobs¶
Given a running Kubernetes cluster, you have several modes for interaction, or submitting jobs:
Submit a MiniCluster custom resource definition (CRD) with a command to run one job. (#ref)
Submit in “interactive” mode, starting a Flux broker and instance, and interacting with your Flux instance. (#ref)
Submit jobs interacting with Flux via ssh-ing to the pod (advanced) (#ref)
Submit batch jobs
The reason we have many options is because we are still testing and understanding what use cases are best matched for each! Each of the strategies above will be discussed here.
1. Single Job MiniCluster CRD¶
a. Submit a MiniCluster yaml via kubectl¶
When you define a command in a MiniCluster CRD, this will submit and run one job using that command, and clean up.
You can find many examples for doing this under our tests
directory. For any file there (e.g., lammps) once you have a Kubernetes cluster with the operator installed and
the flux-operator namespace created, you can do:
# Create the MiniCluster
$ kubectl apply -f examples/tests/lammps/minicluster.yaml
# See the status of the pods
$ kubectl get pods -n flux-operator pods
# View output for lammps (replace with actual pod id)
$ kubectl logs -n flux-operator lammps-0-xxxxx
b. Submit a MiniCluster CRD via the Python SDK¶
We have early in development a python SDK that interacts directly with the Kubernetes API and defines the MiniCluster CRD programmatically. You can see an early directory of examples. Note that using this method you need to be pedantic - we use the openapi generator and not all defaults get correctly carried through. As an example, the default deadline in seconds is a large number but it gets passed as 0, so if you don’t define it, the Job will never start.
2. Submit in interactive mode¶
We have an interactive mode
that means starting a single-user MiniCluster and launching a command.
We start the broker, and essentially wait for you to interact with it. This can mean issuing commands from
the terminal with kubectl exec (as shown in the tutorial above) or writing a Python script using the Flux Operator
Python SDK to proggrammatically interact. This is a really nice solution if you want to automate something.
3. Submit jobs directly to Flux via ssh¶
You can also submit jobs interacting with Flux via ssh-ing to the pod! This is considered advanced, and is good for debugging. As you did before, get your pod listing:
$ kubectl -n flux-operator pods
And then shell in
$ kubectl exec --stdin --tty -n flux-operator ${brokerPod} -- /bin/bash
Note that if you have more than one container, you’ll need to include it’s name with -c <container>.
Also note that if you need to load any custom environments (e.g., something you’d define in the preCommand for a container)
you’ll likely need to do that when you shell in.
4. Submit batch jobs¶
If you are submitting many jobs, you are better off providing them to flux at once as a batch submission.
This way, we won’t stress any Kubernetes APIs to submit multiple. To do this, set batch to the batch
script content. The content is written verbatim to a file and run with flux batch — you provide the
flux submit lines yourself (they are not wrapped or numbered for you), which gives you full control over
flags, output paths, and #flux: directives. Here is the “containers” section of a MiniCluster crd:
containers:
- image: rockylinux:9
# The batch content is written verbatim and run with flux batch.
batch: |
#!/bin/bash
flux submit --output=/tmp/fluxout/job-1.out echo hello world 1
flux submit --output=/tmp/fluxout/job-2.out echo hello world 2
flux submit --output=/tmp/fluxout/job-3.out echo hello world 3
flux queue idle
Because you write the flux submit lines yourself, you control exactly where each job’s output goes
(via --output/--error) and any per-job flags. Set the output to a shared volume so all pods can write
to it. If you can’t use the filesystem for saving output, it’s recommended to have some other service used
in your jobs to send output.