Services

These tutorials will show you how to run a “sidecar” service container (one per Flux node) alongside your flux install, along with a service for the entire cluster (a deployment next to the cluster).

Sidecar Tutorials

Sidecar NGINX Container

Tutorial File

This is one of the simplest examples, implemented as a test, to run a sidecar with NGINX and then curl localhost to get a response from flux. Note that if you are interested in a more complex example that does similar, but also provides a custom nginx.conf and shows how to interact with the flux restful API, see the services/sidecar/nginx example.

apiVersion: flux-framework.org/v1alpha1
kind: MiniCluster
metadata:
  name: flux-sample
  namespace: flux-operator
spec:

  logging:
    quiet: true

  # Number of pods to create for MiniCluster
  size: 2

  # This is a list because a pod can support multiple containers
  containers:
    - image: ghcr.io/flux-framework/flux-restful-api:latest
      runFlux: true
      command: curl -s localhost
      commands:
        pre: apt-get update > /dev/null && apt-get install -y curl > /dev/null
    - image: nginx
      name: nginx
      ports:
        - 80

Create it (after you have the flux-operator namespace):

$ kubectl create -f ./examples/tests/nginx-sidecar-service/minicluster.yaml

See nginx is running:

$ kubectl -n flux-operator logs flux-sample-0-zlpwx -c nginx -f
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/03/18 05:01:31 [notice] 1#1: using the "epoll" event method
2023/03/18 05:01:31 [notice] 1#1: nginx/1.23.3
2023/03/18 05:01:31 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2023/03/18 05:01:31 [notice] 1#1: OS: Linux 5.15.0-67-generic
2023/03/18 05:01:31 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/03/18 05:01:31 [notice] 1#1: start worker processes
2023/03/18 05:01:31 [notice] 1#1: start worker process 29
2023/03/18 05:01:31 [notice] 1#1: start worker process 30
2023/03/18 05:01:31 [notice] 1#1: start worker process 31
...

And then look at the main logs to see the output of curl, run by Flux:

$ kubectl -n flux-operator logs flux-sample-0-zlpwx -c flux-sample -f
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And that’s it! In a real world use case, you’d have some service running alongside an analysis. Clean up:

$ kubectl delete -f ./examples/tests/nginx-sidecar-service/minicluster.yaml

Sidecar Registry with ORAS

Create an interactive MiniCluster with a sidecar registry container

As an example, we will run a local container registry to push/pull artifacts with ORAS. I don’t know why, I just like ORAS :) In all seriousness, you could imagine interesting use cases like needing an API to save and get artifacts for your analysis.

Tutorial File

This example demonstrates bringing up a MiniCluster and then interacting with a service (a registry) to push / pull artifacts. Here is our example custom resource definition:

apiVersion: flux-framework.org/v1alpha1
kind: MiniCluster
metadata:
  name: flux-sample
  namespace: flux-operator
spec:

  # Number of pods to create for MiniCluster
  size: 2

  # Interactive so we can submit commands
  interactive: true

  # This is a list because a pod can support multiple containers
  containers:
    # The container URI to pull (currently needs to be public)
    - image: ghcr.io/flux-framework/flux-restful-api:latest
      runFlux: true
      commands:

        # This is going to install oras for the demo
        pre: |
          apt-get update && apt-get install -y curl
          VERSION="1.0.0-rc.2"
          curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz"
          mkdir -p oras-install/
          tar -zxf oras_${VERSION}_*.tar.gz -C oras-install/
          sudo mv oras-install/oras /usr/local/bin/
          rm -rf oras_${VERSION}_*.tar.gz oras-install/

      # This is our registry we want to run
    - image: ghcr.io/oras-project/registry:latest
      name: registry
      ports:
        - 5000

It’s helpful to pull containers to MiniKube first:

$ minikube ssh docker pull ghcr.io/oras-project/registry:latest
$ minikube ssh docker pull ghcr.io/flux-framework/flux-restful-api:latest

When interactive is true, we tell the Flux broker to start without a command. This means the cluster will remain running until you shutdown Flux or kubectl delete the MiniCluster itself. The container you choose should have the software you are interested in having for each node. Given a running cluster, we can create the namespace and the MiniCluster as follows:

$ kubectl create namespace flux-operator

And apply the MiniCluster CRD:

$ kubectl apply -f examples/services/sidecar/minicluster-registry.yaml

If you are curious, the entrypoint for the service sidecar container is registry serve /etc/docker/registry/config.yml to start the registry. Since it’s not a flux runner, not providing an entrypoint means we use the container’s default entrypoint. We can then wait for our pods to be running

$ kubectl get -n flux-operator pods
NAME                         READY   STATUS      RESTARTS   AGE
flux-sample-0-p5xls          1/1     Running     0          7s
flux-sample-1-nmtt7          1/1     Running     0          7s

To see logs, since we have 2 containers per pod, you can either leave out the pod (and get the first or default) or specify a container with -c:

$ kubectl logs -n flux-operator flux-sample-0-d5jbb -c registry
$ kubectl logs -n flux-operator flux-sample-0-d5jbb -c flux-sample
$ kubectl logs -n flux-operator flux-sample-0-d5jbb

And then shell into the broker pod, index 0, which is “flux-sample”

$ kubectl exec -it  -n flux-operator flux-sample-0-d5jbb -c flux-sample -- bash

Let’s first make and push an artifact. First, just using oras natively (no flux)

cd /tmp

# Assume we would be running from inside the flux instance
sudo -u flux echo "hello dinosaur" > artifact.txt

And push! The registry, by way of being a container in the same pod, is on port 5000:

At this point, remember the broker is running, and we need to connect to it. We do this via flux proxy and targeting the socket, which is a local reference at /run/flux/local:

# Connect to the flux socket at /run/flux/local as the flux instance owner "flux"
$ sudo -u flux flux proxy local:///run/flux/local oras push localhost:5000/dinosaur/artifact:v1 \
   --artifact-type application/vnd.acme.rocket.config \
   ./artifact.txt
Uploading 07f469745bff artifact.txt
Uploaded  07f469745bff artifact.txt
Pushed [registry] localhost:5000/dinosaur/artifact:v1
Digest: sha256:3a6cb1d1d1b1d80d4c4de6abc66a6c9b4f7fef0b117f87be87fea9b725053ead

Now try pulling, deleting the original first, and again without flux:

rm -f artifact.txt
sudo -u flux flux proxy local:///run/flux/local oras pull localhost:5000/dinosaur/artifact:v1
cat artifact.txt
hello dinosaur

We did this under the broker (and flux user) assuming your actual use case will be running in the Flux instance. Feel free to play with oras outside of that context! When you are done, exit from the instance, and exit from the pod, and then delete the MiniCluster.

$ kubectl delete -f examples/services/sidecar/minicluster-registry.yaml

That’s it. Please do something more useful than my terrible example.

Service Containers Alongside the Cluster

Registry Service

Tutorial File

Unlike the example above, it’s more reasonable that you would want a single registry that your pods can access. E.g., perhaps you use it like a pull-through cache - first pulling to this service node (or pushing from another pod) and then having your worker pods pull from there. Let’s do that now. If you are using MiniKube, remember to pull large containers first. Create the namespace:

$ kubectl create namespace flux-operator

And apply the MiniCluster CRD:

$ kubectl apply -f examples/services/single/minicluster-registry.yaml

You’ll see a services pod! The current design deploys one pod to share your services, and the services share the same cluster networking space.

$ kubectl get -n flux-operator pods
NAME                         READY   STATUS      RESTARTS   AGE
flux-sample-0-4wt26          1/1     Running     0          38s
flux-sample-1-sr5zx          1/1     Running     0          38s
flux-sample-services         1/1     Running     0          38s

And then shell into the broker pod, index 0, which is “flux-sample”

$ kubectl exec -it  -n flux-operator flux-sample-0-d5jbb -- bash

The registry hostname should be in the environment (it’s defined in the CRD):

# echo $REGISTRY 
flux-sample-services.flux-service.flux-operator.svc.cluster.local

Both oras and Singularity are installed. Conceptually, we should be able to pull a container with Singularity, and push it to the registry with oras. This is a relatively small container and should be quick:

$ singularity pull docker://vanessa/salad

Give it a run!

$ singularity run salad_latest.sif 
 In Go an array is a slice. Utensil discrimination!  

          _________________  .========
         [_________________>< :======
                             '======== 

Now let’s push to the oras registry

$ oras push $REGISTRY:5000/vanessa/salad:latest --artifact-type appliciation/vnd.sylabs.sif.layer.tar ./salad_latest.sif  --plain-http

Great! Now you could, theoretically, push a single SIF to your registry (as a local cache) and have the other nodes pull it! Here is an example, shelling in to the second worker:

$ oras pull $REGISTRY:5000/vanessa/salad:latest --plain-http

Super cool! We will have more examples in the examples folder of how this can be used for workflow containers.

Development Notes

I did some digging into the logic, and found that the underlying submission was a flux submit -> flux exec to start a celery worker:

$ flux alloc -N 2 --exclusive --job-name=merlin flux exec `which /bin/bash` -c "celery -A merlin worker -l INFO --concurrency 1 --prefetch-multiplier 1 -Ofair -Q \'[merlin]_flux_par\'"

I think this should be changed to:

$ flux alloc -N 2 --exclusive --job-name=merlin /bin/bash -c "celery -A merlin worker -l INFO --concurrency 1 --prefetch-multiplier 1 -Ofair -Q \'[merlin]_flux_par\'"
  • I don’t think we need flux exec

  • Why would there be more than one /bin/bash?

I don’t fully understand the relationship between the celery queue and Flux - I think Flux should be used to submit jobs directly to, as opposed to just using it to start a celery working. It also seems like there is one too many layers of complexity. If we have a Flux queue why do we also need a celery queue?


Last update: Sep 10, 2023