Multi-tenancy¶

“Multi-tenancy” means having multiple tenants, or users, in a cluster! We can accomplish this fairly by creating multiple users for our MiniCluster. This small tutorial will walk through creating an example with MiniKube.

1. Create Cluster¶

Bring up your MiniKube cluster.

$ minikube start

Then choose your step of choice to install the operator

2. Custom Resource Definition¶

You’ll want a custom resource definition that turns on multi-user mode, and defines the users you want to create, in addition to root (sets up the pods) and flux (owns the main flux instance). We have a minicluster.yaml provided as an example:

apiVersion: flux-framework.org/v1alpha1
kind: MiniCluster
metadata:
  name: flux-sample
  namespace: flux-operator
spec:
  # suppress all output except for test run
  logging:
    quiet: false

  # Number of pods to create for MiniCluster
  size: 4

  # Define one or more users for your cluster
  # These users will be created, and user/pass required for auth
  # passwords are optional, if not provided will be generated
  users:
    - name: peenut
    - name: squidward
    - name: avocadosaurus

  containers:
    - image: ghcr.io/flux-framework/flux-restful-api:latest
    # Multi-user mode should not have a command

Note that we don’t provide a command to run the Flux Restful API. It’s recommended to pull the container to MiniKube first:

$ minikube ssh docker pull ghcr.io/flux-framework/flux-restful-api:latest

Then, create the flux-operator namespace and MiniCluster:

$ kubectl create namespace flux-operator
$ kubectl create -f examples/flux-restful/minicluster-multi-tenant.yaml

3. View Logs¶

And then in the logs, we can see the different blocks of creating users (and passwords):

# get pods
$ kubectl get -n flux-operator pods

# get logs for broker pod
$ kubectl logs -n flux-operator flux-sample-0-fz64b

INFO:flux-restful:Initial data created
Adding 'peenut' with password '0ff24df5'
.
INFO:flux-restful:User peenut has been created.
Adding 'squidward' with password '393418c3'
.
INFO:flux-restful:User squidward has been created.
Adding 'avocadosaurus' with password '4ae21fc5'
...

And the Flux Restful Server is ready to go! Note that a secret is generated by the server that would be required for you to use the RESTful API (to encode payloads). If you want to define your own secret, add it to the fluxRestful directive in the MiniCluster yaml file, e.g.,

fluxRestful:
  secretKey: thisisnotsosecretakey

For the web interface, you can use basic auth, and since we aren’t issuing tokens, there is no need for the secret key.

broker.info[0]: quorum-full: quorum->run 0.461457s
🍓 Require auth: True
🍓   Secret key ************************************
🍓    Flux user: ****
🍓   Flux token: ************************************
INFO:     Started server process [217]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)

Make sure that you grab a user name and password from the terminal.

4. Expose Web Interface¶

Next we will want to expose the service.

$ kubectl port-forward -n flux-operator flux-sample-0-zdhkp 5000:5000

And open your browser to http://localhost:5000. You should be able to login with a user and password, which are required when you want to submit or see the jobs table. In more specific terms, when you click on the top left hamburger to “submit” you’ll be prompted for a user name and password (as shown in the logs above). You can also get prompted in the “API” link in the top right (and test the other endpoints).

If you want more programmattic access, look at our Python SDK examples for “port forwarding” that will do the same above, but not require the manual extra work (all is done in Python).

5. Admin Actions¶

Adding a User¶

It is possible to dynamically add a user! You will need to create them inside the container. Shell inside to interact directly:

$ kubectl exec -it -n flux-operator flux-sample-0-bnwnt -- bash

The flux-restful-api is under /flux-restful-api

Add a new user:

$ python3 /flux-restful-api/app/db/init_db.py add-user newuser newuser

INFO:flux-restful:User newuser has been created.

Viewing Instance Jobs¶

If you shell into the broker pod (as shown above) you can connect to the broker’s socket, the FLUX_URI as follows:

$ export FLUX_URI=local:///run/flux/local
$ sudo -u flux flux proxy $FLUX_URI

$ flux jobs -a

       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
   ƒ7EaBwt4w flux     hostname   CD      1      1   0.019s flux-sample-3
   ƒ6X8ua7Fu flux     pwd        CD      1      1   0.042s flux-sample-3
   ƒ6MNGWc9m flux     hostname   CD      1      1   0.047s flux-sample-3
   ƒ5u2QLkgP flux     hostname   CD      1      1   0.075s flux-sample-1

You’ll be the flux user that owns the instance:

$ whoami
flux

We are currently testing different ways to (aside from the id) ensure ownership of a job. This is why multi-user mode is “use at your own risk” for the time being - both jobs and the working space is largely shared.

Frequently Asked Questions¶

What are the different types of multi-tenant use cases?¶

1. A single multi-tenant MiniCluster¶

In the example above, we define our users in advance and bring up a single MiniCluster, meaning one MiniCluster custom resource definition on one Kubernetes cluster, that allows multiple users to submit jobs authenticating with basic (UI) or OAauth-based authentication (RESTFul) to authenticate to the Flux RESTFul API database. This is done via the flux-restful-api, which also has a command line client to submit jobs from the command line, where you export your credentials and secret in the environment instead. The server secret is used to encode the payloads.

This use case is recommended if you have a small group that would be able to submit jobs to the same Flux Instance on a set of container bases with the same needs, and where you largely trust one another. We do not yet support multi-tenant storage (we are still working on simple storage) so the filesystem is largely shared, but still has expected user permissions (e.g., “peenut” has a home). The cluster is persistent, meaning it stays up until the Kubernetes cluster owner destroys it.

2. Multiple MiniClusters for multi-tenancy¶

For this use case, you create a single-user MiniCluster, meaning it’s owned by only one user, and that user has total control of the Flux Instance, to submit their jobs to. Akin to the previous example, it is persistent and can be brought down when the user is finished. Unlike the first, if you have several (separate) MiniClusters running on a Kubernetes cluster, the Flux instances do not communicate. This use case is ideal for users that currently need storage or need a custom base container for the MiniCluster.

What are the non multi-tenant use cases?¶

Bringing up an ephermal MiniCluster to run one job is not considered a multi-user use case, and we don’t even care about creating user accounts. The MiniCluster comes up, runs the job, and the job completes and everything cleans up.

If you have any questions please let us know!