Debugging¶
These sections include common errors that you might run into developing the operator.
Errors¶
Pull Errors¶
I had a problem where the containers I wanted to deploy seemed to time-out during pull. In the pod logs I would see:
Context deadline exceeded
In this case, if you manually pull first with ssh (and then don’t ask to re-pull) you should be able to get around this.
$ minikube ssh docker pull ghcr.io/rse-ops/lammps:flux-sched-focal-v0.24.0
Service Already on Port 8080¶
One time I messed something up and my metrics server was still running (and I couldn’t start again) and I killed it like this:
$ kill $(lsof -t -i:8080)
CRD should be installed¶
If you see something like:
1.6605195805812113e+09 ERROR controller-runtime.source if kind is a CRD, it should be installed before calling Start {"kind": "Flux.flux-framework.org", "error": "no matches for kind \"Flux\" in version \"flux-framework.org/v1alpha1\""}
You need to remove the previous kustomize and install the CRD again:
$ rm bin/kustomize
Configs not taking¶
If your resource (config/samples) don’t seem to be taking, remember you need to apply them for changes to take effect:
$ bin/kustomize build config/samples | kubectl apply -f -
Fields not showing up¶
Since we are providing the instance as a reference (&instance
) to get the fields from that you sometimes need to do:
(*instance).Spec.Field
Otherwise it can show up as an empty string!
Yaml¶
Since many configs are created in the operator, a strategy to debug what you are creating is to write them out verbatim in a yaml
folder alongside the repository. The “describe” commands of kubectl are useful for debugging specific Kubernetes objects:
# Why didn't my statefulset create?
kubectl describe -n flux-operator statefulset
Troubleshooting¶
Here is a general debugging strategy. To view a resource (in the flux-operator namespace):
$ kubectl --namespace flux-operator get pods
I found this helpful for debugging the stateful set - e.g., it was stuck on ContainerCreating:
$ kubectl describe -n flux-operator pods
If you need to clean things up (ensuring you only have this one pod and service running first) I’ve found it easier to do:
$ kubectl delete pod --all
# Service shorthand
$ kubectl delete svc --all
$ kubectl delete statefulset --all
# ConfigMap shorthand
$ kubectl delete cm --all
And take a look at the project Makefile
that has some commands pre-set for you, and feel free to add other
combinations that might be useful.