[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Feedback needed: Automating Kubernetes / OpenShift integrations - i.e. building your own controllers

TL:DR If you've wanted to write a custom controller, try this out and give feedback!

Controllers are a fundamental part of Kubernetes (they're the logic loops that ensure all the declarative APIs make it into reality) but for a long time they have been very difficult to write.  We want to start taking steps to make it easier to build simple integrations with Kubernetes and prototype new declarative patterns (like pet set controllers and extensions to services / ingress).  

We're looking for feedback on a prototype implementation of a "script controller" - a simple command in kubectl / oc (the openshift cli) that makes it easy to solve problems like:

  I want to do Y on every resource of type X

where X is any resource, third party or otherwise, and Y is anything.  What is anything?  Some examples are:

* Ensure every namespace gets a Quota and LimitRange (so that users can't use unlimited cluster resources)
* Ensure every service is registered in my cloud / private / zone file DNS provider
* Send an email alert every time a node transitions to "NotReady"
* On an NFS server, add an NFS export anytime a PVC is created and bind to that PVC (i.e. a custom dynamic provisioner)
* Delete any pod that has restarted more than 100 times and send an email to the owner
* Anytime a pod fails scheduling, write an annotation on the deployment that launched the pod indicating when and why.

In Kubernetes we say controllers are usually level-driven (they depend not on seeing the change happen, but on whatever the current state is) and that they maintain some invariant (if X is true, Y should also be true).  Sometimes you also need to keep an external system in sync - if you crash, have a bug, or someone messes around with the external system you need to be able to reconcile those changes.  This is a lot more complicated than just "watch" - it's a reconciliation loop, and as we've shown over the last few years it's tricky to get right.

## So?

We've created a prototype "observe" command on the OpenShift CLI that works with Kubernetes or OpenShift to explore how we can make building a controller easier.  It's available as an image "openshift/observe:latest" so you can run it on a local machine, or you can download a recent release binary.  This is roughly tracking Allow admins to implement controller loops from the CLI (IFTTT) and is step one: getting feedback.

The goal of the "observe" command is to help you build simple controller loops and help you do them correctly.  That is:

1. Make it easy to run a script once for every X in the system, and to re-run that script when X changes
2. Make it easy to get data off X - you shouldn't have to jump through 'jq' hoops if you want to fetch a service IP or annotation value
3. Make it *possible* to write a correct reconciliation loop against an external system, and help guide you through the gotchas
4. Be friendly for simple integrations, possible to write complex integrations, and eventually be able to tell you when you need to go write some Go code.  Not every problem can be solved here.

If this describes a problem you've had that you've cobbled together a bunch of scripts for, please give the new command a try and see if it helps / hurts / is amazing for your use case, and give us feedback.

## Try it out

Get it locally - download release binaries or run:

    docker run --entrypoint /bin/bash -it openshift/observe:latest
    # copy in a kubeconfig for your cluster or login directly
    $ oc observe -h

Watch everything:

    # for every service, and any time a service changes, print out info
    $ oc observe --all-namespaces services

Add something to do:

    # for every service, and any time a service changes, echo
    $ oc observe --all-namespaces services -- echo

You'll see that this prints out namespace and name for each one as arguments 1 and 2 to echo.  If you create / delete a service in the background, you'll see it show up in this list (the update, at least).

Add the service IP to the output:

    $ oc observe --all-namespaces services -a '{ .spec.clusterIP }' -- echo

We've used '-a' to print a JSONPath style template for each object, which becomes the last argument of the command.   

To turn this out into something practical, create a new script called 'record.sh' in the current directory and make it executable:

    $ cat record.sh
    echo $1 $2 $3 >> services

    $ oc observe --all-namespaces services -a '{ .spec.clusterIP }' -- ./record.sh

All services and their IPs will be recorded in that local file.  You can extend that to anything you can do with bash.

The more complex case is handling deletions.  Say you want to create an ingress for every service, but if the service gets deleted you want to delete the ingress.  To properly cleanup, we need to know the ingresses that were created this way.

    $ cat create.sh
    echo "{\"kind\":\"Ingress\": \"apiVersion\": \"extensions/v1beta1\",\"metadata\":{\"name\":\"$2\"}, ...}' kubectl create -f - --namespace $1
    kubectl annotate ingress/$2 fromservice=true
    $ cat names.sh
    kubectl get ingress --all-namespaces --template '{{ range .items }}{{ if eq .metadata.annotations.fromservice "true" }}{{ .metadata.namespace }}/{{ .metadata.name }}{{"\n"}}{{ end }}{{ end }}'

    $ cat delete.sh
    kubectl delete ingress $2 --namespace=$1

    $ oc observe --all-namespaces services --delete ./delete.sh --names=./names.sh -- ./create.sh

The first script creates an ingress with the same name as the service and sets an annotation.  The second walks every ingress and outputs namespace/name for any that have the annotation fromservice=true (note that the go template here is actually not enough - you have to check for the annotation being empty because it will error otherwise).  

The combination of those allows the observer to detect that a service has been deleted while it was not running - any ingress that has the annotation was created by a service, and since they match names, that must mean that a service was deleted.  If a user deletes a service directly, we'll get the watch notification - but not if we crashed, or on initial sync.

This reconciliation is tricky to get right - but observe is able to use the exact same pattern and code that Kubernetes uses to ensure we only fix critical reconcile bugs once.

There are other options around failure modes, retries, metrics endpoints, and restart behavior.  Please see the help for more.

## Call to action!

The sorts of feedback that are really useful are:

* Can you build something useful with this?
* Does this simplify scripting that you were already doing?
* Is the command obvious / intuitive if you know your way around kube?
* What sort of gotchas have you hit doing this yourself, and what can we do to make this simpler?
* Would you use this to build third party resources?
* What else do you need?

We've tried to base this on real working controllers we've seen people trying to build - our hope is that we can make this a generally useful bit of glue code for all kubernetes clusters (eventually making it into kubectl once we sort out the use cases).

Please file any feedback on Allow admins to implement controller loops from the CLI (IFTTT) 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]