Sorry for the extensive delay, but I’ve been thinking about this more and I’m wondering if it’s safe to remove a master from the endpoint just before restarting it (say in Ansible), so that failures aren’t seen inside the cluster?
Or would something in Kubernetes just go and add the master back to the endpoint?
Alternatively, would it be possible to tell Kubernetes not to add the individual masters to that endpoint and use a load balancer instead? Say a private ELB for example?
Or are there future features in kubernetes that will make master failover more reliable internally?
In OpenShift 3.9, when a master goes down the endpoints object should be updated within 15s (the TTL on the record for the master). You can check the value of "oc get endpoints -n default kubernetes" - if you still see the master IP in that list after 15s then something else is wrong.