I’d like to ask a question regarding the use of evacuating pods and how openshift/kubernetes schedules the replacement.
We have 3 nodes configured to run applications, and we went through a cycle of applying patches. So we’ve created an ansible playbook that goes through, evacuates the pods and restarts that node, one node at a time.
Prior to starting, we had an application running 3 pods, one one each node. When node1 was forced to evac the pods, kubernetes scheduled the replacement pod on node3. Node2 was next in line, when ansible forced the evac of pods, the final pod was placed on node3. So at this point, all pods were on the same physical node.
When ansible forced the evac of pods on node3, I then had an outage. The three pods were put in a “terminating” state, while 3 others were in a “pending” state. It took approximately 30 seconds to terminate the pods. The new ‘pending’ pods sat pending for about 65 seconds, after which they were finally scheduled on nodes 1 and 2 and X time to start the containers.
Is this expected behavior? I was hoping that the replication controller woud recognize this behavior a bit better for scheduling nodes to ensure pods don’t get shifted to the same physical box when there’s two boxes available. I’m also hoping that before pods are term’ed, replacements are brought online.