[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Complete cluster meltdown due to "Kubelet stopped posting node status"



Hello,

we just had our whole Openshift cluster go down hard due to a "feature" in the Openshift master that deletes all pods from a node if the node doesn't report back to the master on a regular basis.

Turns out we're not the only ones who have been bitten by this "feature":
https://github.com/kubernetes/kubernetes/issues/30972#issuecomment-241077740
https://github.com/kubernetes/kubernetes/issues/24200

I am writing here to find out whether it is possible to disable this feature completely. We don't need it and we don't want our master to ever do something like that again.

Please note how easily this feature can be abused: At the moment anyone can bring down your whole Openshift cluster just by DDoSing the master(s) for a few minutes.







The logs (they were the same for all nodes):
Okt 09 21:47:10 openshiftmaster.com origin-master[919215]: I1004 21:47:10.804666  919215 nodecontroller.go:697] node openshiftnode.com hasn't been updated for 5m17.169004459s. Last out of disk condition is: &{Type:OutOfDisk Status:Unknown LastHeartbeatTime:2016-10-04 21:41:53 +0200 CEST LastTransitionTime:2016-10-04 21:42:33 +0200 CEST Reason:NodeStatusUnknown Message:Kubelet stopped posting node status.}
Okt 09 21:47:10 openshiftmaster.com origin-master[919215]: I1004 21:47:10.804742  919215 nodecontroller.go:451] Evicting pods on node openshiftnode.com: 2016-10-04 21:47:10.80472667 +0200 CEST is later than 2016-10-04 21:42:33.779813315 +0200 CEST + 4m20s
Okt 09 21:47:10 openshiftmaster.com origin-master[919215]: I1004 21:47:10.945766  919215 nodecontroller.go:540] Recording Deleting all Pods from Node openshiftnode.com. event message for node openshiftnode.com

Regards
v



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]