[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Origin 3.6 cluster no longer deploying deployemnts



I just tried restarting the controller service a couple more times and those panics dont show anymore and the deployment has started.



On Mon, 3 Dec 2018 at 10:39 Cameron Braid <cameron braid com au> wrote:

Yeah, these look like errors :

Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.150240   89591 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:514
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/signal_unix.go:290
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/daemon/daemoncontroller.go:155
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: <autogenerated>:57
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:2197
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.230853   89591 controller_utils.go:1032] Caches are synced for RC controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: W1203 10:28:44.300741   89591 shared_informer.go:298] resyncPeriod 30000000000 is smaller than resyncCheckPeriod 600000000000 and the informer has already started. Changing it to 600000000000
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.300811   89591 controllermanager.go:466] Started "statefulset"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.300820   89591 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:514
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/signal_unix.go:290
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:365
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:174
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:111
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: <autogenerated>:57
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:2197



Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.750941   89591 controllermanager.go:456] Starting "deployment"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.750997   89591 attach_detach_controller.go:242] Starting attach detach controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.751014   89591 controller_utils.go:1025] Waiting for caches to sync for attach detach controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.900441   89591 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:514
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:489
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/panic.go:63
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/signal_unix.go:290
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:169
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:121
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: <autogenerated>:57
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /builddir/build/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: /usr/lib/golang/src/runtime/asm_amd64.s:2197
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.900445   89591 controllermanager.go:466] Started "deployment"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: W1203 10:28:44.900490   89591 controllermanager.go:450] "ttl" is disabled
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.900454   89591 deployment_controller.go:152] Starting deployment controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.900765   89591 controller_utils.go:1025] Waiting for caches to sync for deployment controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907405   89591 actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed true because nodeName="node03-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907431   89591 actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node03-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907467   89591 actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed true because nodeName="node01-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907478   89591 actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node01-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907521   89591 actual_state_of_world.go:478] Failed to set statusUpdateNeeded to needed true because nodeName="node02-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: E1203 10:28:44.907535   89591 actual_state_of_world.go:492] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node02-2018"  does not exist
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.923823   89591 controller_utils.go:1032] Caches are synced for replica set controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943026   89591 controller_utils.go:1032] Caches are synced for node controller
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943076   89591 taint_controller.go:182] Starting NoExecuteTaintManager
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943092   89591 nodecontroller.go:525] NodeController observed a new Node: "node03-2018"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943113   89591 controller_utils.go:274] Recording Registered Node node03-2018 in NodeController event message for node node03-2018
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943145   89591 nodecontroller.go:542] Initializing eviction metric for zone:
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943159   89591 nodecontroller.go:525] NodeController observed a new Node: "node01-2018"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943167   89591 controller_utils.go:274] Recording Registered Node node01-2018 in NodeController event message for node node01-2018
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943177   89591 nodecontroller.go:525] NodeController observed a new Node: "node02-2018"
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943185   89591 controller_utils.go:274] Recording Registered Node node02-2018 in NodeController event message for node node02-2018
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: W1203 10:28:44.943257   89591 nodecontroller.go:877] Missing timestamp for Node node03-2018. Assuming now as a timestamp.
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943280   89591 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node03-2018", UID:"9d690c08-f0ed-11e7-a024-0cc47a39144c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node node03-2018 event: Registered Node node03-2018 in NodeController
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943307   89591 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node01-2018", UID:"9d638709-f0ed-11e7-a024-0cc47a39144c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node node01-2018 event: Registered Node node01-2018 in NodeController
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943319   89591 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node02-2018", UID:"9d6692f6-f0ed-11e7-a024-0cc47a39144c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node node02-2018 event: Registered Node node02-2018 in NodeController
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: W1203 10:28:44.943296   89591 nodecontroller.go:877] Missing timestamp for Node node01-2018. Assuming now as a timestamp.
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: W1203 10:28:44.943364   89591 nodecontroller.go:877] Missing timestamp for Node node02-2018. Assuming now as a timestamp.
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.943394   89591 nodecontroller.go:793] NodeController detected that zone  is now in state Normal.
Dec 03 10:28:44 node01-2018.drivenow.com.au origin-master-controllers[89591]: I1203 10:28:44.950122   89591 controller_utils.go:1032] Caches are synced for daemon sets controller

Cameron

On Mon, 3 Dec 2018 at 02:48 Clayton Coleman <ccoleman redhat com> wrote:
Are there errors in the controller logs?

On Dec 2, 2018, at 2:42 AM, Cameron Braid <cameron braid com au> wrote:

Sorry, a typo - its a 3.7 cluster not 3.6

 ~> oc version

oc v3.7.2+282e43f

kubernetes v1.7.6+a08f5eeb62

features: Basic-Auth


Server 

openshift v3.7.0+7ed6862

kubernetes v1.7.6+a08f5eeb62


On Sun, 2 Dec 2018 at 18:39 Cameron Braid <cameron braid com au> wrote:
I have a strange issue in my 3.6 cluster.  I create a extensions/v1beta1 Deployment and nothing happens.  No pods are created.  oc describe shows 

 oc -n drivenow-staging-x describe deployment strimzi-cluster-operator
Name: strimzi-cluster-operator
Namespace: drivenow-staging-x
CreationTimestamp: Sun, 02 Dec 2018 01:33:10 +1100
Labels: app=strimzi
Annotations: <none>
Selector: name=strimzi-cluster-operator
Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
  Labels: name=strimzi-cluster-operator
  Service Account: strimzi-cluster-operator
  Containers:
   strimzi-cluster-operator:
    Image: strimzi/cluster-operator:0.6.0
    Port: <none>
    Limits:
      cpu: 1
      memory: 256Mi
    Requests:
      cpu: 200m
      memory: 256Mi
    Liveness: http-get http://:8080/healthy delay=10s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:8080/ready delay=10s timeout=1s period=30s #success=1 #failure=3
    Environment: ...
    Mounts: <none>
  Volumes: <none>
OldReplicaSets: <none>
NewReplicaSet: <none>
Events: <none>

All nodes are ready, schedulable and there doesnt appear to be antyhing in the logs.

I've tried restarting origin-node, origin-master-api and origin-master-controllers on all nodes and that has no impact.

I'm out of ideas.

Cameron
_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]