[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Pod never starts and cannot delete; completely stops openshift from responding



I have tried restarting the physical machine along with restarting openshift and the docker daemon.  Originally I was just seeing a message like this in the deployment logs:

I0126 07:43:30.212175       1 deployer.go:196] Deploying keycloak/keycloakservice-5 for the first time (replicas: 1)
2 I0126 07:43:30.233288       1 recreate.go:105] Scaling keycloak/keycloakservice-5 to 1 before performing acceptance check
3 I0126 07:43:32.260466       1 recreate.go:110] Performing acceptance check of keycloak/keycloakservice-5
4 I0126 07:43:32.261555       1 lifecycle.go:391] Waiting 120 seconds for pods owned by deployment "keycloak/keycloakservice-5" to become ready (checking every 1 seconds; 0 pods previously accepted)
5 F0126 07:45:32.261935       1 deployer.go:65] update acceptor rejected keycloak/keycloakservice-5: pods for deployment "keycloak/keycloakservice-5" took longer than 120 seconds to become ready

Since restarting, my docker registry has become unresponsive and I see messages in the openshift console that say: "container 'registry' is unhealthy, it will be killed and re-created." It just keeps repeating that over and over.

Now I see deployment messages like this:
I0126 07:53:26.557631       1 deployer.go:196] Deploying keycloak/keycloakservice-6 for the first time (replicas: 1)
2 I0126 07:53:26.572359       1 recreate.go:105] Scaling keycloak/keycloakservice-6 to 1 before performing acceptance check
3 I0126 07:53:28.600383       1 recreate.go:110] Performing acceptance check of keycloak/keycloakservice-6
4 I0126 07:53:28.600501       1 lifecycle.go:391] Waiting 120 seconds for pods owned by deployment "keycloak/keycloakservice-6" to become ready (checking every 1 seconds; 0 pods previously accepted)
5 E0126 07:54:36.144407       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
6 E0126 07:54:37.145797       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
7 E0126 07:54:38.146563       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
8 E0126 07:54:39.147084       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
9 E0126 07:54:40.147622       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
10 E0126 07:54:41.148262       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
11 E0126 07:54:42.148894       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
12 E0126 07:54:43.149535       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
13 E0126 07:54:44.150027       1 reflector.go:206] pkg/deploy/strategy/support/lifecycle.go:350: Failed to watch *api.Pod: Get https://23.25.149.227:8443/api/v1/watch/namespaces/keycloak/pods?labelSelector=deployment%3Dkeycloakservice-6%2Cdeploymentconfig%3Dkeycloakservice%2Cname%3Dkeycloakservice&resourceVersion=48423: dial tcp 23.25.149.227:8443: connection refused
14 F0126 07:55:28.600789       1 deployer.go:65] update acceptor rejected keycloak/keycloakservice-6: pods for deployment "keycloak/keycloakservice-6" took longer than 120 seconds to become ready

On Tue, Jan 26, 2016 at 9:01 AM, Clayton Coleman <ccoleman redhat com> wrote:
What was the status of the pod keycloakservice-6-deploy

On Tue, Jan 26, 2016 at 9:55 AM, Dean Peterson <peterson dean gmail com> wrote:
> It appears the node name is correct:
>
> [root openshift openshift]# ./oc get nodes
> NAME                LABELS                                     STATUS    AGE
> openshift.master1   kubernetes.io/hostname=openshift.master1   Ready     4d
>
> [root openshift openshift]# hostname
> openshift.master1
>
> I started openshift the same way I have always started it; with "./openshift
> start" and that creates a master and node on the same machine.
>
>
> On Tue, Jan 26, 2016 at 8:37 AM, Clayton Coleman <ccoleman redhat com>
> wrote:
>>
>> Deletions are graceful by default - you can pass --grace-period=0 to
>> force delete the offending pod.
>>
>> This is almost always related to nodes being misconfigured.  There was
>> an issue where nodes may have gotten their node names changed to their
>> IP in 1.1 can you verify that the nodes the pods are assigned to match
>> actual node names in the system?
>>
>> > On Jan 26, 2016, at 2:14 AM, Dean Peterson <peterson dean gmail com>
>> > wrote:
>> >
>> > If I try to build and deploy a number of pods simultaneously Openshift
>> > seems to go crazy and I am no longer able to get my app up.  I recently
>> > upgraded to v1.1.1.  I think there may be serious locking issues with this
>> > version.  After keycloakservice-5-g75fg pod hangs, I try to delete it
>> > because I can't start anything else in Openshift it seems while that is
>> > stuck terminating.  I try to manually delete the pod but it just keeps
>> > saying deleted, deleted deleted over and over but the pod stays.
>> >
>> > keycloakservice-5-g75fg          0/1       Terminating   0          14m
>> > keycloakservice-6-deploy         0/1       Error         0          7m
>> > keycloakservice-build-1-build    0/1       Completed     0          1d
>> > keycloakservice-build-10-build   0/1       Completed     0          24m
>> > keycloakservice-build-11-build   0/1       Completed     0          17m
>> > keycloakservice-build-2-build    0/1       Completed     0          2h
>> > keycloakservice-build-3-build    0/1       Completed     0          2h
>> > wildfly-jdk-8-build-1-build      0/1       Completed     0          1d
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> > [root openshift openshift]# ./oc delete pod keycloakservice-5-g75fg
>> > pod "keycloakservice-5-g75fg" deleted
>> >
>> > _______________________________________________
>> > users mailing list
>> > users lists openshift redhat com
>> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]