[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: why all pods are redeployed daily?



  1. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008 14:58:08.204117   48577 nodecontroller.go:230] NodeController observed a Node deletion: b16
  2. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008 14:58:08.204147   48577 nodecontroller.go:356] Recording Removing Node b16 from NodeController event message for node b16
  3. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008 14:58:08.204174   48577 nodecontroller.go:160] Delete all pods from b16

Which means something deleted your node via the API. Check whether there is a corresponding log entry at that time in the journal for docker and Openshift-node.  Is this an all in one containerized deployment, an Ansible installed cluster, or down thing else?

On Oct 8, 2015, at 4:20 PM, Viet Nguyen <vnguyen redhat com> wrote:

pastebin correct link:  http://pastebin.com/ky9dmJWB



----- Original Message -----
From: "Viet Nguyen" <vnguyen redhat com>
To: "Clayton Coleman" <ccoleman redhat com>
Cc: users lists openshift redhat com
Sent: Thursday, October 8, 2015 12:47:29 PM
Subject: Re: why all pods are redeployed daily?

Did your node restart 21 hours ago?

No, the node (RHEL7.1 on bare metal) has been up for awhile.  As I was writing my reply all the pods were restarted again.  Master system journal shows a lot "Attempting to schedule..." msgs around that time.

openshift master log:
http://pastebin.com/ky9dmJWBI

I can see inactive containers on the node with 'docker ps -a'




----- Original Message -----
From: "Clayton Coleman" <ccoleman redhat com>
To: "Viet Nguyen" <vnguyen redhat com>
Cc: users lists openshift redhat com
Sent: Thursday, October 8, 2015 10:50:00 AM
Subject: Re: why all pods are redeployed daily?

Did your node restart 21 hours ago?  On the node, do you see the
previous docker container from that time, and is there any other info
in the system journal from that time?

On Oct 8, 2015, at 1:21 PM, Viet Nguyen <vnguyen redhat com> wrote:

Dead cakephp containers have the following message in the log though I don't think it's the cause:

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.1.0.58.

I don't see any other errors in other pods.  The dead containers have either exit code 0 or 137.


[root openshift-master1 ~]# oc describe pod cakephp-example-1-5v9jg
Name:                cakephp-example-1-5v9jg
...
Conditions:
Type        Status
Ready    True
No events.



----- Original Message -----
From: "Clayton Coleman" <ccoleman redhat com>
To: "Viet Nguyen" <vnguyen redhat com>
Cc: users lists openshift redhat com
Sent: Thursday, October 8, 2015 9:59:06 AM
Subject: Re: why all pods are redeployed daily?

That's restarting, not redeployment. What is in the pod logs and what
are the recent pod events (do a describe of each pod)?

On Oct 8, 2015, at 12:38 PM, Viet Nguyen <vnguyen redhat com> wrote:

I have a 1-node cluster with about 30 or so pods running in different projects.  All of my pods (including the registry, router) are automatically redeployed almost daily without my intervention.   Is this a configurable option or something wrong with my cluster?


# oadm manage-node b16 --list-pods

NAME                            READY     STATUS    RESTARTS   AGE
cakephp-example-1-3exdw         1/1       Running   0          21h
cakephp-example-1-5v9jg         1/1       Running   0          21h
cakephp-example-1-dpiag         1/1       Running   0          21h
...


version:

oc v3.0.2.0
kubernetes v1.1.0-alpha.0-1605-g44c91b1

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]