[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: why all pods are redeployed daily?



What is your node name?  Is it by any chance based on something that
dynamically changes every day, like say an IP address assigned by
dhcp?

> On Oct 8, 2015, at 5:49 PM, Viet Nguyen <vnguyen redhat com> wrote:
>
> This is an openshift-ansible install on bring-your-own-VMs.
>
> I can see "Killing unwanted pod..." in the node log.  Not sure about the source of that delete node api call.  Really strange!
>
> node journal around 14:85
>
> http://pasted.co/2dd5fa80
>
>
>
>
> ----- Original Message -----
> From: "Clayton Coleman" <ccoleman redhat com>
> To: "Viet Nguyen" <vnguyen redhat com>
> Cc: users lists openshift redhat com
> Sent: Thursday, October 8, 2015 1:39:06 PM
> Subject: Re: why all pods are redeployed daily?
>
>   1. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008
>   14:58:08.204117   48577 nodecontroller.go:230] NodeController observed a
>   Node deletion: b16
>   2. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008
>   14:58:08.204147   48577 nodecontroller.go:356] Recording Removing Node b16
>   from NodeController event message for node b16
>   3. Oct 08 14:58:08 openshift-master1 openshift-master[48577]: I1008
>   14:58:08.204174   48577 nodecontroller.go:160] Delete all pods from b16
>
>
> Which means something deleted your node via the API. Check whether there is
> a corresponding log entry at that time in the journal for docker and
> Openshift-node.  Is this an all in one containerized deployment, an Ansible
> installed cluster, or down thing else?
>
> On Oct 8, 2015, at 4:20 PM, Viet Nguyen <vnguyen redhat com> wrote:
>
> pastebin correct link:  http://pastebin.com/ky9dmJWB
>
>
>
> ----- Original Message -----
> From: "Viet Nguyen" <vnguyen redhat com>
> To: "Clayton Coleman" <ccoleman redhat com>
> Cc: users lists openshift redhat com
> Sent: Thursday, October 8, 2015 12:47:29 PM
> Subject: Re: why all pods are redeployed daily?
>
> Did your node restart 21 hours ago?
>
>
> No, the node (RHEL7.1 on bare metal) has been up for awhile.  As I was
> writing my reply all the pods were restarted again.  Master system journal
> shows a lot "Attempting to schedule..." msgs around that time.
>
> openshift master log:
> http://pastebin.com/ky9dmJWBI
>
> I can see inactive containers on the node with 'docker ps -a'
>
>
>
>
> ----- Original Message -----
> From: "Clayton Coleman" <ccoleman redhat com>
> To: "Viet Nguyen" <vnguyen redhat com>
> Cc: users lists openshift redhat com
> Sent: Thursday, October 8, 2015 10:50:00 AM
> Subject: Re: why all pods are redeployed daily?
>
> Did your node restart 21 hours ago?  On the node, do you see the
> previous docker container from that time, and is there any other info
> in the system journal from that time?
>
> On Oct 8, 2015, at 1:21 PM, Viet Nguyen <vnguyen redhat com> wrote:
>
>
> Dead cakephp containers have the following message in the log though I
> don't think it's the cause:
>
>
> AH00558: httpd: Could not reliably determine the server's fully qualified
> domain name, using 10.1.0.58.
>
>
> I don't see any other errors in other pods.  The dead containers have
> either exit code 0 or 137.
>
>
>
> [root openshift-master1 ~]# oc describe pod cakephp-example-1-5v9jg
>
> Name:                cakephp-example-1-5v9jg
>
> ...
>
> Conditions:
>
> Type        Status
>
> Ready    True
>
> No events.
>
>
>
>
> ----- Original Message -----
>
> From: "Clayton Coleman" <ccoleman redhat com>
>
> To: "Viet Nguyen" <vnguyen redhat com>
>
> Cc: users lists openshift redhat com
>
> Sent: Thursday, October 8, 2015 9:59:06 AM
>
> Subject: Re: why all pods are redeployed daily?
>
>
> That's restarting, not redeployment. What is in the pod logs and what
>
> are the recent pod events (do a describe of each pod)?
>
>
> On Oct 8, 2015, at 12:38 PM, Viet Nguyen <vnguyen redhat com> wrote:
>
>
> I have a 1-node cluster with about 30 or so pods running in different
> projects.  All of my pods (including the registry, router) are
> automatically redeployed almost daily without my intervention.   Is this a
> configurable option or something wrong with my cluster?
>
>
>
> # oadm manage-node b16 --list-pods
>
>
> NAME                            READY     STATUS    RESTARTS   AGE
>
> cakephp-example-1-3exdw         1/1       Running   0          21h
>
> cakephp-example-1-5v9jg         1/1       Running   0          21h
>
> cakephp-example-1-dpiag         1/1       Running   0          21h
>
> ...
>
>
>
> version:
>
>
> oc v3.0.2.0
>
> kubernetes v1.1.0-alpha.0-1605-g44c91b1
>
>
> _______________________________________________
>
> users mailing list
>
> users lists openshift redhat com
>
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
> _______________________________________________
>
> users mailing list
>
> users lists openshift redhat com
>
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
> _______________________________________________
> users mailing list
> users lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
> _______________________________________________
> users mailing list
> users lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]