[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Failed to remove orphaned pod - device or resource busy



docker run hello-world works.
On Tue., 4 Oct. 2016 at 8:06 pm, Michail Kargakis <mkargaki redhat com> wrote:
Can you run docker containers directly via the docker command?

On Tue, Oct 4, 2016 at 10:18 AM, Lionel Orellana <lionelve gmail com> wrote:
The deployment of the router hangs in Pending status. If I cancel the deployment, wait a little and try to deploy again I get 

"Deployment of version 3 awaiting cancellation of older running deployments"

This shows in the logs:

Oct 04 18:54:58 poc-docker02.aipo.gov.au origin-node[1773]: E1004 18:54:58.805480    1815 kubelet.go:2684] Failed cleaning pods: [remove /var/lib/origin/openshift.local.volumes/pods/0d0c9fa0-8624-11e6-83c1-005056915814/volumes/kubernetes.io~secret/default-token-36qqf: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/10493062-89ce-11e6-827b-005056915814/volumes/kubernetes.io~secret/server-certificate: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/104a80c3-89ce-11e6-827b-005056915814/volumes/kubernetes.io~nfs/pv-registry/.snapshot/hourly.2016-10-04_1505: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/1204b6d2-8556-11e6-83c1-005056915814/volumes/kubernetes.io~secret/deployer-token-ygldd: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/19f736b2-8619-11e6-83c1-005056915814/volumes/kubernetes.io~secret/deployer-token-lhbl7: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/1a16e853-8460-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-token-33845: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/kubernetes.io~secret/router-token-5qacw: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/29857e2d-8554-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-token-ftgao: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/2f33e60b-854d-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-token-ftgao: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/3810b449-89d5-11e6-827b-005056915814/volumes/kubernetes.io~secret/deployer-token-1ehih: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/3e5e918b-85e1-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-dockercfg-4kyom-push: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/3e8cc25e-85ee-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-dockercfg-4kyom-

If I then force-delete the old deployment pod by doing

  oc delete pod router-2-deploy --grace-period=0   

then the next deployment starts and hangs in Pending again. 

A similar but smaller output appears in the logs

Oct 04 18:55:14 poc-docker02.aipo.gov.au origin-node[1773]: I1004 18:55:14.421502    1815 kubelet.go:2117] Failed to remove orphaned pod "1a997bdd-7ed8-11e6-adb7-005056915814" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/kubernetes.io~secret/router-token-5qacw: device or resource busy

This device is a tmpfs mount. 

-bash-4.2$ sudo df -h | grep 1a997bdd
tmpfs                                                                        1.9G     0  1.9G   0% /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/kubernetes.io~secret/server-certificate
tmpfs                                                                        1.9G     0  1.9G   0% /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/kubernetes.io~secret/router-token-5qacw
-bash-4.2$

Restarting the docker daemon doesn't get rid of them. I'm well and truly stuck. 


On 4 October 2016 at 17:40, Lionel Orellana <lionelve gmail com> wrote:
All the "device or resource busy" errors seem related to tmpfs mounts for secret volumes. 



On 4 October 2016 at 17:32, Lionel Orellana <lionelve gmail com> wrote:
Hi All,

I had a v1.3 cluster with a master and a node going. Both servers were rebooted over the weekend and all hell broke loose. 

The registry, the router and all apps I had running have stopped working. 

I see quite a few of these errors in the logs: 

Oct 04 17:14:07 poc-docker02.aipo.gov.au origin-node[1773]: I1004 17:14:07.510615    1815 kubelet.go:2117] Failed to remove orphaned pod "c449d37f-8549-11e6-83c1-005056915814" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/c449d37f-8549-11e6-83c1-005056915814/volumes/kubernetes.io~secret/builder-token-ftgao: device or resource busy

I don't really know what happened, how I got into this state. The registry is stuck in "Container Creating".  If I start a new deployment of the router the deployment pod doesn't get past "Pending". 

Seems like I can't delete pods either. They get stuck in "Terminating". 

Not sure how to narrow this down. Any help greatly appreciated. 

Thanks

Lionel.



_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]