[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Fwd: Using Persistent Volume



Title: HTML podpis
Hello,

I already described a lot of things in another thread ("Images with persistent storage" and "Re: Images with persistent storage"), so fell free to read them. And also reported this issue to github here:
https://github.com/kubernetes/kubernetes/issues/19060

To summarize it:
I have working OpenShift cluster with 3 masters and 5 nodes.
I have working Ceph storage cluster (we are using it for OpenStack and it is working just fine).

I wanted to connect Ceph images via rbd plugin to OpenShift pods. I do not encounter any errors during mounting the persistent volume. Unfortunately the persistent volume is mounted under root owner and group. So for example in mongodb template, user mongodb can not write into that folder and pod will fail into CrashLoopBackOff state in few sec (it will try restart the container after while but the same problem occur again).

The volume is successfully bounded, if I am fast enough and enter my node server and enter container via docker as root user, then I can read/write from the volume. I can also chown the directory to user mongodb, which actually solve the problem. After next pod restart, database will start without problems.
I tried it several times.

This is probably issue only with anything that uses block devices. Since you cannot define permission before the FS is actually created. That part is probably doing kubernetes in the middle of mounting process.


from container:
/dev/rbd0          9.8G   37M  9.2G   1% /var/lib/mongodb/data

/dev/rbd0 on /var/lib/mongodb/data type ext4 (rw,relatime,seclabel,stripe=1024,data=""


bash-4.2$ ls -lha /var/lib/mongodb
total 16K
drwxrwxr-x.  3 mongodb    root   50 Jan  7 03:52 .
drwxr-xr-x. 16 root       root 4.0K Nov 12 14:30 ..
-rw-r--r--.  1 1000030000 root   12 Jan  7 03:52 .address
drwxr-xr-x.  3 root       root 4.0K Jan  7 03:52 data
-rw-r--r--.  1 1000030000 root    3 Jan  7 03:52 mongodb.pid


Logs from container:
[root master-1 ~]# oc logs mongodbcephtest-1-h9ttq
=> Waiting for container IP address ... 192.168.0.26:27017
=> Waiting for MongoDB service startup  ...
note: noprealloc may hurt performance in many applications
Thu Jan  7 03:53:49.805 [initandlisten] MongoDB starting : pid=23 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodbcephtest-1-h9ttq
Thu Jan  7 03:53:49.805 [initandlisten]
Thu Jan  7 03:53:49.805 [initandlisten] ** WARNING: You are running on a NUMA machine.
Thu Jan  7 03:53:49.805 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Thu Jan  7 03:53:49.805 [initandlisten] **              numactl --interleave=all mongod [other options]
Thu Jan  7 03:53:49.806 [initandlisten]
Thu Jan  7 03:53:49.806 [initandlisten] db version v2.4.9
Thu Jan  7 03:53:49.806 [initandlisten] git version: nogitversion
Thu Jan  7 03:53:49.806 [initandlisten] build info: Linux x86-020.build.eng.bos.redhat.com 2.6.32-431.4.1.el6.x86_64 #1 SMP Thu Dec 19 10:26:41 EST 2013 x86_64 BOOST_LIB_VERSION=1_53
Thu Jan  7 03:53:49.806 [initandlisten] allocator: tcmalloc
Thu Jan  7 03:53:49.806 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongodb/data", nohttpinterface: "true", noprealloc: "true", oplogSize: 64, pidfilepath: "/var/lib/mongodb/mongodb.pid", port: 27017, quiet: "true", smallfiles: "true" }
Thu Jan  7 03:53:49.806 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Thu Jan  7 03:53:49.806 dbexit:
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close listening sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to flush diaglog...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: waiting for fs preallocator...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: lock for final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: closing all files...
Thu Jan  7 03:53:49.806 [initandlisten] closeAllFiles() finished
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: removing fs lock...
Thu Jan  7 03:53:49.806 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Thu Jan  7 03:53:49.806 dbexit: really exiting now
=> Waiting for MongoDB service startup  ...



I am attaching all requested outputs as text files.



On 01/07/2016 05:52 AM, Paul Morie wrote:
Vaclav-

Would you provide the following information?

-  output of `oc get -o yaml pod <your-pod-name>`
-  output of `oc get -o yaml pv <your-pv-name>`
-  output of `oc get -o yaml pvc <you-pvc-name>`
-  log of the openshift-node process
-  log of the container

...and we'll take a look at what's happening.

Thanks,

P


On Wed, Jan 6, 2016 at 8:50 AM, Erin Boyd <eboyd redhat com> wrote:

Hi Vaclav,
Sorry you are experiencing difficulty setting the pvs up.
I would be happy to help.
There are several different documents and it would help me if you could tell me what storage you are using so I can point you in the right direction.
Thanks,
Erin

On Jan 6, 2016 8:43 AM, "Mark Turansky" <mturansk redhat com> wrote:
I don't have any information besides what's in this person's email to the user list.  I would have replied but this is not currently my area of expertise and knowledge.



On Wed, Jan 6, 2016 at 8:34 AM, Erin Boyd <eboyd redhat com> wrote:

What type of storage?
Then I can point then to the exact document.
Erin

On Jan 6, 2016 7:36 AM, "Mark Turansky" <mturansk redhat com> wrote:
Are there good guides/docs to point this user towards?  


---------- Forwarded message ----------
From: Vaclav Rozsypalek <rozsypalek master cz>
Date: Wed, Jan 6, 2016 at 3:49 AM
Subject: Using Persistent Volume
To: "users lists openshift redhat com" <users lists openshift redhat com>


Hello,

Lately I have been struggling with use of persistent volumes in OpenShift.
I am using ceph pv, but i think the problem is same with anything that uses block devices.

Not a single pre-created image works, because  all volumes are mounted with root owner and group and with 0755 permissions.  Non-root user can not write there and all Openshift images uses no-nroot users for services (apache, databases etc.).  At this point I am not even sure which technology is the actual issue. First I thought it was kubernetes but then i also found this issue with docker on github:
https://github.com/docker/docker/issues/2259

Does anybody found any workaround this? (NFS is different so anything from nfs work probably work)
I found it kinda interesting that it actually does not work. Looks like nobody tested it at all.




--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email: rozsypalek master cz

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users





--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email:

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 

=> Waiting for container IP address ... 192.168.0.26:27017
=> Waiting for MongoDB service startup  ...
note: noprealloc may hurt performance in many applications
Thu Jan  7 03:53:49.805 [initandlisten] MongoDB starting : pid=23 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodbcephtest-1-h9ttq
Thu Jan  7 03:53:49.805 [initandlisten] 
Thu Jan  7 03:53:49.805 [initandlisten] ** WARNING: You are running on a NUMA machine.
Thu Jan  7 03:53:49.805 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Thu Jan  7 03:53:49.805 [initandlisten] **              numactl --interleave=all mongod [other options]
Thu Jan  7 03:53:49.806 [initandlisten] 
Thu Jan  7 03:53:49.806 [initandlisten] db version v2.4.9
Thu Jan  7 03:53:49.806 [initandlisten] git version: nogitversion
Thu Jan  7 03:53:49.806 [initandlisten] build info: Linux x86-020.build.eng.bos.redhat.com 2.6.32-431.4.1.el6.x86_64 #1 SMP Thu Dec 19 10:26:41 EST 2013 x86_64 BOOST_LIB_VERSION=1_53
Thu Jan  7 03:53:49.806 [initandlisten] allocator: tcmalloc
Thu Jan  7 03:53:49.806 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongodb/data", nohttpinterface: "true", noprealloc: "true", oplogSize: 64, pidfilepath: "/var/lib/mongodb/mongodb.pid", port: 27017, quiet: "true", smallfiles: "true" }
Thu Jan  7 03:53:49.806 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Thu Jan  7 03:53:49.806 dbexit: 
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close listening sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to flush diaglog...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: waiting for fs preallocator...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: lock for final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: closing all files...
Thu Jan  7 03:53:49.806 [initandlisten] closeAllFiles() finished
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: removing fs lock...
Thu Jan  7 03:53:49.806 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Thu Jan  7 03:53:49.806 dbexit: really exiting now
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Waiting for MongoDB service startup  ...
=> Giving up: Failed to start MongoDB service!
Jan  7 09:52:41 node-4 kernel: libceph: client7246946 fsid 4304ab9a-7a21-47ce-bf2c-2adefcda3680
Jan  7 09:52:41 node-4 kernel: libceph: mon0 10.2.70.82:6789 session established
Jan  7 09:52:41 node-4 kernel: rbd0: unknown partition table
Jan  7 09:52:41 node-4 kernel: rbd: rbd0: added with size 0x280000000
Jan  7 09:52:42 node-4 kernel: EXT4-fs (rbd0): VFS: Can't find ext4 filesystem
Jan  7 09:52:43 node-4 kernel: EXT4-fs (rbd0): mounted filesystem with ordered data mode. Opts: (null)
Jan  7 09:52:43 node-4 kernel: EXT4-fs (rbd0): re-mounted. Opts: (null)
Jan  7 09:52:43 node-4 origin-node: I0107 09:52:43.129526    7260 manager.go:1720] Need to restart pod infra container for "mongodbcephtest-1-h9ttq_mai" because it is not found
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Mounting V4 Filesystem
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Ending clean mount
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Mounting V4 Filesystem
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Ending clean mount
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Mounting V4 Filesystem
Jan  7 09:52:43 node-4 kernel: XFS (dm-9): Ending clean mount
Jan  7 09:52:43 node-4 kernel: device veth0549462 entered promiscuous mode
Jan  7 09:52:43 node-4 kernel: IPv6: ADDRCONF(NETDEV_UP): veth0549462: link is not ready
Jan  7 09:52:43 node-4 kernel: IPVS: Creating netns size=2032 id=25
Jan  7 09:52:43 node-4 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0549462: link becomes ready
Jan  7 09:52:43 node-4 kernel: lbr0: port 2(veth0549462) entered forwarding state
Jan  7 09:52:43 node-4 kernel: lbr0: port 2(veth0549462) entered forwarding state
Jan  7 09:52:43 node-4 origin-node: W0107 09:52:43.574507    7260 manager.go:1892] Hairpin setup failed for pod "mongodbcephtest-1-h9ttq_mai": open /sys/devices/virtual/net/veth0549462/brport/hairpin_mode: no such file or directory
...
...
Jan  7 09:52:44 node-4 origin-node: I0107 09:52:44.259843    7260 roundrobin.go:263] LoadBalancerRR: Setting endpoints for mai/mongodbcephtest:mongo to [192.168.0.26:27017]
Jan  7 09:52:49 node-4 origin-node: I0107 09:52:49.397099    7260 kubelet.go:1685] volume "026d1e05-b51c-11e5-9fcc-44a842155c83/ceph-sas-0000-043", still has a container running "026d1e05-b51c-11e5-9fcc-44a842155c83", skipping teardown
Jan  7 09:52:49 node-4 origin-node: I0107 09:52:49.448885    7260 rbd.go:93] ceph secret info: key/{xxxxxxx_key_ommitted_xxxxxxx}==
Jan  7 09:52:57 node-4 origin-node: I0107 09:52:57.706741    7260 node_auth.go:142] Node request attributes: namespace=, user=system:openshift-node-admin, groups=[system:node-admins system:authenticated], attrs=authorizer.DefaultAuthorizationAttributes{Verb:"proxy", APIVersion:"v1", APIGroup:"", Resource:"nodes", ResourceName:"node-4.shift.masterinter.net", RequestAttributes:interface {}(nil), NonResourceURL:false, URL:"/exec/mai/mongodbcephtest-1-h9ttq/mongodb"}
Jan  7 09:52:57 node-4 origin-node: I0107 09:52:57.709216    7260 authorizer.go:74] allowed=true, reason=allowed by cluster rule
Jan  7 09:53:09 node-4 origin-node: I0107 09:53:09.397177    7260 kubelet.go:1685] volume "026d1e05-b51c-11e5-9fcc-44a842155c83/ceph-sas-0000-043", still has a container running "026d1e05-b51c-11e5-9fcc-44a842155c83", skipping teardown
Jan  7 09:53:19 node-4 origin-node: I0107 09:53:19.396987    7260 kubelet.go:1685] volume "026d1e05-b51c-11e5-9fcc-44a842155c83/ceph-sas-0000-043", still has a container running "026d1e05-b51c-11e5-9fcc-44a842155c83", skipping teardown
Jan  7 09:53:29 node-4 origin-node: I0107 09:53:29.450771    7260 rbd.go:93] ceph secret info: key/AQBDoLVUwPLFHRAA+qfOBJ+wN8lmkyhXKbpH6A==
... and so
[root master-1 ~]# oc describe pod mongodbcephtest-1-h9ttq
Name:				mongodbcephtest-1-h9ttq
Namespace:			mai
Image(s):			registry.access.redhat.com/openshift3/mongodb-24-rhel7:latest
Node:				node-4.shift.masterinter.net/80.79.26.42
Start Time:			Thu, 07 Jan 2016 09:52:40 +0100
Labels:				deployment=mongodbcephtest-1,deploymentconfig=mongodbcephtest,name=mongodbcephtest
Status:				Running
Reason:				
Message:			
IP:				192.168.0.26
Replication Controllers:	mongodbcephtest-1 (1/1 replicas created)
Containers:
  mongodb:
    Container ID:	docker://81bd176f25165518433752a573b2bf22715671a1a13a5b94752e8990852ad48d
    Image:		registry.access.redhat.com/openshift3/mongodb-24-rhel7:latest
    Image ID:		docker://82a8a2d5ec208e5d325cf0e80e69a16c3ebbcef7a42dfd129b96c01517b30ef2
    QoS Tier:
      cpu:			BestEffort
      memory:			BestEffort
    State:			Waiting
      Reason:			CrashLoopBackOff
    Last Termination State:	Terminated
      Reason:			Error
      Exit Code:		1
      Started:			Thu, 07 Jan 2016 10:00:59 +0100
      Finished:			Thu, 07 Jan 2016 10:02:01 +0100
    Ready:			False
    Restart Count:		6
    Environment Variables:
      MONGODB_USER:		userD3P
      MONGODB_PASSWORD:		xAvUmXOUFBr1ClcM
      MONGODB_DATABASE:		sampledb
      MONGODB_ADMIN_PASSWORD:	50erylL3J5jBNrB1
Conditions:
  Type		Status
  Ready 	False 
Volumes:
  mongodbcephtest-data:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	mongodbcephtest
    ReadOnly:	false
  default-token-5fdp7:
    Type:	Secret (a secret that should populate this volume)
    SecretName:	default-token-5fdp7
Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath		Reason		Message
  ─────────	────────	─────	────					─────────────		──────		───────
  9m		9m		1	{scheduler }							Scheduled	Successfully assigned mongodbcephtest-1-h9ttq to node-4.shift.masterinter.net
  9m		9m		1	{kubelet node-4.shift.masterinter.net}	implicitly required container POD	Pulled		Container image "openshift/origin-pod:v1.1.0.1" already present on machine
  9m		9m		1	{kubelet node-4.shift.masterinter.net}	implicitly required container POD	Created		Created with docker id 2dfa36911669
  9m		9m		1	{kubelet node-4.shift.masterinter.net}	implicitly required container POD	Started		Started with docker id 2dfa36911669
  9m		9m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id 87451798f12c
  9m		9m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id 87451798f12c
  8m		8m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id 17d4f13bddbb
  8m		8m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id 17d4f13bddbb
  7m		7m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id b17cf837b30b
  7m		7m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id b17cf837b30b
  5m		5m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id 736ecf772323
  5m		5m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id 736ecf772323
  4m		4m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id 926bda0a3271
  4m		4m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id 926bda0a3271
  1m		1m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Started		Started with docker id 81bd176f2516
  9m		1m		6	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Pulled		Container image "registry.access.redhat.com/openshift3/mongodb-24-rhel7:latest" already present on machine
  1m		1m		1	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Created		Created with docker id 81bd176f2516
  7m		5s		18	{kubelet node-4.shift.masterinter.net}	spec.containers{mongodb}		Backoff		Back-off restarting failed docker container

Attachment: pod.yaml
Description: application/yaml

Attachment: pv.yaml
Description: application/yaml

Attachment: pvc.yaml
Description: application/yaml


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]