[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Fwd: Using Persistent Volume



Title: HTML podpis
Hello,
thanks for the response.
changing

fsGroup:
type: RunAsAny

to

fsGroup:
type: MustRunAs

in

oc edit scc restricted

solved the problem.
Is there any documentation that would tell me to do this, when I am using persistent volumes?


On 01/07/2016 04:27 PM, Erin Boyd wrote:
Hi Vaclav,
Thanks for all the info.
In the pod.yaml you need to put in the fsgroup for the ceph group.
  securityContext:
    seLinuxOptions:
      level: s0:c6,c0
   fsGroup
: 0
   runAsUser: 0

I am not sure I saw your scc listed in the email. Depending on whether or not you created a seperate security context or not, lets say you are using restricted.
You would need to udpate the values for FSGROUP to 'MustRunAs' and the RUNASUSER to 'MustRunAsRange'
Looking at your ls, it looks like the group is root but there is a different user: 1000030000

So when you do an
edit occ restricted


openshift.io/sa.scc.supplemental-groups:  0/10000
openshift.io/sa.scc.uid-range:  0/1000050000

fsGroup:
  type: MustRunAs
runAsUser:
  type: MustRunAsRange

Can you please send that so I can see if that also needs to be changed.
I am also copying Scott Creeley from my team as I am in all day meetings and he can respond more quickly than I.

Thanks,
Erin


On Thu, Jan 7, 2016 at 4:18 AM, Vaclav Rozsypalek <rozsypalek master cz> wrote:
Hello,

I already described a lot of things in another thread ("Images with persistent storage" and "Re: Images with persistent storage"), so fell free to read them. And also reported this issue to github here:
https://github.com/kubernetes/kubernetes/issues/19060

To summarize it:
I have working OpenShift cluster with 3 masters and 5 nodes.
I have working Ceph storage cluster (we are using it for OpenStack and it is working just fine).

I wanted to connect Ceph images via rbd plugin to OpenShift pods. I do not encounter any errors during mounting the persistent volume. Unfortunately the persistent volume is mounted under root owner and group. So for example in mongodb template, user mongodb can not write into that folder and pod will fail into CrashLoopBackOff state in few sec (it will try restart the container after while but the same problem occur again).

The volume is successfully bounded, if I am fast enough and enter my node server and enter container via docker as root user, then I can read/write from the volume. I can also chown the directory to user mongodb, which actually solve the problem. After next pod restart, database will start without problems.
I tried it several times.

This is probably issue only with anything that uses block devices. Since you cannot define permission before the FS is actually created. That part is probably doing kubernetes in the middle of mounting process.


from container:
/dev/rbd0          9.8G   37M  9.2G   1% /var/lib/mongodb/data

/dev/rbd0 on /var/lib/mongodb/data type ext4 (rw,relatime,seclabel,stripe=1024,data=""


bash-4.2$ ls -lha /var/lib/mongodb
total 16K
drwxrwxr-x.  3 mongodb    root   50 Jan  7 03:52 .
drwxr-xr-x. 16 root       root 4.0K Nov 12 14:30 ..
-rw-r--r--.  1 1000030000 root   12 Jan  7 03:52 .address
drwxr-xr-x.  3 root       root 4.0K Jan  7 03:52 data
-rw-r--r--.  1 1000030000 root    3 Jan  7 03:52 mongodb.pid


Logs from container:
[root master-1 ~]# oc logs mongodbcephtest-1-h9ttq
=> Waiting for container IP address ... 192.168.0.26:27017
=> Waiting for MongoDB service startup  ...
note: noprealloc may hurt performance in many applications
Thu Jan  7 03:53:49.805 [initandlisten] MongoDB starting : pid=23 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodbcephtest-1-h9ttq
Thu Jan  7 03:53:49.805 [initandlisten]
Thu Jan  7 03:53:49.805 [initandlisten] ** WARNING: You are running on a NUMA machine.
Thu Jan  7 03:53:49.805 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Thu Jan  7 03:53:49.805 [initandlisten] **              numactl --interleave=all mongod [other options]
Thu Jan  7 03:53:49.806 [initandlisten]
Thu Jan  7 03:53:49.806 [initandlisten] db version v2.4.9
Thu Jan  7 03:53:49.806 [initandlisten] git version: nogitversion
Thu Jan  7 03:53:49.806 [initandlisten] build info: Linux x86-020.build.eng.bos.redhat.com 2.6.32-431.4.1.el6.x86_64 #1 SMP Thu Dec 19 10:26:41 EST 2013 x86_64 BOOST_LIB_VERSION=1_53
Thu Jan  7 03:53:49.806 [initandlisten] allocator: tcmalloc
Thu Jan  7 03:53:49.806 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongodb/data", nohttpinterface: "true", noprealloc: "true", oplogSize: 64, pidfilepath: "/var/lib/mongodb/mongodb.pid", port: 27017, quiet: "true", smallfiles: "true" }
Thu Jan  7 03:53:49.806 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Thu Jan  7 03:53:49.806 dbexit:
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close listening sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to flush diaglog...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: waiting for fs preallocator...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: lock for final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: closing all files...
Thu Jan  7 03:53:49.806 [initandlisten] closeAllFiles() finished
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: removing fs lock...
Thu Jan  7 03:53:49.806 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Thu Jan  7 03:53:49.806 dbexit: really exiting now
=> Waiting for MongoDB service startup  ...



I am attaching all requested outputs as text files.




On 01/07/2016 05:52 AM, Paul Morie wrote:
Vaclav-

Would you provide the following information?

-  output of `oc get -o yaml pod <your-pod-name>`
-  output of `oc get -o yaml pv <your-pv-name>`
-  output of `oc get -o yaml pvc <you-pvc-name>`
-  log of the openshift-node process
-  log of the container

...and we'll take a look at what's happening.

Thanks,

P


On Wed, Jan 6, 2016 at 8:50 AM, Erin Boyd <eboyd redhat com> wrote:

Hi Vaclav,
Sorry you are experiencing difficulty setting the pvs up.
I would be happy to help.
There are several different documents and it would help me if you could tell me what storage you are using so I can point you in the right direction.
Thanks,
Erin

On Jan 6, 2016 8:43 AM, "Mark Turansky" <mturansk redhat com> wrote:
I don't have any information besides what's in this person's email to the user list.  I would have replied but this is not currently my area of expertise and knowledge.



On Wed, Jan 6, 2016 at 8:34 AM, Erin Boyd <eboyd redhat com> wrote:

What type of storage?
Then I can point then to the exact document.
Erin

On Jan 6, 2016 7:36 AM, "Mark Turansky" <mturansk redhat com> wrote:
Are there good guides/docs to point this user towards?  


---------- Forwarded message ----------
From: Vaclav Rozsypalek <rozsypalek master cz>
Date: Wed, Jan 6, 2016 at 3:49 AM
Subject: Using Persistent Volume
To: "users lists openshift redhat com" <users lists openshift redhat com>


Hello,

Lately I have been struggling with use of persistent volumes in OpenShift.
I am using ceph pv, but i think the problem is same with anything that uses block devices.

Not a single pre-created image works, because  all volumes are mounted with root owner and group and with 0755 permissions.  Non-root user can not write there and all Openshift images uses no-nroot users for services (apache, databases etc.).  At this point I am not even sure which technology is the actual issue. First I thought it was kubernetes but then i also found this issue with docker on github:
https://github.com/docker/docker/issues/2259

Does anybody found any workaround this? (NFS is different so anything from nfs work probably work)
I found it kinda interesting that it actually does not work. Looks like nobody tested it at all.




--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email: rozsypalek master cz

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users





--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email: rozsypalek master cz

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 



--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email:

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]