[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Using Persistent Volume



Title: HTML podpis
Hello guys,

thanks all for the answers. and examples.
At least I understand scc little bit more :).

Hopefully there will be documentation about this soon, so no more people will run into this problem.

Anyway keep up the good work and have a nice day.
Vaclav

On 01/08/2016 08:45 AM, Jeff Vance wrote:
Hello Vaclav,

The "official" scc/security related doc is here:
  https://github.com/openshift/openshift-docs/blob/master/architecture/additional_concepts/authorization.adoc#understanding-pre-allocated-values-and-security-context-constraints

I have a matrix showing the resulting supplemental GIDs for the containers, using a Ceph PV, based on group id settings in either the pod spec, in the namespace(project) or both, combined with the supported scc settings:

ENV:

Default project IDs:
====================
    openshift.io/sa.scc.supplemental-groups: 1000030000/10000,5000-5999,590-590 #1st number in 1st range is default S-GID
    openshift.io/sa.scc.uid-range: 1000040000/10000,27-27

Test Cases with Ceph-RBD Persistent Volume:
===========================================
(FSG=pod's fsGroup property, SGs=pod's supplementalGroups property)

   -----SCC Policy-------  ---Pod----   --Container's Suppl--
   fsGroup     suppGroups  FSG   SGs          Groups
   =========   ==========  ===== =====  =====================
1. MustRunAs   MustRunAs   none  none   groups=1000030000
2. RunAsAny    RunAsAny    none  none   (none)
3. MustRunAs   RunAsAny    none  none   groups=1000030000
4. RunAsAny    MustRunAs   none  none   groups=1000030000

5. MustRunAs   MustRunAs   5432  none   (error *)
6. RunAsAny    RunAsAny    5432  none   groups=5432
7. RunAsAny    MustRunAs   5432  none   groups=5432,1000030000
8. MustRunAs   RunAsAny    5432  none   (error *)

9. MustRunAs   MustRunAs   none  5678   groups=5678,1000030000
10.RunAsAny    RunAsAny    none  5678   groups=5678
11.RunAsAny    MustRunAs   none  5678   groups=5678
12.MustRunAs   RunAsAny    none  5678   groups=5678,1000030000

13.MustRunAs   MustRunAs   5432  5678   (error *)
14.RunAsAny    RunAsAny    5432  5678   groups=5432,5678
15.RunAsAny    MustRunAs   5432  5678   groups=5432,5678
16.MustRunAs   RunAsAny    5432  5678   (error *)

------------

Container's Mount's GID:
=======================
In all cases where the pod is created the resulting mount's gid is:
   1000030000  for ceph, when the pod's fsGroup is *not* set in the pod spec
   <id>        for ceph, when the pod's fsGroup is set to <id>


* Error from server: error when creating "test-nfs-pod.yaml": Pod "nfs-pod1" is forbidden: unable to
  validate against any security context constraint: [provider restricted: .spec.securityContext.fsGroup:
  invalid value '[5432]', Details: 5432 is not an allowed group]


jeff

----- Original Message -----
+Jeff Vance
As he is working on this security example story

On Thu, Jan 7, 2016 at 2:52 PM, Erin Boyd <eboyd redhat com> wrote:

Hi Vaclav,
So glad this solved your issue.
Here is the documentation around the security context in OSE:

https://docs.openshift.org/latest/install_config/persistent_storage/pod_security_context.html

We are working on a better way of directly tying this into the persistent
volume documentation so it's clearer.

Thanks,
Erin


On Thu, Jan 7, 2016 at 11:37 AM, Vaclav Rozsypalek <rozsypalek master cz>
wrote:

Hello,
thanks for the response.
changing

fsGroup:
type: RunAsAny

to

fsGroup:
type: MustRunAs

in

oc edit scc restricted

solved the problem.
Is there any documentation that would tell me to do this, when I am using
persistent volumes?



On 01/07/2016 04:27 PM, Erin Boyd wrote:

Hi Vaclav,
Thanks for all the info.
In the pod.yaml you need to put in the fsgroup for the ceph group.
  securityContext:
    seLinuxOptions:
      level: s0:c6,c0
   fsGroup: 0
   runAsUser: 0

I am not sure I saw your scc listed in the email. Depending on whether
or not you created a seperate security context or not, lets say you are
using restricted.
You would need to udpate the values for FSGROUP to 'MustRunAs' and the
RUNASUSER to 'MustRunAsRange'
Looking at your ls, it looks like the group is root but there is a
different user: 1000030000

So when you do an
edit occ restricted


openshift.io/sa.scc.supplemental-groups:  0/10000
openshift.io/sa.scc.uid-range:  0/1000050000

fsGroup:
  type: MustRunAs
runAsUser:
  type: MustRunAsRange

Can you please send that so I can see if that also needs to be changed.
I am also copying Scott Creeley from my team as I am in all day meetings
and he can respond more quickly than I.

Thanks,
Erin


On Thu, Jan 7, 2016 at 4:18 AM, Vaclav Rozsypalek <rozsypalek master cz>
wrote:

Hello,

I already described a lot of things in another thread ("Images with
persistent storage" and "Re: Images with persistent storage"), so fell
free
to read them. And also reported this issue to github here:
https://github.com/kubernetes/kubernetes/issues/19060

To summarize it:
I have working OpenShift cluster with 3 masters and 5 nodes.
I have working Ceph storage cluster (we are using it for OpenStack and
it is working just fine).

I wanted to connect Ceph images via rbd plugin to OpenShift pods. I do
not encounter any errors during mounting the persistent volume.
Unfortunately the persistent volume is mounted under *root* owner and
group. So for example in mongodb template, user mongodb can not write
into
that folder and pod will fail into *CrashLoopBackOff *state in few sec
(it will try restart the container after while but the same problem occur
again).

The volume is successfully bounded, if I am fast enough and enter my
node server and enter container via docker as root user, then I can
read/write from the volume. I can also chown the directory to user
mongodb,
which actually solve the problem. After next pod restart, database will
start without problems.
I tried it several times.

This is probably issue only with anything that uses block devices. Since
you cannot define permission before the FS is actually created. That part
is probably doing kubernetes in the middle of mounting process.


from container:

/dev/rbd0          9.8G   37M  9.2G   1% /var/lib/mongodb/data

/dev/rbd0 on /var/lib/mongodb/data type ext4
(rw,relatime,seclabel,stripe=1024,data=""


bash-4.2$ ls -lha /var/lib/mongodb
total 16K
drwxrwxr-x.  3 mongodb    root   50 Jan  7 03:52 .
drwxr-xr-x. 16 root       root 4.0K Nov 12 14:30 ..
-rw-r--r--.  1 1000030000 root   12 Jan  7 03:52 .address
drwxr-xr-x.  3 root       root 4.0K Jan  7 03:52 data
-rw-r--r--.  1 1000030000 root    3 Jan  7 03:52 mongodb.pid



Logs from container:
[root master-1 ~]# oc logs mongodbcephtest-1-h9ttq
=> Waiting for container IP address ... 192.168.0.26:27017
=> Waiting for MongoDB service startup  ...
note: noprealloc may hurt performance in many applications
Thu Jan  7 03:53:49.805 [initandlisten] MongoDB starting : pid=23
port=27017 dbpath=/var/lib/mongodb/data 64-bit
host=mongodbcephtest-1-h9ttq
Thu Jan  7 03:53:49.805 [initandlisten]
Thu Jan  7 03:53:49.805 [initandlisten] ** WARNING: You are running on a
NUMA machine.
Thu Jan  7 03:53:49.805 [initandlisten] **          We suggest launching
mongod like this to avoid performance problems:
Thu Jan  7 03:53:49.805 [initandlisten] **              numactl
--interleave=all mongod [other options]
Thu Jan  7 03:53:49.806 [initandlisten]
Thu Jan  7 03:53:49.806 [initandlisten] db version v2.4.9
Thu Jan  7 03:53:49.806 [initandlisten] git version: nogitversion
Thu Jan  7 03:53:49.806 [initandlisten] build info: Linux
x86-020.build.eng.bos.redhat.com 2.6.32-431.4.1.el6.x86_64 #1 SMP Thu
Dec 19 10:26:41 EST 2013 x86_64 BOOST_LIB_VERSION=1_53
Thu Jan  7 03:53:49.806 [initandlisten] allocator: tcmalloc
Thu Jan  7 03:53:49.806 [initandlisten] options: { config:
"/etc/mongod.conf", dbpath: "/var/lib/mongodb/data", nohttpinterface:
"true", noprealloc: "true", oplogSize: 64, pidfilepath:
"/var/lib/mongodb/mongodb.pid", port: 27017, quiet: "true", smallfiles:
"true" }
*Thu Jan  7 03:53:49.806 [initandlisten] exception in initAndListen:
10309 Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock
errno:13 Permission denied Is a mongod instance already running?,
terminating*
Thu Jan  7 03:53:49.806 dbexit:
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close
listening sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to flush
diaglog...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: going to close
sockets...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: waiting for fs
preallocator...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: lock for final
commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: final commit...
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: closing all files...
Thu Jan  7 03:53:49.806 [initandlisten] closeAllFiles() finished
Thu Jan  7 03:53:49.806 [initandlisten] shutdown: removing fs lock...
Thu Jan  7 03:53:49.806 [initandlisten] couldn't remove fs lock errno:9
Bad file descriptor
Thu Jan  7 03:53:49.806 dbexit: really exiting now
=> Waiting for MongoDB service startup  ...



*I am attaching all requested outputs as text files.*




On 01/07/2016 05:52 AM, Paul Morie wrote:

Vaclav-

Would you provide the following information?

-  output of `oc get -o yaml pod <your-pod-name>`
-  output of `oc get -o yaml pv <your-pv-name>`
-  output of `oc get -o yaml pvc <you-pvc-name>`
-  log of the openshift-node process
-  log of the container

...and we'll take a look at what's happening.

Thanks,

P


On Wed, Jan 6, 2016 at 8:50 AM, Erin Boyd < <eboyd redhat com>
eboyd redhat com> wrote:

Hi Vaclav,
Sorry you are experiencing difficulty setting the pvs up.
I would be happy to help.
There are several different documents and it would help me if you could
tell me what storage you are using so I can point you in the right
direction.
Thanks,
Erin
On Jan 6, 2016 8:43 AM, "Mark Turansky" < <mturansk redhat com>
mturansk redhat com> wrote:

I don't have any information besides what's in this person's email to
the user list.  I would have replied but this is not currently my area
of
expertise and knowledge.



On Wed, Jan 6, 2016 at 8:34 AM, Erin Boyd < <eboyd redhat com>
eboyd redhat com> wrote:

What type of storage?
Then I can point then to the exact document.
Erin
On Jan 6, 2016 7:36 AM, "Mark Turansky" < <mturansk redhat com>
mturansk redhat com> wrote:

Are there good guides/docs to point this user towards?


---------- Forwarded message ----------
From: Vaclav Rozsypalek < <rozsypalek master cz>rozsypalek master cz

                      
Date: Wed, Jan 6, 2016 at 3:49 AM
Subject: Using Persistent Volume
To: " <users lists openshift redhat com>
users lists openshift redhat com" <
<users lists openshift redhat com>users lists openshift redhat com>


Hello,

Lately I have been struggling with use of persistent volumes in
OpenShift.
I am using ceph pv, but i think the problem is same with anything
that uses block devices.

Not a single pre-created image works, because  all volumes are
mounted with root owner and group and with 0755 permissions.
Non-root user
can not write there and all Openshift images uses no-nroot users for
services (apache, databases etc.).  At this point I am not even sure
which
technology is the actual issue. First I thought it was kubernetes but
then
i also found this issue with docker on github:
<https://github.com/docker/docker/issues/2259>
https://github.com/docker/docker/issues/2259

Does anybody found any workaround this? (NFS is different so
anything from nfs work probably work)
I found it kinda interesting that it actually does not work. Looks
like nobody tested it at all.




--

Vaclav Rozsypalek

Linux System Administrator

*Master Internet, s.r.o.*

Email: <rozsypalek master cz> <rozsypalek master cz>
rozsypalek master cz

Office: Cejl 20, 602 00  Brno

Web <http://www.master.cz/> | Facebook
<https://www.facebook.com/MasterDC/> | Twitter
<https://twitter.com/masterdc> | Google+
<https://plus.google.com/u/0/102162817669708232782/posts> | Linkedin
<https://www.linkedin.com/company/masterdc>

MasterDC Praha <http://www.master.cz/datacentrum-praha/> | Kodaňská
46, 101 00  Praha 10

MasterDC Brno <http://www.master.cz/datacentrum-brno/> | Cejl 20,
602 00  Brno

Support: +420 515 919 805 <+420%20515%20919%20805>



_______________________________________________
users mailing list
<users lists openshift redhat com>users lists openshift redhat com
<http://lists.openshift.redhat.com/openshiftmm/listinfo/users>
http://lists.openshift.redhat.com/openshiftmm/listinfo/users




                
--

Vaclav Rozsypalek

Linux System Administrator

*Master Internet, s.r.o.*

Email: rozsypalek master cz

Office: Cejl 20, 602 00  Brno

Web <http://www.master.cz/> | Facebook
<https://www.facebook.com/MasterDC/> | Twitter
<https://twitter.com/masterdc> | Google+
<https://plus.google.com/u/0/102162817669708232782/posts> | Linkedin
<https://www.linkedin.com/company/masterdc>

MasterDC Praha <http://www.master.cz/datacentrum-praha/> | Kodaňská 46,
101 00  Praha 10

MasterDC Brno <http://www.master.cz/datacentrum-brno/> | Cejl 20, 602
00  Brno

Support: +420 515 919 805 <+420%20515%20919%20805>




--

Vaclav Rozsypalek

Linux System Administrator

*Master Internet, s.r.o.*

Email: rozsypalek master cz

Office: Cejl 20, 602 00  Brno

Web <http://www.master.cz/> | Facebook
<https://www.facebook.com/MasterDC/> | Twitter
<https://twitter.com/masterdc> | Google+
<https://plus.google.com/u/0/102162817669708232782/posts> | Linkedin
<https://www.linkedin.com/company/masterdc>

MasterDC Praha <http://www.master.cz/datacentrum-praha/> | Kodaňská 46,
101 00  Praha 10

MasterDC Brno <http://www.master.cz/datacentrum-brno/> | Cejl 20, 602
00  Brno

Support: +420 515 919 805 <+420%20515%20919%20805>





      

--

Vaclav Rozsypalek

Linux System Administrator


Master Internet, s.r.o.

Email:

Office: Cejl 20, 602 00  Brno


Web | Facebook | Twitter | Google+ | Linkedin

MasterDC Praha | Kodaňská 46, 101 00  Praha 10

MasterDC Brno | Cejl 20, 602 00  Brno

Support: +420 515 919 805

 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]