[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Aggregating container logs using Kibana

On Wed, Apr 13, 2016 at 3:16 AM, Lorenz Vanthillo <lorenz vanthillo outlook com> wrote:
I saw on https://github.com/openshift/origin/issues/8358:

$ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current
Debugging with pod/debug-logging-fluentd-80xzt, original command: <image entrypoint>
Waiting for pod to start ...

Removing debug pod ...

Yup. The problem was what I thought: it's being run under the svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t. If you don't want to disable SELinux, you'll need to follow the instructions for creating a new SELinux type that I posted above.

So I understand what's wrong but I don't see why the workaround (changing the service account permissions from anyuid to privileged) isn't working for me + I don't want to create a new selinuxtype.

Sorry about that, we had missed a step.  You'll need to delete your daemonset, edit your logging-fluentd-template to add a property to your container spec and recreate your daemonset to let it properly run as privileged to escape the SELinux enforcing.

$ oc delete daemonset logging-fluentd

$ oc edit template/logging-fluentd-template

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
apiVersion: v1
kind: Template
  component: fluentd
. . .
- apiVersion: extensions/v1beta1
  kind: DaemonSet
. . .
        component: fluentd
        provider: openshift
          component: fluentd
          provider: openshift
        name: fluentd-elasticsearch
. . .
          name: fluentd-elasticsearch

# insert below here
            privileged: true             
# insert above here

              cpu: 100m
. . .

$ oc process logging-fluentd-template | oc create -f -

From: lorenz vanthillo outlook com
To: ewolinet redhat com
CC: users lists openshift redhat com
Subject: RE: Aggregating container logs using Kibana
Date: Wed, 13 Apr 2016 09:30:48 +0200

Fixed the issue with nodeselectormismatching:
So now I have 3 fluentd pods on my 2 normal nodes and my infranode:
But still the same permission issue:
NAME                          READY     STATUS      RESTARTS   AGE
logging-curator-1-j7mz0       1/1       Running     0          17m
logging-deployer-39qcz        0/1       Completed   0          47m
logging-es-605u5g7g-1-36owl   1/1       Running     0          17m
logging-fluentd-4uqx1         1/1       Running     0          46m
logging-fluentd-dez5r         1/1       Running     0          2m
logging-fluentd-m50nj         1/1       Running     0          46m
logging-kibana-1-wfog2        2/2       Running     0          16m

From: lorenz vanthillo outlook com
To: ewolinet redhat com
CC: users lists openshift redhat com
Subject: RE: Aggregating container logs using Kibana
Date: Wed, 13 Apr 2016 09:21:47 +0200

Hi Eric,

Thanks for your reply and the follow up of this issue.
I've created a new origin 1.1.6 cluster (2 days ago) but still have the same issue:
My environment is one master (with node) non schedulable, 2 'normal' nodes and one infra node.
I still got the permission denied (The documentation is up to date so I even don't had to perform the workaround manually).
- system:serviceaccount:logging:aggregated-logging-fluentd is in scc privileged by default.

The logging-deployer-template creates services and 2 pods of fluentd (on the normal nodes).
The pods appear after performing this command:
oc label nodes --all logging-infra-fluentd=true
So my nodes got that label. also the unschedulable node on my master. So that's normal that it failed but why it fails on my infra-node I don't know. (I defined in my master-config that projects are by default on the other 2 nodes, maybe that's why but I don't know it's relevant for my issue).
I also don't really understand why 'oc process logging-support-tempalte | oc create -f -' is only be cited at the troubleshooting part.
Still the error: [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>

oc get is
NAME                    DOCKER REPO                                        TAGS            UPDATED
logging-auth-proxy      docker.io/openshift/origin-logging-auth-proxy      latest,v0.0.1   4 minutes ago
logging-curator         docker.io/openshift/origin-logging-curator         latest          4 minutes ago
logging-elasticsearch   docker.io/openshift/origin-logging-elasticsearch   latest          4 minutes ago
logging-fluentd         docker.io/openshift/origin-logging-fluentd         latest          4 minutes ago
logging-kibana          docker.io/openshift/origin-logging-kibana          latest          4 minutes ago

oc get svc
NAME                     CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
logging-es               172.30.68.xx     <none>        9200/TCP   33m
logging-es-cluster       None             <none>        9300/TCP   33m
logging-es-ops           172.30.18.xx    <none>        9200/TCP   33m
logging-es-ops-cluster   None             <none>        9300/TCP   33m
logging-kibana           172.30.216.xx   <none>        443/TCP    33m
logging-kibana-ops       172.30.186.xx   <none>        443/TCP    33m

oc get pods
NAME                          READY     STATUS                    RESTARTS   AGE
logging-curator-1-j7mz0       1/1       Running                   0          4m
logging-deployer-39qcz        0/1       Completed                 0          34m
logging-es-605u5g7g-1-36owl   1/1       Running                   0          4m
logging-fluentd-4uqx1         1/1       Running                   0          33m
logging-fluentd-ex34j         0/1       NodeSelectorMismatching   0          33m
logging-fluentd-injz7         0/1       NodeSelectorMismatching   0          33m
logging-fluentd-m50nj         1/1       Running                   0          33m
logging-kibana-1-wfog2        2/2       Running                   0          4m

oc get daemonset
NAME              DESIRED   CURRENT   NODE-SELECTOR                AGE
logging-fluentd   4         4         logging-infra-fluentd=true   34m

oc get dc
logging-curator       1          1          config,image(logging-curator:latest)
logging-es-605u5g7g   1          1          config,image(logging-elasticsearch:latest)
logging-kibana        1          1          config,image(logging-auth-proxy:latest),image(logging-kibana:latest)

oc get routes
[centos ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops)
NAME         HOST/PORT                PATH      SERVICE              TERMINATION   LABELS
kibana       kibana.test.xxx.eu               logging-kibana       passthrough   component=support,logging-infra=support,provider=openshift
kibana-ops   kibana-ops.example.com             logging-kibana-ops   passthrough   component=support,logging-infra=support,provider=openshift

oc get oauthclient
NAME                           SECRET                                                             WWW-CHALLENGE   REDIRECT URIS
kibana-proxy                   j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX   FALSE           https://kibana.xxx.eu,https://kibana-ops.example.com
openshift-browser-client       71724303-b823-4435-8568-bcafxxxx4                               FALSE           https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display
openshift-challenging-client   ac7c9942-9a55-4e1e-8e5f-9fxxxxx                              TRUE            https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit
openshift-web-console          6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx                            FALSE           https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000

From: ewolinet redhat com
Date: Tue, 12 Apr 2016 17:27:06 -0500
Subject: Re: Aggregating container logs using Kibana
To: lorenz vanthillo outlook com
CC: lmeyer redhat com; users lists openshift redhat com

On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo <lorenz vanthillo outlook com> wrote:
This are all the steps I'm performing:

oc new-project logging

$ oc secrets new logging-deployer nothing=/dev/null

$ oc process logging-deployer-account-template -n openshift \ | oc create -f -

$ oc policy add-role-to-user edit --serviceaccount logging-deployer $ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer $ oadm policy add-cluster-role-to-user oauth-editor \ system:serviceaccount:logging:logging-deployer

$ oadm policy add-scc-to-user \ privileged system:serviceaccount:logging:aggregated-logging-fluentd

$ oadm policy add-cluster-role-to-user cluster-reader \ system:serviceaccount:logging:aggregated-logging-fluentd

Than I execute the deployer template:

$ oc process logging-deployer-template -n openshift \ -v KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 \ | oc create -f -

This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on master machine) and some empty services (the logs of the pods are telling me the permission error)
When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all fine.

$ oc label nodes --all logging-infra-fluentd=true

I've edited /master/master-config.yaml + restart
$ oc scale dc/logging-kibana --replicas=2

$ oc delete oauthclient/kibana-proxy $ oc process logging-support-template | oc create -f -

The last step creates also some pods. It's a bit weird for me that this step is only mentioned for troubleshooting or is
it an issue that I don't have those pods after executing the deployer-template?
The template 'logging-support-template' creates your ImageStreams (along with your routes and oauthclient) so it shouldn't be creating your pods.  There may have been a delay in scheduling your pods initially or the image stream tags could have been in the processes of being fetched.

What does the following output?
oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging

And do you still see the same permission denied errors in the Fluentd logs?

From: lorenz vanthillo outlook com
To: lmeyer redhat com
CC: users lists openshift redhat com
Subject: RE: Aggregating container logs using Kibana
Date: Tue, 5 Apr 2016 18:00:02 +0200

I still have the same issue:

I've deleted it from scc hostmount-anyuid and added it on scc privileged.
I've deleted all fluentd pods but still the same issue. Even after recreating the project.

From: lmeyer redhat com
Date: Tue, 5 Apr 2016 10:29:04 -0400
Subject: Re: Aggregating container logs using Kibana
To: lorenz vanthillo outlook com
CC: users lists openshift redhat com

On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <lmeyer redhat com> wrote:

2016-04-05 10:55:13 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>

This looks like https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps fluentd from reading any logs on the node.

You should be able to resolve this by adding the fluentd service account to the privileged SCC, then having fluentd restart everywhere.

 oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd

Oh; probably need to also remove them from the  hostmount-anyuid SCC.

users mailing list
users lists openshift redhat com

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]