[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: The cluster-logging pods (Elasticsearch, Kibana, Fluentd) don't start - Openshift 4.1



Thank you Rich for your prompt reply.

After viewing  the "manifests/4.2/cluster-logging.v4.2.0.clusterserviceversion.yaml " on the  cluster-logging-operator pod, I confirm that the added (minKubeVersion: 1.16.0) line in GITHUB  is missing in the manifest file on the CLO pod on my Cluster.

I tried to edit the manifest file thru "oc rsh vi"  but the file is in ReadOnly and I can't get root access to this pod.

What is the good method of editing the manifest yaml file to update it with the missing minKubeversion ?

Thank you.


-----Original Message-----
From: "Rich Megginson" [rmeggins redhat com]
Date: 11/06/2019 01:21 PM
To: users lists openshift redhat com
Subject: Re: The cluster-logging pods (Elasticsearch, Kibana, Fluentd) don't
	start - Openshift 4.1

are you running into https://bugzilla.redhat.com/show_bug.cgi?id=1766343 ?

On 11/6/19 9:19 AM, Full Name wrote:
> Hi all,
> 
> I'm trying to deploy logging on Openshift cluster 4.1.21 using the procedure described in the following link https://docs.openshift.com/container-platform/4.1/logging/efk-logging.html.
> Everything is going fine but the logging pods don't want to start and stay at pending state.  I have the following error (0/7 nodes are available: 7 node(s) didn't match node selector) for all the 5 logging pods (2 x elasticsearch,  2 x kibana,  1x curator).
> 
> The logging pods don't start  with or without nodeSelector in the Cluster-Logging instance.
> 
> -----------------------------------------------------------
> the Cluster-Logging instance YAML file:
> -------
> apiVersion: logging.openshift.io/v1
> kind: ClusterLogging
> metadata:
>    creationTimestamp: '2019-11-04T21:20:57Z'
>    generation: 37
>    name: instance
>    namespace: openshift-logging
>    resourceVersion: '569806'
>    selfLink: >-
>      /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance
>    uid: fdc0e971-ff48-11e9-a3f8-0af5a0903ee4
> spec:
>    collection:
>      logs:
>        fluentd:
>          nodeSelector:
>            kubernetes.io/os: linux
>            node-role.kubernetes.io/infra: ''
>          resources: null
>        rsyslog:
>          resources: null
> type: fluentd
>    curation:
>      curator:
>        nodeSelector:
>          kubernetes.io/os: linux
>          node-role.kubernetes.io/infra: ''
>        resources: null
>        schedule: 30 3 * * *
>      type: curator
>    logStore:
>      elasticsearch:
>        nodeCount: 2
>        nodeSelector:
>          node-role.kubernetes.io/infra: ''
>        redundancyPolicy: SingleRedundancy
>        resources:
>          requests:
>            cpu: 500m
>            memory: 4Gi
>        storage:
>         size: 20G
>          storageClassName: gp2
>      type: elasticsearch
>    managementState: Managed
>    visualization:
>      kibana:
>        nodeSelector:
>          kubernetes.io/os: linux
>          node-role.kubernetes.io/infra: ''
>        proxy:
>          resources: null
>        replicas: 1
>        resources: null
>      type: kibana
> status:
>    collection:
>      logs:
>        fluentdStatus:
>          daemonSet: fluentd
>          nodes: {}
>          pods:
>            failed: []
>            notReady: []
>            ready: []
>        rsyslogStatus:
>          Nodes: null
>        daemonSet: ''
>          pods: null
>    curation:
>      curatorStatus:
>        - clusterCondition:
>            curator-1572924600-pwbf8:
>              - lastTransitionTime: '2019-11-05T03:30:01Z'
>                message: '0/7 nodes are available: 7 node(s) didn''t match node selector.'
>                reason: Unschedulable
>                status: 'True'
>                type: Unschedulable
>          cronJobs: curator
>          schedules: 30 3 * * *
>          suspended: false
>    logStore:
>     elasticsearchStatus:
>        - ShardAllocationEnabled: shard allocation unknown
>          cluster:
>            numDataNodes: 0
>            initializingShards: 0
>            numNodes: 0
>            activePrimaryShards: 0
>            status: cluster health unknown
>            pendingTasks: 0
>            relocatingShards: 0
>            activeShards: 0
>            unassignedShards: 0
>          clusterName: elasticsearch
>          nodeConditions:
>            elasticsearch-cdm-wgsf9ygw-1:
>         - lastTransitionTime: '2019-11-04T22:33:32Z'
>                message: '0/7 nodes are available: 7 node(s) didn''t match node selector.'
>                reason: Unschedulable
>                status: 'True'
>                type: Unschedulable
>          elasticsearch-cdm-wgsf9ygw-2:
>              - lastTransitionTime: '2019-11-04T22:33:33Z'
>                message: '0/7 nodes are available: 7 node(s) didn''t match node selector.'
>                reason: Unschedulable
>                status: 'True'
>                type: Unschedulable
>          nodeCount: 2
>          pods:
>            client:
>              failed: []
>              notReady:
>                - elasticsearch-cdm-wgsf9ygw-1-6f49f466dc-57dbk
>                - elasticsearch-cdm-wgsf9ygw-2-5777666679-2z4ph
>              ready: []
>            data:
>              failed: []
>              notReady:
>                - elasticsearch-cdm-wgsf9ygw-1-6f49f466dc-57dbk
>                - elasticsearch-cdm-wgsf9ygw-2-5777666679-2z4ph
>              ready: []
>            master:
>              failed: []
>              notReady:
>                - elasticsearch-cdm-wgsf9ygw-1-6f49f466dc-57dbk
>                - elasticsearch-cdm-wgsf9ygw-2-5777666679-2z4ph
>              ready: []
>    visualization:
>      kibanaStatus:
>        - clusterCondition:
>            kibana-99dc6bb95-5848h:
>              - lastTransitionTime: '2019-11-04T22:00:49Z'
>                message: '0/7 nodes are available: 7 node(s) didn''t match node selector.'
>                reason: Unschedulable
>                status: 'True'
>                type: Unschedulable
>            kibana-fb96dc875-wk4w5:
>              - lastTransitionTime: '2019-11-04T22:33:26Z'
>                message: '0/7 nodes are available: 7 node(s) didn''t match node selector.'
>                reason: Unschedulable
>                status: 'True'
>          type: Unschedulable
>          deployment: kibana
>          pods:
>            failed: []
>            notReady:
>             - kibana-99dc6bb95-5848h
>              - kibana-fb96dc875-wk4w5
>            ready: []
>          replicaSets:
>            - kibana-5d77fb4b85
>            - kibana-99dc6bb95
>            - kibana-fb96dc875
>          replicas: 1
> -------
> 
> The 2 Infra nodes are labeled corrcetly  : node-role.kubernetes.io/infra: ''.
> -------------
> [mohamed hamouch-capgemini com clientvm 0 ~]$ oc get nodes --show-labels
> NAME                                            STATUS   ROLES          AGE VERSION             LABELS
> ip-10-0-130-209.eu-central-1.compute.internal   Ready    master         33h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1a,kubernetes.io/hostname=ip-10-0-130-209,node-role.kubernetes.io/master=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-134-187.eu-central-1.compute.internal   Ready    worker         33h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1a,kubernetes.io/hostname=ip-10-0-134-187,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-141-221.eu-central-1.compute.internal   Ready    infra,worker   31h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1a,infra=infra,kubernetes.io/hostname=ip-10-0-141-221,node-role.kubernetes.io/infra=,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-150-157.eu-central-1.compute.internal   Ready    worker         33h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1b,kubernetes.io/hostname=ip-10-0-150-157,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-152-34.eu-central-1.compute.internal    Ready    master         33h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1b,kubernetes.io/hostname=ip-10-0-152-34,node-role.kubernetes.io/master=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-159-5.eu-central-1.compute.internal     Ready    infra,worker   31h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1b,infra=infra,kubernetes.io/hostname=ip-10-0-159-5,node-role.kubernetes.io/infra=,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> ip-10-0-165-162.eu-central-1.compute.internal   Ready    master         33h   v1.13.4+a80aad556   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1c,kubernetes.io/hostname=ip-10-0-165-162,node-role.kubernetes.io/master=,node.openshift.io/os_id=rhcos,node.openshift.io/os_version=4.1
> -------------
> 
> The logging pods (elasticsearch, kibana and curator):
> [mohamed hamouch-capgemini com clientvm 0 ~]$ oc get pods --show-labels -n openshift-logging
> NAME                                            READY   STATUS    RESTARTS   AGE LABELS
> cluster-logging-operator-bd64d698d-8xzxw        1/1     Running   0          25h   name=cluster-logging-operator,pod-template-hash=bd64d698d
> curator-1572924600-pwbf8                        0/1     Pending   0          18h   component=curator,controller-uid=8cc4c661-ff7c-11e9-b9e8-0226c8b0ff44,job-name=curator-1572924600,logging-infra=curator,provider=openshift
> elasticsearch-cdm-wgsf9ygw-1-6f49f466dc-57dbk   0/2     Pending   0          23h   cluster-name=elasticsearch,component=elasticsearch,es-node-client=true,es-node-data=true,es-node-master=true,node-name=elasticsearch-cdm-wgsf9ygw-1,pod-template-hash=6f49f466dc,tuned.openshift.io/elasticsearch=true
> elasticsearch-cdm-wgsf9ygw-2-5777666679-2z4ph   0/2     Pending   0          23h   cluster-name=elasticsearch,component=elasticsearch,es-node-client=true,es-node-data=true,es-node-master=true,node-name=elasticsearch-cdm-wgsf9ygw-2,pod-template-hash=5777666679,tuned.openshift.io/elasticsearch=true
> kibana-99dc6bb95-5848h                          0/2     Pending   0          24h   component=kibana,logging-infra=kibana,pod-template-hash=99dc6bb95,provider=openshift
> kibana-fb96dc875-wk4w5                          0/2     Pending   0          23h   component=kibana,logging-infra=kibana,pod-template-hash=fb96dc875,provider=openshift
> ---------
> 
> At what part should I look  to fix this issue?
> 
> Thank you very much for your help.
> 
> haed98.
> 
> _______________________________________________
> users mailing list
> users lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> 

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]