[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: problem with installation of Okd 3.11



Yeah, i did not included in installiation grafana, hawkular etc. I have ELK stack with okd integration..

I setup cluster on CentOS 7.5 with checked out openshift-ansible playbook’s repo “release-3.11” and inventory file looks like this (attached text file).

 

-- 

 

Best,

 

Alexander Kozhemyakin

 

From: <users-bounces lists openshift redhat com> on behalf of Bobby Corpus <bobby corpus icloud com>
Date: Tuesday, 27 November 2018 at 15:46
To: Erekle Magradze <erekle magradze recogizer de>
Cc: "users lists openshift redhat com" <users lists openshift redhat com>
Subject: Re: problem with installation of Okd 3.11

 

I managed to install OKD 3.11 as of today. I'll share my experience:

 

My setup:

 

NAME    STATUS    ROLES          AGE       VERSION

 

node1   Ready     infra,master   12h       v1.11.0+d4cacc0

 

node2   Ready     infra          12h       v1.11.0+d4cacc0

 

node3   Ready     compute        12h       v1.11.0+d4cacc0

 

 

I needed to do a lot of workarounds including restarting all nodes.

 

I bumped into this error:

 

- FAILED - RETRYING: Wait for sync DS to set annotations on master nodes (180 retries left).

- FAILED - RETRYING: Wait for sync DS to set annotations on master nodes (179 retries left).

 

and into this error

 

- TASK [openshift_cluster_monitoring_operator : Wait for the ServiceMonitor CRD to be created] ********************************

- Monday 26 November 2018  19:31:50 -0500 (0:00:01.251)       0:11:17.509 *******

- FAILED - RETRYING: Wait for the ServiceMonitor CRD to be created (30 retries left).

- FAILED - RETRYING: Wait for the ServiceMonitor CRD to be created (29 retries left).

 

and i noticed that 

 

- No networks found in /etc/cni/net.d

 

There was no file in this directory. In my other successful installations of an OpenShift enterprise cluster v3.11, I have a file at that directory called 80-openshift-network.conf

 

To workaround this, I created the file /etc/cni/net.d/80-openshift-network.conf with contents

 

{

  "cniVersion": "0.2.0",

  "name": "openshift-sdn",

  "type": "openshift-sdn"

}

 

 

 

I then copied this file to all nodes in the cluster.

 

Then I restarted all the nodes and I did ansible-playbook deploy_cluster again. This time I was lucky. The install proceeded.

 

However, I have to check that:

 

1. The file

/etc/dnsmasq.d/origin-upstream-dns.conf

 

is present and has the information of my upstream dns:

 

For example,

 

cat  /etc/dnsmasq.d/origin-upstream-dns.conf

 

server=8.8.8.8

 

 

 

If not present, create I created it and restarted dnsmasq

 

2. That i'm able to resolve using the dnsmasq in all nodes,

For example,

 

ansible nodes -a 'dig yahoo.com'

 

3. The file /etc/resolv.conf is correct

 

cat /etc/resolv.conf

 

# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

 

search cluster.local

nameserver xxx.xxx.xxx.xxx

 

where xxx.xxx.xxx.xxx is the ip of the current node.

 

4. The file /etc/cni/net.d/80-openshift-network.conf is present and has contents

 

{

  "cniVersion": "0.2.0",

  "name": "openshift-sdn",

  "type": "openshift-sdn"

}

 

5. My grafana pod did not run. The error was "No API token found for service account "grafana", retry after the token is automatically created and added to the service account"

 

oc describe sa grafana -n openshift-monitoring

Name:                grafana

Namespace:           openshift-monitoring

Labels:              <none>

Annotations:         serviceaccounts.openshift.io/oauth-redirectreference.grafana={"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana"}}

Image pull secrets:  grafana-dockercfg-dnvvm

Mountable secrets:   grafana-dockercfg-dnvvm

                     grafana-token-6sw2j

Tokens:              grafana-token-6sw2j

Events:              <none>

 

There was one token and I deleted it using:

 

oc delete secret grafana-token-6sw2j

 

Two new tokens were generated and grafana pod was successfully run.

 

Best regards,

 

Bobby

 

 

On Nov 26, 2018, at 09:46 PM, Erekle Magradze <erekle magradze recogizer de> wrote:

Hello Guys,

Did anyone face the similar problem? it says that network component of K8S has some problem in installation.

So, I am failing at the final steps of command

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

problem looks like this

TASK [openshift_node_group : Wait for sync DS to set annotations on master nodes] ***********************************************************************************************************************************************************
FAILED - RETRYING: Wait for sync DS to set annotations on master nodes (180 retries left).
FAILED - RETRYING: Wait for sync DS to set annotations on master nodes (179 retries left).
...
...
...

The final message looks like this

fatal: [os-master.apps.mydomain.net]: FAILED! => {"attempts": 180, "changed": false, "results": {"cmd": "/usr/bin/oc get node --selector= -o json -n default", "results": [{"apiVersion": "v1", "items": [{"apiVersion": "v1", "kind": "Node", "metadata": {"annotations": {"volumes.kubernetes.io/controller-managed-attach-detach": "true"}, "creationTimestamp": "2018-11-25T21:49:08Z", "labels": {"beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "os-master.apps.mydomain.net"}, "name": "os-master.apps.mydomain.net", "namespace": "", "resourceVersion": "33274", "selfLink": "/api/v1/nodes/os-master.apps.mydomain.net", "uid": "efd782f8-f0fb-11e8-a72f-001a4a160102"}, "spec": {}, "status": {"addresses": [{"address": "172.31.1.71", "type": "InternalIP"}, {"address": "os-master.apps.mydomain.net", "type": "Hostname"}], "allocatable": {"cpu": "16", "hugepages-2Mi": "0", "memory": "32676788Ki", "pods": "250"}, "capacity": {"cpu": "16", "hugepages-2Mi": "0", "memory": "32779188Ki", "pods": "250"}, "conditions": [{"lastHeartbeatTime": "2018-11-26T05:44:12Z", "lastTransitionTime": "2018-11-25T21:49:08Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk"}, {"lastHeartbeatTime": "2018-11-26T05:44:12Z", "lastTransitionTime": "2018-11-25T21:49:08Z", "message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure"}, {"lastHeartbeatTime": "2018-11-26T05:44:12Z", "lastTransitionTime": "2018-11-25T21:49:08Z", "message": "kubelet has no disk pressure", "reason": "KubeletHasNoDiskPressure", "status": "False", "type": "DiskPressure"}, {"lastHeartbeatTime": "2018-11-26T05:44:12Z", "lastTransitionTime": "2018-11-25T21:49:08Z", "message": "kubelet has sufficient PID available", "reason": "KubeletHasSufficientPID", "status": "False", "type": "PIDPressure"}, {"lastHeartbeatTime": "2018-11-26T05:44:12Z", "lastTransitionTime": "2018-11-25T21:49:08Z", "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", "reason": "KubeletNotReady", "status": "False", "type": "Ready"}], "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"names": ["docker.io/openshift/origin-node sha256:8a8e6341cc3af32953ee2a313a5f0973bed538f0326591c48188f4de9617c992", "docker.io/openshift/origin-node:v3.11.0"], "sizeBytes": 1157253404}, {"names": ["docker.io/openshift/origin-control-plane sha256:181069f5d67cc2ba8d9b3c80efab8eda107eb24140f2d9f4a394cdb164c28a86", "docker.io/openshift/origin-control-plane:v3.11.0"], "sizeBytes": 818390387}, {"names": ["docker.io/openshift/origin-pod sha256:1641b78e32c100938b2db51088e284568a056a3716492db78335a3e35be03853", "docker.io/openshift/origin-pod:v3.11.0"], "sizeBytes": 253795602}, {"names": ["quay.io/coreos/etcd sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948", "quay.io/coreos/etcd:v3.2.22"], "sizeBytes": 37269372}], "nodeInfo": {"architecture": "amd64", "bootID": "2c873547-4546-45ae-af3b-bd685a7d556f", "containerRuntimeVersion": "docker://1.13.1", "kernelVersion": "3.10.0-862.14.4.el7.x86_64", "kubeProxyVersion": "v1.11.0+d4cacc0", "kubeletVersion": "v1.11.0+d4cacc0", "machineID": "159ec545080a4b849d70b5b10694bd1a", "operatingSystem": "linux", "osImage": "CentOS Linux 7 (Core)", "systemUUID": "159EC545-080A-4B84-9D70-B5B10694BD1A"}}}], "kind": "List", "metadata": {"resourceVersion": "", "selfLink": ""}}], "returncode": 0}, "state": "list"}

In /var/logs/messages of the master node I see the following

Nov 26 06:55:55 os-master origin-node: E1126 06:55:53.495294    5353 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 26 06:55:58 os-master origin-node: W1126 06:55:58.496250    5353 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 26 06:55:58 os-master origin-node: E1126 06:55:58.496391    5353 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 26 06:56:03 os-master origin-node: W1126 06:56:03.497529    5353 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 26 06:56:03 os-master origin-node: E1126 06:56:03.497667    5353 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 26 06:56:09 os-master origin-node: I1126 06:56:07.931551    5353 container_manager_linux.go:428] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Nov 26 06:56:09 os-master origin-node: W1126 06:56:08.498812    5353 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 26 06:56:09 os-master origin-node: E1126 06:56:08.498929    5353 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 26 06:56:13 os-master origin-node: W1126 06:56:13.499781    5353 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 26 06:56:13 os-master origin-node: E1126 06:56:13.500567    5353 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Can you please advise what to do in this case and how to solve the problem?

Many Thanks in advance

Best Regards

Erekle

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf100
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww22980\viewh15080\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0

\f0\fs24 \cf0 [masters]\
dca-master[1:3]\
\
[etcd]\
dca-master[1:3]\
\
[nodes]\
dca-master[1:3] openshift_node_group_name='node-config-master'\
dca-infra[1:2] openshift_node_group_name='node-config-infra'\
dca-node[1:3] openshift_node_group_name='node-config-compute'\
\
[nfs]\
\
[lb]\
\
[new_masters]\
\
[new_nodes]\
\
\
[OSEv3:children]\
masters\
nodes\
etcd\
#lb\
#nfs\
#new_nodes\
#new_masters\
\
\
[OSEv3:vars]\
openshift_master_default_subdomain=XXXX\
openshift_install_examples=true\
openshift_master_named_certificates=[\{"certfile": "/opt/origin/clusters/XX/certs/XXX.crt", "keyfile": "/opt/origin/clusters/XX/XXX.key", "cafile": "/opt/origin/certs/ca-web.crt", "names": ['XXXXX']\}]\
openshift_release='3.11'\
openshift_pkg_version='-3.11.0'\
openshift_image_tag='v3.11.0'\
debug_level=2\
openshift_master_cluster_public_hostname=XXXX\
openshift_use_openshift_sdn=true\
os_sdn_network_plugin_name='redhat/openshift-ovs-networkpolicy'\
openshift_hosted_router_replicas=2\
openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'\
openshift_hosted_routers=XXXX\
openshift_hosted_manage_registry=false\
openshift_hosted_manage_registry_console=false\
openshift_metrics_install_metrics=false\
openshift_master_cluster_hostname=XXX\
ansible_become=true\
openshift_master_overwrite_named_certificates=true\
openshift_master_open_ports=[\{"service":"zabbix","port":"10050/tcp"\}]\
openshift_node_open_ports=[\{"service":"zabbix","port":"10050/tcp"\}]\
openshift_node_dnsmasq_additional_config_file='/opt/origin/files/dnsmasq-additional-config'\
openshift_deployment_type=origin\
openshift_master_identity_providers=XXXX\
openshift_master_ldap_ca_file='/opt/origin/certs/DigiCertCA.crt'\
ansible_ssh_user=XXXXX\
osn_storage_plugin_deps=['nfs','iscsi']\
openshift_master_cluster_method=native\
openshift_master_api_port=8443\
openshift_master_console_port=8443\
openshift_clock_enabled=true\
osm_use_cockpit=true\
osm_cockpit_plugins=['cockpit-kubernetes']\
openshift_docker_options="--log-driver=journald --signature-verification=false -l warn --ipv6=false --storage-driver=overlay2"\
openshift_docker_selinux_enabled=True\
openshift_enable_service_catalog=false\
template_service_broker_install=false\
openshift_cluster_monitoring_operator_install=false\
penshift_additional_repos=[\{'id': 'centos-openshift-origin-311', 'name': 'CentOS-OpenShift-Origin-311', 'baseurl': 'http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin311/', 'enabled': 1, 'gpgcheck': 1, 'gpgkey': 'https://raw.githubusercontent.com/CentOS-PaaS-SIG/centos-release-paas-common/master/RPM-GPG-KEY-CentOS-SIG-PaaS'\}]}

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]