[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: 3.10 openshift-ansible install is failing - cni not configured



Thanks to Alexander, I found out that a major part of my problem is that my nodes have a poor internet connection and pulling images from docker.io is either slow or docker.io reports that the manifest is not found. Pulling the images locally, pushing them to a local registry and changing system_images_registry to my local registry helped a lot.

However, it seems to consistently fail the first time I run deploy_cluster.yml (the control plane pods do not come up completely - they come up, become ready and then are deleted and started over again in a cycle every 5 seconds or so). If I run deploy_cluster.yml again (without changing anything) the deploy seems to go better the second time.

I am unable to enable metrics. First, the ansible installer seems to want to get the metrics images with the tag v3.10.0 which doesn't exist. I tried pulling them down, labeling latest as v3.10.0 and pushing them to my local registry, but the image for openshift/origin-metrics-schema-installer doesn't seem to exist with any label.

Anyway, thanks again Alexander - this is significant progress even though I'm definitely not ready to move off of 3.9.0 yet

On Tue, Sep 11, 2018 at 1:42 PM Peter Heitman <peter heitman us> wrote:
Thanks for the reply. I was pinning the release only because I was updating a working inventory from 3.9 and forgot that I had pinned that release to avoid upgrading to 3.10. I've updated the inventory to set openshift_release="3.10" and commented out openshift_image_tag and openshift_pkg_version so that the ansible scripts will derive the correct values. I have re-run the installer using a fresh version of the master and minion VMs (CentOS 7.5 with docker installed). I get the same error. The output of systemctl status origin-node on the master is:

● origin-node.service - OpenShift Node
   Loaded: loaded (/etc/systemd/system/origin-node.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-09-11 10:31:51 PDT; 3min 29s ago
     Docs: https://github.com/openshift/origin
 Main PID: 21183 (hyperkube)
   CGroup: /system.slice/origin-node.service
           └─21183 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 --allow-privileged=true --anonymous-auth=true --authentication-token-webhook=true --authentication-token-webhook-cache-ttl=5m --authorization-mode=Webhook --authorization-webhook-cache-authorized-ttl=5m --authorization-webhook-cache-unauthorized-ttl=5m --bootstrap-kubeconfig=/etc/origin/node/bootstrap.kubeconfig --cadvisor-port=0 --cert-dir=/etc/origin/node/certificates --cgroup-driver=systemd --client-ca-file=/etc/origin/node/client-ca.crt --cluster-dns=10.93.233.126 --cluster-domain=cluster.local --container-runtime-endpoint=/var/run/dockershim.sock --containerized=false --enable-controller-attach-detach=true --experimental-dockershim-root-directory=/var/lib/dockershim --fail-swap-on=false --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --file-check-frequency=0s --healthz-bind-address= --healthz-port=0 --host-ipc-sources=api --host-ipc-sources=file --host-network-sources=api --host-network-sources=file --host-pid-sources=api --host-pid-sources=file --hostname-override= --http-check-frequency=0s --image-service-endpoint=/var/run/dockershim.sock --iptables-masquerade-bit=0 --kubeconfig=/etc/origin/node/node.kubeconfig --max-pods=250 --network-plugin=cni --node-ip= --pod-infra-container-image=docker.io/openshift/origin-pod:v3.10.0 --pod-manifest-path=/etc/origin/node/pods --port=10250 --read-only-port=0 --register-node=true --root-dir=/var/lib/origin/openshift.local.volumes --rotate-certificates=true --tls-cert-file= --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA --tls-min-version=VersionTLS12 --tls-private-key-file=

Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:17.667696   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:17.668264   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: W0911 10:35:18.518516   21183 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.518716   21183 kubelet.go:2143] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.667894   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.669169   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:18.670127   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:19.669734   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:19.670769   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused
Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 10:35:19.671644   21183 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Get https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: dial tcp 10.93.233.126:8443: getsockopt: connection refused


On Tue, Sep 11, 2018 at 10:41 AM Alexander Bartilla <alexander bartilla cloudwerkstatt com> wrote:
Hi Peter,

Is there a reason behind pinning the release, image_tag and pkg_version variables to this release version? I would recommend you use just 3.10, this will ensure that you get the latest version of Openshift installed

Futhermore I found several bugreports with this issue:


Some more logs from the node would help to troubleshoot the problem.

Best regards,
Alexander

On Tue, Sep 11, 2018 at 3:50 PM, Peter Heitman <peter heitman us> wrote:
I am attempting to use the openshift-ansible installer for 3.10 to deploy openshift on 1 master and 3 minions. I am using the same inventory I have been using for 3.9 with the changes shown below. I'm consistently hitting a problem with the control plane pods not appearing. Looking in to it, it seems that the cni plugin is not being configured properly. From systemctl status origin-node, I see the following:

E0911 06:19:25.821170   18922 kubelet.go:2143] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Is there something I need to add to my 3.10 inventory to address this? Are there other workarounds?

- openshift_release=v3.9.0
+ openshift_release=v3.10.0

- openshift_image_tag=v3.9.0
- openshift_image_tag=v3.10.0
+ openshift_pkg_version=-3.10.0
+ openshift_pkg_version=-3.9.0

- openshift_metrics_image_version=v3.9
+ openshift_metrics_image_version=v3.10

- [masters]
- <master fqdn> openshift_node_labels="{'node-role.kubernetes.io/master': 'true', 'node-role.kubernetes.io/infra': 'true'}" openshift_schedulable=true

+ [masters]
+ <master fqdn>

+ [masters:vars]
+ #openshift_node_group_name="node-config-master"
+ openshift_node_group_name="node-config-master-infra"
+ openshift_schedulable=true

- [compute-nodes]
- <minion1 fqdn> openshift_node_labels="{'node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true
- <minion2 fqdn> openshift_node_labels="{'node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true
- <minion3 fqdn> openshift_node_labels="{'node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true

+ [compute-nodes]
+ <minion1 fqdn>
+ <minion2 fqdn>
+ <minion3 fqdn>

+ [compute-nodes:vars]
+ openshift_node_group_name="node-config-compute"
+ openshift_schedulable=true


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users




--

Alexander Bartilla

IT-Consultant

Cloudwerkstatt GmbH - Lassallestraße 7b – A-1020 Wien

+43-660-8989058

alexander bartilla cloudwerkstatt com


id:image001 png 01D24B57 D1D08F70


Cloudwerkstatt GmbH - Lassallestraße 7b - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]