[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ETCD no longer starting during install



From log of POD k8s_controllers_master-controllers-host-10-1-241-74_kube-system,  it seemed that master config wasn't compatible with images pullled so that etcd wasn't started.

The detailed log is as below.

E1030 14:26:51.590478       1 helpers.go:134] Encountered config error json: unknown field "masterCount" in object *config.MasterConfig, raw JSON:
{"admissionConfig":{"pluginConfig":{"BuildDefaults":{"configuration":{"apiVersion":"v1","env":[],"kind":"BuildDefaultsConfig","resources":{"limits":{},"requests":{}}}},"BuildOverrides":{"configuration":{"apiVersion":"v1","kind":"BuildOverridesConfig"}},"openshift.io/ImagePolicy":{"configuration":{"apiVersion":"v1","executionRules":[{"matchImageAnnotations":[{"key":"images.openshift.io/deny-execution","value":"true"}],"name":"execution-denied","onResources":[{"resource":"pods"},{"resource":"builds"}],"reject":true,"skipOnResolutionFailure":true}],"kind":"ImagePolicyConfig"}}}},"aggregatorConfig":{"proxyClientInfo":{"certFile":"aggregator-front-proxy.crt","keyFile":"aggregator-front-proxy.key"}},"apiLevels":["v1"],"apiVersion":"v1","authConfig":{"requestHeader":{"clientCA":"front-proxy-ca.crt","clientCommonNames":["aggregator-front-proxy"],"extraHeaderPrefixes":["X-Remote-Extra-"],"groupHeaders":["X-Remote-Group"],"usernameHeaders":["X-Remote-User"]}},"controllerConfig":{"election":{"lockName":"openshift-master-controllers"},"serviceServingCert":{"signer":{"certFile":"service-signer.crt","keyFile":"service-signer.key"}}},"controllers":"*","corsAllowedOrigins":["(?i)//127\\.0\\.0\\.1(:|\\z)","(?i)//localhost(:|\\z)","(?i)//10\\.1\\.241\\.74(:|\\z)","(?i)//kubernetes\\.default(:|\\z)","(?i)//kubernetes\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//kubernetes(:|\\z)","(?i)//openshift\\.default(:|\\z)","(?i)//openshift\\.default\\.svc(:|\\z)","(?i)//172\\.30\\.0\\.1(:|\\z)","(?i)//paas\\-dev\\.dataos\\.io(:|\\z)","(?i)//openshift\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//host\\-10\\-1\\-241\\-74(:|\\z)","(?i)//kubernetes\\.default\\.svc(:|\\z)","(?i)//openshift(:|\\z)"],"dnsConfig":{"bindAddress":"0.0.0.0:8053","bindNetwork":"tcp4"},"etcdClientInfo":{"ca":"master.etcd-ca.crt","certFile":"master.etcd-client.crt","keyFile":"master.etcd-client.key","urls":["https://host-10-1-241-74:2379"]},"etcdStorageConfig":{"kubernetesStoragePrefix":"kubernetes.io","kubernetesStorageVersion":"v1","openShiftStoragePrefix":"openshift.io","openShiftStorageVersion":"v1"},"imageConfig":{"format":"docker.io/openshift/origin-${component}:${version}","latest":false},"imagePolicyConfig":{"internalRegistryHostname":"docker-registry.default.svc:5000"},"kind":"MasterConfig","kubeletClientInfo":{"ca":"ca-bundle.crt","certFile":"master.kubelet-client.crt","keyFile":"master.kubelet-client.key","port":10250},"kubernetesMasterConfig":{"apiServerArguments":{"storage-backend":["etcd3"],"storage-media-type":["application/vnd.kubernetes.protobuf"]},"controllerArguments":{"cluster-signing-cert-file":["/etc/origin/master/ca.crt"],"cluster-signing-key-file":["/etc/origin/master/ca.key"],"pv-recycler-pod-template-filepath-hostpath":["/etc/origin/master/recycler_pod.yaml"],"pv-recycler-pod-template-filepath-nfs":["/etc/origin/master/recycler_pod.yaml"]},"masterCount":1,"masterIP":"10.1.241.74","podEvictionTimeout":null,"proxyClientInfo":{"certFile":"master.proxy-client.crt","keyFile":"master.proxy-client.key"},"schedulerArguments":null,"schedulerConfigFile":"/etc/origin/master/scheduler.json","servicesNodePortRange":"","servicesSubnet":"172.30.0.0/16","staticNodeNames":[]},"masterClients":{"externalKubernetesClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":400,"contentType":"application/vnd.kubernetes.protobuf","qps":200},"externalKubernetesKubeConfig":"","openshiftLoopbackClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":600,"contentType":"application/vnd.kubernetes.protobuf","qps":300},"openshiftLoopbackKubeConfig":"openshift-master.kubeconfig"},"masterPublicURL":"https://paas-dev.dataos.io:8443","networkConfig":{"clusterNetworks":[{"cidr":"10.128.0.0/14","hostSubnetLength":9}],"externalIPNetworkCIDRs":["0.0.0.0/0"],"networkPluginName":"redhat/openshift-ovs-networkpolicy","serviceNetworkCIDR":"172.30.0.0/16"},"oauthConfig":{"assetPublicURL":"https://paas-dev.dataos.io:8443/console/","grantConfig":{"method":"auto"},"identityProviders":[{"challenge":true,"login":true,"mappingMethod":"claim","name":"allow_all","provider":{"apiVersion":"v1","kind":"AllowAllPasswordIdentityProvider"}}],"masterCA":"ca-bundle.crt","masterPublicURL":"https://paas-dev.dataos.io:8443","masterURL":"https://paas-dev.dataos.io:8443","sessionConfig":{"sessionMaxAgeSeconds":3600,"sessionName":"ssn","sessionSecretsFile":"/etc/origin/master/session-secrets.yaml"},"tokenConfig":{"accessTokenMaxAgeSeconds":86400,"authorizeTokenMaxAgeSeconds":500}},"pauseControllers":false,"policyConfig":{"bootstrapPolicyFile":"/etc/origin/master/policy.json","openshiftInfrastructureNamespace":"openshift-infra","openshiftSharedResourcesNamespace":"openshift"},"projectConfig":{"defaultNodeSelector":"node-role.kubernetes.io/compute=true","projectRequestMessage":"","projectRequestTemplate":"","securityAllocator":{"mcsAllocatorRange":"s0:/2","mcsLabelsPerProject":5,"uidAllocatorRange":"1000000000-1999999999/10000"}},"routingConfig":{"subdomain":"paas-dev.dataos.io"},"serviceAccountConfig":{"limitSecretReferences":false,"managedNames":["default","builder","deployer"],"masterCA":"ca-bundle.crt","privateKeyFile":"serviceaccounts.private.key","publicKeyFiles":["serviceaccounts.public.key"]},"servingInfo":{"bindAddress":"0.0.0.0:8443","bindNetwork":"tcp4","certFile":"master.server.crt","clientCA":"ca.crt","keyFile":"master.server.key","maxRequestsInFlight":500,"requestTimeoutSeconds":3600},"volumeConfig":{"dynamicProvisioningEnabled":true}}
I1030 14:26:51.597367       1 plugins.go:84] Registered admission plugin "NamespaceLifecycle"
I1030 14:26:51.597442       1 plugins.go:84] Registered admission plugin "Initializers"
I1030 14:26:51.597504       1 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I1030 14:26:51.597566       1 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I1030 14:26:51.597628       1 plugins.go:84] Registered admission plugin "AlwaysAdmit"
I1030 14:26:51.597674       1 plugins.go:84] Registered admission plugin "AlwaysPullImages"
I1030 14:26:51.597699       1 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I1030 14:26:51.597727       1 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds"
I1030 14:26:51.597815       1 plugins.go:84] Registered admission plugin "AlwaysDeny"
I1030 14:26:51.597868       1 plugins.go:84] Registered admission plugin "EventRateLimit"
I1030 14:26:51.597895       1 plugins.go:84] Registered admission plugin "DenyEscalatingExec"
I1030 14:26:51.597912       1 plugins.go:84] Registered admission plugin "DenyExecOnPrivileged"
I1030 14:26:51.597931       1 plugins.go:84] Registered admission plugin "ExtendedResourceToleration"
I1030 14:26:51.597959       1 plugins.go:84] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I1030 14:26:51.597989       1 plugins.go:84] Registered admission plugin "ImagePolicyWebhook"
I1030 14:26:51.598008       1 plugins.go:84] Registered admission plugin "LimitRanger"
I1030 14:26:51.598041       1 plugins.go:84] Registered admission plugin "NamespaceAutoProvision"
I1030 14:26:51.598062       1 plugins.go:84] Registered admission plugin "NamespaceExists"
I1030 14:26:51.598084       1 plugins.go:84] Registered admission plugin "NodeRestriction"
I1030 14:26:51.598111       1 plugins.go:84] Registered admission plugin "PersistentVolumeLabel"
I1030 14:26:51.598135       1 plugins.go:84] Registered admission plugin "PodNodeSelector"
I1030 14:26:51.598161       1 plugins.go:84] Registered admission plugin "PodPreset"
I1030 14:26:51.598181       1 plugins.go:84] Registered admission plugin "PodTolerationRestriction"
I1030 14:26:51.598200       1 plugins.go:84] Registered admission plugin "ResourceQuota"
I1030 14:26:51.598228       1 plugins.go:84] Registered admission plugin "PodSecurityPolicy"
I1030 14:26:51.598258       1 plugins.go:84] Registered admission plugin "Priority"
I1030 14:26:51.598285       1 plugins.go:84] Registered admission plugin "SecurityContextDeny"
I1030 14:26:51.598318       1 plugins.go:84] Registered admission plugin "ServiceAccount"
I1030 14:26:51.598340       1 plugins.go:84] Registered admission plugin "DefaultStorageClass"
I1030 14:26:51.598358       1 plugins.go:84] Registered admission plugin "PersistentVolumeClaimResize"
I1030 14:26:51.598397       1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection"
Invalid MasterConfig /etc/origin/master/master-config.yaml
  etcdClientInfo.ca: Invalid value: "/etc/origin/master/master.etcd-ca.crt": could not read file: stat /etc/origin/master/master.etcd-ca.crt: no such file or directory



On 2018/10/30 17:23, Yu Wei wrote:

I met the same problem and found that etcd was skipped as below,

TASK [openshift_control_plane : Establish the default bootstrap kubeconfig for masters] ******************************************************
changed: [host-10-1-241-74] => (item=/etc/origin/node/bootstrap.kubeconfig)
changed: [host-10-1-241-74] => (item=/etc/origin/node/node.kubeconfig)

TASK [openshift_control_plane : Check status of control plane image pre-pull] ****************************************************************
changed: [host-10-1-241-74]

TASK [openshift_control_plane : Check status of etcd image pre-pull] *************************************************************************
skipping: [host-10-1-241-74]

TASK [openshift_control_plane : Start and enable self-hosting node] **************************************************************************
changed: [host-10-1-241-74]


Is this playbooks issue?
Thanks,
Jared
Interested in big data, cloud computing
On 2018/10/30 15:47, marc schlegel sdv-it de wrote:
Hello everyone

I am facing an issue with the installer for 3.10 (and 3.11 has the same problem)

It started around 2-3 weeks ago, since I wasnt able to run the Ansible installer successfully...even when using a tag from 3.10 in the installer-repo that worked before.
The control-plane is not starting and what I could figure out is, that etcd is not started anywhere. The last time it was working, when running "docker ps" on the master (single master, multi node system) I saw about 4 running containers...one of them was the etcd.
Now, there are only 2 of them and no etcd anywhere.

https://github.com/lostiniceland/devops/tree/master/openshift
This is my current Vagrant-Setup which uses a simple script to check-out the openshift-installer, prepare Vagrant and run the Ansible files.

I thought that I might have broken my inventory or script but I double checked everything and I new that this setup was working before.
Now at work, the collegue who is maining our test-cluster has the same problem when upgrading from 3.9 to 3.10...no etcd anywhere. It seems restarting the docker-daemon fixes for our test-cluster.

If anyone could look into this would be very appreciated.
What I find odd is the fact that even a before working tag like openshit-ansible-3.10.53-1 is now broken. The only reasons I can think of, is the used Dockerimages have been updated or the installed Version of Docker is somewhat broken.


best regards
Marc

_______________________________________________
users mailing list
users lists openshift redhat com
https://nam02.safelinks.protection.outlook.com/?url="">

_______________________________________________
users mailing list
users lists openshift redhat com
https://nam04.safelinks.protection.outlook.com/?url="">

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]