[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: Unable to connect with service using mysql-ephemeral template



But OpenShift uses it's own DNS inside isn't it? So has this something to do with my cluster.
How do I have to debug/fix this issue?
Some more logs

Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923436   13766 proxier.go:421] Adding new service "default/kubernetes:https" at 172.30.0.1:443/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923524   13766 proxier.go:421] Adding new service "default/kubernetes:dns" at 172.30.0.1:53/UDP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923552   13766 proxier.go:421] Adding new service "default/kubernetes:dns-tcp" at 172.30.0.1:53/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923576   13766 proxier.go:421] Adding new service "default/router:80-tcp" at 172.30.44.225:80/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923602   13766 proxier.go:421] Adding new service "default/router:443-tcp" at 172.30.44.225:443/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923625   13766 proxier.go:421] Adding new service "default/router:1936-tcp" at 172.30.44.225:1936/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923647   13766 proxier.go:421] Adding new service "test/mysql:mysql" at 172.30.222.94:3306/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923685   13766 proxier.go:421] Adding new service "test/postgresql-94-centos7:5432-tcp" at 172.30.200.135:5432/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923713   13766 proxier.go:421] Adding new service "default/docker-registry:5000-tcp" at 172.30.44.216:5000/TCP
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.923750   13766 proxier.go:565] Not syncing iptables until Services and Endpoints have been received from master
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926911   13766 proxier.go:484] Setting endpoints for "default/router:80-tcp" to [10.0.0.5:80]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926947   13766 proxier.go:484] Setting endpoints for "default/router:443-tcp" to [10.0.0.5:443]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926957   13766 proxier.go:484] Setting endpoints for "default/router:1936-tcp" to [10.0.0.5:1936]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926968   13766 proxier.go:484] Setting endpoints for "default/docker-registry:5000-tcp" to [10.1.0.3:5000]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926978   13766 proxier.go:484] Setting endpoints for "default/kubernetes:dns-tcp" to [178.xx.xx.xx:8053]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926986   13766 proxier.go:484] Setting endpoints for "default/kubernetes:https" to [178.xx.xx.xx:8443]
Jul  6 19:46:23 node01 origin-node: I0706 19:46:23.926995   13766 proxier.go:484] Setting endpoints for "default/kubernetes:dns" to [178.xx.xx.xx:8053]

My public master-server IP:
178.xx.xx.xx

My internal node-server IP:
10.0.0.5

Service IP of router:
172.30.44.225

Service IP of registry:
172.30.44.216




From: bparees redhat com
Date: Wed, 6 Jul 2016 13:45:31 -0400
Subject: Re: Unable to connect with service using mysql-ephemeral template
To: dencowboy hotmail com; ccoleman redhat com
CC: users lists openshift redhat com



On Wed, Jul 6, 2016 at 1:12 PM, Den Cowboy <dencowboy hotmail com> wrote:
I have an older environment on amazon (older images + 1.1.6v) and there it works for mysql:
I perform a lookup of my service IP:

nslookup 172.30.177.4                                                   
Server:         172.30.0.1                                                      
Address:        172.30.0.1#53                                                   
                                                                                
Non-authoritative answer:                                                       
4.177.30.172.in-addr.arpa       name = mysql.dev-activiti.svc.cluster.local.    
                                                                                
Authoritative answers can be found from:                                        
                                                                                
But when I perform the same on my environment on OVH (newer version ofcourse)
sh-4.2$ nslookup 172.30.222.94                                                  
Server:         213.186.33.xx                                                  
Address:        213.186.33.xx#53                                                
                                                                                
** server can't find 94.222.30.172.in-addr.arpa.: NXDOMAIN                                                                  

Than it's pointing to a wrong server (not the 172.30... of OpenShift)

​it sounds like your cluster DNS server/service is not functioning correctly.

 
I didn't saw an issue pending the installation and also /var/log/messages are telling nothing.

logs:
Jul  6 19:11:17 node01 origin-node: I0706 19:11:17.989899    4926 manager.go:1024] Using docker native exec to run cmd [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'] inside container {docker b4958c468b643b7ec7dc239569f73e2ea8568b6c6d7e4151cffd621c58db5778}
Jul  6 19:11:17 node01 journal: time="2016-07-06T19:11:17.990407769+02:00" level=info msg="{Action="" ID=b4958c468b643b7ec7dc239569f73e2ea8568b6c6d7e4151cffd621c58db5778, LoginUID=4294967295, PID=4926}"
Jul  6 19:11:17 node01 journal: time="2016-07-06T19:11:17.991021609+02:00" level=info msg="{Action="" LoginUID=4294967295, PID=4926}"
Jul  6 19:11:18 node01 origin-node: I0706 19:11:18.042239    4926 proxier.go:484] Setting endpoints for "test/mysql:mysql" to [10.1.0.2:3306]

It keeps showing this log every 10 seconds:
sing docker native exec to run cmd [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'] inside container ...

From: dencowboy hotmail com
To: bparees redhat com
Subject: RE: Unable to connect with service using mysql-ephemeral template
Date: Wed, 6 Jul 2016 17:03:56 +0000
CC: users lists openshift redhat com


I seem to have the same issue for my postgresdb:
nslookup 172.30.200.135
** server can't find 135.200.30.172.in-addr.arpa.: NXDOMAIN




From: bparees redhat com
Date: Wed, 6 Jul 2016 12:53:06 -0400
Subject: Re: Unable to connect with service using mysql-ephemeral template
To: dencowboy hotmail com
CC: users lists openshift redhat com



On Wed, Jul 6, 2016 at 12:44 PM, Den Cowboy <dencowboy hotmail com> wrote:
I don't know the best way to check:

​I was wondering if you had other apps deployed in your cluster that were accessing this, or other services by service hostname.
 
I see this error in my events after the deploy:
Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

I also saw this: https://github.com/openshift/origin/blob/master/docs/debugging-openshift.md
I putted also 8.8.8.8 as ns in my /etc/resolv.conf and rebooted but didn't work. Also not after scaling down and up the pod


From: bparees redhat com
Date: Wed, 6 Jul 2016 12:31:14 -0400

Subject: Re: Unable to connect with service using mysql-ephemeral template
To: dencowboy hotmail com
CC: users lists openshift redhat com

is service hostname resolution otherwise working in your cluster?


On Wed, Jul 6, 2016 at 12:20 PM, Den Cowboy <dencowboy hotmail com> wrote:
ping mysql: unknown host mysql
nslookup mysql:                                                         
Server:         213.186.33.xx                                                  
Address:        213.186.33.xx#53                                                
                                                                                
** server can't find mysql: NXDOMAIN 
dig: answer 0

content of /etc/resolv.conf:

search test.svc.cluster.local svc.cluster.local cluster.local ovh.net           
nameserver 178.32.27.xx                                                       
nameserver 213.186.33.xx                                                       
options ndots:5          

This works fine (IP = service IP):
mysql -utest -ptest -h172.30.222.94          

ping 172.30.222.94                                                      
PING 172.30.222.94 (172.30.222.94) 56(84) bytes of data.                        
From 10.1.0.1 icmp_seq=1 Destination Host Unreachable                           
From 10.1.0.1 icmp_seq=2 Destination Host Unreachable                           
From 10.1.0.1 icmp_seq=3 Destination Host Unreachable                           
From 10.1.0.1 icmp_seq=4 Destination Host Unreachable                                   
~             


From: bparees redhat com
Date: Wed, 6 Jul 2016 12:02:20 -0400
Subject: Re: Unable to connect with service using mysql-ephemeral template
To: dencowboy hotmail com
CC: users lists openshift redhat com


Is service DNS resolution otherwise working in your cluster?

if you just enter the container w/o starting the mysql shell are you able to dig/nslookup/ping the mysql hostname?

can you check the /etc/resolv.conf settings within the container to ensure the cluster DNS server is listed?


On Wed, Jul 6, 2016 at 11:49 AM, Den Cowboy <dencowboy hotmail com> wrote:
I'm on:
oc v1.2.0
kubernetes v1.2.0-36-g4a3f9c5

I've deployed the mysql-template which went fine:
Now I've a running mysql container. I go to the terminal inside my webconsole to authenticate on my mysql container:

mysql -utest -ptest -h127.0.0.1
> mysql

Fine, but when I try my service as my host:
mysql -utest -ptest -hmysql

Error 2005 (HY000): Unknown MySQL server host 'mysql' (0)

My service above my container is called 'mysql'
Can someone explain this issue?

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users




--
Ben Parees | OpenShift




--
Ben Parees | OpenShift




--
Ben Parees | OpenShift


_______________________________________________ users mailing list users lists openshift redhat com http://lists.openshift.redhat.com/openshiftmm/listinfo/users



--
Ben Parees | OpenShift


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]