[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Pod unable to connect with service IP



I am unable to connect to the service from this new pod too. Here is the output when I run curl inside that shell:

# curl http://couchbase-cluster:8091 -v
* Rebuilt URL to: http://couchbase-cluster:8091/
*   Trying 172.30.118.19...
* TCP_NODELAY set
* connect to 172.30.118.19 port 8091 failed: Operation timed out
* Failed to connect to couchbase-cluster port 8091: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to couchbase-cluster port 8091: Operation timed out

Thanks
Vignesh

On Wed, Mar 8, 2017 at 2:51 PM, Marko Lukša <marko luksa gmail com> wrote:

OK, so the problem is inside the pod. Have you tried accessing the service from a different pod (using some other container image), e.g.:

oc run -it --image=appropriate/curl --restart Never --rm curl -- sh

and then using the curl command inside that shell?




On 08. 03. 2017 20:12, Vignesh M.P.N. wrote:
As I have setup Openshift origin all-in-one setup, I don't have a dedicated worker node. So I tried them all in the Openshift host.

Yes, all the IP:port combinations worked for me.

# curl -k https://172.30.0.1
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/controllers",
    "/healthz",
    "/healthz/ping",
    "/healthz/ready",
    "/metrics",
    "/oapi",
    "/oapi/v1",
    "/swaggerapi/"
  ]


Thanks
Vignesh


On Wed, Mar 8, 2017 at 2:04 PM, Marko Lukša <marko luksa gmail com> wrote:

I don't see anything wrong. Have you tried curl on the worker node itself (not inside the pod)?

These IP:port combinations should all work (try if any of them work for you):

curl localhost:30279
curl 172.30.118.19:8091
curl 172.46.103.138:8091

and of course

curl 172.17.0.22:8091

What happens when you try to connect to the API server (curl -k https://172.30.0.1)?

M.


On 08. 03. 2017 18:49, Vignesh M.P.N. wrote:
No, it was not the whole output. I had too many projects running, so I tried to filter them. Now I've deleted other projects and here is the whole output. http://pastebin.com/w2Led1vc

Thanks
Vignesh



On Wed, Mar 8, 2017 at 11:34 AM, Marko Lukša <marko luksa gmail com> wrote:

Is this the whole output? Some rows seem to be missing.


On 08. 03. 2017 16:29, Vignesh M.P.N. wrote:
Yes your summary is correct. I have Openshift origin all-in-one setup using VirtualBox, so I don't really have a specific node running worker pod.

Following output is from my all-in-one node:
# oc get svc
NAME                CLUSTER-IP      EXTERNAL-IP                     PORT(S)    AGE
couchbase-cluster   172.30.118.19   172.46.103.138,172.46.103.138   8091/TCP   12m

# iptables --list -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination        
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-PORTALS-CONTAINER  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-CONTAINER  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-PORTALS-HOST  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-HOST  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination        
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        anywhere           

Chain DOCKER (2 references)
target     prot opt source               destination        
RETURN     all  --  anywhere             anywhere           

Chain KUBE-MARK-MASQ (29 references)
target     prot opt source               destination        
MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORT-CONTAINER (1 references)
target     prot opt source               destination        

Chain KUBE-NODEPORT-HOST (1 references)
target     prot opt source               destination        

Chain KUBE-NODEPORTS (1 references)
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* cbcluster/couchbase-cluster:http-ui */ tcp dpt:30279
KUBE-SVC-77YHHSEL5NKLMS25  tcp  --  anywhere             anywhere             /* cbcluster/couchbase-cluster:http-ui */ tcp dpt:30279

Chain KUBE-SEP-BZ77QZDWKYFJXEH5 (1 references)
target     prot opt source               destination        
KUBE-MARK-MASQ  all  --  172.17.0.22          anywhere             /* cbcluster/couchbase-cluster:http-ui */
DNAT       tcp  --  anywhere             anywhere             /* cbcluster/couchbase-cluster:http-ui */ tcp to:172.17.0.22:8091

KUBE-SVC-77YHHSEL5NKLMS25  tcp  --  anywhere             172.30.118.19        /* cbcluster/couchbase-cluster:http-ui cluster IP */ tcp dpt:jamlink
KUBE-MARK-MASQ  tcp  --  anywhere             172.46.103.138       /* cbcluster/couchbase-cluster:http-ui external IP */ tcp dpt:jamlink
KUBE-SVC-77YHHSEL5NKLMS25  tcp  --  anywhere             172.46.103.138       /* cbcluster/couchbase-cluster:http-ui external IP */ tcp dpt:jamlink PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
KUBE-SVC-77YHHSEL5NKLMS25  tcp  --  anywhere             172.46.103.138       /* cbcluster/couchbase-cluster:http-ui external IP */ tcp dpt:jamlink ADDRTYPE match dst-type LOCAL
KUBE-MARK-MASQ  tcp  --  anywhere             172.46.103.138       /* cbcluster/couchbase-cluster:http-ui loadbalancer IP */ tcp dpt:jamlink
KUBE-SVC-77YHHSEL5NKLMS25  tcp  --  anywhere             172.46.103.138       /* cbcluster/couchbase-cluster:http-ui loadbalancer IP */ tcp dpt:jamlink


Thanks
Vignesh

On Wed, Mar 8, 2017 at 9:58 AM, Marko Lukša <marko luksa gmail com> wrote:

OK, to summarize:

- from inside the worker pod, you are able to connect to the master pod through its IP.
- the endpoints object shows that the pod is in fact an endpoint of the service
- from inside the worker pod, you are not able to connect to the service IP.

Is that correct? If yes, please send the output of iptables --list -t nat (on the node running the worker pod)

M.


On 08. 03. 2017 15:41, Vignesh M.P.N. wrote:
Hello Clayton & Marko

I am stuck at this point. I am unable to move ahead without being able to access the Service Name/Service IP from the pod. The DNS seems to be working fine. Is there anything else I can try to troubleshoot this issue?

Any help is much appreciated.

Thanks
Vignesh



On Mon, Mar 6, 2017 at 9:45 AM, Vignesh M.P.N. <vigneshb4u gmail com> wrote:
Correction: I switched the IPs. Here is the correct one.

couchbase-cluster-rc-6ulm8 is the first pod (172.17.0.19) and is defined as part of service couchbase-cluster. couchbase-index-rc-wj46z is the second pod (172.17.0.20) and not part of the service.

Thanks
Vignesh



On Mon, Mar 6, 2017 at 9:42 AM, Vignesh M.P.N. <vigneshb4u gmail com> wrote:
Thanks for the reply.

Here is my setup
# oc get svc
NAME                CLUSTER-IP      EXTERNAL-IP                     PORT(S)    AGE
couchbase-cluster   172.30.138.16   172.46.243.177,172.46.243.177   8091/TCP   4m

# oc get pods
NAME                         READY     STATUS    RESTARTS   AGE
couchbase-cluster-rc-6ulm8   1/1       Running   0          5m
couchbase-index-rc-wj46z     1/1       Running   0          3m

couchbase-cluster-rc-6ulm8 is the first pod (172.17.0.20) and is defined as part of service couchbase-cluster. couchbase-index-rc-wj46z is the second pod (172.17.0.19) and not part of the service. To add the second pod the couchbase-cluster, I am trying to access Service IP of couchbase-cluster from second pod.

# oc get endpoints
NAME                ENDPOINTS          AGE
couchbase-cluster   172.17.0.19:8091   6m

The big picture of this setup is explained here http://blog.kubernetes.io/2016/08/create-couchbase-cluster-using-kubernetes.html Though blog uses raw Kubernetes, I am trying it in an openshift setup.

Any help is appreciated.

Thanks
Vignesh




On Mon, Mar 6, 2017 at 9:15 AM, Clayton Coleman <ccoleman redhat com> wrote:
Has the pod showed up in the service endpoints list?

  oc get endpoints

Should return at least one ip+port combo.

The other steps look correct (the dns name is likely resolving correctly, you don't have to use the env var for the ip)

On Mar 6, 2017, at 6:07 AM, Vignesh M.P.N. <vigneshb4u gmail com> wrote:

Thanks for the response. That explains why the service IP was not pingable.

I would like to step up and ask the original problem I am facing here. I am trying to setup a couchbase cluster on openshift, I have deployed a pod, service (name: couchbase-master-service) and route in openshift environment following the example provided here https://github.com/arun-gupta/couchbase-kubernetes/tree/master/cluster

I create another pod, which is not part of the service and in the Dockerfile of the pod, I have a command that refers to the service using Service Name. To be specific, I am trying to add the second pod to the couchbase cluster instance of the first pod. I won't be aware of the first Pod IP address or the Service IP address. Hence referring by the service name, hoping to resolve it to the right IP address. The command goes like this

couchbase-cli server-add --cluster=couchbase-master-service:8091

Here is the script being invoked by Dockerfile (https://github.com/arun-gupta/couchbase-kubernetes/blob/master/cluster/configure-node.sh)

But during the pod creation I get this error:
Unable to connect to host at http://couchbase-master-service:8091

Initially I thought the Service Name "couchbase-master-service" was not resolving correctly to the service IP by openshift's built-in DNS cluster add-on. So I tried to troubleshoot using ping. But I did notice the pod was able to resolve the Service IP address but it was unable to reach the host. As you explained, that was not the right way to troubleshoot this issue.

Is there any alternative approach on how I could refer to an existing service without using it Service IP?

Thanks
Vignesh

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users











_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]