[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



9042 firewall created across all nodes to node communication. But issue still persist. Port 53 already open between all nodes to masters.

Almost dead struck, appreciated any pointers to get fresh thoughts 

-- 
Srinivas Kotaru

On 6/14/16, 3:16 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:

>Ok. Let me open the port 9043. This port should be open between node to node right? 
>
>
>-- 
>Srinivas Kotaru
>
>On 6/14/16, 3:02 PM, "Matt Wringe" <mwringe redhat com> wrote:
>
>>----- Original Message -----
>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> To: "Matt Wringe" <mwringe redhat com>
>>> Cc: users lists openshift redhat com
>>> Sent: Tuesday, June 14, 2016 5:40:36 PM
>>> Subject: Re: Metrics deployment
>>> 
>>> Matt.
>>> 
>>> Sure, let us figure it out Hawkular side. Am here pasting 2 logs
>>> 
>>> 1. oc logs –f
>>> 2. cat /opt/eap/standalone/log/server.log
>>
>>Hmm, its getting an 'unknown error' when trying to connect to Cassandra, which doesn't really tell us anything :/
>>
>>The port that Hawkular Metrics uses to connect to Cassandra is 9042, you may also want to make sure that the DNS port is also open.
>>
>>> 
>>> Srinivas Kotaru
>>> 
>>>  
>>> --
>>> Srinivas Kotaru
>>> 
>>> On 6/14/16, 2:28 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>> 
>>> >
>>> >
>>> >----- Original Message -----
>>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> >> To: "Matt Wringe" <mwringe redhat com>
>>> >> Cc: users lists openshift redhat com
>>> >> Sent: Tuesday, June 14, 2016 4:44:04 PM
>>> >> Subject: Re: Metrics deployment
>>> >> 
>>> >> 
>>> >> I’m still struck with this issue. It is kind of chicken and egg problem.
>>> >> Heapster health probes failing since it is waiting for hawkular to start.
>>> >> Hawkular health probes are failing since it is unable to connect Casandra.
>>> >> Cansandra health probe also failing.
>>> >
>>> >The first step is to get Cassandra running, ignore Hawkular Metrics and
>>> >Heapster until you have Cassandra running properly. Without Cassandra being
>>> >able to run, those other components will not fully start.
>>> >
>>> >> 
>>> >> @Matt:  Internal DNS looks ups working. I’m able to create apps, build and
>>> >> deploy code. Router and registry components also working as expected
>>> >> 
>>> >> 
>>> >> oc get pods
>>> >> NAME                         READY     STATUS      RESTARTS   AGE
>>> >> hawkular-cassandra-1-mxd2m   1/1       Running     0          1h
>>> >> hawkular-metrics-gvp9k       0/1       Running     4          11m
>>> >> heapster-uleul               0/1       Running     4          11m
>>> >> metrics-deployer-2z75w       0/1       Completed   0          1h
>>> >
>>> >Cassandra being in "READY 1/1" means that it started up properly. So
>>> >Cassandra is running. Why do you think its not running? Things like
>>> >readiness probes are expected to fail until the pod is ready. Just because
>>> >there is a failure in the events doesn't mean its an error condition.
>>> >
>>> >Hawkular Metrics is not running here. So lets figure out why. Ignore
>>> >Heapster until Hawkular Metrics is started.
>>> >
>>> >> 
>>> >> heapster events:
>>> >> ------------------------
>>> >> 
>>> >> 
>>> >> Events:
>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason		Message
>>> >>   ---------	--------	-----	----					-------------			--------	------
>>> >>   		-------
>>> >>   1m		1m		1	{default-scheduler }							Normal		Scheduled	Successfully
>>> >>   assigned heapster-uleul to l3inpn-id2-004.cisco.com
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>> >>   	Normal		Pulling		pulling image
>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>> >>   	Normal		Pulled		Successfully pulled image
>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>> >>   	Normal		Created		Created container with docker id a22bb9a246ca
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>> >>   	Normal		Started		Started container with docker id a22bb9a246ca
>>> >>   1m		5s		10	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>> >>   	Warning		Unhealthy	Readiness probe failed: The heapster process is not
>>> >>   	yet
>>> >>   started, it is waiting for the Hawkular Metrics to start.
>>> >
>>> >Readiness probe failing here is expected, and we know why from the error
>>> >message:
>>> >"Readiness probe failed: The heapster process is not yet started, it is
>>> >waiting for the Hawkular Metrics to start."
>>> >
>>> >Once Hawkular Metrics is running then Heapster should automatically start
>>> >functioning.
>>> >
>>> >> 
>>> >> Hawkular events :
>>> >> ===============
>>> >> 
>>> >> Events:
>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
>>> >>   ---------	--------	-----	----					-------------				--------	------
>>> >>   		-------
>>> >>   1m		1m		1	{default-scheduler }								Normal		Scheduled	Successfully
>>> >>   assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulling		pulling image
>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulled		Successfully pulled
>>> >>   image
>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Normal		Created		Created container
>>> >>   	with
>>> >>   docker id 6e32459624d7
>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Normal		Started		Started container
>>> >>   	with
>>> >>   docker id 6e32459624d7
>>> >>   1m		57s		4	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Warning		Unhealthy	Readiness probe
>>> >>   failed: Failed to access the status endpoint : <urlopen error [Errno
>>> >>   111]
>>> >>   Connection refused>.
>>> >> 
>>> >>   46s	7s	5	{kubelet l3inpn-id2-004.cisco.com}
>>> >>   	spec.containers{hawkular-metrics}	Warning	Unhealthy	Readiness probe
>>> >>   failed:
>>> >> 
>>> >> 
>>> >> Casandra events
>>> >> ==============
>>> >> 
>>> >> Events:
>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
>>> >>   ---------	--------	-----	----					-------------				--------	------
>>> >>   		-------
>>> >>   1h		1h		1	{default-scheduler }								Normal		Scheduled	Successfully
>>> >>   assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image
>>> >>   "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully
>>> >>   	pulled
>>> >>   image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>>> >>   	container
>>> >>   with docker id 2a1390edd4f1
>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>>> >>   	container
>>> >>   with docker id 2a1390edd4f1
>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness
>>> >>   	probe
>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>> >> 'Connection refused'.
>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>>> >> operator expected
>>> >> Cassandra not in the up and normal state. Current state is
>>> >> 
>>> >>   1h	1h	2	{kubelet l3inpn-id2-003.cisco.com}
>>> >>   	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe
>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>> >> 'Connection refused'.
>>> >> Cassandra not in the up and normal state. Current state is
>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>>> >> operator expected
>>> >> 
>>> >> 
>>> >> Logs:
>>> >> ====
>>> >> 
>>> >> # oc logs -f heapster-uleul
>>> >> Endpoint Check in effect. Checking
>>> >> https://hawkular-metrics:443/hawkular/metrics/status
>>> >> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> Curl exit code: 6. Status Code 000
>>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >
>>> >Heapster is not starting, as expected, the problem is that it cannot access
>>> >that URL because it cannot resolve the host (error code 6). This is
>>> >expected, that hostname will not be resolvable until Hawkular Metrics
>>> >starts.
>>> >
>>> >> # oc logs -f hawkular-metrics-gvp9k
>>> >> 
>>> >> 16:42:43,201 INFO  [org.jboss.as.server] (Controller Boot Thread)
>>> >> JBAS015888:
>>> >> Creating http management service using socket-binding (management-http)
>>> >> 16:42:43,202 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version
>>> >> 3.0.14.GA-redhat-1
>>> >> 16:42:43,208 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
>>> >> Implementation Version 3.0.14.GA-redhat-1
>>> >> 16:42:43,223 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss
>>> >> Remoting version 3.3.5.Final-redhat-1
>>> >
>>> >I need the full logs for Hawkular Metrics, if you could get me the logs for
>>> >the last container that was run that would be helpful (eg oc get logs -p
>>> >....)
>>> >
>>> >> 
>>> >> # oc logs -f hawkular-cassandra-1-mxd2m
>>> >> 
>>> >> INFO  20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4 sstables
>>> >> to
>>> >> [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
>>> >> to level=0.  1,643 bytes to 1,155 (~70% of original) in 207ms =
>>> >> 0.005321MB/s.  0 total partitions merged to 11.  Partition merge counts
>>> >> were
>>> >> {1:11, }
>>> >> INFO  20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
>>> >> ]
>>> >> INFO  20:15:20 Writing Memtable-schema_columns 672348153(49.749KiB
>>> >> serialized
>>> >> bytes, 967 ops, 0%/0% of on/off-heap limit)
>>> >> INFO  20:15:20 Completed flushing
>>> >> /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
>>> >> (11.212KiB) for commitlog position ReplayPosition(segmentId=1465931718336,
>>> >> position=159511)
>>> >> INFO  20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4 sstables
>>> >> to
>>> >> [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
>>> >> to level=0.  128 bytes to 32 (~25% of original) in 146ms = 0.000209MB/s.
>>> >> 0
>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>> >> INFO  20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
>>> >> ]
>>> >> INFO  20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4 sstables
>>> >> to
>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
>>> >> to level=0.  128 bytes to 32 (~25% of original) in 155ms = 0.000197MB/s.
>>> >> 0
>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>> >> INFO  20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
>>> >> ]
>>> >> INFO  20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4 sstables
>>> >> to
>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
>>> >> to level=0.  128 bytes to 32 (~25% of original) in 83ms = 0.000368MB/s.  0
>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>> >> INFO  20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4 sstables
>>> >> to
>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
>>> >> to level=0.  2,538 bytes to 527 (~20% of original) in 176ms =
>>> >> 0.002856MB/s.
>>> >> 0 total partitions merged to 3.  Partition merge counts were {4:3, }
>>> >> 
>>> >> 
>>> >> 
>>> >> almost struck here. Any points to look for? Any ports need to be open
>>> >> explicitly ?
>>> >> 
>>> >> 
>>> >> 
>>> >> 
>>> >> 
>>> >> 
>>> >> 
>>> >> --
>>> >> Srinivas Kotaru
>>> >> 
>>> >> On 6/14/16, 11:27 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>> >> 
>>> >> >----- Original Message -----
>>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> >> >> To: "Matt Wringe" <mwringe redhat com>
>>> >> >> Cc: users lists openshift redhat com
>>> >> >> Sent: Tuesday, June 14, 2016 2:09:49 PM
>>> >> >> Subject: Re: Metrics deployment
>>> >> >> 
>>> >> >> Not sure what do you mean by installation. Am just running oc new-app
>>> >> >> -f
>>> >> >> metrics-deployer.yaml with default values except HOST_NAME and PV
>>> >> >> storage.
>>> >> >
>>> >> >I would suspect something wrong with your node or cluster installation.
>>> >> >Or
>>> >> >you have firewall rules blocking connections between your nodes so that
>>> >> >pod
>>> >> >cannot connect with each other or access the OpenShift DNS server.
>>> >> >
>>> >> >The lifecycle hooks exist to make sure that components only enter the
>>> >> >ready
>>> >> >state when they are fully started and ready.
>>> >> >
>>> >> >Can you check the Hawkular Metrics status page and see what that outputs?
>>> >> >eg
>>> >> >https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>>> >> >
>>> >> >> 
>>> >> >> I just deleted entire metrics setup and re running. But not sure this
>>> >> >> will
>>> >> >> fix the issue.
>>> >> >> 
>>> >> >> $ ./delete_metrics-infra.sh
>>> >> >> replicationcontroller "hawkular-cassandra-1" deleted
>>> >> >> replicationcontroller "hawkular-metrics" deleted
>>> >> >> replicationcontroller "heapster" deleted
>>> >> >> route "hawkular-metrics" deleted
>>> >> >> service "hawkular-cassandra" deleted
>>> >> >> service "hawkular-cassandra-nodes" deleted
>>> >> >> service "hawkular-metrics" deleted
>>> >> >> service "heapster" deleted
>>> >> >> pod "heapster-lyf65" deleted
>>> >> >> serviceaccount "cassandra" deleted
>>> >> >> serviceaccount "hawkular" deleted
>>> >> >> serviceaccount "heapster" deleted
>>> >> >> template "hawkular-cassandra-node-emptydir" deleted
>>> >> >> template "hawkular-cassandra-node-pv" deleted
>>> >> >> template "hawkular-cassandra-services" deleted
>>> >> >> template "hawkular-heapster" deleted
>>> >> >> template "hawkular-metrics" deleted
>>> >> >> template "hawkular-support" deleted
>>> >> >> secret "hawkular-cassandra-certificate" deleted
>>> >> >> secret "hawkular-cassandra-secrets" deleted
>>> >> >> secret "hawkular-metrics-account" deleted
>>> >> >> secret "hawkular-metrics-certificate" deleted
>>> >> >> secret "hawkular-metrics-secrets" deleted
>>> >> >> secret "heapster-secrets" deleted
>>> >> >> 
>>> >> >> --
>>> >> >> Srinivas Kotaru
>>> >> >> 
>>> >> >> On 6/14/16, 10:53 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>> >> >> 
>>> >> >> >----- Original Message -----
>>> >> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> >> >> >> To: "Matt Wringe" <mwringe redhat com>
>>> >> >> >> Cc: users lists openshift redhat com
>>> >> >> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>>> >> >> >> Subject: Re: Metrics deployment
>>> >> >> >> 
>>> >> >> >> I removed readiness probes from both hawkular-cassandra-1 &
>>> >> >> >> hawkular-metrics
>>> >> >> >> as both status shows probes failed.
>>> >> >> >
>>> >> >> >You should not have to remove the probes, this indicates that
>>> >> >> >something
>>> >> >> >is
>>> >> >> >wrong with your installation.
>>> >> >> >
>>> >> >> >> 
>>> >> >> >> It looks good now. Both containers looks and running
>>> >> >> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>>> >> >> >> heapster-7yl34
>>> >> >> >> logs still shows Could not connect to
>>> >> >> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit
>>> >> >> >> code:
>>> >> >> >> 6.
>>> >> >> >> Status Code 000.
>>> >> >> >> 
>>> >> >> >> Are we good or still had issues?
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> >> # oc get pods
>>> >> >> >> NAME                         READY     STATUS    RESTARTS   AGE
>>> >> >> >> hawkular-cassandra-1-kr8ka   1/1       Running   0          6m
>>> >> >> >> hawkular-metrics-vhe3u       1/1       Running   2          5m
>>> >> >> >> heapster-7yl34               0/1       Running   2          5m
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> >> --
>>> >> >> >> Srinivas Kotaru
>>> >> >> >> 
>>> >> >> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)"
>>> >> >> >> <skotaru cisco com>
>>> >> >> >> wrote:
>>> >> >> >> 
>>> >> >> >> >Matt
>>> >> >> >> >
>>> >> >> >> >Just want to share more info by running describe pod.
>>> >> >> >> >
>>> >> >> >> >It seems to be health probe failing. Do you think it is the issue?
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> ># oc describe pod hawkular-cassandra-1-it5uh
>>> >> >> >> >Name:		hawkular-cassandra-1-it5uh
>>> >> >> >> >Namespace:	openshift-infra
>>> >> >> >> >Node:		l3inpn-id2-003.cisco.com/173.36.96.16
>>> >> >> >> >Start Time:	Tue, 14 Jun 2016 16:36:21 +0000
>>> >> >> >> >Labels:
>>> >> >> >> >		metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>>> >> >> >> >Status:		Running
>>> >> >> >> >IP:		10.1.9.2
>>> >> >> >> >Controllers:	ReplicationController/hawkular-cassandra-1
>>> >> >> >> >Containers:
>>> >> >> >> >  hawkular-cassandra-1:
>>> >> >> >> >    Container ID:
>>> >> >> >> >    	docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>>> >> >> >> >    Image:
>>> >> >> >> >    		registry.access.redhat.com/openshift3/metrics-cassandra:latest
>>> >> >> >> >    Image ID:
>>> >> >> >> >    		docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>>> >> >> >> >    Ports:		9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>>> >> >> >> >    Command:
>>> >> >> >> >      /opt/apache-cassandra/bin/cassandra-docker.sh
>>> >> >> >> >      --cluster_name=hawkular-metrics
>>> >> >> >> >      --data_volume=/cassandra_data
>>> >> >> >> >      --internode_encryption=all
>>> >> >> >> >      --require_node_auth=true
>>> >> >> >> >      --enable_client_encryption=true
>>> >> >> >> >      --require_client_auth=true
>>> >> >> >> >      --keystore_file=/secret/cassandra.keystore
>>> >> >> >> >      --keystore_password_file=/secret/cassandra.keystore.password
>>> >> >> >> >      --truststore_file=/secret/cassandra.truststore
>>> >> >> >> >      --truststore_password_file=/secret/cassandra.truststore.password
>>> >> >> >> >      --cassandra_pem_file=/secret/cassandra.pem
>>> >> >> >> >    QoS Tier:
>>> >> >> >> >      cpu:		BestEffort
>>> >> >> >> >      memory:		BestEffort
>>> >> >> >> >    State:		Running
>>> >> >> >> >      Started:		Tue, 14 Jun 2016 16:37:01 +0000
>>> >> >> >> >    Ready:		True
>>> >> >> >> >    Restart Count:	0
>>> >> >> >> >    Readiness:		exec
>>> >> >> >> >    [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>>> >> >> >> >    delay=0s timeout=1s period=10s #success=1 #failure=3
>>> >> >> >> >    Environment Variables:
>>> >> >> >> >      CASSANDRA_MASTER:	true
>>> >> >> >> >      POD_NAMESPACE:	openshift-infra (v1:metadata.namespace)
>>> >> >> >> >Conditions:
>>> >> >> >> >  Type		Status
>>> >> >> >> >  Ready 	True
>>> >> >> >> >Volumes:
>>> >> >> >> >  cassandra-data:
>>> >> >> >> >    Type:	PersistentVolumeClaim (a reference to a
>>> >> >> >> >    PersistentVolumeClaim
>>> >> >> >> >    in
>>> >> >> >> >    the same namespace)
>>> >> >> >> >    ClaimName:	metrics-cassandra-1
>>> >> >> >> >    ReadOnly:	false
>>> >> >> >> >  hawkular-cassandra-secrets:
>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>>> >> >> >> >    SecretName:	hawkular-cassandra-secrets
>>> >> >> >> >  cassandra-token-4urfd:
>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>>> >> >> >> >    SecretName:	cassandra-token-4urfd
>>> >> >> >> >Events:
>>> >> >> >> >  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason
>>> >> >> >> >  		Message
>>> >> >> >> >  ---------	--------	-----	----					-------------				--------
>>> >> >> >> >  	------
>>> >> >> >> >  		-------
>>> >> >> >> >  27m		27m		1	{default-scheduler }								Normal		Scheduled
>>> >> >> >> >  	Successfully
>>> >> >> >> >  assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling
>>> >> >> >> >  	image
>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulled
>>> >> >> >> >  			Successfully
>>> >> >> >> >  	pulled
>>> >> >> >> >  image
>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>>> >> >> >> >  	container
>>> >> >> >> >  with docker id 17a9575eb655
>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>>> >> >> >> >  	container
>>> >> >> >> >  with docker id 17a9575eb655
>>> >> >> >> >  27m		26m		3	{kubelet l3inpn-id2-003.cisco.com}
>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy
>>> >> >> >> >  		Readiness
>>> >> >> >> >  	probe
>>> >> >> >> >  failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>> >> >> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>> >> >> >> >'Connection refused'.
>>> >> >> >> >Cassandra not in the up and normal state. Current state is
>>> >> >> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
>>> >> >> >> >unary
>>> >> >> >> >operator expected
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >--
>>> >> >> >> >Srinivas Kotaru
>>> >> >> >> >
>>> >> >> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>>> >> >> >> ><skotaru cisco com>
>>> >> >> >> >wrote:
>>> >> >> >> >
>>> >> >> >> >>Matt
>>> >> >> >> >>
>>> >> >> >> >>DNS service is working
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>>> >> >> >> >>172.30.0.1
>>> >> >> >> >>~  dig +short @master-01  jenkins.alln-test.svc.cluster.local
>>> >> >> >> >>172.30.85.148
>>> >> >> >> >>~  dig +short @master-01
>>> >> >> >> >>cakephp-example.alln-test.svc.cluster.local
>>> >> >> >> >>172.30.31.6
>>> >> >> >> >>
>>> >> >> >> >>I captured hawkular-metrics, it shows the problem. It seems to be
>>> >> >> >> >>unable
>>> >> >> >> >>to
>>> >> >> >> >>connect Cassandra cluster
>>> >> >> >> >>
>>> >> >> >> >># oc exec hawkular-metrics-qldd8 cat
>>> >> >> >> >>/opt/eap/standalone/log/server.log
>>> >> >> >> >>
>>> >> >> >> >>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan]
>>> >> >> >> >>(ServerService
>>> >> >> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>>> >> >> >> >>12:40:58,898 INFO  [org.jboss.as.webservices] (ServerService
>>> >> >> >> >>Thread
>>> >> >> >> >>Pool
>>> >> >> >> >>--
>>> >> >> >> >>48) JBAS015537: Activating WebServices Extension
>>> >> >> >> >>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread Pool
>>> >> >> >> >>--
>>> >> >> >> >>46)
>>> >> >> >> >>JBAS010153: Node identifier property is set to the default value.
>>> >> >> >> >>Please
>>> >> >> >> >>make sure it is unique.
>>> >> >> >> >>12:40:58,902 INFO  [org.jboss.as.security] (ServerService Thread
>>> >> >> >> >>Pool
>>> >> >> >> >>--
>>> >> >> >> >>44) JBAS013371: Activating Security Subsystem
>>> >> >> >> >>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService Thread
>>> >> >> >> >>Pool
>>> >> >> >> >>--
>>> >> >> >> >>39)
>>> >> >> >> >>JBAS011800: Activating Naming Subsystem
>>> >> >> >> >>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread Pool
>>> >> >> >> >>--
>>> >> >> >> >>37)
>>> >> >> >> >>JBAS012615: Activated the following JSF Implementations: [main,
>>> >> >> >> >>1.2]
>>> >> >> >> >>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>>> >> >> >> >>1.0.32.Final-redhat-1)
>>> >> >> >> >>12:40:58,981 INFO  [org.jboss.as.security] (MSC service thread
>>> >> >> >> >>1-2)
>>> >> >> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>>> >> >> >> >>12:40:58,961 INFO  [org.jboss.as.connector.subsystems.datasources]
>>> >> >> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying
>>> >> >> >> >>JDBC-compliant
>>> >> >> >> >>driver class org.h2.Driver (version 1.3)
>>> >> >> >> >>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service thread 1-2)
>>> >> >> >> >>JBAS011802: Starting Naming Service
>>> >> >> >> >>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-4)
>>> >> >> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>>> >> >> >> >>12:40:59,614 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>> >> >> >> >>service
>>> >> >> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>>> >> >> >> >>http-/0.0.0.0:8444
>>> >> >> >> >>12:40:59,620 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>> >> >> >> >>service
>>> >> >> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>>> >> >> >> >>http-/0.0.0.0:8080
>>> >> >> >> >>12:40:59,621 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>> >> >> >> >>service
>>> >> >> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on:
>>> >> >> >> >>http-/0.0.0.0:8444
>>> >> >> >> >>12:40:59,627 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>> >> >> >> >>service
>>> >> >> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on:
>>> >> >> >> >>http-/0.0.0.0:8080
>>> >> >> >> >>12:40:59,654 INFO  [org.jboss.as.connector.subsystems.datasources]
>>> >> >> >> >>(MSC
>>> >> >> >> >>service thread 1-3) JBAS010400: Bound data source
>>> >> >> >> >>[java:jboss/datasources/ExampleDS]
>>> >> >> >> >>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>>> >> >> >> >>(runtime-name:
>>> >> >> >> >>"activemq-rar.rar")
>>> >> >> >> >>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-1) JBAS015876: Starting deployment of
>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>>> >> >> >> >>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner] (MSC
>>> >> >> >> >>service
>>> >> >> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
>>> >> >> >> >>directory
>>> >> >> >> >>/opt/eap/standalone/deployments
>>> >> >> >> >>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service thread
>>> >> >> >> >>1-2)
>>> >> >> >> >>JBAS017100: Listening on 127.0.0.1:9999
>>> >> >> >> >>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-3)
>>> >> >> >> >>JBAS015102: Unable to load console module for slot eap, disabling
>>> >> >> >> >>console
>>> >> >> >> >>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
>>> >> >> >> >>4.3.5.Final-redhat-3
>>> >> >> >> >>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service thread
>>> >> >> >> >>1-4)
>>> >> >> >> >>JBAS017100: Listening on 0.0.0.0:4447
>>> >> >> >> >>12:41:01,361 INFO  [org.jboss.as.connector.deployers.RADeployer]
>>> >> >> >> >>(MSC
>>> >> >> >> >>service thread 1-3) IJ020001: Required license terms for
>>> >> >> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>>> >> >> >> >>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-1) JBAS015867: Deployment
>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>is
>>> >> >> >> >>using a private module
>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-1) JBAS015867: Deployment
>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>is
>>> >> >> >> >>using a private module
>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-1) JBAS015867: Deployment
>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>is
>>> >> >> >> >>using a private module
>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>> >> >> >> >>thread
>>> >> >> >> >>1-1) JBAS015867: Deployment
>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>is
>>> >> >> >> >>using a private module
>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>> >> >> >> >>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service thread
>>> >> >> >> >>1-1)
>>> >> >> >> >>JBAS016002: Processing weld deployment
>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>> >> >> >> >>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service thread
>>> >> >> >> >>1-1)
>>> >> >> >> >>JBAS016005: Starting Services for CDI deployment:
>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>> >> >> >> >>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service thread
>>> >> >> >> >>1-1)
>>> >> >> >> >>WELD-000900 1.1.31 (redhat)
>>> >> >> >> >>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service thread
>>> >> >> >> >>1-3)
>>> >> >> >> >>JBAS016008: Starting weld service for deployment
>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>> >> >> >> >>12:41:02,820 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread Pool --
>>> >> >> >> >>51)
>>> >> >> >> >>JBAS018210: Register web context: /hawkular/metrics
>>> >> >> >> >>12:41:02,958 INFO  [org.jboss.as.server] (ServerService Thread
>>> >> >> >> >>Pool
>>> >> >> >> >>--
>>> >> >> >> >>28)
>>> >> >> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war"
>>> >> >> >> >>(runtime-name :
>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war")
>>> >> >> >> >>12:41:02,959 INFO  [org.jboss.as.server] (ServerService Thread
>>> >> >> >> >>Pool
>>> >> >> >> >>--
>>> >> >> >> >>28)
>>> >> >> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>>> >> >> >> >>"activemq-rar.rar")
>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>> >> >> >> >>JBAS015961:
>>> >> >> >> >>Http
>>> >> >> >> >>management interface listening on http://127.0.0.1:9990/management
>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>> >> >> >> >>JBAS015951:
>>> >> >> >> >>Admin console listening on http://127.0.0.1:9990
>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>> >> >> >> >>JBAS015874:
>>> >> >> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms -
>>> >> >> >> >>Started
>>> >> >> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
>>> >> >> >> >>12:41:02,999 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>> >> >> >> >>unknown
>>> >> >> >> >>error
>>> >> >> >> >>12:41:02,999 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>> >> >> >> >>12:41:04,000 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:04,001 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:04,001 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>> >> >> >> >>12:41:06,002 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:06,002 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:06,003 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>> >> >> >> >>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n]
>>> >> >> >> >>(http-/0.0.0.0:8080-2)
>>> >> >> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>>> >> >> >> >>12:41:08,299 INFO  [org.hibernate.validator.internal.util.Version]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>>> >> >> >> >>4.3.2.Final-redhat-2
>>> >> >> >> >>12:41:08,418 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>>> >> >> >> >>starting
>>> >> >> >> >>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>>> >> >> >> >>javax.ws.rs.core.Application: class
>>> >> >> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>>> >> >> >> >>12:41:08,421 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>>> >> >> >> >>starting
>>> >> >> >> >>12:41:08,422 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>>> >> >> >> >>12:41:08,422 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>>> >> >> >> >>12:41:09,003 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:09,004 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:09,004 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>> >> >> >> >>12:41:13,004 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:13,123 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>> >> >> >> >>unknown
>>> >> >> >> >>error
>>> >> >> >> >>12:41:13,124 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>> >> >> >> >>12:41:14,124 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:14,124 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:14,125 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>> >> >> >> >>12:41:16,125 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:16,126 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:16,126 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>> >> >> >> >>12:41:19,126 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:19,127 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:19,127 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>> >> >> >> >>12:41:23,128 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:23,249 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>> >> >> >> >>unknown
>>> >> >> >> >>error
>>> >> >> >> >>12:41:23,249 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>> >> >> >> >>12:41:24,250 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:24,250 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:24,250 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>> >> >> >> >>12:41:26,251 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:26,251 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:26,251 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>> >> >> >> >>12:41:29,252 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:29,252 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:29,253 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>> >> >> >> >>12:41:33,253 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:33,376 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>> >> >> >> >>unknown
>>> >> >> >> >>error
>>> >> >> >> >>12:41:33,377 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>> >> >> >> >>12:41:34,377 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:34,378 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:34,378 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>> >> >> >> >>12:41:36,378 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:36,379 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:36,379 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>> >> >> >> >>12:41:39,379 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:39,380 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:39,380 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>> >> >> >> >>12:41:43,380 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:43,503 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>> >> >> >> >>unknown
>>> >> >> >> >>error
>>> >> >> >> >>12:41:43,504 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>> >> >> >> >>12:41:44,504 INFO
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>> >> >> >> >>metrics
>>> >> >> >> >>service
>>> >> >> >> >>12:41:44,505 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>> >> >> >> >>connect
>>> >> >> >> >>to
>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>> >> >> >> >>12:41:44,505 WARN
>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>> >> >> >> >>
>>> >> >> >> >>--
>>> >> >> >> >>Srinivas Kotaru
>>> >> >> >> >>
>>> >> >> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>> >> >> >> >>
>>> >> >> >> >>>----- Original Message -----
>>> >> >> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> >> >> >> >>>> To: "Matt Wringe" <mwringe redhat com>
>>> >> >> >> >>>> Cc: users lists openshift redhat com
>>> >> >> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>>> >> >> >> >>>> Subject: Re: Metrics deployment
>>> >> >> >> >>>> 
>>> >> >> >> >>>> Matt
>>> >> >> >> >>>> 
>>> >> >> >> >>>> PV issue resolved. Was able to to see PV successfully bounded
>>> >> >> >> >>>> and
>>> >> >> >> >>>> Casandra
>>> >> >> >> >>>> container has been running. However, it seems puzzle not fully
>>> >> >> >> >>>> yet
>>> >> >> >> >>>> solved.
>>> >> >> >> >>>
>>> >> >> >> >>>Are you sure the OpenShift DNS server is running?
>>> >> >> >> >>>
>>> >> >> >> >>>If you are running OSE 3.1, can you please follow this
>>> >> >> >> >>>https://access.redhat.com/solutions/2329131 and see if you are
>>> >> >> >> >>>now
>>> >> >> >> >>>seeing
>>> >> >> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc
>>> >> >> >> >>>exec
>>> >> >> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>>> >> >> >> >>>
>>> >> >> >> >>>> 
>>> >> >> >> >>>> I could see other container(heapster) not coming up, and seeing
>>> >> >> >> >>>> below
>>> >> >> >> >>>> errors
>>> >> >> >> >>>> 
>>> >> >> >> >>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>>> >> >> >> >>>> Endpoint Check in effect. Checking
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>>> >> >> >> >>>> Could not connect to
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>> >> >> >> >>>> accessible
>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> Could not connect to
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>> >> >> >> >>>> accessible
>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> 
>>> >> >> >> >>>> 
>>> >> >> >> >>>> # oc get pv
>>> >> >> >> >>>> pv-5gb-0011   5Gi        RWO           Bound
>>> >> >> >> >>>> openshift-infra/metrics-cassandra-1             22m
>>> >> >> >> >>>> 
>>> >> >> >> >>>> 
>>> >> >> >> >>>> $ oc get pods
>>> >> >> >> >>>> NAME                         READY     STATUS      RESTARTS
>>> >> >> >> >>>> AGE
>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7   1/1       Running     0
>>> >> >> >> >>>> 20m
>>> >> >> >> >>>> hawkular-metrics-mf5qf       0/1       Running     7
>>> >> >> >> >>>> 20m
>>> >> >> >> >>>> heapster-fnkdc               0/1       Error       6
>>> >> >> >> >>>> 20m
>>> >> >> >> >>>> metrics-deployer-cvep0       0/1       Completed   0
>>> >> >> >> >>>> 21m
>>> >> >> >> >>>> 
>>> >> >> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>>> >> >> >> >>>> 
>>> >> >> >> >>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2) XNIO
>>> >> >> >> >>>> Version
>>> >> >> >> >>>> 3.0.14.GA-redhat-1
>>> >> >> >> >>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller Boot
>>> >> >> >> >>>> Thread)
>>> >> >> >> >>>> JBAS015888:
>>> >> >> >> >>>> Creating http management service using socket-binding
>>> >> >> >> >>>> (management-http)
>>> >> >> >> >>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO
>>> >> >> >> >>>> NIO
>>> >> >> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>>> >> >> >> >>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service thread
>>> >> >> >> >>>> 1-2)
>>> >> >> >> >>>> JBoss
>>> >> >> >> >>>> Remoting version 3.3.5.Final-redhat-1
>>> >> >> >> >>>> 
>>> >> >> >> >>>> $ oc logs -f heapster-fnkdc
>>> >> >> >> >>>> Endpoint Check in effect. Checking
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>>> >> >> >> >>>> Could not connect to
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>> >> >> >> >>>> accessible
>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> Could not connect to
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>> >> >> >> >>>> accessible
>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> Could not connect to
>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> 
>>> >> >> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>>> >> >> >> >>>> INFO  23:00:24 Starting listening for CQL clients on
>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>>> >> >> >> >>>> INFO  23:00:24 Binding thrift service to
>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>>> >> >> >> >>>> INFO  23:00:24 enabling encrypted thrift connections between
>>> >> >> >> >>>> client
>>> >> >> >> >>>> and
>>> >> >> >> >>>> server
>>> >> >> >> >>>> INFO  23:00:24 Listening for thrift clients...
>>> >> >> >> >>>> INFO  23:00:26 Created default superuser role 'cassandra'
>>> >> >> >> >>>> 
>>> >> >> >> >>>> # oc get svc
>>> >> >> >> >>>> NAME                       CLUSTER-IP       EXTERNAL-IP
>>> >> >> >> >>>> PORT(S)
>>> >> >> >> >>>> AGE
>>> >> >> >> >>>> hawkular-cassandra         172.30.2.13      <none>
>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>> >> >> >> >>>> hawkular-cassandra-nodes   None             <none>
>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>> >> >> >> >>>> hawkular-metrics           172.30.117.176   <none>
>>> >> >> >> >>>> 443/TCP
>>> >> >> >> >>>> 25m
>>> >> >> >> >>>> heapster                   172.30.107.135   <none>
>>> >> >> >> >>>> 80/TCP
>>> >> >> >> >>>> 25m
>>> >> >> >> >>>> 
>>> >> >> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>>> >> >> >> >>>> 
>>> >> >> >> >>>> HTTP/1.1 504 Gateway Timeout
>>> >> >> >> >>>> Mime-Version: 1.0
>>> >> >> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>>> >> >> >> >>>> Content-Type: text/html
>>> >> >> >> >>>> Connection: keep-alive
>>> >> >> >> >>>> Proxy-Connection: keep-alive
>>> >> >> >> >>>> Content-Length: 1572
>>> >> >> >> >>>> 
>>> >> >> >> >>>> --
>>> >> >> >> >>>> Srinivas Kotaru
>>> >> >> >> >>>> 
>>> >> >> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>>> >> >> >> >>>> <skotaru cisco com>
>>> >> >> >> >>>> wrote:
>>> >> >> >> >>>> 
>>> >> >> >> >>>> >Matt
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false
>>> >> >> >> >>>> >and
>>> >> >> >> >>>> >working
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >Thanks you for your help so far.
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >--
>>> >> >> >> >>>> >Srinivas Kotaru
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>> >> >> >> >>>> >
>>> >> >> >> >>>> >>
>>> >> >> >> >>>> >>
>>> >> >> >> >>>> >>----- Original Message -----
>>> >> >> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>> >> >> >> >>>> >>> To: "Matt Wringe" <mwringe redhat com>
>>> >> >> >> >>>> >>> Cc: users lists openshift redhat com
>>> >> >> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>>> >> >> >> >>>> >>> Subject: Re: Metrics deployment
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> Oh ok
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> Am using PV for metrics
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> description: "The persistent volume size for each of the
>>> >> >> >> >>>> >>> Cassandra
>>> >> >> >> >>>> >>> nodes"
>>> >> >> >> >>>> >>>   name: CASSANDRA_PV_SIZE
>>> >> >> >> >>>> >>>   value: "10Gi"
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> oc get pv
>>> >> >> >> >>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS      CLAIM
>>> >> >> >> >>>> >>> REASON
>>> >> >> >> >>>> >>> AGE
>>> >> >> >> >>>> >>> pv-1gb-001    1Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-1gb-002    1Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-1gb-003    1Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-1gb-004    1Gi        RWO           Bound
>>> >> >> >> >>>> >>> thlatt/mongodb
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-1gb-005    1Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-2gb-0010   2Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-2gb-006    2Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-2gb-007    2Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-2gb-008    2Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-2gb-009    2Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-5gb-0011   5Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-5gb-0012   5Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-5gb-0013   5Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-5gb-0014   5Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> pv-5gb-0015   5Gi        RWO           Available
>>> >> >> >> >>>> >>> 4d
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> am running with below command
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded
>>> >> >> >> >>>> >>> HOSTNAME,
>>> >> >> >> >>>> >>> MASTER_API
>>> >> >> >> >>>> >>> and
>>> >> >> >> >>>> >>> PV info so not passing any parameters)
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>
>>> >> >> >> >>>> >>I would suspect that Cassandra is blocked because its waiting
>>> >> >> >> >>>> >>for
>>> >> >> >> >>>> >>10Gi
>>> >> >> >> >>>> >>PV
>>> >> >> >> >>>> >>to become available, and none of the PV listed above are big
>>> >> >> >> >>>> >>enough.
>>> >> >> >> >>>> >>
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> --
>>> >> >> >> >>>> >>> Srinivas Kotaru
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwringe redhat com>
>>> >> >> >> >>>> >>> wrote:
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> >----- Original Message -----
>>> >> >> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)"
>>> >> >> >> >>>> >>> >> <skotaru cisco com>
>>> >> >> >> >>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
>>> >> >> >> >>>> >>> >> Cc: users lists openshift redhat com
>>> >> >> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>>> >> >> >> >>>> >>> >> Subject: Re: Metrics deployment
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> Matt
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the
>>> >> >> >> >>>> >>> >> same
>>> >> >> >> >>>> >>> >> issue
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> # oc get pods
>>> >> >> >> >>>> >>> >> NAME                         READY     STATUS
>>> >> >> >> >>>> >>> >> RESTARTS
>>> >> >> >> >>>> >>> >> AGE
>>> >> >> >> >>>> >>> >> hawkular-cassandra-1-y2egy   0/1       ContainerCreating
>>> >> >> >> >>>> >>> >> 0
>>> >> >> >> >>>> >>> >> 5m
>>> >> >> >> >>>> >>> >> hawkular-metrics-4b16f       0/1       Running
>>> >> >> >> >>>> >>> >> 1
>>> >> >> >> >>>> >>> >> 4m
>>> >> >> >> >>>> >>> >> heapster-x2gj2               0/1       Running
>>> >> >> >> >>>> >>> >> 2
>>> >> >> >> >>>> >>> >> 4m
>>> >> >> >> >>>> >>> >> metrics-deployer-9v7vc       0/1       Completed
>>> >> >> >> >>>> >>> >> 0
>>> >> >> >> >>>> >>> >> 6m
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>>> >> >> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in
>>> >> >> >> >>>> >>> >> pod
>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>>> >> >> >> >>>> >>> >> ContainerCreating
>>> >> >> >> >>>> >>> >
>>> >> >> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra
>>> >> >> >> >>>> >>> >pod
>>> >> >> >> >>>> >>> >from
>>> >> >> >> >>>> >>> >starting.
>>> >> >> >> >>>> >>> >
>>> >> >> >> >>>> >>> >If you are using persistent storage, Cassandra will not
>>> >> >> >> >>>> >>> >start
>>> >> >> >> >>>> >>> >until
>>> >> >> >> >>>> >>> >the
>>> >> >> >> >>>> >>> >PV
>>> >> >> >> >>>> >>> >is available. There may be some more information about
>>> >> >> >> >>>> >>> >Cassandra
>>> >> >> >> >>>> >>> >in
>>> >> >> >> >>>> >>> >the
>>> >> >> >> >>>> >>> >pod
>>> >> >> >> >>>> >>> >section of the console under events.
>>> >> >> >> >>>> >>> >
>>> >> >> >> >>>> >>> >What command did you use when deploying the deployer?
>>> >> >> >> >>>> >>> >
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service
>>> >> >> >> >>>> >>> >> thread
>>> >> >> >> >>>> >>> >> 1-4)
>>> >> >> >> >>>> >>> >> VM
>>> >> >> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
>>> >> >> >> >>>> >>> >> -D[Standalone]
>>> >> >> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>>> >> >> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>>> >> >> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>>> >> >> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>>> >> >> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>>> >> >> >> >>>> >>> >> -XX:-TraceClassUnloading
>>> >> >> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>>> >> >> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>>> >> >> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>>> >> >> >> >>>> >>> >> -Djava.awt.headless=true
>>> >> >> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>>> >> >> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>>> >> >> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>>> >> >> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>>> >> >> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>>> >> >> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>>> >> >> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>>> >> >> >> >>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3)
>>> >> >> >> >>>> >>> >> XNIO
>>> >> >> >> >>>> >>> >> Version
>>> >> >> >> >>>> >>> >> 3.0.14.GA-redhat-1
>>> >> >> >> >>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread
>>> >> >> >> >>>> >>> >> 1-3)
>>> >> >> >> >>>> >>> >> XNIO
>>> >> >> >> >>>> >>> >> NIO
>>> >> >> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>>> >> >> >> >>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server] (Controller
>>> >> >> >> >>>> >>> >> Boot
>>> >> >> >> >>>> >>> >> Thread)
>>> >> >> >> >>>> >>> >> JBAS015888:
>>> >> >> >> >>>> >>> >> Creating http management service using socket-binding
>>> >> >> >> >>>> >>> >> (management-http)
>>> >> >> >> >>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service
>>> >> >> >> >>>> >>> >> thread
>>> >> >> >> >>>> >>> >> 1-3)
>>> >> >> >> >>>> >>> >> JBoss
>>> >> >> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>>> >> >> >> >>>> >>> >> Endpoint Check in effect. Checking
>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>>> >> >> >> >>>> >>> >> Could not connect to
>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>> >> >> >> >>>> >>> >> is
>>> >> >> >> >>>> >>> >> not
>>> >> >> >> >>>> >>> >> accessible
>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> >>> >> Could not connect to
>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>> >> >> >> >>>> >>> >> is
>>> >> >> >> >>>> >>> >> not
>>> >> >> >> >>>> >>> >> accessible
>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> >>> >> Could not connect to
>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> ++ oc create -f -
>>> >> >> >> >>>> >>> >> serviceaccount "heapster" created
>>> >> >> >> >>>> >>> >> service "heapster" created
>>> >> >> >> >>>> >>> >> replicationcontroller "heapster" created
>>> >> >> >> >>>> >>> >> + echo 'Success!'
>>> >> >> >> >>>> >>> >> Success!
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> --
>>> >> >> >> >>>> >>> >> Srinivas Kotaru
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com>
>>> >> >> >> >>>> >>> >> wrote:
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >----- Original Message -----
>>> >> >> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)"
>>> >> >> >> >>>> >>> >> >> <skotaru cisco com>
>>> >> >> >> >>>> >>> >> >> To: users lists openshift redhat com
>>> >> >> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>>> >> >> >> >>>> >>> >> >> Subject: Metrics deployment
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Hi
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
>>> >> >> >> >>>> >>> >> >> clusters. Am
>>> >> >> >> >>>> >>> >> >> seeing
>>> >> >> >> >>>> >>> >> >> below errors once metrics-deploy script was
>>> >> >> >> >>>> >>> >> >> successful.
>>> >> >> >> >>>> >>> >> >> I
>>> >> >> >> >>>> >>> >> >> used
>>> >> >> >> >>>> >>> >> >> our
>>> >> >> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
>>> >> >> >> >>>> >>> >> >> MASTER_URL
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>>> >> >> >> >>>> >>> >> >> (
>>> >> >> >> >>>> >>> >> >> hard coded as values)
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> template "hawkular-heapster" created
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Deploying the Heapster component
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> ++ '[' -n '' ']'
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> ++ oc create -f -
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>>> >> >> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> serviceaccount "heapster" created
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> service "heapster" created
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> replicationcontroller "heapster" created
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> + echo 'Success!'
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Success!
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> # oc get pods
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >It looks like none of your containers are fully up and
>>> >> >> >> >>>> >>> >> >running
>>> >> >> >> >>>> >>> >> >yet.
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not
>>> >> >> >> >>>> >>> >> >run,
>>> >> >> >> >>>> >>> >> >and
>>> >> >> >> >>>> >>> >> >Heapster
>>> >> >> >> >>>> >>> >> >will
>>> >> >> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first
>>> >> >> >> >>>> >>> >> >step
>>> >> >> >> >>>> >>> >> >will
>>> >> >> >> >>>> >>> >> >be to
>>> >> >> >> >>>> >>> >> >get
>>> >> >> >> >>>> >>> >> >Cassandra running properly.
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Could not connect to
>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>> >> >> >> >>>> >>> >> >> is
>>> >> >> >> >>>> >>> >> >> not
>>> >> >> >> >>>> >>> >> >> accessible
>>> >> >> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Could not connect to
>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
>>> >> >> >> >>>> >>> >> >trying
>>> >> >> >> >>>> >>> >> >to
>>> >> >> >> >>>> >>> >> >push
>>> >> >> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
>>> >> >> >> >>>> >>> >> >Heapster
>>> >> >> >> >>>> >>> >> >could
>>> >> >> >> >>>> >>> >> >not
>>> >> >> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the
>>> >> >> >> >>>> >>> >> >Hawkular
>>> >> >> >> >>>> >>> >> >Metrics
>>> >> >> >> >>>> >>> >> >service
>>> >> >> >> >>>> >>> >> >is
>>> >> >> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> What is the wrong? Why it checking just
>>> >> >> >> >>>> >>> >> >> hawkular-metrics
>>> >> >> >> >>>> >>> >> >> rather
>>> >> >> >> >>>> >>> >> >> full
>>> >> >> >> >>>> >>> >> >> routing
>>> >> >> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the
>>> >> >> >> >>>> >>> >> >internal
>>> >> >> >> >>>> >>> >> >hostname
>>> >> >> >> >>>> >>> >> >used
>>> >> >> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and
>>> >> >> >> >>>> >>> >> >the
>>> >> >> >> >>>> >>> >> >external
>>> >> >> >> >>>> >>> >> >hostname (eg what is configured via
>>> >> >> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>>> >> >> >> >>>> >>> >> >The
>>> >> >> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name
>>> >> >> >> >>>> >>> >> >of
>>> >> >> >> >>>> >>> >> >services,
>>> >> >> >> >>>> >>> >> >which
>>> >> >> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>>> >> >> >> >>>> >>> >> >
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> --
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> Srinivas Kotaru
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> >> _______________________________________________
>>> >> >> >> >>>> >>> >> >> users mailing list
>>> >> >> >> >>>> >>> >> >> users lists openshift redhat com
>>> >> >> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>> >> >> >> >>>> >>> >> >> 
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> >> 
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >>> 
>>> >> >> >> >>>> >
>>> >> >> >> >>>> 
>>> >> >> >> >>>> 
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >> 
>>> >> >> >> 
>>> >> >> 
>>> >> >> 
>>> >> 
>>> >> 
>>> 
>>> 
>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]