[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



18:34:03,414 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:03,532 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
18:34:03,533 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying connecting to Cassandra cluster in [1]s...
18:34:04,534 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:04,534 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
18:34:04,535 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying connecting to Cassandra cluster in [2]s...
18:34:06,535 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:06,535 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
18:34:06,536 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying connecting to Cassandra cluster in [3]s...
18:34:09,536 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:09,537 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
18:34:09,537 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying connecting to Cassandra cluster in [4]s...
18:34:13,537 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:13,653 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
18:34:13,653 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying connecting to Cassandra cluster in [1]s...
18:34:14,654 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
18:34:14,654 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
18:34:14,654 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying connecting to Cassandra cluster in [2]s...
18:34:16,655 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service



-- 
Srinivas Kotaru

On 6/14/16, 4:02 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:

>9042 firewall created across all nodes to node communication. But issue still persist. Port 53 already open between all nodes to masters.
>
>Almost dead struck, appreciated any pointers to get fresh thoughts 
>
>-- 
>Srinivas Kotaru
>
>On 6/14/16, 3:16 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:
>
>>Ok. Let me open the port 9043. This port should be open between node to node right? 
>>
>>
>>-- 
>>Srinivas Kotaru
>>
>>On 6/14/16, 3:02 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>
>>>----- Original Message -----
>>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> To: "Matt Wringe" <mwringe redhat com>
>>>> Cc: users lists openshift redhat com
>>>> Sent: Tuesday, June 14, 2016 5:40:36 PM
>>>> Subject: Re: Metrics deployment
>>>> 
>>>> Matt.
>>>> 
>>>> Sure, let us figure it out Hawkular side. Am here pasting 2 logs
>>>> 
>>>> 1. oc logs –f
>>>> 2. cat /opt/eap/standalone/log/server.log
>>>
>>>Hmm, its getting an 'unknown error' when trying to connect to Cassandra, which doesn't really tell us anything :/
>>>
>>>The port that Hawkular Metrics uses to connect to Cassandra is 9042, you may also want to make sure that the DNS port is also open.
>>>
>>>> 
>>>> Srinivas Kotaru
>>>> 
>>>>  
>>>> --
>>>> Srinivas Kotaru
>>>> 
>>>> On 6/14/16, 2:28 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>>> 
>>>> >
>>>> >
>>>> >----- Original Message -----
>>>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> >> To: "Matt Wringe" <mwringe redhat com>
>>>> >> Cc: users lists openshift redhat com
>>>> >> Sent: Tuesday, June 14, 2016 4:44:04 PM
>>>> >> Subject: Re: Metrics deployment
>>>> >> 
>>>> >> 
>>>> >> I’m still struck with this issue. It is kind of chicken and egg problem.
>>>> >> Heapster health probes failing since it is waiting for hawkular to start.
>>>> >> Hawkular health probes are failing since it is unable to connect Casandra.
>>>> >> Cansandra health probe also failing.
>>>> >
>>>> >The first step is to get Cassandra running, ignore Hawkular Metrics and
>>>> >Heapster until you have Cassandra running properly. Without Cassandra being
>>>> >able to run, those other components will not fully start.
>>>> >
>>>> >> 
>>>> >> @Matt:  Internal DNS looks ups working. I’m able to create apps, build and
>>>> >> deploy code. Router and registry components also working as expected
>>>> >> 
>>>> >> 
>>>> >> oc get pods
>>>> >> NAME                         READY     STATUS      RESTARTS   AGE
>>>> >> hawkular-cassandra-1-mxd2m   1/1       Running     0          1h
>>>> >> hawkular-metrics-gvp9k       0/1       Running     4          11m
>>>> >> heapster-uleul               0/1       Running     4          11m
>>>> >> metrics-deployer-2z75w       0/1       Completed   0          1h
>>>> >
>>>> >Cassandra being in "READY 1/1" means that it started up properly. So
>>>> >Cassandra is running. Why do you think its not running? Things like
>>>> >readiness probes are expected to fail until the pod is ready. Just because
>>>> >there is a failure in the events doesn't mean its an error condition.
>>>> >
>>>> >Hawkular Metrics is not running here. So lets figure out why. Ignore
>>>> >Heapster until Hawkular Metrics is started.
>>>> >
>>>> >> 
>>>> >> heapster events:
>>>> >> ------------------------
>>>> >> 
>>>> >> 
>>>> >> Events:
>>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason		Message
>>>> >>   ---------	--------	-----	----					-------------			--------	------
>>>> >>   		-------
>>>> >>   1m		1m		1	{default-scheduler }							Normal		Scheduled	Successfully
>>>> >>   assigned heapster-uleul to l3inpn-id2-004.cisco.com
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>>> >>   	Normal		Pulling		pulling image
>>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>>> >>   	Normal		Pulled		Successfully pulled image
>>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>>> >>   	Normal		Created		Created container with docker id a22bb9a246ca
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>>> >>   	Normal		Started		Started container with docker id a22bb9a246ca
>>>> >>   1m		5s		10	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}
>>>> >>   	Warning		Unhealthy	Readiness probe failed: The heapster process is not
>>>> >>   	yet
>>>> >>   started, it is waiting for the Hawkular Metrics to start.
>>>> >
>>>> >Readiness probe failing here is expected, and we know why from the error
>>>> >message:
>>>> >"Readiness probe failed: The heapster process is not yet started, it is
>>>> >waiting for the Hawkular Metrics to start."
>>>> >
>>>> >Once Hawkular Metrics is running then Heapster should automatically start
>>>> >functioning.
>>>> >
>>>> >> 
>>>> >> Hawkular events :
>>>> >> ===============
>>>> >> 
>>>> >> Events:
>>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
>>>> >>   ---------	--------	-----	----					-------------				--------	------
>>>> >>   		-------
>>>> >>   1m		1m		1	{default-scheduler }								Normal		Scheduled	Successfully
>>>> >>   assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulling		pulling image
>>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulled		Successfully pulled
>>>> >>   image
>>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Normal		Created		Created container
>>>> >>   	with
>>>> >>   docker id 6e32459624d7
>>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Normal		Started		Started container
>>>> >>   	with
>>>> >>   docker id 6e32459624d7
>>>> >>   1m		57s		4	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Warning		Unhealthy	Readiness probe
>>>> >>   failed: Failed to access the status endpoint : <urlopen error [Errno
>>>> >>   111]
>>>> >>   Connection refused>.
>>>> >> 
>>>> >>   46s	7s	5	{kubelet l3inpn-id2-004.cisco.com}
>>>> >>   	spec.containers{hawkular-metrics}	Warning	Unhealthy	Readiness probe
>>>> >>   failed:
>>>> >> 
>>>> >> 
>>>> >> Casandra events
>>>> >> ==============
>>>> >> 
>>>> >> Events:
>>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
>>>> >>   ---------	--------	-----	----					-------------				--------	------
>>>> >>   		-------
>>>> >>   1h		1h		1	{default-scheduler }								Normal		Scheduled	Successfully
>>>> >>   assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
>>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image
>>>> >>   "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully
>>>> >>   	pulled
>>>> >>   image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>>>> >>   	container
>>>> >>   with docker id 2a1390edd4f1
>>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>>>> >>   	container
>>>> >>   with docker id 2a1390edd4f1
>>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness
>>>> >>   	probe
>>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>>> >> 'Connection refused'.
>>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>>>> >> operator expected
>>>> >> Cassandra not in the up and normal state. Current state is
>>>> >> 
>>>> >>   1h	1h	2	{kubelet l3inpn-id2-003.cisco.com}
>>>> >>   	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe
>>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>>> >> 'Connection refused'.
>>>> >> Cassandra not in the up and normal state. Current state is
>>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>>>> >> operator expected
>>>> >> 
>>>> >> 
>>>> >> Logs:
>>>> >> ====
>>>> >> 
>>>> >> # oc logs -f heapster-uleul
>>>> >> Endpoint Check in effect. Checking
>>>> >> https://hawkular-metrics:443/hawkular/metrics/status
>>>> >> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> Curl exit code: 6. Status Code 000
>>>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >
>>>> >Heapster is not starting, as expected, the problem is that it cannot access
>>>> >that URL because it cannot resolve the host (error code 6). This is
>>>> >expected, that hostname will not be resolvable until Hawkular Metrics
>>>> >starts.
>>>> >
>>>> >> # oc logs -f hawkular-metrics-gvp9k
>>>> >> 
>>>> >> 16:42:43,201 INFO  [org.jboss.as.server] (Controller Boot Thread)
>>>> >> JBAS015888:
>>>> >> Creating http management service using socket-binding (management-http)
>>>> >> 16:42:43,202 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version
>>>> >> 3.0.14.GA-redhat-1
>>>> >> 16:42:43,208 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
>>>> >> Implementation Version 3.0.14.GA-redhat-1
>>>> >> 16:42:43,223 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss
>>>> >> Remoting version 3.3.5.Final-redhat-1
>>>> >
>>>> >I need the full logs for Hawkular Metrics, if you could get me the logs for
>>>> >the last container that was run that would be helpful (eg oc get logs -p
>>>> >....)
>>>> >
>>>> >> 
>>>> >> # oc logs -f hawkular-cassandra-1-mxd2m
>>>> >> 
>>>> >> INFO  20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4 sstables
>>>> >> to
>>>> >> [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
>>>> >> to level=0.  1,643 bytes to 1,155 (~70% of original) in 207ms =
>>>> >> 0.005321MB/s.  0 total partitions merged to 11.  Partition merge counts
>>>> >> were
>>>> >> {1:11, }
>>>> >> INFO  20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
>>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
>>>> >> ]
>>>> >> INFO  20:15:20 Writing Memtable-schema_columns 672348153(49.749KiB
>>>> >> serialized
>>>> >> bytes, 967 ops, 0%/0% of on/off-heap limit)
>>>> >> INFO  20:15:20 Completed flushing
>>>> >> /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
>>>> >> (11.212KiB) for commitlog position ReplayPosition(segmentId=1465931718336,
>>>> >> position=159511)
>>>> >> INFO  20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4 sstables
>>>> >> to
>>>> >> [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
>>>> >> to level=0.  128 bytes to 32 (~25% of original) in 146ms = 0.000209MB/s.
>>>> >> 0
>>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>>> >> INFO  20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
>>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
>>>> >> ]
>>>> >> INFO  20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4 sstables
>>>> >> to
>>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
>>>> >> to level=0.  128 bytes to 32 (~25% of original) in 155ms = 0.000197MB/s.
>>>> >> 0
>>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>>> >> INFO  20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
>>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
>>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
>>>> >> ]
>>>> >> INFO  20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4 sstables
>>>> >> to
>>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
>>>> >> to level=0.  128 bytes to 32 (~25% of original) in 83ms = 0.000368MB/s.  0
>>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>>>> >> INFO  20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4 sstables
>>>> >> to
>>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
>>>> >> to level=0.  2,538 bytes to 527 (~20% of original) in 176ms =
>>>> >> 0.002856MB/s.
>>>> >> 0 total partitions merged to 3.  Partition merge counts were {4:3, }
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> almost struck here. Any points to look for? Any ports need to be open
>>>> >> explicitly ?
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> --
>>>> >> Srinivas Kotaru
>>>> >> 
>>>> >> On 6/14/16, 11:27 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>>> >> 
>>>> >> >----- Original Message -----
>>>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> >> >> To: "Matt Wringe" <mwringe redhat com>
>>>> >> >> Cc: users lists openshift redhat com
>>>> >> >> Sent: Tuesday, June 14, 2016 2:09:49 PM
>>>> >> >> Subject: Re: Metrics deployment
>>>> >> >> 
>>>> >> >> Not sure what do you mean by installation. Am just running oc new-app
>>>> >> >> -f
>>>> >> >> metrics-deployer.yaml with default values except HOST_NAME and PV
>>>> >> >> storage.
>>>> >> >
>>>> >> >I would suspect something wrong with your node or cluster installation.
>>>> >> >Or
>>>> >> >you have firewall rules blocking connections between your nodes so that
>>>> >> >pod
>>>> >> >cannot connect with each other or access the OpenShift DNS server.
>>>> >> >
>>>> >> >The lifecycle hooks exist to make sure that components only enter the
>>>> >> >ready
>>>> >> >state when they are fully started and ready.
>>>> >> >
>>>> >> >Can you check the Hawkular Metrics status page and see what that outputs?
>>>> >> >eg
>>>> >> >https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>>>> >> >
>>>> >> >> 
>>>> >> >> I just deleted entire metrics setup and re running. But not sure this
>>>> >> >> will
>>>> >> >> fix the issue.
>>>> >> >> 
>>>> >> >> $ ./delete_metrics-infra.sh
>>>> >> >> replicationcontroller "hawkular-cassandra-1" deleted
>>>> >> >> replicationcontroller "hawkular-metrics" deleted
>>>> >> >> replicationcontroller "heapster" deleted
>>>> >> >> route "hawkular-metrics" deleted
>>>> >> >> service "hawkular-cassandra" deleted
>>>> >> >> service "hawkular-cassandra-nodes" deleted
>>>> >> >> service "hawkular-metrics" deleted
>>>> >> >> service "heapster" deleted
>>>> >> >> pod "heapster-lyf65" deleted
>>>> >> >> serviceaccount "cassandra" deleted
>>>> >> >> serviceaccount "hawkular" deleted
>>>> >> >> serviceaccount "heapster" deleted
>>>> >> >> template "hawkular-cassandra-node-emptydir" deleted
>>>> >> >> template "hawkular-cassandra-node-pv" deleted
>>>> >> >> template "hawkular-cassandra-services" deleted
>>>> >> >> template "hawkular-heapster" deleted
>>>> >> >> template "hawkular-metrics" deleted
>>>> >> >> template "hawkular-support" deleted
>>>> >> >> secret "hawkular-cassandra-certificate" deleted
>>>> >> >> secret "hawkular-cassandra-secrets" deleted
>>>> >> >> secret "hawkular-metrics-account" deleted
>>>> >> >> secret "hawkular-metrics-certificate" deleted
>>>> >> >> secret "hawkular-metrics-secrets" deleted
>>>> >> >> secret "heapster-secrets" deleted
>>>> >> >> 
>>>> >> >> --
>>>> >> >> Srinivas Kotaru
>>>> >> >> 
>>>> >> >> On 6/14/16, 10:53 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>>> >> >> 
>>>> >> >> >----- Original Message -----
>>>> >> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> >> >> >> To: "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> Cc: users lists openshift redhat com
>>>> >> >> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>>>> >> >> >> Subject: Re: Metrics deployment
>>>> >> >> >> 
>>>> >> >> >> I removed readiness probes from both hawkular-cassandra-1 &
>>>> >> >> >> hawkular-metrics
>>>> >> >> >> as both status shows probes failed.
>>>> >> >> >
>>>> >> >> >You should not have to remove the probes, this indicates that
>>>> >> >> >something
>>>> >> >> >is
>>>> >> >> >wrong with your installation.
>>>> >> >> >
>>>> >> >> >> 
>>>> >> >> >> It looks good now. Both containers looks and running
>>>> >> >> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>>>> >> >> >> heapster-7yl34
>>>> >> >> >> logs still shows Could not connect to
>>>> >> >> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit
>>>> >> >> >> code:
>>>> >> >> >> 6.
>>>> >> >> >> Status Code 000.
>>>> >> >> >> 
>>>> >> >> >> Are we good or still had issues?
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> >> # oc get pods
>>>> >> >> >> NAME                         READY     STATUS    RESTARTS   AGE
>>>> >> >> >> hawkular-cassandra-1-kr8ka   1/1       Running   0          6m
>>>> >> >> >> hawkular-metrics-vhe3u       1/1       Running   2          5m
>>>> >> >> >> heapster-7yl34               0/1       Running   2          5m
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> >> --
>>>> >> >> >> Srinivas Kotaru
>>>> >> >> >> 
>>>> >> >> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)"
>>>> >> >> >> <skotaru cisco com>
>>>> >> >> >> wrote:
>>>> >> >> >> 
>>>> >> >> >> >Matt
>>>> >> >> >> >
>>>> >> >> >> >Just want to share more info by running describe pod.
>>>> >> >> >> >
>>>> >> >> >> >It seems to be health probe failing. Do you think it is the issue?
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> ># oc describe pod hawkular-cassandra-1-it5uh
>>>> >> >> >> >Name:		hawkular-cassandra-1-it5uh
>>>> >> >> >> >Namespace:	openshift-infra
>>>> >> >> >> >Node:		l3inpn-id2-003.cisco.com/173.36.96.16
>>>> >> >> >> >Start Time:	Tue, 14 Jun 2016 16:36:21 +0000
>>>> >> >> >> >Labels:
>>>> >> >> >> >		metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>>>> >> >> >> >Status:		Running
>>>> >> >> >> >IP:		10.1.9.2
>>>> >> >> >> >Controllers:	ReplicationController/hawkular-cassandra-1
>>>> >> >> >> >Containers:
>>>> >> >> >> >  hawkular-cassandra-1:
>>>> >> >> >> >    Container ID:
>>>> >> >> >> >    	docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>>>> >> >> >> >    Image:
>>>> >> >> >> >    		registry.access.redhat.com/openshift3/metrics-cassandra:latest
>>>> >> >> >> >    Image ID:
>>>> >> >> >> >    		docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>>>> >> >> >> >    Ports:		9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>>>> >> >> >> >    Command:
>>>> >> >> >> >      /opt/apache-cassandra/bin/cassandra-docker.sh
>>>> >> >> >> >      --cluster_name=hawkular-metrics
>>>> >> >> >> >      --data_volume=/cassandra_data
>>>> >> >> >> >      --internode_encryption=all
>>>> >> >> >> >      --require_node_auth=true
>>>> >> >> >> >      --enable_client_encryption=true
>>>> >> >> >> >      --require_client_auth=true
>>>> >> >> >> >      --keystore_file=/secret/cassandra.keystore
>>>> >> >> >> >      --keystore_password_file=/secret/cassandra.keystore.password
>>>> >> >> >> >      --truststore_file=/secret/cassandra.truststore
>>>> >> >> >> >      --truststore_password_file=/secret/cassandra.truststore.password
>>>> >> >> >> >      --cassandra_pem_file=/secret/cassandra.pem
>>>> >> >> >> >    QoS Tier:
>>>> >> >> >> >      cpu:		BestEffort
>>>> >> >> >> >      memory:		BestEffort
>>>> >> >> >> >    State:		Running
>>>> >> >> >> >      Started:		Tue, 14 Jun 2016 16:37:01 +0000
>>>> >> >> >> >    Ready:		True
>>>> >> >> >> >    Restart Count:	0
>>>> >> >> >> >    Readiness:		exec
>>>> >> >> >> >    [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>>>> >> >> >> >    delay=0s timeout=1s period=10s #success=1 #failure=3
>>>> >> >> >> >    Environment Variables:
>>>> >> >> >> >      CASSANDRA_MASTER:	true
>>>> >> >> >> >      POD_NAMESPACE:	openshift-infra (v1:metadata.namespace)
>>>> >> >> >> >Conditions:
>>>> >> >> >> >  Type		Status
>>>> >> >> >> >  Ready 	True
>>>> >> >> >> >Volumes:
>>>> >> >> >> >  cassandra-data:
>>>> >> >> >> >    Type:	PersistentVolumeClaim (a reference to a
>>>> >> >> >> >    PersistentVolumeClaim
>>>> >> >> >> >    in
>>>> >> >> >> >    the same namespace)
>>>> >> >> >> >    ClaimName:	metrics-cassandra-1
>>>> >> >> >> >    ReadOnly:	false
>>>> >> >> >> >  hawkular-cassandra-secrets:
>>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>>>> >> >> >> >    SecretName:	hawkular-cassandra-secrets
>>>> >> >> >> >  cassandra-token-4urfd:
>>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>>>> >> >> >> >    SecretName:	cassandra-token-4urfd
>>>> >> >> >> >Events:
>>>> >> >> >> >  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason
>>>> >> >> >> >  		Message
>>>> >> >> >> >  ---------	--------	-----	----					-------------				--------
>>>> >> >> >> >  	------
>>>> >> >> >> >  		-------
>>>> >> >> >> >  27m		27m		1	{default-scheduler }								Normal		Scheduled
>>>> >> >> >> >  	Successfully
>>>> >> >> >> >  assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling
>>>> >> >> >> >  	image
>>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulled
>>>> >> >> >> >  			Successfully
>>>> >> >> >> >  	pulled
>>>> >> >> >> >  image
>>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>>>> >> >> >> >  	container
>>>> >> >> >> >  with docker id 17a9575eb655
>>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>>>> >> >> >> >  	container
>>>> >> >> >> >  with docker id 17a9575eb655
>>>> >> >> >> >  27m		26m		3	{kubelet l3inpn-id2-003.cisco.com}
>>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy
>>>> >> >> >> >  		Readiness
>>>> >> >> >> >  	probe
>>>> >> >> >> >  failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>>> >> >> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>>>> >> >> >> >'Connection refused'.
>>>> >> >> >> >Cassandra not in the up and normal state. Current state is
>>>> >> >> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
>>>> >> >> >> >unary
>>>> >> >> >> >operator expected
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >--
>>>> >> >> >> >Srinivas Kotaru
>>>> >> >> >> >
>>>> >> >> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>>>> >> >> >> ><skotaru cisco com>
>>>> >> >> >> >wrote:
>>>> >> >> >> >
>>>> >> >> >> >>Matt
>>>> >> >> >> >>
>>>> >> >> >> >>DNS service is working
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>>>> >> >> >> >>172.30.0.1
>>>> >> >> >> >>~  dig +short @master-01  jenkins.alln-test.svc.cluster.local
>>>> >> >> >> >>172.30.85.148
>>>> >> >> >> >>~  dig +short @master-01
>>>> >> >> >> >>cakephp-example.alln-test.svc.cluster.local
>>>> >> >> >> >>172.30.31.6
>>>> >> >> >> >>
>>>> >> >> >> >>I captured hawkular-metrics, it shows the problem. It seems to be
>>>> >> >> >> >>unable
>>>> >> >> >> >>to
>>>> >> >> >> >>connect Cassandra cluster
>>>> >> >> >> >>
>>>> >> >> >> >># oc exec hawkular-metrics-qldd8 cat
>>>> >> >> >> >>/opt/eap/standalone/log/server.log
>>>> >> >> >> >>
>>>> >> >> >> >>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan]
>>>> >> >> >> >>(ServerService
>>>> >> >> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>>>> >> >> >> >>12:40:58,898 INFO  [org.jboss.as.webservices] (ServerService
>>>> >> >> >> >>Thread
>>>> >> >> >> >>Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>48) JBAS015537: Activating WebServices Extension
>>>> >> >> >> >>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>46)
>>>> >> >> >> >>JBAS010153: Node identifier property is set to the default value.
>>>> >> >> >> >>Please
>>>> >> >> >> >>make sure it is unique.
>>>> >> >> >> >>12:40:58,902 INFO  [org.jboss.as.security] (ServerService Thread
>>>> >> >> >> >>Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>44) JBAS013371: Activating Security Subsystem
>>>> >> >> >> >>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService Thread
>>>> >> >> >> >>Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>39)
>>>> >> >> >> >>JBAS011800: Activating Naming Subsystem
>>>> >> >> >> >>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>37)
>>>> >> >> >> >>JBAS012615: Activated the following JSF Implementations: [main,
>>>> >> >> >> >>1.2]
>>>> >> >> >> >>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>>>> >> >> >> >>1.0.32.Final-redhat-1)
>>>> >> >> >> >>12:40:58,981 INFO  [org.jboss.as.security] (MSC service thread
>>>> >> >> >> >>1-2)
>>>> >> >> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>>>> >> >> >> >>12:40:58,961 INFO  [org.jboss.as.connector.subsystems.datasources]
>>>> >> >> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying
>>>> >> >> >> >>JDBC-compliant
>>>> >> >> >> >>driver class org.h2.Driver (version 1.3)
>>>> >> >> >> >>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service thread 1-2)
>>>> >> >> >> >>JBAS011802: Starting Naming Service
>>>> >> >> >> >>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-4)
>>>> >> >> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>>>> >> >> >> >>12:40:59,614 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>>> >> >> >> >>service
>>>> >> >> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>>>> >> >> >> >>http-/0.0.0.0:8444
>>>> >> >> >> >>12:40:59,620 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>>> >> >> >> >>service
>>>> >> >> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>>>> >> >> >> >>http-/0.0.0.0:8080
>>>> >> >> >> >>12:40:59,621 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>>> >> >> >> >>service
>>>> >> >> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on:
>>>> >> >> >> >>http-/0.0.0.0:8444
>>>> >> >> >> >>12:40:59,627 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>>>> >> >> >> >>service
>>>> >> >> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on:
>>>> >> >> >> >>http-/0.0.0.0:8080
>>>> >> >> >> >>12:40:59,654 INFO  [org.jboss.as.connector.subsystems.datasources]
>>>> >> >> >> >>(MSC
>>>> >> >> >> >>service thread 1-3) JBAS010400: Bound data source
>>>> >> >> >> >>[java:jboss/datasources/ExampleDS]
>>>> >> >> >> >>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>>>> >> >> >> >>(runtime-name:
>>>> >> >> >> >>"activemq-rar.rar")
>>>> >> >> >> >>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-1) JBAS015876: Starting deployment of
>>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>>>> >> >> >> >>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner] (MSC
>>>> >> >> >> >>service
>>>> >> >> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
>>>> >> >> >> >>directory
>>>> >> >> >> >>/opt/eap/standalone/deployments
>>>> >> >> >> >>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service thread
>>>> >> >> >> >>1-2)
>>>> >> >> >> >>JBAS017100: Listening on 127.0.0.1:9999
>>>> >> >> >> >>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-3)
>>>> >> >> >> >>JBAS015102: Unable to load console module for slot eap, disabling
>>>> >> >> >> >>console
>>>> >> >> >> >>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
>>>> >> >> >> >>4.3.5.Final-redhat-3
>>>> >> >> >> >>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service thread
>>>> >> >> >> >>1-4)
>>>> >> >> >> >>JBAS017100: Listening on 0.0.0.0:4447
>>>> >> >> >> >>12:41:01,361 INFO  [org.jboss.as.connector.deployers.RADeployer]
>>>> >> >> >> >>(MSC
>>>> >> >> >> >>service thread 1-3) IJ020001: Required license terms for
>>>> >> >> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>>>> >> >> >> >>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-1) JBAS015867: Deployment
>>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>is
>>>> >> >> >> >>using a private module
>>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-1) JBAS015867: Deployment
>>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>is
>>>> >> >> >> >>using a private module
>>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-1) JBAS015867: Deployment
>>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>is
>>>> >> >> >> >>using a private module
>>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
>>>> >> >> >> >>thread
>>>> >> >> >> >>1-1) JBAS015867: Deployment
>>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>is
>>>> >> >> >> >>using a private module
>>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>>>> >> >> >> >>which may be changed or removed in future versions without notice.
>>>> >> >> >> >>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service thread
>>>> >> >> >> >>1-1)
>>>> >> >> >> >>JBAS016002: Processing weld deployment
>>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>>> >> >> >> >>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service thread
>>>> >> >> >> >>1-1)
>>>> >> >> >> >>JBAS016005: Starting Services for CDI deployment:
>>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>>> >> >> >> >>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service thread
>>>> >> >> >> >>1-1)
>>>> >> >> >> >>WELD-000900 1.1.31 (redhat)
>>>> >> >> >> >>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service thread
>>>> >> >> >> >>1-3)
>>>> >> >> >> >>JBAS016008: Starting weld service for deployment
>>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>>>> >> >> >> >>12:41:02,820 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread Pool --
>>>> >> >> >> >>51)
>>>> >> >> >> >>JBAS018210: Register web context: /hawkular/metrics
>>>> >> >> >> >>12:41:02,958 INFO  [org.jboss.as.server] (ServerService Thread
>>>> >> >> >> >>Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>28)
>>>> >> >> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war"
>>>> >> >> >> >>(runtime-name :
>>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war")
>>>> >> >> >> >>12:41:02,959 INFO  [org.jboss.as.server] (ServerService Thread
>>>> >> >> >> >>Pool
>>>> >> >> >> >>--
>>>> >> >> >> >>28)
>>>> >> >> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>>>> >> >> >> >>"activemq-rar.rar")
>>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>>> >> >> >> >>JBAS015961:
>>>> >> >> >> >>Http
>>>> >> >> >> >>management interface listening on http://127.0.0.1:9990/management
>>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>>> >> >> >> >>JBAS015951:
>>>> >> >> >> >>Admin console listening on http://127.0.0.1:9990
>>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>>>> >> >> >> >>JBAS015874:
>>>> >> >> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms -
>>>> >> >> >> >>Started
>>>> >> >> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
>>>> >> >> >> >>12:41:02,999 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>>> >> >> >> >>unknown
>>>> >> >> >> >>error
>>>> >> >> >> >>12:41:02,999 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>>> >> >> >> >>12:41:04,000 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:04,001 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:04,001 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>>> >> >> >> >>12:41:06,002 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:06,002 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:06,003 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>>> >> >> >> >>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2)
>>>> >> >> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>>>> >> >> >> >>12:41:08,299 INFO  [org.hibernate.validator.internal.util.Version]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>>>> >> >> >> >>4.3.2.Final-redhat-2
>>>> >> >> >> >>12:41:08,418 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>>>> >> >> >> >>starting
>>>> >> >> >> >>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>>>> >> >> >> >>javax.ws.rs.core.Application: class
>>>> >> >> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>>>> >> >> >> >>12:41:08,421 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>>>> >> >> >> >>starting
>>>> >> >> >> >>12:41:08,422 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>>>> >> >> >> >>12:41:08,422 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>>>> >> >> >> >>12:41:09,003 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:09,004 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:09,004 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>>> >> >> >> >>12:41:13,004 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:13,123 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>>> >> >> >> >>unknown
>>>> >> >> >> >>error
>>>> >> >> >> >>12:41:13,124 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>>> >> >> >> >>12:41:14,124 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:14,124 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:14,125 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>>> >> >> >> >>12:41:16,125 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:16,126 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:16,126 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>>> >> >> >> >>12:41:19,126 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:19,127 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:19,127 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>>> >> >> >> >>12:41:23,128 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:23,249 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>>> >> >> >> >>unknown
>>>> >> >> >> >>error
>>>> >> >> >> >>12:41:23,249 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>>> >> >> >> >>12:41:24,250 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:24,250 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:24,250 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>>> >> >> >> >>12:41:26,251 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:26,251 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:26,251 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>>> >> >> >> >>12:41:29,252 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:29,252 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:29,253 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>>> >> >> >> >>12:41:33,253 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:33,376 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>>> >> >> >> >>unknown
>>>> >> >> >> >>error
>>>> >> >> >> >>12:41:33,377 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>>> >> >> >> >>12:41:34,377 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:34,378 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:34,378 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>>> >> >> >> >>12:41:36,378 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:36,379 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:36,379 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>>>> >> >> >> >>12:41:39,379 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:39,380 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:39,380 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>>>> >> >> >> >>12:41:43,380 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:43,503 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
>>>> >> >> >> >>unknown
>>>> >> >> >> >>error
>>>> >> >> >> >>12:41:43,504 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>>>> >> >> >> >>12:41:44,504 INFO
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>>>> >> >> >> >>metrics
>>>> >> >> >> >>service
>>>> >> >> >> >>12:41:44,505 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not
>>>> >> >> >> >>connect
>>>> >> >> >> >>to
>>>> >> >> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>> >> >> >> >>12:41:44,505 WARN
>>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
>>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>>>> >> >> >> >>
>>>> >> >> >> >>--
>>>> >> >> >> >>Srinivas Kotaru
>>>> >> >> >> >>
>>>> >> >> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>>> >> >> >> >>
>>>> >> >> >> >>>----- Original Message -----
>>>> >> >> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> >> >> >> >>>> To: "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> >>>> Cc: users lists openshift redhat com
>>>> >> >> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>>>> >> >> >> >>>> Subject: Re: Metrics deployment
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> Matt
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> PV issue resolved. Was able to to see PV successfully bounded
>>>> >> >> >> >>>> and
>>>> >> >> >> >>>> Casandra
>>>> >> >> >> >>>> container has been running. However, it seems puzzle not fully
>>>> >> >> >> >>>> yet
>>>> >> >> >> >>>> solved.
>>>> >> >> >> >>>
>>>> >> >> >> >>>Are you sure the OpenShift DNS server is running?
>>>> >> >> >> >>>
>>>> >> >> >> >>>If you are running OSE 3.1, can you please follow this
>>>> >> >> >> >>>https://access.redhat.com/solutions/2329131 and see if you are
>>>> >> >> >> >>>now
>>>> >> >> >> >>>seeing
>>>> >> >> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc
>>>> >> >> >> >>>exec
>>>> >> >> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>>>> >> >> >> >>>
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> I could see other container(heapster) not coming up, and seeing
>>>> >> >> >> >>>> below
>>>> >> >> >> >>>> errors
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>>>> >> >> >> >>>> Endpoint Check in effect. Checking
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>>>> >> >> >> >>>> Could not connect to
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>> >> >> >> >>>> accessible
>>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> Could not connect to
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>> >> >> >> >>>> accessible
>>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> # oc get pv
>>>> >> >> >> >>>> pv-5gb-0011   5Gi        RWO           Bound
>>>> >> >> >> >>>> openshift-infra/metrics-cassandra-1             22m
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> $ oc get pods
>>>> >> >> >> >>>> NAME                         READY     STATUS      RESTARTS
>>>> >> >> >> >>>> AGE
>>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7   1/1       Running     0
>>>> >> >> >> >>>> 20m
>>>> >> >> >> >>>> hawkular-metrics-mf5qf       0/1       Running     7
>>>> >> >> >> >>>> 20m
>>>> >> >> >> >>>> heapster-fnkdc               0/1       Error       6
>>>> >> >> >> >>>> 20m
>>>> >> >> >> >>>> metrics-deployer-cvep0       0/1       Completed   0
>>>> >> >> >> >>>> 21m
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2) XNIO
>>>> >> >> >> >>>> Version
>>>> >> >> >> >>>> 3.0.14.GA-redhat-1
>>>> >> >> >> >>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller Boot
>>>> >> >> >> >>>> Thread)
>>>> >> >> >> >>>> JBAS015888:
>>>> >> >> >> >>>> Creating http management service using socket-binding
>>>> >> >> >> >>>> (management-http)
>>>> >> >> >> >>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO
>>>> >> >> >> >>>> NIO
>>>> >> >> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>>>> >> >> >> >>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service thread
>>>> >> >> >> >>>> 1-2)
>>>> >> >> >> >>>> JBoss
>>>> >> >> >> >>>> Remoting version 3.3.5.Final-redhat-1
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> $ oc logs -f heapster-fnkdc
>>>> >> >> >> >>>> Endpoint Check in effect. Checking
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>>>> >> >> >> >>>> Could not connect to
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>> >> >> >> >>>> accessible
>>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> Could not connect to
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>> >> >> >> >>>> accessible
>>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> Could not connect to
>>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>>>> >> >> >> >>>> INFO  23:00:24 Starting listening for CQL clients on
>>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>>>> >> >> >> >>>> INFO  23:00:24 Binding thrift service to
>>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>>>> >> >> >> >>>> INFO  23:00:24 enabling encrypted thrift connections between
>>>> >> >> >> >>>> client
>>>> >> >> >> >>>> and
>>>> >> >> >> >>>> server
>>>> >> >> >> >>>> INFO  23:00:24 Listening for thrift clients...
>>>> >> >> >> >>>> INFO  23:00:26 Created default superuser role 'cassandra'
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> # oc get svc
>>>> >> >> >> >>>> NAME                       CLUSTER-IP       EXTERNAL-IP
>>>> >> >> >> >>>> PORT(S)
>>>> >> >> >> >>>> AGE
>>>> >> >> >> >>>> hawkular-cassandra         172.30.2.13      <none>
>>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>>> >> >> >> >>>> hawkular-cassandra-nodes   None             <none>
>>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>>> >> >> >> >>>> hawkular-metrics           172.30.117.176   <none>
>>>> >> >> >> >>>> 443/TCP
>>>> >> >> >> >>>> 25m
>>>> >> >> >> >>>> heapster                   172.30.107.135   <none>
>>>> >> >> >> >>>> 80/TCP
>>>> >> >> >> >>>> 25m
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> HTTP/1.1 504 Gateway Timeout
>>>> >> >> >> >>>> Mime-Version: 1.0
>>>> >> >> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>>>> >> >> >> >>>> Content-Type: text/html
>>>> >> >> >> >>>> Connection: keep-alive
>>>> >> >> >> >>>> Proxy-Connection: keep-alive
>>>> >> >> >> >>>> Content-Length: 1572
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> --
>>>> >> >> >> >>>> Srinivas Kotaru
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>>>> >> >> >> >>>> <skotaru cisco com>
>>>> >> >> >> >>>> wrote:
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> >Matt
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false
>>>> >> >> >> >>>> >and
>>>> >> >> >> >>>> >working
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >Thanks you for your help so far.
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >--
>>>> >> >> >> >>>> >Srinivas Kotaru
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> >>
>>>> >> >> >> >>>> >>
>>>> >> >> >> >>>> >>----- Original Message -----
>>>> >> >> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>> >> >> >> >>>> >>> To: "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> >>>> >>> Cc: users lists openshift redhat com
>>>> >> >> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>>>> >> >> >> >>>> >>> Subject: Re: Metrics deployment
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> Oh ok
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> Am using PV for metrics
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> description: "The persistent volume size for each of the
>>>> >> >> >> >>>> >>> Cassandra
>>>> >> >> >> >>>> >>> nodes"
>>>> >> >> >> >>>> >>>   name: CASSANDRA_PV_SIZE
>>>> >> >> >> >>>> >>>   value: "10Gi"
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> oc get pv
>>>> >> >> >> >>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS      CLAIM
>>>> >> >> >> >>>> >>> REASON
>>>> >> >> >> >>>> >>> AGE
>>>> >> >> >> >>>> >>> pv-1gb-001    1Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-1gb-002    1Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-1gb-003    1Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-1gb-004    1Gi        RWO           Bound
>>>> >> >> >> >>>> >>> thlatt/mongodb
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-1gb-005    1Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-2gb-0010   2Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-2gb-006    2Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-2gb-007    2Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-2gb-008    2Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-2gb-009    2Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-5gb-0011   5Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-5gb-0012   5Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-5gb-0013   5Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-5gb-0014   5Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> pv-5gb-0015   5Gi        RWO           Available
>>>> >> >> >> >>>> >>> 4d
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> am running with below command
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded
>>>> >> >> >> >>>> >>> HOSTNAME,
>>>> >> >> >> >>>> >>> MASTER_API
>>>> >> >> >> >>>> >>> and
>>>> >> >> >> >>>> >>> PV info so not passing any parameters)
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>
>>>> >> >> >> >>>> >>I would suspect that Cassandra is blocked because its waiting
>>>> >> >> >> >>>> >>for
>>>> >> >> >> >>>> >>10Gi
>>>> >> >> >> >>>> >>PV
>>>> >> >> >> >>>> >>to become available, and none of the PV listed above are big
>>>> >> >> >> >>>> >>enough.
>>>> >> >> >> >>>> >>
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> --
>>>> >> >> >> >>>> >>> Srinivas Kotaru
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> >>>> >>> wrote:
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> >----- Original Message -----
>>>> >> >> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)"
>>>> >> >> >> >>>> >>> >> <skotaru cisco com>
>>>> >> >> >> >>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> >>>> >>> >> Cc: users lists openshift redhat com
>>>> >> >> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>>>> >> >> >> >>>> >>> >> Subject: Re: Metrics deployment
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> Matt
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the
>>>> >> >> >> >>>> >>> >> same
>>>> >> >> >> >>>> >>> >> issue
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> # oc get pods
>>>> >> >> >> >>>> >>> >> NAME                         READY     STATUS
>>>> >> >> >> >>>> >>> >> RESTARTS
>>>> >> >> >> >>>> >>> >> AGE
>>>> >> >> >> >>>> >>> >> hawkular-cassandra-1-y2egy   0/1       ContainerCreating
>>>> >> >> >> >>>> >>> >> 0
>>>> >> >> >> >>>> >>> >> 5m
>>>> >> >> >> >>>> >>> >> hawkular-metrics-4b16f       0/1       Running
>>>> >> >> >> >>>> >>> >> 1
>>>> >> >> >> >>>> >>> >> 4m
>>>> >> >> >> >>>> >>> >> heapster-x2gj2               0/1       Running
>>>> >> >> >> >>>> >>> >> 2
>>>> >> >> >> >>>> >>> >> 4m
>>>> >> >> >> >>>> >>> >> metrics-deployer-9v7vc       0/1       Completed
>>>> >> >> >> >>>> >>> >> 0
>>>> >> >> >> >>>> >>> >> 6m
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>>>> >> >> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in
>>>> >> >> >> >>>> >>> >> pod
>>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>>>> >> >> >> >>>> >>> >> ContainerCreating
>>>> >> >> >> >>>> >>> >
>>>> >> >> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra
>>>> >> >> >> >>>> >>> >pod
>>>> >> >> >> >>>> >>> >from
>>>> >> >> >> >>>> >>> >starting.
>>>> >> >> >> >>>> >>> >
>>>> >> >> >> >>>> >>> >If you are using persistent storage, Cassandra will not
>>>> >> >> >> >>>> >>> >start
>>>> >> >> >> >>>> >>> >until
>>>> >> >> >> >>>> >>> >the
>>>> >> >> >> >>>> >>> >PV
>>>> >> >> >> >>>> >>> >is available. There may be some more information about
>>>> >> >> >> >>>> >>> >Cassandra
>>>> >> >> >> >>>> >>> >in
>>>> >> >> >> >>>> >>> >the
>>>> >> >> >> >>>> >>> >pod
>>>> >> >> >> >>>> >>> >section of the console under events.
>>>> >> >> >> >>>> >>> >
>>>> >> >> >> >>>> >>> >What command did you use when deploying the deployer?
>>>> >> >> >> >>>> >>> >
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service
>>>> >> >> >> >>>> >>> >> thread
>>>> >> >> >> >>>> >>> >> 1-4)
>>>> >> >> >> >>>> >>> >> VM
>>>> >> >> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
>>>> >> >> >> >>>> >>> >> -D[Standalone]
>>>> >> >> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>>>> >> >> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>>>> >> >> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>>>> >> >> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>>>> >> >> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>>>> >> >> >> >>>> >>> >> -XX:-TraceClassUnloading
>>>> >> >> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>>>> >> >> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>>>> >> >> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>>>> >> >> >> >>>> >>> >> -Djava.awt.headless=true
>>>> >> >> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>>>> >> >> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>>>> >> >> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>>>> >> >> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>>>> >> >> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>>>> >> >> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>>>> >> >> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>>>> >> >> >> >>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3)
>>>> >> >> >> >>>> >>> >> XNIO
>>>> >> >> >> >>>> >>> >> Version
>>>> >> >> >> >>>> >>> >> 3.0.14.GA-redhat-1
>>>> >> >> >> >>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread
>>>> >> >> >> >>>> >>> >> 1-3)
>>>> >> >> >> >>>> >>> >> XNIO
>>>> >> >> >> >>>> >>> >> NIO
>>>> >> >> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>>>> >> >> >> >>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server] (Controller
>>>> >> >> >> >>>> >>> >> Boot
>>>> >> >> >> >>>> >>> >> Thread)
>>>> >> >> >> >>>> >>> >> JBAS015888:
>>>> >> >> >> >>>> >>> >> Creating http management service using socket-binding
>>>> >> >> >> >>>> >>> >> (management-http)
>>>> >> >> >> >>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service
>>>> >> >> >> >>>> >>> >> thread
>>>> >> >> >> >>>> >>> >> 1-3)
>>>> >> >> >> >>>> >>> >> JBoss
>>>> >> >> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>>>> >> >> >> >>>> >>> >> Endpoint Check in effect. Checking
>>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>>>> >> >> >> >>>> >>> >> Could not connect to
>>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>>> >> >> >> >>>> >>> >> is
>>>> >> >> >> >>>> >>> >> not
>>>> >> >> >> >>>> >>> >> accessible
>>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> >>> >> Could not connect to
>>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>>> >> >> >> >>>> >>> >> is
>>>> >> >> >> >>>> >>> >> not
>>>> >> >> >> >>>> >>> >> accessible
>>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> >>> >> Could not connect to
>>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> ++ oc create -f -
>>>> >> >> >> >>>> >>> >> serviceaccount "heapster" created
>>>> >> >> >> >>>> >>> >> service "heapster" created
>>>> >> >> >> >>>> >>> >> replicationcontroller "heapster" created
>>>> >> >> >> >>>> >>> >> + echo 'Success!'
>>>> >> >> >> >>>> >>> >> Success!
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> --
>>>> >> >> >> >>>> >>> >> Srinivas Kotaru
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com>
>>>> >> >> >> >>>> >>> >> wrote:
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >----- Original Message -----
>>>> >> >> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)"
>>>> >> >> >> >>>> >>> >> >> <skotaru cisco com>
>>>> >> >> >> >>>> >>> >> >> To: users lists openshift redhat com
>>>> >> >> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>>>> >> >> >> >>>> >>> >> >> Subject: Metrics deployment
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Hi
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
>>>> >> >> >> >>>> >>> >> >> clusters. Am
>>>> >> >> >> >>>> >>> >> >> seeing
>>>> >> >> >> >>>> >>> >> >> below errors once metrics-deploy script was
>>>> >> >> >> >>>> >>> >> >> successful.
>>>> >> >> >> >>>> >>> >> >> I
>>>> >> >> >> >>>> >>> >> >> used
>>>> >> >> >> >>>> >>> >> >> our
>>>> >> >> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
>>>> >> >> >> >>>> >>> >> >> MASTER_URL
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>>>> >> >> >> >>>> >>> >> >> (
>>>> >> >> >> >>>> >>> >> >> hard coded as values)
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> template "hawkular-heapster" created
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Deploying the Heapster component
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> ++ '[' -n '' ']'
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> ++ oc create -f -
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>>>> >> >> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> serviceaccount "heapster" created
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> service "heapster" created
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> replicationcontroller "heapster" created
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> + echo 'Success!'
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Success!
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> # oc get pods
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >It looks like none of your containers are fully up and
>>>> >> >> >> >>>> >>> >> >running
>>>> >> >> >> >>>> >>> >> >yet.
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not
>>>> >> >> >> >>>> >>> >> >run,
>>>> >> >> >> >>>> >>> >> >and
>>>> >> >> >> >>>> >>> >> >Heapster
>>>> >> >> >> >>>> >>> >> >will
>>>> >> >> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first
>>>> >> >> >> >>>> >>> >> >step
>>>> >> >> >> >>>> >>> >> >will
>>>> >> >> >> >>>> >>> >> >be to
>>>> >> >> >> >>>> >>> >> >get
>>>> >> >> >> >>>> >>> >> >Cassandra running properly.
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Could not connect to
>>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>>>> >> >> >> >>>> >>> >> >> is
>>>> >> >> >> >>>> >>> >> >> not
>>>> >> >> >> >>>> >>> >> >> accessible
>>>> >> >> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Could not connect to
>>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
>>>> >> >> >> >>>> >>> >> >trying
>>>> >> >> >> >>>> >>> >> >to
>>>> >> >> >> >>>> >>> >> >push
>>>> >> >> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
>>>> >> >> >> >>>> >>> >> >Heapster
>>>> >> >> >> >>>> >>> >> >could
>>>> >> >> >> >>>> >>> >> >not
>>>> >> >> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the
>>>> >> >> >> >>>> >>> >> >Hawkular
>>>> >> >> >> >>>> >>> >> >Metrics
>>>> >> >> >> >>>> >>> >> >service
>>>> >> >> >> >>>> >>> >> >is
>>>> >> >> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> What is the wrong? Why it checking just
>>>> >> >> >> >>>> >>> >> >> hawkular-metrics
>>>> >> >> >> >>>> >>> >> >> rather
>>>> >> >> >> >>>> >>> >> >> full
>>>> >> >> >> >>>> >>> >> >> routing
>>>> >> >> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the
>>>> >> >> >> >>>> >>> >> >internal
>>>> >> >> >> >>>> >>> >> >hostname
>>>> >> >> >> >>>> >>> >> >used
>>>> >> >> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and
>>>> >> >> >> >>>> >>> >> >the
>>>> >> >> >> >>>> >>> >> >external
>>>> >> >> >> >>>> >>> >> >hostname (eg what is configured via
>>>> >> >> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>>>> >> >> >> >>>> >>> >> >The
>>>> >> >> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name
>>>> >> >> >> >>>> >>> >> >of
>>>> >> >> >> >>>> >>> >> >services,
>>>> >> >> >> >>>> >>> >> >which
>>>> >> >> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>>>> >> >> >> >>>> >>> >> >
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> --
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> Srinivas Kotaru
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> >> _______________________________________________
>>>> >> >> >> >>>> >>> >> >> users mailing list
>>>> >> >> >> >>>> >>> >> >> users lists openshift redhat com
>>>> >> >> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>> >> >> >> >>>> >>> >> >> 
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> >> 
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >>> 
>>>> >> >> >> >>>> >
>>>> >> >> >> >>>> 
>>>> >> >> >> >>>> 
>>>> >> >> >> >>
>>>> >> >> >> >
>>>> >> >> >> 
>>>> >> >> >> 
>>>> >> >> 
>>>> >> >> 
>>>> >> 
>>>> >> 
>>>> 
>>>> 
>>
>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]