[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



Matt

This issue fixed. Worked with RedHat support team yesterday night. It was narrow down to dnsmasq which was stopped on all nodes. As you knew, dnsmasq responsible for forwarding node DNS requests to master for service resolution. Since it was stopped, hawkular service unable to resolve to Metrics-Casandra service. Am not sure at this point, Ansible based install suppose to enable and restart dnsmasq across all worker nodes. 

All good now. Metrics service is up and running, able to visualize graphs on console. 

Thanks for your help. It was a great co operation and collaboration.  

-- 
Srinivas Kotaru

On 6/15/16, 8:45 AM, "Matt Wringe" <mwringe redhat com> wrote:

>The 'unknown error' is most likely because Hawkular Metrics cannot resolve the 'hawkular-cassandra' hostname.
>
>You should be able to exec into the Hawkular Metrics pod before it get shutdown and try to dig, curl or ping the 'hawkular-cassandra' hostname. If you get an error message here about the hostname not being resolveable then that is the problem.
>
>----- Original Message -----
>> From: "Matt Wringe" <mwringe redhat com>
>> To: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> Cc: users lists openshift redhat com
>> Sent: Wednesday, June 15, 2016 11:36:15 AM
>> Subject: Re: Metrics deployment
>> 
>> Can you please list the secrets used and the deployment options used when
>> deploying metrics?
>> 
>> If you redeploy everything, does the error still exist?
>> 
>> I am trying to check a few error conditions to see if I can reproduce the
>> 'unknown error' here, but not displaying the full error is a bug in Hawkular
>> Metrics which needs to be fixed.
>> 
>> ----- Original Message -----
>> > From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > To: "Matt Wringe" <mwringe redhat com>
>> > Cc: users lists openshift redhat com
>> > Sent: Tuesday, June 14, 2016 7:14:46 PM
>> > Subject: Re: Metrics deployment
>> > 
>> > 18:34:03,414 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:03,532 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> > error
>> > 18:34:03,533 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>> > connecting
>> > to Cassandra cluster in [1]s...
>> > 18:34:04,534 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:04,534 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> > 18:34:04,535 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>> > connecting
>> > to Cassandra cluster in [2]s...
>> > 18:34:06,535 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:06,535 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> > 18:34:06,536 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>> > connecting
>> > to Cassandra cluster in [3]s...
>> > 18:34:09,536 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:09,537 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> > 18:34:09,537 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>> > connecting
>> > to Cassandra cluster in [4]s...
>> > 18:34:13,537 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:13,653 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> > error
>> > 18:34:13,653 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>> > connecting
>> > to Cassandra cluster in [1]s...
>> > 18:34:14,654 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 18:34:14,654 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
>> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> > 18:34:14,654 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>> > connecting to Cassandra cluster in [2]s...
>> > 18:34:16,655 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
>> > service
>> > 
>> > 
>> > 
>> > --
>> > Srinivas Kotaru
>> > 
>> > On 6/14/16, 4:02 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > wrote:
>> > 
>> > >9042 firewall created across all nodes to node communication. But issue
>> > >still persist. Port 53 already open between all nodes to masters.
>> > >
>> > >Almost dead struck, appreciated any pointers to get fresh thoughts
>> > >
>> > >--
>> > >Srinivas Kotaru
>> > >
>> > >On 6/14/16, 3:16 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > >wrote:
>> > >
>> > >>Ok. Let me open the port 9043. This port should be open between node to
>> > >>node right?
>> > >>
>> > >>
>> > >>--
>> > >>Srinivas Kotaru
>> > >>
>> > >>On 6/14/16, 3:02 PM, "Matt Wringe" <mwringe redhat com> wrote:
>> > >>
>> > >>>----- Original Message -----
>> > >>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > >>>> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> Cc: users lists openshift redhat com
>> > >>>> Sent: Tuesday, June 14, 2016 5:40:36 PM
>> > >>>> Subject: Re: Metrics deployment
>> > >>>> 
>> > >>>> Matt.
>> > >>>> 
>> > >>>> Sure, let us figure it out Hawkular side. Am here pasting 2 logs
>> > >>>> 
>> > >>>> 1. oc logs –f
>> > >>>> 2. cat /opt/eap/standalone/log/server.log
>> > >>>
>> > >>>Hmm, its getting an 'unknown error' when trying to connect to Cassandra,
>> > >>>which doesn't really tell us anything :/
>> > >>>
>> > >>>The port that Hawkular Metrics uses to connect to Cassandra is 9042, you
>> > >>>may also want to make sure that the DNS port is also open.
>> > >>>
>> > >>>> 
>> > >>>> Srinivas Kotaru
>> > >>>> 
>> > >>>>  
>> > >>>> --
>> > >>>> Srinivas Kotaru
>> > >>>> 
>> > >>>> On 6/14/16, 2:28 PM, "Matt Wringe" <mwringe redhat com> wrote:
>> > >>>> 
>> > >>>> >
>> > >>>> >
>> > >>>> >----- Original Message -----
>> > >>>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > >>>> >> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> Cc: users lists openshift redhat com
>> > >>>> >> Sent: Tuesday, June 14, 2016 4:44:04 PM
>> > >>>> >> Subject: Re: Metrics deployment
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> I’m still struck with this issue. It is kind of chicken and egg
>> > >>>> >> problem.
>> > >>>> >> Heapster health probes failing since it is waiting for hawkular to
>> > >>>> >> start.
>> > >>>> >> Hawkular health probes are failing since it is unable to connect
>> > >>>> >> Casandra.
>> > >>>> >> Cansandra health probe also failing.
>> > >>>> >
>> > >>>> >The first step is to get Cassandra running, ignore Hawkular Metrics
>> > >>>> >and
>> > >>>> >Heapster until you have Cassandra running properly. Without Cassandra
>> > >>>> >being
>> > >>>> >able to run, those other components will not fully start.
>> > >>>> >
>> > >>>> >> 
>> > >>>> >> @Matt:  Internal DNS looks ups working. I’m able to create apps,
>> > >>>> >> build and
>> > >>>> >> deploy code. Router and registry components also working as
>> > >>>> >> expected
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> oc get pods
>> > >>>> >> NAME                         READY     STATUS      RESTARTS   AGE
>> > >>>> >> hawkular-cassandra-1-mxd2m   1/1       Running     0          1h
>> > >>>> >> hawkular-metrics-gvp9k       0/1       Running     4          11m
>> > >>>> >> heapster-uleul               0/1       Running     4          11m
>> > >>>> >> metrics-deployer-2z75w       0/1       Completed   0          1h
>> > >>>> >
>> > >>>> >Cassandra being in "READY 1/1" means that it started up properly. So
>> > >>>> >Cassandra is running. Why do you think its not running? Things like
>> > >>>> >readiness probes are expected to fail until the pod is ready. Just
>> > >>>> >because
>> > >>>> >there is a failure in the events doesn't mean its an error condition.
>> > >>>> >
>> > >>>> >Hawkular Metrics is not running here. So lets figure out why. Ignore
>> > >>>> >Heapster until Hawkular Metrics is started.
>> > >>>> >
>> > >>>> >> 
>> > >>>> >> heapster events:
>> > >>>> >> ------------------------
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> Events:
>> > >>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason
>> > >>>> >>   		Message
>> > >>>> >>   ---------	--------	-----	----					-------------			--------	------
>> > >>>> >>   		-------
>> > >>>> >>   1m		1m		1	{default-scheduler }							Normal		Scheduled
>> > >>>> >>   	Successfully
>> > >>>> >>   assigned heapster-uleul to l3inpn-id2-004.cisco.com
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{heapster}
>> > >>>> >>   	Normal		Pulling		pulling image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{heapster}
>> > >>>> >>   	Normal		Pulled		Successfully pulled image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-heapster:latest"
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{heapster}
>> > >>>> >>   	Normal		Created		Created container with docker id a22bb9a246ca
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{heapster}
>> > >>>> >>   	Normal		Started		Started container with docker id a22bb9a246ca
>> > >>>> >>   1m		5s		10	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{heapster}
>> > >>>> >>   	Warning		Unhealthy	Readiness probe failed: The heapster process
>> > >>>> >>   	is
>> > >>>> >>   	not
>> > >>>> >>   	yet
>> > >>>> >>   started, it is waiting for the Hawkular Metrics to start.
>> > >>>> >
>> > >>>> >Readiness probe failing here is expected, and we know why from the
>> > >>>> >error
>> > >>>> >message:
>> > >>>> >"Readiness probe failed: The heapster process is not yet started, it
>> > >>>> >is
>> > >>>> >waiting for the Hawkular Metrics to start."
>> > >>>> >
>> > >>>> >Once Hawkular Metrics is running then Heapster should automatically
>> > >>>> >start
>> > >>>> >functioning.
>> > >>>> >
>> > >>>> >> 
>> > >>>> >> Hawkular events :
>> > >>>> >> ===============
>> > >>>> >> 
>> > >>>> >> Events:
>> > >>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason
>> > >>>> >>   		Message
>> > >>>> >>   ---------	--------	-----	----					-------------				--------
>> > >>>> >>   	------
>> > >>>> >>   		-------
>> > >>>> >>   1m		1m		1	{default-scheduler }								Normal		Scheduled
>> > >>>> >>   	Successfully
>> > >>>> >>   assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulling		pulling image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Normal		Pulled		Successfully
>> > >>>> >>   	pulled
>> > >>>> >>   image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Normal		Created		Created
>> > >>>> >>   	container
>> > >>>> >>   	with
>> > >>>> >>   docker id 6e32459624d7
>> > >>>> >>   1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Normal		Started		Started
>> > >>>> >>   	container
>> > >>>> >>   	with
>> > >>>> >>   docker id 6e32459624d7
>> > >>>> >>   1m		57s		4	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Warning		Unhealthy	Readiness
>> > >>>> >>   	probe
>> > >>>> >>   failed: Failed to access the status endpoint : <urlopen error
>> > >>>> >>   [Errno
>> > >>>> >>   111]
>> > >>>> >>   Connection refused>.
>> > >>>> >> 
>> > >>>> >>   46s	7s	5	{kubelet l3inpn-id2-004.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-metrics}	Warning	Unhealthy	Readiness
>> > >>>> >>   	probe
>> > >>>> >>   failed:
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> Casandra events
>> > >>>> >> ==============
>> > >>>> >> 
>> > >>>> >> Events:
>> > >>>> >>   FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason
>> > >>>> >>   		Message
>> > >>>> >>   ---------	--------	-----	----					-------------				--------
>> > >>>> >>   	------
>> > >>>> >>   		-------
>> > >>>> >>   1h		1h		1	{default-scheduler }								Normal		Scheduled
>> > >>>> >>   	Successfully
>> > >>>> >>   assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
>> > >>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling
>> > >>>> >>   	image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> > >>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Pulled
>> > >>>> >>   			Successfully
>> > >>>> >>   	pulled
>> > >>>> >>   image
>> > >>>> >>   "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> > >>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>> > >>>> >>   	container
>> > >>>> >>   with docker id 2a1390edd4f1
>> > >>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>> > >>>> >>   	container
>> > >>>> >>   with docker id 2a1390edd4f1
>> > >>>> >>   1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy
>> > >>>> >>   		Readiness
>> > >>>> >>   	probe
>> > >>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>> > >>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>> > >>>> >> 'Connection refused'.
>> > >>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
>> > >>>> >> unary
>> > >>>> >> operator expected
>> > >>>> >> Cassandra not in the up and normal state. Current state is
>> > >>>> >> 
>> > >>>> >>   1h	1h	2	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >>   	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy
>> > >>>> >>   		Readiness
>> > >>>> >>   	probe
>> > >>>> >>   failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>> > >>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>> > >>>> >> 'Connection refused'.
>> > >>>> >> Cassandra not in the up and normal state. Current state is
>> > >>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
>> > >>>> >> unary
>> > >>>> >> operator expected
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> Logs:
>> > >>>> >> ====
>> > >>>> >> 
>> > >>>> >> # oc logs -f heapster-uleul
>> > >>>> >> Endpoint Check in effect. Checking
>> > >>>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> > >>>> >> Could not connect to
>> > >>>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> Curl exit code: 6. Status Code 000
>> > >>>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> > >>>> >> accessible
>> > >>>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> > >>>> >
>> > >>>> >Heapster is not starting, as expected, the problem is that it cannot
>> > >>>> >access
>> > >>>> >that URL because it cannot resolve the host (error code 6). This is
>> > >>>> >expected, that hostname will not be resolvable until Hawkular Metrics
>> > >>>> >starts.
>> > >>>> >
>> > >>>> >> # oc logs -f hawkular-metrics-gvp9k
>> > >>>> >> 
>> > >>>> >> 16:42:43,201 INFO  [org.jboss.as.server] (Controller Boot Thread)
>> > >>>> >> JBAS015888:
>> > >>>> >> Creating http management service using socket-binding
>> > >>>> >> (management-http)
>> > >>>> >> 16:42:43,202 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version
>> > >>>> >> 3.0.14.GA-redhat-1
>> > >>>> >> 16:42:43,208 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
>> > >>>> >> Implementation Version 3.0.14.GA-redhat-1
>> > >>>> >> 16:42:43,223 INFO  [org.jboss.remoting] (MSC service thread 1-1)
>> > >>>> >> JBoss
>> > >>>> >> Remoting version 3.3.5.Final-redhat-1
>> > >>>> >
>> > >>>> >I need the full logs for Hawkular Metrics, if you could get me the
>> > >>>> >logs
>> > >>>> >for
>> > >>>> >the last container that was run that would be helpful (eg oc get logs
>> > >>>> >-p
>> > >>>> >....)
>> > >>>> >
>> > >>>> >> 
>> > >>>> >> # oc logs -f hawkular-cassandra-1-mxd2m
>> > >>>> >> 
>> > >>>> >> INFO  20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4
>> > >>>> >> sstables
>> > >>>> >> to
>> > >>>> >> [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
>> > >>>> >> to level=0.  1,643 bytes to 1,155 (~70% of original) in 207ms =
>> > >>>> >> 0.005321MB/s.  0 total partitions merged to 11.  Partition merge
>> > >>>> >> counts
>> > >>>> >> were
>> > >>>> >> {1:11, }
>> > >>>> >> INFO  20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
>> > >>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
>> > >>>> >> ]
>> > >>>> >> INFO  20:15:20 Writing Memtable-schema_columns 672348153(49.749KiB
>> > >>>> >> serialized
>> > >>>> >> bytes, 967 ops, 0%/0% of on/off-heap limit)
>> > >>>> >> INFO  20:15:20 Completed flushing
>> > >>>> >> /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
>> > >>>> >> (11.212KiB) for commitlog position
>> > >>>> >> ReplayPosition(segmentId=1465931718336,
>> > >>>> >> position=159511)
>> > >>>> >> INFO  20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4
>> > >>>> >> sstables
>> > >>>> >> to
>> > >>>> >> [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
>> > >>>> >> to level=0.  128 bytes to 32 (~25% of original) in 146ms =
>> > >>>> >> 0.000209MB/s.
>> > >>>> >> 0
>> > >>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>> > >>>> >> INFO  20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
>> > >>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
>> > >>>> >> ]
>> > >>>> >> INFO  20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4
>> > >>>> >> sstables
>> > >>>> >> to
>> > >>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
>> > >>>> >> to level=0.  128 bytes to 32 (~25% of original) in 155ms =
>> > >>>> >> 0.000197MB/s.
>> > >>>> >> 0
>> > >>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>> > >>>> >> INFO  20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
>> > >>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
>> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
>> > >>>> >> ]
>> > >>>> >> INFO  20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4
>> > >>>> >> sstables
>> > >>>> >> to
>> > >>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
>> > >>>> >> to level=0.  128 bytes to 32 (~25% of original) in 83ms =
>> > >>>> >> 0.000368MB/s.  0
>> > >>>> >> total partitions merged to 1.  Partition merge counts were {4:1, }
>> > >>>> >> INFO  20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4
>> > >>>> >> sstables
>> > >>>> >> to
>> > >>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
>> > >>>> >> to level=0.  2,538 bytes to 527 (~20% of original) in 176ms =
>> > >>>> >> 0.002856MB/s.
>> > >>>> >> 0 total partitions merged to 3.  Partition merge counts were {4:3,
>> > >>>> >> }
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> almost struck here. Any points to look for? Any ports need to be
>> > >>>> >> open
>> > >>>> >> explicitly ?
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> >> --
>> > >>>> >> Srinivas Kotaru
>> > >>>> >> 
>> > >>>> >> On 6/14/16, 11:27 AM, "Matt Wringe" <mwringe redhat com> wrote:
>> > >>>> >> 
>> > >>>> >> >----- Original Message -----
>> > >>>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > >>>> >> >> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> Cc: users lists openshift redhat com
>> > >>>> >> >> Sent: Tuesday, June 14, 2016 2:09:49 PM
>> > >>>> >> >> Subject: Re: Metrics deployment
>> > >>>> >> >> 
>> > >>>> >> >> Not sure what do you mean by installation. Am just running oc
>> > >>>> >> >> new-app
>> > >>>> >> >> -f
>> > >>>> >> >> metrics-deployer.yaml with default values except HOST_NAME and
>> > >>>> >> >> PV
>> > >>>> >> >> storage.
>> > >>>> >> >
>> > >>>> >> >I would suspect something wrong with your node or cluster
>> > >>>> >> >installation.
>> > >>>> >> >Or
>> > >>>> >> >you have firewall rules blocking connections between your nodes so
>> > >>>> >> >that
>> > >>>> >> >pod
>> > >>>> >> >cannot connect with each other or access the OpenShift DNS server.
>> > >>>> >> >
>> > >>>> >> >The lifecycle hooks exist to make sure that components only enter
>> > >>>> >> >the
>> > >>>> >> >ready
>> > >>>> >> >state when they are fully started and ready.
>> > >>>> >> >
>> > >>>> >> >Can you check the Hawkular Metrics status page and see what that
>> > >>>> >> >outputs?
>> > >>>> >> >eg
>> > >>>> >> >https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>> > >>>> >> >
>> > >>>> >> >> 
>> > >>>> >> >> I just deleted entire metrics setup and re running. But not sure
>> > >>>> >> >> this
>> > >>>> >> >> will
>> > >>>> >> >> fix the issue.
>> > >>>> >> >> 
>> > >>>> >> >> $ ./delete_metrics-infra.sh
>> > >>>> >> >> replicationcontroller "hawkular-cassandra-1" deleted
>> > >>>> >> >> replicationcontroller "hawkular-metrics" deleted
>> > >>>> >> >> replicationcontroller "heapster" deleted
>> > >>>> >> >> route "hawkular-metrics" deleted
>> > >>>> >> >> service "hawkular-cassandra" deleted
>> > >>>> >> >> service "hawkular-cassandra-nodes" deleted
>> > >>>> >> >> service "hawkular-metrics" deleted
>> > >>>> >> >> service "heapster" deleted
>> > >>>> >> >> pod "heapster-lyf65" deleted
>> > >>>> >> >> serviceaccount "cassandra" deleted
>> > >>>> >> >> serviceaccount "hawkular" deleted
>> > >>>> >> >> serviceaccount "heapster" deleted
>> > >>>> >> >> template "hawkular-cassandra-node-emptydir" deleted
>> > >>>> >> >> template "hawkular-cassandra-node-pv" deleted
>> > >>>> >> >> template "hawkular-cassandra-services" deleted
>> > >>>> >> >> template "hawkular-heapster" deleted
>> > >>>> >> >> template "hawkular-metrics" deleted
>> > >>>> >> >> template "hawkular-support" deleted
>> > >>>> >> >> secret "hawkular-cassandra-certificate" deleted
>> > >>>> >> >> secret "hawkular-cassandra-secrets" deleted
>> > >>>> >> >> secret "hawkular-metrics-account" deleted
>> > >>>> >> >> secret "hawkular-metrics-certificate" deleted
>> > >>>> >> >> secret "hawkular-metrics-secrets" deleted
>> > >>>> >> >> secret "heapster-secrets" deleted
>> > >>>> >> >> 
>> > >>>> >> >> --
>> > >>>> >> >> Srinivas Kotaru
>> > >>>> >> >> 
>> > >>>> >> >> On 6/14/16, 10:53 AM, "Matt Wringe" <mwringe redhat com> wrote:
>> > >>>> >> >> 
>> > >>>> >> >> >----- Original Message -----
>> > >>>> >> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> > >>>> >> >> >> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> Cc: users lists openshift redhat com
>> > >>>> >> >> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>> > >>>> >> >> >> Subject: Re: Metrics deployment
>> > >>>> >> >> >> 
>> > >>>> >> >> >> I removed readiness probes from both hawkular-cassandra-1 &
>> > >>>> >> >> >> hawkular-metrics
>> > >>>> >> >> >> as both status shows probes failed.
>> > >>>> >> >> >
>> > >>>> >> >> >You should not have to remove the probes, this indicates that
>> > >>>> >> >> >something
>> > >>>> >> >> >is
>> > >>>> >> >> >wrong with your installation.
>> > >>>> >> >> >
>> > >>>> >> >> >> 
>> > >>>> >> >> >> It looks good now. Both containers looks and running
>> > >>>> >> >> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>> > >>>> >> >> >> heapster-7yl34
>> > >>>> >> >> >> logs still shows Could not connect to
>> > >>>> >> >> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl
>> > >>>> >> >> >> exit
>> > >>>> >> >> >> code:
>> > >>>> >> >> >> 6.
>> > >>>> >> >> >> Status Code 000.
>> > >>>> >> >> >> 
>> > >>>> >> >> >> Are we good or still had issues?
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> >> # oc get pods
>> > >>>> >> >> >> NAME                         READY     STATUS    RESTARTS
>> > >>>> >> >> >> AGE
>> > >>>> >> >> >> hawkular-cassandra-1-kr8ka   1/1       Running   0
>> > >>>> >> >> >> 6m
>> > >>>> >> >> >> hawkular-metrics-vhe3u       1/1       Running   2
>> > >>>> >> >> >> 5m
>> > >>>> >> >> >> heapster-7yl34               0/1       Running   2
>> > >>>> >> >> >> 5m
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> >> --
>> > >>>> >> >> >> Srinivas Kotaru
>> > >>>> >> >> >> 
>> > >>>> >> >> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> <skotaru cisco com>
>> > >>>> >> >> >> wrote:
>> > >>>> >> >> >> 
>> > >>>> >> >> >> >Matt
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >Just want to share more info by running describe pod.
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >It seems to be health probe failing. Do you think it is the
>> > >>>> >> >> >> >issue?
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> ># oc describe pod hawkular-cassandra-1-it5uh
>> > >>>> >> >> >> >Name:		hawkular-cassandra-1-it5uh
>> > >>>> >> >> >> >Namespace:	openshift-infra
>> > >>>> >> >> >> >Node:		l3inpn-id2-003.cisco.com/173.36.96.16
>> > >>>> >> >> >> >Start Time:	Tue, 14 Jun 2016 16:36:21 +0000
>> > >>>> >> >> >> >Labels:
>> > >>>> >> >> >> >		metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>> > >>>> >> >> >> >Status:		Running
>> > >>>> >> >> >> >IP:		10.1.9.2
>> > >>>> >> >> >> >Controllers:	ReplicationController/hawkular-cassandra-1
>> > >>>> >> >> >> >Containers:
>> > >>>> >> >> >> >  hawkular-cassandra-1:
>> > >>>> >> >> >> >    Container ID:
>> > >>>> >> >> >> >    	docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>> > >>>> >> >> >> >    Image:
>> > >>>> >> >> >> >    		registry.access.redhat.com/openshift3/metrics-cassandra:latest
>> > >>>> >> >> >> >    Image ID:
>> > >>>> >> >> >> >    		docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>> > >>>> >> >> >> >    Ports:		9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>> > >>>> >> >> >> >    Command:
>> > >>>> >> >> >> >      /opt/apache-cassandra/bin/cassandra-docker.sh
>> > >>>> >> >> >> >      --cluster_name=hawkular-metrics
>> > >>>> >> >> >> >      --data_volume=/cassandra_data
>> > >>>> >> >> >> >      --internode_encryption=all
>> > >>>> >> >> >> >      --require_node_auth=true
>> > >>>> >> >> >> >      --enable_client_encryption=true
>> > >>>> >> >> >> >      --require_client_auth=true
>> > >>>> >> >> >> >      --keystore_file=/secret/cassandra.keystore
>> > >>>> >> >> >> >      --keystore_password_file=/secret/cassandra.keystore.password
>> > >>>> >> >> >> >      --truststore_file=/secret/cassandra.truststore
>> > >>>> >> >> >> >      --truststore_password_file=/secret/cassandra.truststore.password
>> > >>>> >> >> >> >      --cassandra_pem_file=/secret/cassandra.pem
>> > >>>> >> >> >> >    QoS Tier:
>> > >>>> >> >> >> >      cpu:		BestEffort
>> > >>>> >> >> >> >      memory:		BestEffort
>> > >>>> >> >> >> >    State:		Running
>> > >>>> >> >> >> >      Started:		Tue, 14 Jun 2016 16:37:01 +0000
>> > >>>> >> >> >> >    Ready:		True
>> > >>>> >> >> >> >    Restart Count:	0
>> > >>>> >> >> >> >    Readiness:		exec
>> > >>>> >> >> >> >    [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>> > >>>> >> >> >> >    delay=0s timeout=1s period=10s #success=1 #failure=3
>> > >>>> >> >> >> >    Environment Variables:
>> > >>>> >> >> >> >      CASSANDRA_MASTER:	true
>> > >>>> >> >> >> >      POD_NAMESPACE:	openshift-infra (v1:metadata.namespace)
>> > >>>> >> >> >> >Conditions:
>> > >>>> >> >> >> >  Type		Status
>> > >>>> >> >> >> >  Ready 	True
>> > >>>> >> >> >> >Volumes:
>> > >>>> >> >> >> >  cassandra-data:
>> > >>>> >> >> >> >    Type:	PersistentVolumeClaim (a reference to a
>> > >>>> >> >> >> >    PersistentVolumeClaim
>> > >>>> >> >> >> >    in
>> > >>>> >> >> >> >    the same namespace)
>> > >>>> >> >> >> >    ClaimName:	metrics-cassandra-1
>> > >>>> >> >> >> >    ReadOnly:	false
>> > >>>> >> >> >> >  hawkular-cassandra-secrets:
>> > >>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>> > >>>> >> >> >> >    SecretName:	hawkular-cassandra-secrets
>> > >>>> >> >> >> >  cassandra-token-4urfd:
>> > >>>> >> >> >> >    Type:	Secret (a volume populated by a Secret)
>> > >>>> >> >> >> >    SecretName:	cassandra-token-4urfd
>> > >>>> >> >> >> >Events:
>> > >>>> >> >> >> >  FirstSeen	LastSeen	Count	From					SubobjectPath				Type
>> > >>>> >> >> >> >  		Reason
>> > >>>> >> >> >> >  		Message
>> > >>>> >> >> >> >  ---------	--------	-----	----					-------------
>> > >>>> >> >> >> >  				--------
>> > >>>> >> >> >> >  	------
>> > >>>> >> >> >> >  		-------
>> > >>>> >> >> >> >  27m		27m		1	{default-scheduler }								Normal		Scheduled
>> > >>>> >> >> >> >  	Successfully
>> > >>>> >> >> >> >  assigned hawkular-cassandra-1-it5uh to
>> > >>>> >> >> >> >  l3inpn-id2-003.cisco.com
>> > >>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulling
>> > >>>> >> >> >> >  			pulling
>> > >>>> >> >> >> >  	image
>> > >>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> > >>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulled
>> > >>>> >> >> >> >  			Successfully
>> > >>>> >> >> >> >  	pulled
>> > >>>> >> >> >> >  image
>> > >>>> >> >> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> > >>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Created
>> > >>>> >> >> >> >  			Created
>> > >>>> >> >> >> >  	container
>> > >>>> >> >> >> >  with docker id 17a9575eb655
>> > >>>> >> >> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Started
>> > >>>> >> >> >> >  			Started
>> > >>>> >> >> >> >  	container
>> > >>>> >> >> >> >  with docker id 17a9575eb655
>> > >>>> >> >> >> >  27m		26m		3	{kubelet l3inpn-id2-003.cisco.com}
>> > >>>> >> >> >> >  	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy
>> > >>>> >> >> >> >  		Readiness
>> > >>>> >> >> >> >  	probe
>> > >>>> >> >> >> >  failed: cat: /etc/ld.so.conf.d/*.conf: No such file or
>> > >>>> >> >> >> >  directory
>> > >>>> >> >> >> >nodetool: Failed to connect to '127.0.0.1:7199' -
>> > >>>> >> >> >> >ConnectException:
>> > >>>> >> >> >> >'Connection refused'.
>> > >>>> >> >> >> >Cassandra not in the up and normal state. Current state is
>> > >>>> >> >> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line
>> > >>>> >> >> >> >28:
>> > >>>> >> >> >> >[: =:
>> > >>>> >> >> >> >unary
>> > >>>> >> >> >> >operator expected
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >--
>> > >>>> >> >> >> >Srinivas Kotaru
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> ><skotaru cisco com>
>> > >>>> >> >> >> >wrote:
>> > >>>> >> >> >> >
>> > >>>> >> >> >> >>Matt
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>DNS service is working
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>~ dig +short @master-01
>> > >>>> >> >> >> >>kubernetes.default.svc.cluster.local
>> > >>>> >> >> >> >>172.30.0.1
>> > >>>> >> >> >> >>~  dig +short @master-01
>> > >>>> >> >> >> >>jenkins.alln-test.svc.cluster.local
>> > >>>> >> >> >> >>172.30.85.148
>> > >>>> >> >> >> >>~  dig +short @master-01
>> > >>>> >> >> >> >>cakephp-example.alln-test.svc.cluster.local
>> > >>>> >> >> >> >>172.30.31.6
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>I captured hawkular-metrics, it shows the problem. It seems
>> > >>>> >> >> >> >>to be
>> > >>>> >> >> >> >>unable
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>connect Cassandra cluster
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >># oc exec hawkular-metrics-qldd8 cat
>> > >>>> >> >> >> >>/opt/eap/standalone/log/server.log
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan]
>> > >>>> >> >> >> >>(ServerService
>> > >>>> >> >> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan
>> > >>>> >> >> >> >>subsystem.
>> > >>>> >> >> >> >>12:40:58,898 INFO  [org.jboss.as.webservices]
>> > >>>> >> >> >> >>(ServerService
>> > >>>> >> >> >> >>Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>48) JBAS015537: Activating WebServices Extension
>> > >>>> >> >> >> >>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>46)
>> > >>>> >> >> >> >>JBAS010153: Node identifier property is set to the default
>> > >>>> >> >> >> >>value.
>> > >>>> >> >> >> >>Please
>> > >>>> >> >> >> >>make sure it is unique.
>> > >>>> >> >> >> >>12:40:58,902 INFO  [org.jboss.as.security] (ServerService
>> > >>>> >> >> >> >>Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>44) JBAS013371: Activating Security Subsystem
>> > >>>> >> >> >> >>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService
>> > >>>> >> >> >> >>Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>39)
>> > >>>> >> >> >> >>JBAS011800: Activating Naming Subsystem
>> > >>>> >> >> >> >>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>37)
>> > >>>> >> >> >> >>JBAS012615: Activated the following JSF Implementations:
>> > >>>> >> >> >> >>[main,
>> > >>>> >> >> >> >>1.2]
>> > >>>> >> >> >> >>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>> > >>>> >> >> >> >>1.0.32.Final-redhat-1)
>> > >>>> >> >> >> >>12:40:58,981 INFO  [org.jboss.as.security] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-2)
>> > >>>> >> >> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>> > >>>> >> >> >> >>12:40:58,961 INFO
>> > >>>> >> >> >> >>[org.jboss.as.connector.subsystems.datasources]
>> > >>>> >> >> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying
>> > >>>> >> >> >> >>JDBC-compliant
>> > >>>> >> >> >> >>driver class org.h2.Driver (version 1.3)
>> > >>>> >> >> >> >>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-2)
>> > >>>> >> >> >> >>JBAS011802: Starting Naming Service
>> > >>>> >> >> >> >>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-4)
>> > >>>> >> >> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>> > >>>> >> >> >> >>12:40:59,614 INFO
>> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> > >>>> >> >> >> >>http-/0.0.0.0:8444
>> > >>>> >> >> >> >>12:40:59,620 INFO
>> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> > >>>> >> >> >> >>http-/0.0.0.0:8080
>> > >>>> >> >> >> >>12:40:59,621 INFO
>> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on:
>> > >>>> >> >> >> >>http-/0.0.0.0:8444
>> > >>>> >> >> >> >>12:40:59,627 INFO
>> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on:
>> > >>>> >> >> >> >>http-/0.0.0.0:8080
>> > >>>> >> >> >> >>12:40:59,654 INFO
>> > >>>> >> >> >> >>[org.jboss.as.connector.subsystems.datasources]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service thread 1-3) JBAS010400: Bound data source
>> > >>>> >> >> >> >>[java:jboss/datasources/ExampleDS]
>> > >>>> >> >> >> >>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>> > >>>> >> >> >> >>(runtime-name:
>> > >>>> >> >> >> >>"activemq-rar.rar")
>> > >>>> >> >> >> >>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1) JBAS015876: Starting deployment of
>> > >>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>> > >>>> >> >> >> >>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService
>> > >>>> >> >> >> >>for
>> > >>>> >> >> >> >>directory
>> > >>>> >> >> >> >>/opt/eap/standalone/deployments
>> > >>>> >> >> >> >>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-2)
>> > >>>> >> >> >> >>JBAS017100: Listening on 127.0.0.1:9999
>> > >>>> >> >> >> >>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-3)
>> > >>>> >> >> >> >>JBAS015102: Unable to load console module for slot eap,
>> > >>>> >> >> >> >>disabling
>> > >>>> >> >> >> >>console
>> > >>>> >> >> >> >>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF
>> > >>>> >> >> >> >>Server
>> > >>>> >> >> >> >>4.3.5.Final-redhat-3
>> > >>>> >> >> >> >>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-4)
>> > >>>> >> >> >> >>JBAS017100: Listening on 0.0.0.0:4447
>> > >>>> >> >> >> >>12:41:01,361 INFO
>> > >>>> >> >> >> >>[org.jboss.as.connector.deployers.RADeployer]
>> > >>>> >> >> >> >>(MSC
>> > >>>> >> >> >> >>service thread 1-3) IJ020001: Required license terms for
>> > >>>> >> >> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>> > >>>> >> >> >> >>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
>> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>is
>> > >>>> >> >> >> >>using a private module
>> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>> > >>>> >> >> >> >>which may be changed or removed in future versions without
>> > >>>> >> >> >> >>notice.
>> > >>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
>> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>is
>> > >>>> >> >> >> >>using a private module
>> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
>> > >>>> >> >> >> >>which may be changed or removed in future versions without
>> > >>>> >> >> >> >>notice.
>> > >>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
>> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>is
>> > >>>> >> >> >> >>using a private module
>> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>> > >>>> >> >> >> >>which may be changed or removed in future versions without
>> > >>>> >> >> >> >>notice.
>> > >>>> >> >> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
>> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>is
>> > >>>> >> >> >> >>using a private module
>> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
>> > >>>> >> >> >> >>which may be changed or removed in future versions without
>> > >>>> >> >> >> >>notice.
>> > >>>> >> >> >> >>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1)
>> > >>>> >> >> >> >>JBAS016002: Processing weld deployment
>> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>> > >>>> >> >> >> >>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1)
>> > >>>> >> >> >> >>JBAS016005: Starting Services for CDI deployment:
>> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>> > >>>> >> >> >> >>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-1)
>> > >>>> >> >> >> >>WELD-000900 1.1.31 (redhat)
>> > >>>> >> >> >> >>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service
>> > >>>> >> >> >> >>thread
>> > >>>> >> >> >> >>1-3)
>> > >>>> >> >> >> >>JBAS016008: Starting weld service for deployment
>> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
>> > >>>> >> >> >> >>12:41:02,820 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>51)
>> > >>>> >> >> >> >>JBAS018210: Register web context: /hawkular/metrics
>> > >>>> >> >> >> >>12:41:02,958 INFO  [org.jboss.as.server] (ServerService
>> > >>>> >> >> >> >>Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>28)
>> > >>>> >> >> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war"
>> > >>>> >> >> >> >>(runtime-name :
>> > >>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war")
>> > >>>> >> >> >> >>12:41:02,959 INFO  [org.jboss.as.server] (ServerService
>> > >>>> >> >> >> >>Thread
>> > >>>> >> >> >> >>Pool
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>28)
>> > >>>> >> >> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>> > >>>> >> >> >> >>"activemq-rar.rar")
>> > >>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>> > >>>> >> >> >> >>JBAS015961:
>> > >>>> >> >> >> >>Http
>> > >>>> >> >> >> >>management interface listening on
>> > >>>> >> >> >> >>http://127.0.0.1:9990/management
>> > >>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>> > >>>> >> >> >> >>JBAS015951:
>> > >>>> >> >> >> >>Admin console listening on http://127.0.0.1:9990
>> > >>>> >> >> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
>> > >>>> >> >> >> >>JBAS015874:
>> > >>>> >> >> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in
>> > >>>> >> >> >> >>6593ms -
>> > >>>> >> >> >> >>Started
>> > >>>> >> >> >> >>267 of 303 services (62 services are lazy, passive or
>> > >>>> >> >> >> >>on-demand)
>> > >>>> >> >> >> >>12:41:02,999 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra:
>> > >>>> >> >> >> >>unknown
>> > >>>> >> >> >> >>error
>> > >>>> >> >> >> >>12:41:02,999 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>> > >>>> >> >> >> >>12:41:04,000 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:04,001 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:04,001 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>> > >>>> >> >> >> >>12:41:06,002 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:06,002 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:06,003 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>> > >>>> >> >> >> >>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2)
>> > >>>> >> >> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>> > >>>> >> >> >> >>12:41:08,299 INFO
>> > >>>> >> >> >> >>[org.hibernate.validator.internal.util.Version]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>> > >>>> >> >> >> >>4.3.2.Final-redhat-2
>> > >>>> >> >> >> >>12:41:08,418 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>> > >>>> >> >> >> >>starting
>> > >>>> >> >> >> >>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>> > >>>> >> >> >> >>javax.ws.rs.core.Application: class
>> > >>>> >> >> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>> > >>>> >> >> >> >>12:41:08,421 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
>> > >>>> >> >> >> >>starting
>> > >>>> >> >> >> >>12:41:08,422 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> > >>>> >> >> >> >>12:41:08,422 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> > >>>> >> >> >> >>12:41:09,003 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:09,004 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:09,004 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>> > >>>> >> >> >> >>12:41:13,004 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:13,123 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra:
>> > >>>> >> >> >> >>unknown
>> > >>>> >> >> >> >>error
>> > >>>> >> >> >> >>12:41:13,124 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>> > >>>> >> >> >> >>12:41:14,124 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:14,124 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:14,125 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>> > >>>> >> >> >> >>12:41:16,125 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:16,126 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:16,126 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>> > >>>> >> >> >> >>12:41:19,126 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:19,127 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:19,127 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>> > >>>> >> >> >> >>12:41:23,128 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:23,249 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra:
>> > >>>> >> >> >> >>unknown
>> > >>>> >> >> >> >>error
>> > >>>> >> >> >> >>12:41:23,249 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>> > >>>> >> >> >> >>12:41:24,250 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:24,250 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:24,250 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>> > >>>> >> >> >> >>12:41:26,251 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:26,251 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:26,251 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>> > >>>> >> >> >> >>12:41:29,252 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:29,252 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:29,253 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>> > >>>> >> >> >> >>12:41:33,253 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:33,376 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra:
>> > >>>> >> >> >> >>unknown
>> > >>>> >> >> >> >>error
>> > >>>> >> >> >> >>12:41:33,377 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>> > >>>> >> >> >> >>12:41:34,377 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:34,378 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:34,378 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>> > >>>> >> >> >> >>12:41:36,378 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:36,379 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:36,379 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
>> > >>>> >> >> >> >>12:41:39,379 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:39,380 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:39,380 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
>> > >>>> >> >> >> >>12:41:43,380 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:43,503 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra:
>> > >>>> >> >> >> >>unknown
>> > >>>> >> >> >> >>error
>> > >>>> >> >> >> >>12:41:43,504 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
>> > >>>> >> >> >> >>12:41:44,504 INFO
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
>> > >>>> >> >> >> >>Initializing
>> > >>>> >> >> >> >>metrics
>> > >>>> >> >> >> >>service
>> > >>>> >> >> >> >>12:41:44,505 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
>> > >>>> >> >> >> >>not
>> > >>>> >> >> >> >>connect
>> > >>>> >> >> >> >>to
>> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
>> > >>>> >> >> >> >>hawkular-cassandra
>> > >>>> >> >> >> >>12:41:44,505 WARN
>> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18]
>> > >>>> >> >> >> >>Retrying
>> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>--
>> > >>>> >> >> >> >>Srinivas Kotaru
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> >>wrote:
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >>>----- Original Message -----
>> > >>>> >> >> >> >>>> From: "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> >>>> <skotaru cisco com>
>> > >>>> >> >> >> >>>> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> >>>> Cc: users lists openshift redhat com
>> > >>>> >> >> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>> > >>>> >> >> >> >>>> Subject: Re: Metrics deployment
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> Matt
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> PV issue resolved. Was able to to see PV successfully
>> > >>>> >> >> >> >>>> bounded
>> > >>>> >> >> >> >>>> and
>> > >>>> >> >> >> >>>> Casandra
>> > >>>> >> >> >> >>>> container has been running. However, it seems puzzle not
>> > >>>> >> >> >> >>>> fully
>> > >>>> >> >> >> >>>> yet
>> > >>>> >> >> >> >>>> solved.
>> > >>>> >> >> >> >>>
>> > >>>> >> >> >> >>>Are you sure the OpenShift DNS server is running?
>> > >>>> >> >> >> >>>
>> > >>>> >> >> >> >>>If you are running OSE 3.1, can you please follow this
>> > >>>> >> >> >> >>>https://access.redhat.com/solutions/2329131 and see if you
>> > >>>> >> >> >> >>>are
>> > >>>> >> >> >> >>>now
>> > >>>> >> >> >> >>>seeing
>> > >>>> >> >> >> >>>errors in the Hawkular Metrics logs (essentially just run
>> > >>>> >> >> >> >>>`oc
>> > >>>> >> >> >> >>>exec
>> > >>>> >> >> >> >>>hawkular-metrics-xxxxx cat
>> > >>>> >> >> >> >>>/opt/eap/standalone/log/server.log`)
>> > >>>> >> >> >> >>>
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> I could see other container(heapster) not coming up, and
>> > >>>> >> >> >> >>>> seeing
>> > >>>> >> >> >> >>>> below
>> > >>>> >> >> >> >>>> errors
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f
>> > >>>> >> >> >> >>>> heapster-fnkdc
>> > >>>> >> >> >> >>>> Endpoint Check in effect. Checking
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> > >>>> >> >> >> >>>> Could not connect to
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> is
>> > >>>> >> >> >> >>>> not
>> > >>>> >> >> >> >>>> accessible
>> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> > >>>> >> >> >> >>>> Could not connect to
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> is
>> > >>>> >> >> >> >>>> not
>> > >>>> >> >> >> >>>> accessible
>> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> # oc get pv
>> > >>>> >> >> >> >>>> pv-5gb-0011   5Gi        RWO           Bound
>> > >>>> >> >> >> >>>> openshift-infra/metrics-cassandra-1             22m
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> $ oc get pods
>> > >>>> >> >> >> >>>> NAME                         READY     STATUS
>> > >>>> >> >> >> >>>> RESTARTS
>> > >>>> >> >> >> >>>> AGE
>> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7   1/1       Running     0
>> > >>>> >> >> >> >>>> 20m
>> > >>>> >> >> >> >>>> hawkular-metrics-mf5qf       0/1       Running     7
>> > >>>> >> >> >> >>>> 20m
>> > >>>> >> >> >> >>>> heapster-fnkdc               0/1       Error       6
>> > >>>> >> >> >> >>>> 20m
>> > >>>> >> >> >> >>>> metrics-deployer-cvep0       0/1       Completed   0
>> > >>>> >> >> >> >>>> 21m
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2)
>> > >>>> >> >> >> >>>> XNIO
>> > >>>> >> >> >> >>>> Version
>> > >>>> >> >> >> >>>> 3.0.14.GA-redhat-1
>> > >>>> >> >> >> >>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller
>> > >>>> >> >> >> >>>> Boot
>> > >>>> >> >> >> >>>> Thread)
>> > >>>> >> >> >> >>>> JBAS015888:
>> > >>>> >> >> >> >>>> Creating http management service using socket-binding
>> > >>>> >> >> >> >>>> (management-http)
>> > >>>> >> >> >> >>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread
>> > >>>> >> >> >> >>>> 1-2)
>> > >>>> >> >> >> >>>> XNIO
>> > >>>> >> >> >> >>>> NIO
>> > >>>> >> >> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>> > >>>> >> >> >> >>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service
>> > >>>> >> >> >> >>>> thread
>> > >>>> >> >> >> >>>> 1-2)
>> > >>>> >> >> >> >>>> JBoss
>> > >>>> >> >> >> >>>> Remoting version 3.3.5.Final-redhat-1
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> $ oc logs -f heapster-fnkdc
>> > >>>> >> >> >> >>>> Endpoint Check in effect. Checking
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> > >>>> >> >> >> >>>> Could not connect to
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> is
>> > >>>> >> >> >> >>>> not
>> > >>>> >> >> >> >>>> accessible
>> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> > >>>> >> >> >> >>>> Could not connect to
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> is
>> > >>>> >> >> >> >>>> not
>> > >>>> >> >> >> >>>> accessible
>> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> > >>>> >> >> >> >>>> Could not connect to
>> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>> > >>>> >> >> >> >>>> INFO  23:00:24 Starting listening for CQL clients on
>> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>> > >>>> >> >> >> >>>> INFO  23:00:24 Binding thrift service to
>> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>> > >>>> >> >> >> >>>> INFO  23:00:24 enabling encrypted thrift connections
>> > >>>> >> >> >> >>>> between
>> > >>>> >> >> >> >>>> client
>> > >>>> >> >> >> >>>> and
>> > >>>> >> >> >> >>>> server
>> > >>>> >> >> >> >>>> INFO  23:00:24 Listening for thrift clients...
>> > >>>> >> >> >> >>>> INFO  23:00:26 Created default superuser role
>> > >>>> >> >> >> >>>> 'cassandra'
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> # oc get svc
>> > >>>> >> >> >> >>>> NAME                       CLUSTER-IP       EXTERNAL-IP
>> > >>>> >> >> >> >>>> PORT(S)
>> > >>>> >> >> >> >>>> AGE
>> > >>>> >> >> >> >>>> hawkular-cassandra         172.30.2.13      <none>
>> > >>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>> > >>>> >> >> >> >>>> hawkular-cassandra-nodes   None             <none>
>> > >>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>> > >>>> >> >> >> >>>> hawkular-metrics           172.30.117.176   <none>
>> > >>>> >> >> >> >>>> 443/TCP
>> > >>>> >> >> >> >>>> 25m
>> > >>>> >> >> >> >>>> heapster                   172.30.107.135   <none>
>> > >>>> >> >> >> >>>> 80/TCP
>> > >>>> >> >> >> >>>> 25m
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> HTTP/1.1 504 Gateway Timeout
>> > >>>> >> >> >> >>>> Mime-Version: 1.0
>> > >>>> >> >> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>> > >>>> >> >> >> >>>> Content-Type: text/html
>> > >>>> >> >> >> >>>> Connection: keep-alive
>> > >>>> >> >> >> >>>> Proxy-Connection: keep-alive
>> > >>>> >> >> >> >>>> Content-Length: 1572
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> --
>> > >>>> >> >> >> >>>> Srinivas Kotaru
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> >>>> <skotaru cisco com>
>> > >>>> >> >> >> >>>> wrote:
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> >Matt
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >That is good catch. I ran without
>> > >>>> >> >> >> >>>> >USE_PERSISTENT_STORAGE=false
>> > >>>> >> >> >> >>>> >and
>> > >>>> >> >> >> >>>> >working
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >Thanks you for your help so far.
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >--
>> > >>>> >> >> >> >>>> >Srinivas Kotaru
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> >>>> >wrote:
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> >>
>> > >>>> >> >> >> >>>> >>
>> > >>>> >> >> >> >>>> >>----- Original Message -----
>> > >>>> >> >> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> >>>> >>> <skotaru cisco com>
>> > >>>> >> >> >> >>>> >>> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> >>>> >>> Cc: users lists openshift redhat com
>> > >>>> >> >> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>> > >>>> >> >> >> >>>> >>> Subject: Re: Metrics deployment
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> Oh ok
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> Am using PV for metrics
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> description: "The persistent volume size for each of
>> > >>>> >> >> >> >>>> >>> the
>> > >>>> >> >> >> >>>> >>> Cassandra
>> > >>>> >> >> >> >>>> >>> nodes"
>> > >>>> >> >> >> >>>> >>>   name: CASSANDRA_PV_SIZE
>> > >>>> >> >> >> >>>> >>>   value: "10Gi"
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> oc get pv
>> > >>>> >> >> >> >>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS
>> > >>>> >> >> >> >>>> >>> CLAIM
>> > >>>> >> >> >> >>>> >>> REASON
>> > >>>> >> >> >> >>>> >>> AGE
>> > >>>> >> >> >> >>>> >>> pv-1gb-001    1Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-1gb-002    1Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-1gb-003    1Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-1gb-004    1Gi        RWO           Bound
>> > >>>> >> >> >> >>>> >>> thlatt/mongodb
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-1gb-005    1Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-2gb-0010   2Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-2gb-006    2Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-2gb-007    2Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-2gb-008    2Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-2gb-009    2Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-5gb-0011   5Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-5gb-0012   5Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-5gb-0013   5Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-5gb-0014   5Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> pv-5gb-0015   5Gi        RWO           Available
>> > >>>> >> >> >> >>>> >>> 4d
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> am running with below command
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded
>> > >>>> >> >> >> >>>> >>> HOSTNAME,
>> > >>>> >> >> >> >>>> >>> MASTER_API
>> > >>>> >> >> >> >>>> >>> and
>> > >>>> >> >> >> >>>> >>> PV info so not passing any parameters)
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>
>> > >>>> >> >> >> >>>> >>I would suspect that Cassandra is blocked because its
>> > >>>> >> >> >> >>>> >>waiting
>> > >>>> >> >> >> >>>> >>for
>> > >>>> >> >> >> >>>> >>10Gi
>> > >>>> >> >> >> >>>> >>PV
>> > >>>> >> >> >> >>>> >>to become available, and none of the PV listed above
>> > >>>> >> >> >> >>>> >>are
>> > >>>> >> >> >> >>>> >>big
>> > >>>> >> >> >> >>>> >>enough.
>> > >>>> >> >> >> >>>> >>
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> --
>> > >>>> >> >> >> >>>> >>> Srinivas Kotaru
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe"
>> > >>>> >> >> >> >>>> >>> <mwringe redhat com>
>> > >>>> >> >> >> >>>> >>> wrote:
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> >----- Original Message -----
>> > >>>> >> >> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> >>>> >>> >> <skotaru cisco com>
>> > >>>> >> >> >> >>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
>> > >>>> >> >> >> >>>> >>> >> Cc: users lists openshift redhat com
>> > >>>> >> >> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>> > >>>> >> >> >> >>>> >>> >> Subject: Re: Metrics deployment
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> Matt
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but
>> > >>>> >> >> >> >>>> >>> >> had
>> > >>>> >> >> >> >>>> >>> >> the
>> > >>>> >> >> >> >>>> >>> >> same
>> > >>>> >> >> >> >>>> >>> >> issue
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> # oc get pods
>> > >>>> >> >> >> >>>> >>> >> NAME                         READY     STATUS
>> > >>>> >> >> >> >>>> >>> >> RESTARTS
>> > >>>> >> >> >> >>>> >>> >> AGE
>> > >>>> >> >> >> >>>> >>> >> hawkular-cassandra-1-y2egy   0/1
>> > >>>> >> >> >> >>>> >>> >> ContainerCreating
>> > >>>> >> >> >> >>>> >>> >> 0
>> > >>>> >> >> >> >>>> >>> >> 5m
>> > >>>> >> >> >> >>>> >>> >> hawkular-metrics-4b16f       0/1       Running
>> > >>>> >> >> >> >>>> >>> >> 1
>> > >>>> >> >> >> >>>> >>> >> 4m
>> > >>>> >> >> >> >>>> >>> >> heapster-x2gj2               0/1       Running
>> > >>>> >> >> >> >>>> >>> >> 2
>> > >>>> >> >> >> >>>> >>> >> 4m
>> > >>>> >> >> >> >>>> >>> >> metrics-deployer-9v7vc       0/1       Completed
>> > >>>> >> >> >> >>>> >>> >> 0
>> > >>>> >> >> >> >>>> >>> >> 6m
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>> > >>>> >> >> >> >>>> >>> >> Error from server: container
>> > >>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1"
>> > >>>> >> >> >> >>>> >>> >> in
>> > >>>> >> >> >> >>>> >>> >> pod
>> > >>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>> > >>>> >> >> >> >>>> >>> >> ContainerCreating
>> > >>>> >> >> >> >>>> >>> >
>> > >>>> >> >> >> >>>> >>> >Ok, so it looks like something is blocking the
>> > >>>> >> >> >> >>>> >>> >Cassandra
>> > >>>> >> >> >> >>>> >>> >pod
>> > >>>> >> >> >> >>>> >>> >from
>> > >>>> >> >> >> >>>> >>> >starting.
>> > >>>> >> >> >> >>>> >>> >
>> > >>>> >> >> >> >>>> >>> >If you are using persistent storage, Cassandra will
>> > >>>> >> >> >> >>>> >>> >not
>> > >>>> >> >> >> >>>> >>> >start
>> > >>>> >> >> >> >>>> >>> >until
>> > >>>> >> >> >> >>>> >>> >the
>> > >>>> >> >> >> >>>> >>> >PV
>> > >>>> >> >> >> >>>> >>> >is available. There may be some more information
>> > >>>> >> >> >> >>>> >>> >about
>> > >>>> >> >> >> >>>> >>> >Cassandra
>> > >>>> >> >> >> >>>> >>> >in
>> > >>>> >> >> >> >>>> >>> >the
>> > >>>> >> >> >> >>>> >>> >pod
>> > >>>> >> >> >> >>>> >>> >section of the console under events.
>> > >>>> >> >> >> >>>> >>> >
>> > >>>> >> >> >> >>>> >>> >What command did you use when deploying the
>> > >>>> >> >> >> >>>> >>> >deployer?
>> > >>>> >> >> >> >>>> >>> >
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC
>> > >>>> >> >> >> >>>> >>> >> service
>> > >>>> >> >> >> >>>> >>> >> thread
>> > >>>> >> >> >> >>>> >>> >> 1-4)
>> > >>>> >> >> >> >>>> >>> >> VM
>> > >>>> >> >> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss
>> > >>>> >> >> >> >>>> >>> >> -Duser.name=jboss
>> > >>>> >> >> >> >>>> >>> >> -D[Standalone]
>> > >>>> >> >> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>> > >>>> >> >> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>> > >>>> >> >> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> > >>>> >> >> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>> > >>>> >> >> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>> > >>>> >> >> >> >>>> >>> >> -XX:-TraceClassUnloading
>> > >>>> >> >> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>> > >>>> >> >> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>> > >>>> >> >> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>> > >>>> >> >> >> >>>> >>> >> -Djava.awt.headless=true
>> > >>>> >> >> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>> > >>>> >> >> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>> > >>>> >> >> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>> > >>>> >> >> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>> > >>>> >> >> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>> > >>>> >> >> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>> > >>>> >> >> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>> > >>>> >> >> >> >>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread
>> > >>>> >> >> >> >>>> >>> >> 1-3)
>> > >>>> >> >> >> >>>> >>> >> XNIO
>> > >>>> >> >> >> >>>> >>> >> Version
>> > >>>> >> >> >> >>>> >>> >> 3.0.14.GA-redhat-1
>> > >>>> >> >> >> >>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service
>> > >>>> >> >> >> >>>> >>> >> thread
>> > >>>> >> >> >> >>>> >>> >> 1-3)
>> > >>>> >> >> >> >>>> >>> >> XNIO
>> > >>>> >> >> >> >>>> >>> >> NIO
>> > >>>> >> >> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>> > >>>> >> >> >> >>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server]
>> > >>>> >> >> >> >>>> >>> >> (Controller
>> > >>>> >> >> >> >>>> >>> >> Boot
>> > >>>> >> >> >> >>>> >>> >> Thread)
>> > >>>> >> >> >> >>>> >>> >> JBAS015888:
>> > >>>> >> >> >> >>>> >>> >> Creating http management service using
>> > >>>> >> >> >> >>>> >>> >> socket-binding
>> > >>>> >> >> >> >>>> >>> >> (management-http)
>> > >>>> >> >> >> >>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC
>> > >>>> >> >> >> >>>> >>> >> service
>> > >>>> >> >> >> >>>> >>> >> thread
>> > >>>> >> >> >> >>>> >>> >> 1-3)
>> > >>>> >> >> >> >>>> >>> >> JBoss
>> > >>>> >> >> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>> > >>>> >> >> >> >>>> >>> >> Endpoint Check in effect. Checking
>> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> > >>>> >> >> >> >>>> >>> >> Could not connect to
>> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> >>> >> is
>> > >>>> >> >> >> >>>> >>> >> not
>> > >>>> >> >> >> >>>> >>> >> accessible
>> > >>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6].
>> > >>>> >> >> >> >>>> >>> >> Retrying.
>> > >>>> >> >> >> >>>> >>> >> Could not connect to
>> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> >>> >> is
>> > >>>> >> >> >> >>>> >>> >> not
>> > >>>> >> >> >> >>>> >>> >> accessible
>> > >>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6].
>> > >>>> >> >> >> >>>> >>> >> Retrying.
>> > >>>> >> >> >> >>>> >>> >> Could not connect to
>> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> ++ oc create -f -
>> > >>>> >> >> >> >>>> >>> >> serviceaccount "heapster" created
>> > >>>> >> >> >> >>>> >>> >> service "heapster" created
>> > >>>> >> >> >> >>>> >>> >> replicationcontroller "heapster" created
>> > >>>> >> >> >> >>>> >>> >> + echo 'Success!'
>> > >>>> >> >> >> >>>> >>> >> Success!
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> --
>> > >>>> >> >> >> >>>> >>> >> Srinivas Kotaru
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe"
>> > >>>> >> >> >> >>>> >>> >> <mwringe redhat com>
>> > >>>> >> >> >> >>>> >>> >> wrote:
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >----- Original Message -----
>> > >>>> >> >> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)"
>> > >>>> >> >> >> >>>> >>> >> >> <skotaru cisco com>
>> > >>>> >> >> >> >>>> >>> >> >> To: users lists openshift redhat com
>> > >>>> >> >> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>> > >>>> >> >> >> >>>> >>> >> >> Subject: Metrics deployment
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Hi
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Am trying to configure metrics in our newly
>> > >>>> >> >> >> >>>> >>> >> >> installed
>> > >>>> >> >> >> >>>> >>> >> >> clusters. Am
>> > >>>> >> >> >> >>>> >>> >> >> seeing
>> > >>>> >> >> >> >>>> >>> >> >> below errors once metrics-deploy script was
>> > >>>> >> >> >> >>>> >>> >> >> successful.
>> > >>>> >> >> >> >>>> >>> >> >> I
>> > >>>> >> >> >> >>>> >>> >> >> used
>> > >>>> >> >> >> >>>> >>> >> >> our
>> > >>>> >> >> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME
>> > >>>> >> >> >> >>>> >>> >> >> and
>> > >>>> >> >> >> >>>> >>> >> >> MASTER_URL
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL,
>> > >>>> >> >> >> >>>> >>> >> >> and
>> > >>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> > >>>> >> >> >> >>>> >>> >> >> (
>> > >>>> >> >> >> >>>> >>> >> >> hard coded as values)
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> template "hawkular-heapster" created
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Deploying the Heapster component
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> ++ '[' -n '' ']'
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> ++ oc create -f -
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>> > >>>> >> >> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> serviceaccount "heapster" created
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> service "heapster" created
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> replicationcontroller "heapster" created
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> + echo 'Success!'
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Success!
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> # oc get pods
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1
>> > >>>> >> >> >> >>>> >>> >> >> ContainerCreating
>> > >>>> >> >> >> >>>> >>> >> >> 0 4m
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0
>> > >>>> >> >> >> >>>> >>> >> >> 3s
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >It looks like none of your containers are fully
>> > >>>> >> >> >> >>>> >>> >> >up
>> > >>>> >> >> >> >>>> >>> >> >and
>> > >>>> >> >> >> >>>> >>> >> >running
>> > >>>> >> >> >> >>>> >>> >> >yet.
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will
>> > >>>> >> >> >> >>>> >>> >> >not
>> > >>>> >> >> >> >>>> >>> >> >run,
>> > >>>> >> >> >> >>>> >>> >> >and
>> > >>>> >> >> >> >>>> >>> >> >Heapster
>> > >>>> >> >> >> >>>> >>> >> >will
>> > >>>> >> >> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The
>> > >>>> >> >> >> >>>> >>> >> >first
>> > >>>> >> >> >> >>>> >>> >> >step
>> > >>>> >> >> >> >>>> >>> >> >will
>> > >>>> >> >> >> >>>> >>> >> >be to
>> > >>>> >> >> >> >>>> >>> >> >get
>> > >>>> >> >> >> >>>> >>> >> >Cassandra running properly.
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Could not connect to
>> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
>> > >>>> >> >> >> >>>> >>> >> >> is
>> > >>>> >> >> >> >>>> >>> >> >> not
>> > >>>> >> >> >> >>>> >>> >> >> accessible
>> > >>>> >> >> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6].
>> > >>>> >> >> >> >>>> >>> >> >> Retrying.
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Could not connect to
>> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> > >>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started
>> > >>>> >> >> >> >>>> >>> >> >before
>> > >>>> >> >> >> >>>> >>> >> >trying
>> > >>>> >> >> >> >>>> >>> >> >to
>> > >>>> >> >> >> >>>> >>> >> >push
>> > >>>> >> >> >> >>>> >>> >> >metrics to it. The issue that you are seeing is
>> > >>>> >> >> >> >>>> >>> >> >because
>> > >>>> >> >> >> >>>> >>> >> >Heapster
>> > >>>> >> >> >> >>>> >>> >> >could
>> > >>>> >> >> >> >>>> >>> >> >not
>> > >>>> >> >> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the
>> > >>>> >> >> >> >>>> >>> >> >Hawkular
>> > >>>> >> >> >> >>>> >>> >> >Metrics
>> > >>>> >> >> >> >>>> >>> >> >service
>> > >>>> >> >> >> >>>> >>> >> >is
>> > >>>> >> >> >> >>>> >>> >> >fully up, Heapster will not be able to connect
>> > >>>> >> >> >> >>>> >>> >> >to
>> > >>>> >> >> >> >>>> >>> >> >it.
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> What is the wrong? Why it checking just
>> > >>>> >> >> >> >>>> >>> >> >> hawkular-metrics
>> > >>>> >> >> >> >>>> >>> >> >> rather
>> > >>>> >> >> >> >>>> >>> >> >> full
>> > >>>> >> >> >> >>>> >>> >> >> routing
>> > >>>> >> >> >> >>>> >>> >> >> URL which was provided as
>> > >>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames:
>> > >>>> >> >> >> >>>> >>> >> >the
>> > >>>> >> >> >> >>>> >>> >> >internal
>> > >>>> >> >> >> >>>> >>> >> >hostname
>> > >>>> >> >> >> >>>> >>> >> >used
>> > >>>> >> >> >> >>>> >>> >> >by the internal components (eg
>> > >>>> >> >> >> >>>> >>> >> >'hawkular-metrics')
>> > >>>> >> >> >> >>>> >>> >> >and
>> > >>>> >> >> >> >>>> >>> >> >the
>> > >>>> >> >> >> >>>> >>> >> >external
>> > >>>> >> >> >> >>>> >>> >> >hostname (eg what is configured via
>> > >>>> >> >> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>> > >>>> >> >> >> >>>> >>> >> >The
>> > >>>> >> >> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to
>> > >>>> >> >> >> >>>> >>> >> >the
>> > >>>> >> >> >> >>>> >>> >> >name
>> > >>>> >> >> >> >>>> >>> >> >of
>> > >>>> >> >> >> >>>> >>> >> >services,
>> > >>>> >> >> >> >>>> >>> >> >which
>> > >>>> >> >> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes
>> > >>>> >> >> >> >>>> >>> >> >from.
>> > >>>> >> >> >> >>>> >>> >> >
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> --
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> Srinivas Kotaru
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> >> _______________________________________________
>> > >>>> >> >> >> >>>> >>> >> >> users mailing list
>> > >>>> >> >> >> >>>> >>> >> >> users lists openshift redhat com
>> > >>>> >> >> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> > >>>> >> >> >> >>>> >>> >> >> 
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> >> 
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >>> 
>> > >>>> >> >> >> >>>> >
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>>> 
>> > >>>> >> >> >> >>
>> > >>>> >> >> >> >
>> > >>>> >> >> >> 
>> > >>>> >> >> >> 
>> > >>>> >> >> 
>> > >>>> >> >> 
>> > >>>> >> 
>> > >>>> >> 
>> > >>>> 
>> > >>>> 
>> > >>
>> > >
>> > 
>> > 
>> 



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]