[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



I’m still struck with this issue. It is kind of chicken and egg problem.  Heapster health probes failing since it is waiting for hawkular to start. Hawkular health probes are failing since it is unable to connect Casandra. Cansandra health probe also failing.

@Matt:  Internal DNS looks ups working. I’m able to create apps, build and deploy code. Router and registry components also working as expected 


oc get pods
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-mxd2m   1/1       Running     0          1h
hawkular-metrics-gvp9k       0/1       Running     4          11m
heapster-uleul               0/1       Running     4          11m
metrics-deployer-2z75w       0/1       Completed   0          1h


heapster events:
------------------------


Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason		Message
  ---------	--------	-----	----					-------------			--------	------		-------
  1m		1m		1	{default-scheduler }							Normal		Scheduled	Successfully assigned heapster-uleul to l3inpn-id2-004.cisco.com
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}	Normal		Pulling		pulling image "registry.access.redhat.com/openshift3/metrics-heapster:latest"
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}	Normal		Pulled		Successfully pulled image "registry.access.redhat.com/openshift3/metrics-heapster:latest"
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}	Normal		Created		Created container with docker id a22bb9a246ca
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}	Normal		Started		Started container with docker id a22bb9a246ca
  1m		5s		10	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{heapster}	Warning		Unhealthy	Readiness probe failed: The heapster process is not yet started, it is waiting for the Hawkular Metrics to start.

Hawkular events :
===============

Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
  ---------	--------	-----	----					-------------				--------	------		-------
  1m		1m		1	{default-scheduler }								Normal		Scheduled	Successfully assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Normal		Pulling		pulling image "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Normal		Pulled		Successfully pulled image "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Normal		Created		Created container with docker id 6e32459624d7
  1m		1m		1	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Normal		Started		Started container with docker id 6e32459624d7
  1m		57s		4	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Warning		Unhealthy	Readiness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>.

  46s	7s	5	{kubelet l3inpn-id2-004.cisco.com}	spec.containers{hawkular-metrics}	Warning	Unhealthy	Readiness probe failed:


Casandra events
==============

Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
  ---------	--------	-----	----					-------------				--------	------		-------
  1h		1h		1	{default-scheduler }								Normal		Scheduled	Successfully assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
  1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
  1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully pulled image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
  1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Created		Created container with docker id 2a1390edd4f1
  1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Started		Started container with docker id 2a1390edd4f1
  1h		1h		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected
Cassandra not in the up and normal state. Current state is

  1h	1h	2	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
Cassandra not in the up and normal state. Current state is
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected


Logs:
====

# oc logs -f heapster-uleul
Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 6]. Retrying.

# oc logs -f hawkular-metrics-gvp9k

16:42:43,201 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
16:42:43,202 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.14.GA-redhat-1
16:42:43,208 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.14.GA-redhat-1
16:42:43,223 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.3.5.Final-redhat-1

# oc logs -f hawkular-cassandra-1-mxd2m

INFO  20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4 sstables to [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,] to level=0.  1,643 bytes to 1,155 (~70% of original) in 207ms = 0.005321MB/s.  0 total partitions merged to 11.  Partition merge counts were {1:11, }
INFO  20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1) [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0, /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0, /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0, /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0, ]
INFO  20:15:20 Writing Memtable-schema_columns 672348153(49.749KiB serialized bytes, 967 ops, 0%/0% of on/off-heap limit)
INFO  20:15:20 Completed flushing /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db (11.212KiB) for commitlog position ReplayPosition(segmentId=1465931718336, position=159511)
INFO  20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4 sstables to [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,] to level=0.  128 bytes to 32 (~25% of original) in 146ms = 0.000209MB/s.  0 total partitions merged to 1.  Partition merge counts were {4:1, }
INFO  20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1) [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0, /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0, /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0, /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0, ]
INFO  20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4 sstables to [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,] to level=0.  128 bytes to 32 (~25% of original) in 155ms = 0.000197MB/s.  0 total partitions merged to 1.  Partition merge counts were {4:1, }
INFO  20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1) [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0, /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0, /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0, /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0, ]
INFO  20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4 sstables to [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,] to level=0.  128 bytes to 32 (~25% of original) in 83ms = 0.000368MB/s.  0 total partitions merged to 1.  Partition merge counts were {4:1, }
INFO  20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4 sstables to [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,] to level=0.  2,538 bytes to 527 (~20% of original) in 176ms = 0.002856MB/s.  0 total partitions merged to 3.  Partition merge counts were {4:3, }



almost struck here. Any points to look for? Any ports need to be open explicitly ? 







-- 
Srinivas Kotaru

On 6/14/16, 11:27 AM, "Matt Wringe" <mwringe redhat com> wrote:

>----- Original Message -----
>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> To: "Matt Wringe" <mwringe redhat com>
>> Cc: users lists openshift redhat com
>> Sent: Tuesday, June 14, 2016 2:09:49 PM
>> Subject: Re: Metrics deployment
>> 
>> Not sure what do you mean by installation. Am just running oc new-app -f
>> metrics-deployer.yaml with default values except HOST_NAME and PV storage.
>
>I would suspect something wrong with your node or cluster installation. Or you have firewall rules blocking connections between your nodes so that pod cannot connect with each other or access the OpenShift DNS server.
>
>The lifecycle hooks exist to make sure that components only enter the ready state when they are fully started and ready.
>
>Can you check the Hawkular Metrics status page and see what that outputs? eg https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
>
>> 
>> I just deleted entire metrics setup and re running. But not sure this will
>> fix the issue.
>> 
>> $ ./delete_metrics-infra.sh
>> replicationcontroller "hawkular-cassandra-1" deleted
>> replicationcontroller "hawkular-metrics" deleted
>> replicationcontroller "heapster" deleted
>> route "hawkular-metrics" deleted
>> service "hawkular-cassandra" deleted
>> service "hawkular-cassandra-nodes" deleted
>> service "hawkular-metrics" deleted
>> service "heapster" deleted
>> pod "heapster-lyf65" deleted
>> serviceaccount "cassandra" deleted
>> serviceaccount "hawkular" deleted
>> serviceaccount "heapster" deleted
>> template "hawkular-cassandra-node-emptydir" deleted
>> template "hawkular-cassandra-node-pv" deleted
>> template "hawkular-cassandra-services" deleted
>> template "hawkular-heapster" deleted
>> template "hawkular-metrics" deleted
>> template "hawkular-support" deleted
>> secret "hawkular-cassandra-certificate" deleted
>> secret "hawkular-cassandra-secrets" deleted
>> secret "hawkular-metrics-account" deleted
>> secret "hawkular-metrics-certificate" deleted
>> secret "hawkular-metrics-secrets" deleted
>> secret "heapster-secrets" deleted
>> 
>> --
>> Srinivas Kotaru
>> 
>> On 6/14/16, 10:53 AM, "Matt Wringe" <mwringe redhat com> wrote:
>> 
>> >----- Original Message -----
>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> To: "Matt Wringe" <mwringe redhat com>
>> >> Cc: users lists openshift redhat com
>> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
>> >> Subject: Re: Metrics deployment
>> >> 
>> >> I removed readiness probes from both hawkular-cassandra-1 &
>> >> hawkular-metrics
>> >> as both status shows probes failed.
>> >
>> >You should not have to remove the probes, this indicates that something is
>> >wrong with your installation.
>> >
>> >> 
>> >> It looks good now. Both containers looks and running
>> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
>> >> heapster-7yl34
>> >> logs still shows Could not connect to
>> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6.
>> >> Status Code 000.
>> >> 
>> >> Are we good or still had issues?
>> >> 
>> >> 
>> >> # oc get pods
>> >> NAME                         READY     STATUS    RESTARTS   AGE
>> >> hawkular-cassandra-1-kr8ka   1/1       Running   0          6m
>> >> hawkular-metrics-vhe3u       1/1       Running   2          5m
>> >> heapster-7yl34               0/1       Running   2          5m
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> --
>> >> Srinivas Kotaru
>> >> 
>> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> wrote:
>> >> 
>> >> >Matt
>> >> >
>> >> >Just want to share more info by running describe pod.
>> >> >
>> >> >It seems to be health probe failing. Do you think it is the issue?
>> >> >
>> >> >
>> >> >
>> >> ># oc describe pod hawkular-cassandra-1-it5uh
>> >> >Name:		hawkular-cassandra-1-it5uh
>> >> >Namespace:	openshift-infra
>> >> >Node:		l3inpn-id2-003.cisco.com/173.36.96.16
>> >> >Start Time:	Tue, 14 Jun 2016 16:36:21 +0000
>> >> >Labels:
>> >> >		metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>> >> >Status:		Running
>> >> >IP:		10.1.9.2
>> >> >Controllers:	ReplicationController/hawkular-cassandra-1
>> >> >Containers:
>> >> >  hawkular-cassandra-1:
>> >> >    Container ID:
>> >> >    	docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>> >> >    Image:
>> >> >    		registry.access.redhat.com/openshift3/metrics-cassandra:latest
>> >> >    Image ID:
>> >> >    		docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>> >> >    Ports:		9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>> >> >    Command:
>> >> >      /opt/apache-cassandra/bin/cassandra-docker.sh
>> >> >      --cluster_name=hawkular-metrics
>> >> >      --data_volume=/cassandra_data
>> >> >      --internode_encryption=all
>> >> >      --require_node_auth=true
>> >> >      --enable_client_encryption=true
>> >> >      --require_client_auth=true
>> >> >      --keystore_file=/secret/cassandra.keystore
>> >> >      --keystore_password_file=/secret/cassandra.keystore.password
>> >> >      --truststore_file=/secret/cassandra.truststore
>> >> >      --truststore_password_file=/secret/cassandra.truststore.password
>> >> >      --cassandra_pem_file=/secret/cassandra.pem
>> >> >    QoS Tier:
>> >> >      cpu:		BestEffort
>> >> >      memory:		BestEffort
>> >> >    State:		Running
>> >> >      Started:		Tue, 14 Jun 2016 16:37:01 +0000
>> >> >    Ready:		True
>> >> >    Restart Count:	0
>> >> >    Readiness:		exec
>> >> >    [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
>> >> >    delay=0s timeout=1s period=10s #success=1 #failure=3
>> >> >    Environment Variables:
>> >> >      CASSANDRA_MASTER:	true
>> >> >      POD_NAMESPACE:	openshift-infra (v1:metadata.namespace)
>> >> >Conditions:
>> >> >  Type		Status
>> >> >  Ready 	True
>> >> >Volumes:
>> >> >  cassandra-data:
>> >> >    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim
>> >> >    in
>> >> >    the same namespace)
>> >> >    ClaimName:	metrics-cassandra-1
>> >> >    ReadOnly:	false
>> >> >  hawkular-cassandra-secrets:
>> >> >    Type:	Secret (a volume populated by a Secret)
>> >> >    SecretName:	hawkular-cassandra-secrets
>> >> >  cassandra-token-4urfd:
>> >> >    Type:	Secret (a volume populated by a Secret)
>> >> >    SecretName:	cassandra-token-4urfd
>> >> >Events:
>> >> >  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason
>> >> >  		Message
>> >> >  ---------	--------	-----	----					-------------				--------	------
>> >> >  		-------
>> >> >  27m		27m		1	{default-scheduler }								Normal		Scheduled	Successfully
>> >> >  assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image
>> >> >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully
>> >> >  	pulled
>> >> >  image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Created		Created
>> >> >  	container
>> >> >  with docker id 17a9575eb655
>> >> >  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}
>> >> >  	spec.containers{hawkular-cassandra-1}	Normal		Started		Started
>> >> >  	container
>> >> >  with docker id 17a9575eb655
>> >> >  27m		26m		3	{kubelet l3inpn-id2-003.cisco.com}
>> >> >  	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness
>> >> >  	probe
>> >> >  failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>> >> >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
>> >> >'Connection refused'.
>> >> >Cassandra not in the up and normal state. Current state is
>> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary
>> >> >operator expected
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >Srinivas Kotaru
>> >> >
>> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
>> >> ><skotaru cisco com>
>> >> >wrote:
>> >> >
>> >> >>Matt
>> >> >>
>> >> >>DNS service is working
>> >> >>
>> >> >>
>> >> >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>> >> >>172.30.0.1
>> >> >>~  dig +short @master-01  jenkins.alln-test.svc.cluster.local
>> >> >>172.30.85.148
>> >> >>~  dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
>> >> >>172.30.31.6
>> >> >>
>> >> >>I captured hawkular-metrics, it shows the problem. It seems to be unable
>> >> >>to
>> >> >>connect Cassandra cluster
>> >> >>
>> >> >># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
>> >> >>
>> >> >>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan] (ServerService
>> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>> >> >>12:40:58,898 INFO  [org.jboss.as.webservices] (ServerService Thread Pool
>> >> >>--
>> >> >>48) JBAS015537: Activating WebServices Extension
>> >> >>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 46)
>> >> >>JBAS010153: Node identifier property is set to the default value. Please
>> >> >>make sure it is unique.
>> >> >>12:40:58,902 INFO  [org.jboss.as.security] (ServerService Thread Pool --
>> >> >>44) JBAS013371: Activating Security Subsystem
>> >> >>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService Thread Pool --
>> >> >>39)
>> >> >>JBAS011800: Activating Naming Subsystem
>> >> >>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 37)
>> >> >>JBAS012615: Activated the following JSF Implementations: [main, 1.2]
>> >> >>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC service thread
>> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
>> >> >>1.0.32.Final-redhat-1)
>> >> >>12:40:58,981 INFO  [org.jboss.as.security] (MSC service thread 1-2)
>> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>> >> >>12:40:58,961 INFO  [org.jboss.as.connector.subsystems.datasources]
>> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant
>> >> >>driver class org.h2.Driver (version 1.3)
>> >> >>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service thread 1-2)
>> >> >>JBAS011802: Starting Naming Service
>> >> >>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC service thread
>> >> >>1-4)
>> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
>> >> >>12:40:59,614 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8444
>> >> >>12:40:59,620 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
>> >> >>http-/0.0.0.0:8080
>> >> >>12:40:59,621 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444
>> >> >>12:40:59,627 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
>> >> >>service
>> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080
>> >> >>12:40:59,654 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC
>> >> >>service thread 1-3) JBAS010400: Bound data source
>> >> >>[java:jboss/datasources/ExampleDS]
>> >> >>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
>> >> >>(runtime-name:
>> >> >>"activemq-rar.rar")
>> >> >>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC service thread
>> >> >>1-1) JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war"
>> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
>> >> >>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner] (MSC service
>> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
>> >> >>directory
>> >> >>/opt/eap/standalone/deployments
>> >> >>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service thread 1-2)
>> >> >>JBAS017100: Listening on 127.0.0.1:9999
>> >> >>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC service thread
>> >> >>1-3)
>> >> >>JBAS015102: Unable to load console module for slot eap, disabling
>> >> >>console
>> >> >>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC service thread
>> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
>> >> >>4.3.5.Final-redhat-3
>> >> >>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service thread 1-4)
>> >> >>JBAS017100: Listening on 0.0.0.0:4447
>> >> >>12:41:01,361 INFO  [org.jboss.as.connector.deployers.RADeployer] (MSC
>> >> >>service thread 1-3) IJ020001: Required license terms for
>> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>> >> >>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread
>> >> >>1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war"
>> >> >>is
>> >> >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
>> >> >>which may be changed or removed in future versions without notice.
>> >> >>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1)
>> >> >>JBAS016005: Starting Services for CDI deployment:
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service thread 1-1)
>> >> >>WELD-000900 1.1.31 (redhat)
>> >> >>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service thread 1-3)
>> >> >>JBAS016008: Starting weld service for deployment
>> >> >>hawkular-metrics-api-jaxrs.war
>> >> >>12:41:02,820 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread Pool -- 51)
>> >> >>JBAS018210: Register web context: /hawkular/metrics
>> >> >>12:41:02,958 INFO  [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
>> >> >>"hawkular-metrics-api-jaxrs.war")
>> >> >>12:41:02,959 INFO  [org.jboss.as.server] (ServerService Thread Pool --
>> >> >>28)
>> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
>> >> >>"activemq-rar.rar")
>> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961:
>> >> >>Http
>> >> >>management interface listening on http://127.0.0.1:9990/management
>> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951:
>> >> >>Admin console listening on http://127.0.0.1:9990
>> >> >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874:
>> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started
>> >> >>267 of 303 services (62 services are lazy, passive or on-demand)
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:02,999 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:04,000 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:04,001 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:06,002 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:06,002 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:06,003 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2)
>> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
>> >> >>12:41:08,299 INFO  [org.hibernate.validator.internal.util.Version]
>> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
>> >> >>4.3.2.Final-redhat-2
>> >> >>12:41:08,418 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n]
>> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
>> >> >>javax.ws.rs.core.Application: class
>> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>> >> >>12:41:08,421 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:08,422 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
>> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
>> >> >>12:41:09,003 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:09,004 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:13,004 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:13,123 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:13,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:14,124 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:14,124 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:14,125 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:16,125 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:16,126 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:19,126 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:19,127 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:23,128 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:23,249 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:24,250 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:24,250 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:26,251 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:26,251 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:29,252 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:29,252 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:29,253 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:33,253 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:33,376 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:33,377 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:34,377 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:34,378 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>12:41:36,378 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:36,379 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
>> >> >>connecting to Cassandra cluster in [3]s...
>> >> >>12:41:39,379 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:39,380 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
>> >> >>connecting to Cassandra cluster in [4]s...
>> >> >>12:41:43,380 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:43,503 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
>> >> >>error
>> >> >>12:41:43,504 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
>> >> >>connecting to Cassandra cluster in [1]s...
>> >> >>12:41:44,504 INFO
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
>> >> >>metrics
>> >> >>service
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
>> >> >>to
>> >> >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
>> >> >>12:41:44,505 WARN
>> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
>> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
>> >> >>connecting to Cassandra cluster in [2]s...
>> >> >>
>> >> >>--
>> >> >>Srinivas Kotaru
>> >> >>
>> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com> wrote:
>> >> >>
>> >> >>>----- Original Message -----
>> >> >>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> >>>> To: "Matt Wringe" <mwringe redhat com>
>> >> >>>> Cc: users lists openshift redhat com
>> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
>> >> >>>> Subject: Re: Metrics deployment
>> >> >>>> 
>> >> >>>> Matt
>> >> >>>> 
>> >> >>>> PV issue resolved. Was able to to see PV successfully bounded and
>> >> >>>> Casandra
>> >> >>>> container has been running. However, it seems puzzle not fully yet
>> >> >>>> solved.
>> >> >>>
>> >> >>>Are you sure the OpenShift DNS server is running?
>> >> >>>
>> >> >>>If you are running OSE 3.1, can you please follow this
>> >> >>>https://access.redhat.com/solutions/2329131 and see if you are now
>> >> >>>seeing
>> >> >>>errors in the Hawkular Metrics logs (essentially just run `oc exec
>> >> >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>> >> >>>
>> >> >>>> 
>> >> >>>> I could see other container(heapster) not coming up, and seeing below
>> >> >>>> errors
>> >> >>>> 
>> >> >>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> 
>> >> >>>> 
>> >> >>>> # oc get pv
>> >> >>>> pv-5gb-0011   5Gi        RWO           Bound
>> >> >>>> openshift-infra/metrics-cassandra-1             22m
>> >> >>>> 
>> >> >>>> 
>> >> >>>> $ oc get pods
>> >> >>>> NAME                         READY     STATUS      RESTARTS   AGE
>> >> >>>> hawkular-cassandra-1-2pzd7   1/1       Running     0          20m
>> >> >>>> hawkular-metrics-mf5qf       0/1       Running     7          20m
>> >> >>>> heapster-fnkdc               0/1       Error       6          20m
>> >> >>>> metrics-deployer-cvep0       0/1       Completed   0          21m
>> >> >>>> 
>> >> >>>> # oc logs -f hawkular-metrics-mf5qf
>> >> >>>> 
>> >> >>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version
>> >> >>>> 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller Boot Thread)
>> >> >>>> JBAS015888:
>> >> >>>> Creating http management service using socket-binding
>> >> >>>> (management-http)
>> >> >>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
>> >> >>>> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service thread 1-2)
>> >> >>>> JBoss
>> >> >>>> Remoting version 3.3.5.Final-redhat-1
>> >> >>>> 
>> >> >>>> $ oc logs -f heapster-fnkdc
>> >> >>>> Endpoint Check in effect. Checking
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> accessible
>> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> Could not connect to
>> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> Curl exit code: 6. Status Code 000
>> >> >>>> 
>> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>> >> >>>> INFO  23:00:24 Starting listening for CQL clients on
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>> >> >>>> INFO  23:00:24 Binding thrift service to
>> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>> >> >>>> INFO  23:00:24 enabling encrypted thrift connections between client
>> >> >>>> and
>> >> >>>> server
>> >> >>>> INFO  23:00:24 Listening for thrift clients...
>> >> >>>> INFO  23:00:26 Created default superuser role 'cassandra'
>> >> >>>> 
>> >> >>>> # oc get svc
>> >> >>>> NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)
>> >> >>>> AGE
>> >> >>>> hawkular-cassandra         172.30.2.13      <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>> >> >>>> hawkular-cassandra-nodes   None             <none>
>> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>> >> >>>> hawkular-metrics           172.30.117.176   <none>        443/TCP
>> >> >>>> 25m
>> >> >>>> heapster                   172.30.107.135   <none>        80/TCP
>> >> >>>> 25m
>> >> >>>> 
>> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>> >> >>>> 
>> >> >>>> HTTP/1.1 504 Gateway Timeout
>> >> >>>> Mime-Version: 1.0
>> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>> >> >>>> Content-Type: text/html
>> >> >>>> Connection: keep-alive
>> >> >>>> Proxy-Connection: keep-alive
>> >> >>>> Content-Length: 1572
>> >> >>>> 
>> >> >>>> --
>> >> >>>> Srinivas Kotaru
>> >> >>>> 
>> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
>> >> >>>> <skotaru cisco com>
>> >> >>>> wrote:
>> >> >>>> 
>> >> >>>> >Matt
>> >> >>>> >
>> >> >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and
>> >> >>>> >working
>> >> >>>> >
>> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
>> >> >>>> >
>> >> >>>> >Thanks you for your help so far.
>> >> >>>> >
>> >> >>>> >--
>> >> >>>> >Srinivas Kotaru
>> >> >>>> >
>> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com> wrote:
>> >> >>>> >
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >>----- Original Message -----
>> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> >>>> >>> To: "Matt Wringe" <mwringe redhat com>
>> >> >>>> >>> Cc: users lists openshift redhat com
>> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>> >> >>>> >>> Subject: Re: Metrics deployment
>> >> >>>> >>> 
>> >> >>>> >>> Oh ok
>> >> >>>> >>> 
>> >> >>>> >>> Am using PV for metrics
>> >> >>>> >>> 
>> >> >>>> >>> description: "The persistent volume size for each of the
>> >> >>>> >>> Cassandra
>> >> >>>> >>> nodes"
>> >> >>>> >>>   name: CASSANDRA_PV_SIZE
>> >> >>>> >>>   value: "10Gi"
>> >> >>>> >>> 
>> >> >>>> >>> oc get pv
>> >> >>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS      CLAIM
>> >> >>>> >>> REASON
>> >> >>>> >>> AGE
>> >> >>>> >>> pv-1gb-001    1Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-002    1Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-003    1Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-004    1Gi        RWO           Bound       thlatt/mongodb
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-1gb-005    1Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-0010   2Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-006    2Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-007    2Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-008    2Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-2gb-009    2Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0011   5Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0012   5Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0013   5Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0014   5Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> pv-5gb-0015   5Gi        RWO           Available
>> >> >>>> >>> 4d
>> >> >>>> >>> 
>> >> >>>> >>> am running with below command
>> >> >>>> >>> 
>> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded HOSTNAME,
>> >> >>>> >>> MASTER_API
>> >> >>>> >>> and
>> >> >>>> >>> PV info so not passing any parameters)
>> >> >>>> >>> 
>> >> >>>> >>
>> >> >>>> >>I would suspect that Cassandra is blocked because its waiting for
>> >> >>>> >>10Gi
>> >> >>>> >>PV
>> >> >>>> >>to become available, and none of the PV listed above are big
>> >> >>>> >>enough.
>> >> >>>> >>
>> >> >>>> >>> 
>> >> >>>> >>> --
>> >> >>>> >>> Srinivas Kotaru
>> >> >>>> >>> 
>> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwringe redhat com> wrote:
>> >> >>>> >>> 
>> >> >>>> >>> >----- Original Message -----
>> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> >>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
>> >> >>>> >>> >> Cc: users lists openshift redhat com
>> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>> >> >>>> >>> >> Subject: Re: Metrics deployment
>> >> >>>> >>> >> 
>> >> >>>> >>> >> Matt
>> >> >>>> >>> >> 
>> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but had the same
>> >> >>>> >>> >> issue
>> >> >>>> >>> >> 
>> >> >>>> >>> >> # oc get pods
>> >> >>>> >>> >> NAME                         READY     STATUS
>> >> >>>> >>> >> RESTARTS
>> >> >>>> >>> >> AGE
>> >> >>>> >>> >> hawkular-cassandra-1-y2egy   0/1       ContainerCreating   0
>> >> >>>> >>> >> 5m
>> >> >>>> >>> >> hawkular-metrics-4b16f       0/1       Running             1
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> heapster-x2gj2               0/1       Running             2
>> >> >>>> >>> >> 4m
>> >> >>>> >>> >> metrics-deployer-9v7vc       0/1       Completed           0
>> >> >>>> >>> >> 6m
>> >> >>>> >>> >> 
>> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>> >> >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
>> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
>> >> >>>> >>> >> ContainerCreating
>> >> >>>> >>> >
>> >> >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod
>> >> >>>> >>> >from
>> >> >>>> >>> >starting.
>> >> >>>> >>> >
>> >> >>>> >>> >If you are using persistent storage, Cassandra will not start
>> >> >>>> >>> >until
>> >> >>>> >>> >the
>> >> >>>> >>> >PV
>> >> >>>> >>> >is available. There may be some more information about Cassandra
>> >> >>>> >>> >in
>> >> >>>> >>> >the
>> >> >>>> >>> >pod
>> >> >>>> >>> >section of the console under events.
>> >> >>>> >>> >
>> >> >>>> >>> >What command did you use when deploying the deployer?
>> >> >>>> >>> >
>> >> >>>> >>> >> 
>> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>> >> >>>> >>> >> 
>> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread
>> >> >>>> >>> >> 1-4)
>> >> >>>> >>> >> VM
>> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
>> >> >>>> >>> >> -D[Standalone]
>> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> >> >>>> >>> >> -XX:+UseGCLogFileRotation
>> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
>> >> >>>> >>> >> -XX:-TraceClassUnloading
>> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
>> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>> >> >>>> >>> >> -Djava.awt.headless=true
>> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
>> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>> >> >>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3) XNIO
>> >> >>>> >>> >> Version
>> >> >>>> >>> >> 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread 1-3)
>> >> >>>> >>> >> XNIO
>> >> >>>> >>> >> NIO
>> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>> >> >>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server] (Controller Boot
>> >> >>>> >>> >> Thread)
>> >> >>>> >>> >> JBAS015888:
>> >> >>>> >>> >> Creating http management service using socket-binding
>> >> >>>> >>> >> (management-http)
>> >> >>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service thread
>> >> >>>> >>> >> 1-3)
>> >> >>>> >>> >> JBoss
>> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>> >> >>>> >>> >> 
>> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
>> >> >>>> >>> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>> >> >>>> >>> >> accessible
>> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> Could not connect to
>> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> 
>> >> >>>> >>> >> 
>> >> >>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
>> >> >>>> >>> >> 
>> >> >>>> >>> >> ++ oc create -f -
>> >> >>>> >>> >> serviceaccount "heapster" created
>> >> >>>> >>> >> service "heapster" created
>> >> >>>> >>> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> + echo 'Success!'
>> >> >>>> >>> >> Success!
>> >> >>>> >>> >> 
>> >> >>>> >>> >> --
>> >> >>>> >>> >> Srinivas Kotaru
>> >> >>>> >>> >> 
>> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com> wrote:
>> >> >>>> >>> >> 
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >----- Original Message -----
>> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>> >> >>>> >>> >> >> To: users lists openshift redhat com
>> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>> >> >>>> >>> >> >> Subject: Metrics deployment
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Hi
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Am trying to configure metrics in our newly installed
>> >> >>>> >>> >> >> clusters. Am
>> >> >>>> >>> >> >> seeing
>> >> >>>> >>> >> >> below errors once metrics-deploy script was successful. I
>> >> >>>> >>> >> >> used
>> >> >>>> >>> >> >> our
>> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
>> >> >>>> >>> >> >> MASTER_URL
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >> (
>> >> >>>> >>> >> >> hard coded as values)
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> template "hawkular-heapster" created
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Deploying the Heapster component
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> ++ '[' -n '' ']'
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> ++ oc create -f -
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
>> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> serviceaccount "heapster" created
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> service "heapster" created
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> replicationcontroller "heapster" created
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> + echo 'Success!'
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Success!
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> # oc get pods
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >It looks like none of your containers are fully up and
>> >> >>>> >>> >> >running
>> >> >>>> >>> >> >yet.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The first step
>> >> >>>> >>> >> >will
>> >> >>>> >>> >> >be to
>> >> >>>> >>> >> >get
>> >> >>>> >>> >> >Cassandra running properly.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Endpoint Check in effect. Checking
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
>> >> >>>> >>> >> >> not
>> >> >>>> >>> >> >> accessible
>> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Could not connect to
>> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
>> >> >>>> >>> >> >trying
>> >> >>>> >>> >> >to
>> >> >>>> >>> >> >push
>> >> >>>> >>> >> >metrics to it. The issue that you are seeing is because
>> >> >>>> >>> >> >Heapster
>> >> >>>> >>> >> >could
>> >> >>>> >>> >> >not
>> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular
>> >> >>>> >>> >> >Metrics
>> >> >>>> >>> >> >service
>> >> >>>> >>> >> >is
>> >> >>>> >>> >> >fully up, Heapster will not be able to connect to it.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics
>> >> >>>> >>> >> >> rather
>> >> >>>> >>> >> >> full
>> >> >>>> >>> >> >> routing
>> >> >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal
>> >> >>>> >>> >> >hostname
>> >> >>>> >>> >> >used
>> >> >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the
>> >> >>>> >>> >> >external
>> >> >>>> >>> >> >hostname (eg what is configured via
>> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
>> >> >>>> >>> >> >The
>> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of
>> >> >>>> >>> >> >services,
>> >> >>>> >>> >> >which
>> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>> >> >>>> >>> >> >
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> --
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> Srinivas Kotaru
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> >> _______________________________________________
>> >> >>>> >>> >> >> users mailing list
>> >> >>>> >>> >> >> users lists openshift redhat com
>> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >> >>>> >>> >> >> 
>> >> >>>> >>> >> 
>> >> >>>> >>> >> 
>> >> >>>> >>> 
>> >> >>>> >>> 
>> >> >>>> >
>> >> >>>> 
>> >> >>>> 
>> >> >>
>> >> >
>> >> 
>> >> 
>> 
>> 



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]