[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



I can see internal DNS server also resolvable

dig +short @master-01 hawkular-metrics.openshift-infra.svc.cluster.local
172.30.117.176

so all looks good but still seems to be some issues. 

-- 
Srinivas Kotaru

On 6/14/16, 10:37 AM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:

>I removed readiness probes from both hawkular-cassandra-1 & hawkular-metrics as both status shows probes failed. 
>
>It looks good now. Both containers looks and running (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however heapster-7yl34 logs still shows Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. Status Code 000. 
>
>Are we good or still had issues? 
>
>
># oc get pods
>NAME                         READY     STATUS    RESTARTS   AGE
>hawkular-cassandra-1-kr8ka   1/1       Running   0          6m
>hawkular-metrics-vhe3u       1/1       Running   2          5m
>heapster-7yl34               0/1       Running   2          5m
>
>
>
>
>
>-- 
>Srinivas Kotaru
>
>On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:
>
>>Matt
>>
>>Just want to share more info by running describe pod.
>>
>>It seems to be health probe failing. Do you think it is the issue? 
>>
>>
>>
>># oc describe pod hawkular-cassandra-1-it5uh
>>Name:		hawkular-cassandra-1-it5uh
>>Namespace:	openshift-infra
>>Node:		l3inpn-id2-003.cisco.com/173.36.96.16
>>Start Time:	Tue, 14 Jun 2016 16:36:21 +0000
>>Labels:		metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
>>Status:		Running
>>IP:		10.1.9.2
>>Controllers:	ReplicationController/hawkular-cassandra-1
>>Containers:
>>  hawkular-cassandra-1:
>>    Container ID:	docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
>>    Image:		registry.access.redhat.com/openshift3/metrics-cassandra:latest
>>    Image ID:		docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
>>    Ports:		9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>>    Command:
>>      /opt/apache-cassandra/bin/cassandra-docker.sh
>>      --cluster_name=hawkular-metrics
>>      --data_volume=/cassandra_data
>>      --internode_encryption=all
>>      --require_node_auth=true
>>      --enable_client_encryption=true
>>      --require_client_auth=true
>>      --keystore_file=/secret/cassandra.keystore
>>      --keystore_password_file=/secret/cassandra.keystore.password
>>      --truststore_file=/secret/cassandra.truststore
>>      --truststore_password_file=/secret/cassandra.truststore.password
>>      --cassandra_pem_file=/secret/cassandra.pem
>>    QoS Tier:
>>      cpu:		BestEffort
>>      memory:		BestEffort
>>    State:		Running
>>      Started:		Tue, 14 Jun 2016 16:37:01 +0000
>>    Ready:		True
>>    Restart Count:	0
>>    Readiness:		exec [/opt/apache-cassandra/bin/cassandra-docker-ready.sh] delay=0s timeout=1s period=10s #success=1 #failure=3
>>    Environment Variables:
>>      CASSANDRA_MASTER:	true
>>      POD_NAMESPACE:	openshift-infra (v1:metadata.namespace)
>>Conditions:
>>  Type		Status
>>  Ready 	True
>>Volumes:
>>  cassandra-data:
>>    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
>>    ClaimName:	metrics-cassandra-1
>>    ReadOnly:	false
>>  hawkular-cassandra-secrets:
>>    Type:	Secret (a volume populated by a Secret)
>>    SecretName:	hawkular-cassandra-secrets
>>  cassandra-token-4urfd:
>>    Type:	Secret (a volume populated by a Secret)
>>    SecretName:	cassandra-token-4urfd
>>Events:
>>  FirstSeen	LastSeen	Count	From					SubobjectPath				Type		Reason		Message
>>  ---------	--------	-----	----					-------------				--------	------		-------
>>  27m		27m		1	{default-scheduler }								Normal		Scheduled	Successfully assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
>>  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully pulled image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
>>  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Created		Created container with docker id 17a9575eb655
>>  27m		27m		1	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Normal		Started		Started container with docker id 17a9575eb655
>>  27m		26m		3	{kubelet l3inpn-id2-003.cisco.com}	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
>>Cassandra not in the up and normal state. Current state is
>>/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected
>>
>>
>>
>>
>>
>>-- 
>>Srinivas Kotaru
>>
>>On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com> wrote:
>>
>>>Matt
>>>
>>>DNS service is working 
>>>
>>>
>>>~ dig +short @master-01 kubernetes.default.svc.cluster.local
>>>172.30.0.1
>>>~  dig +short @master-01  jenkins.alln-test.svc.cluster.local
>>>172.30.85.148
>>>~  dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
>>>172.30.31.6
>>>
>>>I captured hawkular-metrics, it shows the problem. It seems to be unable to connect Cassandra cluster
>>>
>>># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
>>>
>>>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
>>>12:40:58,898 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
>>>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 46) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.
>>>12:40:58,902 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013371: Activating Security Subsystem
>>>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 39) JBAS011800: Activating Naming Subsystem
>>>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 37) JBAS012615: Activated the following JSF Implementations: [main, 1.2]
>>>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (IronJacamar 1.0.32.Final-redhat-1)
>>>12:40:58,981 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
>>>12:40:58,961 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
>>>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: Starting Naming Service
>>>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
>>>12:40:59,614 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8444
>>>12:40:59,620 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/0.0.0.0:8080
>>>12:40:59,621 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8444
>>>12:40:59,627 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/0.0.0.0:8080
>>>12:40:59,654 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
>>>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "activemq-rar.rar" (runtime-name: "activemq-rar.rar")
>>>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "hawkular-metrics-api-jaxrs.war" (runtime-name: "hawkular-metrics-api-jaxrs.war")
>>>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) JBAS015012: Started FileSystemDeploymentService for directory /opt/eap/standalone/deployments
>>>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on 127.0.0.1:9999
>>>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC service thread 1-3) JBAS015102: Unable to load console module for slot eap, disabling console
>>>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.3.5.Final-redhat-3
>>>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 0.0.0.0:4447
>>>12:41:01,361 INFO  [org.jboss.as.connector.deployers.RADeployer] (MSC service thread 1-3) IJ020001: Required license terms for file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
>>>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a private module ("org.codehaus.jackson.jackson-core-asl:main") which may be changed or removed in future versions without notice.
>>>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a private module ("org.codehaus.jackson.jackson-core-asl:main") which may be changed or removed in future versions without notice.
>>>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be changed or removed in future versions without notice.
>>>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) JBAS015867: Deployment "deployment.hawkular-metrics-api-jaxrs.war" is using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be changed or removed in future versions without notice.
>>>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
>>>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) JBAS016005: Starting Services for CDI deployment: hawkular-metrics-api-jaxrs.war
>>>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service thread 1-1) WELD-000900 1.1.31 (redhat)
>>>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service thread 1-3) JBAS016008: Starting weld service for deployment hawkular-metrics-api-jaxrs.war
>>>12:41:02,820 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread Pool -- 51) JBAS018210: Register web context: /hawkular/metrics
>>>12:41:02,958 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 28) JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name : "hawkular-metrics-api-jaxrs.war")
>>>12:41:02,959 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 28) JBAS015859: Deployed "activemq-rar.rar" (runtime-name : "activemq-rar.rar")
>>>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management
>>>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
>>>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - Started 267 of 303 services (62 services are lazy, passive or on-demand)
>>>12:41:02,999 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
>>>12:41:02,999 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying connecting to Cassandra cluster in [1]s...
>>>12:41:04,000 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:04,001 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:04,001 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying connecting to Cassandra cluster in [2]s...
>>>12:41:06,002 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:06,002 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:06,003 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying connecting to Cassandra cluster in [3]s...
>>>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080-2) RESTEASY006050: Found BeanManager at java:comp/BeanManager
>>>12:41:08,299 INFO  [org.hibernate.validator.internal.util.Version] (http-/0.0.0.0:8080-2) HV000001: Hibernate Validator 4.3.2.Final-redhat-2
>>>12:41:08,418 INFO  [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] (http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>>>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (http-/0.0.0.0:8080-2) RESTEASY000320: Deploying javax.ws.rs.core.Application: class org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
>>>12:41:08,421 INFO  [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] (http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
>>>12:41:08,422 INFO  [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] (http-/0.0.0.0:8080-2) Virtual clock is disabled
>>>12:41:08,422 INFO  [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] (http-/0.0.0.0:8080-2) Virtual clock is disabled
>>>12:41:09,003 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:09,004 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:09,004 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying connecting to Cassandra cluster in [4]s...
>>>12:41:13,004 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:13,123 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
>>>12:41:13,124 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying connecting to Cassandra cluster in [1]s...
>>>12:41:14,124 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:14,124 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:14,125 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying connecting to Cassandra cluster in [2]s...
>>>12:41:16,125 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:16,126 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:16,126 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying connecting to Cassandra cluster in [3]s...
>>>12:41:19,126 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:19,127 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:19,127 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying connecting to Cassandra cluster in [4]s...
>>>12:41:23,128 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:23,249 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
>>>12:41:23,249 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying connecting to Cassandra cluster in [1]s...
>>>12:41:24,250 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:24,250 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:24,250 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying connecting to Cassandra cluster in [2]s...
>>>12:41:26,251 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:26,251 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:26,251 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying connecting to Cassandra cluster in [3]s...
>>>12:41:29,252 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:29,252 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:29,253 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying connecting to Cassandra cluster in [4]s...
>>>12:41:33,253 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:33,376 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
>>>12:41:33,377 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying connecting to Cassandra cluster in [1]s...
>>>12:41:34,377 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:34,378 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:34,378 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying connecting to Cassandra cluster in [2]s...
>>>12:41:36,378 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:36,379 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:36,379 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying connecting to Cassandra cluster in [3]s...
>>>12:41:39,379 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:39,380 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:39,380 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying connecting to Cassandra cluster in [4]s...
>>>12:41:43,380 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:43,503 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown error
>>>12:41:43,504 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying connecting to Cassandra cluster in [1]s...
>>>12:41:44,504 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
>>>12:41:44,505 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: hawkular-cassandra
>>>12:41:44,505 WARN  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying connecting to Cassandra cluster in [2]s...
>>>
>>>-- 
>>>Srinivas Kotaru
>>>
>>>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com> wrote:
>>>
>>>>----- Original Message -----
>>>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>>> To: "Matt Wringe" <mwringe redhat com>
>>>>> Cc: users lists openshift redhat com
>>>>> Sent: Monday, June 13, 2016 7:26:06 PM
>>>>> Subject: Re: Metrics deployment
>>>>> 
>>>>> Matt
>>>>> 
>>>>> PV issue resolved. Was able to to see PV successfully bounded and Casandra
>>>>> container has been running. However, it seems puzzle not fully yet solved.
>>>>
>>>>Are you sure the OpenShift DNS server is running?
>>>>
>>>>If you are running OSE 3.1, can you please follow this https://access.redhat.com/solutions/2329131 and see if you are now seeing errors in the Hawkular Metrics logs (essentially just run `oc exec hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
>>>>
>>>>> 
>>>>> I could see other container(heapster) not coming up, and seeing below errors
>>>>> 
>>>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
>>>>> Endpoint Check in effect. Checking
>>>>> https://hawkular-metrics:443/hawkular/metrics/status
>>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> Curl exit code: 6. Status Code 000
>>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> Curl exit code: 6. Status Code 000
>>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> 
>>>>> 
>>>>> # oc get pv
>>>>> pv-5gb-0011   5Gi        RWO           Bound
>>>>> openshift-infra/metrics-cassandra-1             22m
>>>>> 
>>>>> 
>>>>> $ oc get pods
>>>>> NAME                         READY     STATUS      RESTARTS   AGE
>>>>> hawkular-cassandra-1-2pzd7   1/1       Running     0          20m
>>>>> hawkular-metrics-mf5qf       0/1       Running     7          20m
>>>>> heapster-fnkdc               0/1       Error       6          20m
>>>>> metrics-deployer-cvep0       0/1       Completed   0          21m
>>>>> 
>>>>> # oc logs -f hawkular-metrics-mf5qf
>>>>> 
>>>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version
>>>>> 3.0.14.GA-redhat-1
>>>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888:
>>>>> Creating http management service using socket-binding (management-http)
>>>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
>>>>> Implementation Version 3.0.14.GA-redhat-1
>>>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss
>>>>> Remoting version 3.3.5.Final-redhat-1
>>>>> 
>>>>> $ oc logs -f heapster-fnkdc
>>>>> Endpoint Check in effect. Checking
>>>>> https://hawkular-metrics:443/hawkular/metrics/status
>>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> Curl exit code: 6. Status Code 000
>>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> Curl exit code: 6. Status Code 000
>>>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
>>>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> Curl exit code: 6. Status Code 000
>>>>> 
>>>>> $ oc logs -f hawkular-cassandra-1-2pzd7
>>>>> INFO  23:00:24 Starting listening for CQL clients on
>>>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
>>>>> INFO  23:00:24 Binding thrift service to
>>>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
>>>>> INFO  23:00:24 enabling encrypted thrift connections between client and
>>>>> server
>>>>> INFO  23:00:24 Listening for thrift clients...
>>>>> INFO  23:00:26 Created default superuser role 'cassandra'
>>>>> 
>>>>> # oc get svc
>>>>> NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)
>>>>> AGE
>>>>> hawkular-cassandra         172.30.2.13      <none>
>>>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>>>> hawkular-cassandra-nodes   None             <none>
>>>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
>>>>> hawkular-metrics           172.30.117.176   <none>        443/TCP
>>>>> 25m
>>>>> heapster                   172.30.107.135   <none>        80/TCP
>>>>> 25m
>>>>> 
>>>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
>>>>> 
>>>>> HTTP/1.1 504 Gateway Timeout
>>>>> Mime-Version: 1.0
>>>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
>>>>> Content-Type: text/html
>>>>> Connection: keep-alive
>>>>> Proxy-Connection: keep-alive
>>>>> Content-Length: 1572
>>>>> 
>>>>> --
>>>>> Srinivas Kotaru
>>>>> 
>>>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>>> wrote:
>>>>> 
>>>>> >Matt
>>>>> >
>>>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and working
>>>>> >
>>>>> >I adjusted PV to 5Gi and reran. Will update progress.
>>>>> >
>>>>> >Thanks you for your help so far.
>>>>> >
>>>>> >--
>>>>> >Srinivas Kotaru
>>>>> >
>>>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>>>> >
>>>>> >>
>>>>> >>
>>>>> >>----- Original Message -----
>>>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>>> >>> To: "Matt Wringe" <mwringe redhat com>
>>>>> >>> Cc: users lists openshift redhat com
>>>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
>>>>> >>> Subject: Re: Metrics deployment
>>>>> >>> 
>>>>> >>> Oh ok
>>>>> >>> 
>>>>> >>> Am using PV for metrics
>>>>> >>> 
>>>>> >>> description: "The persistent volume size for each of the Cassandra nodes"
>>>>> >>>   name: CASSANDRA_PV_SIZE
>>>>> >>>   value: "10Gi"
>>>>> >>> 
>>>>> >>> oc get pv
>>>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS      CLAIM
>>>>> >>> REASON
>>>>> >>> AGE
>>>>> >>> pv-1gb-001    1Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-1gb-002    1Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-1gb-003    1Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-1gb-004    1Gi        RWO           Bound       thlatt/mongodb
>>>>> >>> 4d
>>>>> >>> pv-1gb-005    1Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-2gb-0010   2Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-2gb-006    2Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-2gb-007    2Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-2gb-008    2Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-2gb-009    2Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-5gb-0011   5Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-5gb-0012   5Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-5gb-0013   5Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-5gb-0014   5Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> pv-5gb-0015   5Gi        RWO           Available
>>>>> >>> 4d
>>>>> >>> 
>>>>> >>> am running with below command
>>>>> >>> 
>>>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded HOSTNAME, MASTER_API
>>>>> >>> and
>>>>> >>> PV info so not passing any parameters)
>>>>> >>> 
>>>>> >>
>>>>> >>I would suspect that Cassandra is blocked because its waiting for 10Gi PV
>>>>> >>to become available, and none of the PV listed above are big enough.
>>>>> >>
>>>>> >>> 
>>>>> >>> --
>>>>> >>> Srinivas Kotaru
>>>>> >>> 
>>>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>>>> >>> 
>>>>> >>> >----- Original Message -----
>>>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
>>>>> >>> >> Cc: users lists openshift redhat com
>>>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
>>>>> >>> >> Subject: Re: Metrics deployment
>>>>> >>> >> 
>>>>> >>> >> Matt
>>>>> >>> >> 
>>>>> >>> >> Thanks for looking into. I rerun the setup, but had the same issue
>>>>> >>> >> 
>>>>> >>> >> # oc get pods
>>>>> >>> >> NAME                         READY     STATUS              RESTARTS
>>>>> >>> >> AGE
>>>>> >>> >> hawkular-cassandra-1-y2egy   0/1       ContainerCreating   0
>>>>> >>> >> 5m
>>>>> >>> >> hawkular-metrics-4b16f       0/1       Running             1
>>>>> >>> >> 4m
>>>>> >>> >> heapster-x2gj2               0/1       Running             2
>>>>> >>> >> 4m
>>>>> >>> >> metrics-deployer-9v7vc       0/1       Completed           0
>>>>> >>> >> 6m
>>>>> >>> >> 
>>>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
>>>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
>>>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start: ContainerCreating
>>>>> >>> >
>>>>> >>> >Ok, so it looks like something is blocking the Cassandra pod from
>>>>> >>> >starting.
>>>>> >>> >
>>>>> >>> >If you are using persistent storage, Cassandra will not start until the
>>>>> >>> >PV
>>>>> >>> >is available. There may be some more information about Cassandra in the
>>>>> >>> >pod
>>>>> >>> >section of the console under events.
>>>>> >>> >
>>>>> >>> >What command did you use when deploying the deployer?
>>>>> >>> >
>>>>> >>> >> 
>>>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
>>>>> >>> >> 
>>>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread 1-4) VM
>>>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss -D[Standalone]
>>>>> >>> >> -XX:+UseCompressedOops -verbose:gc
>>>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
>>>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
>>>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading
>>>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
>>>>> >>> >> -Djava.net.preferIPv4Stack=true
>>>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
>>>>> >>> >> -Djava.awt.headless=true
>>>>> >>> >> -Djboss.modules.policy-permissions=true
>>>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
>>>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
>>>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
>>>>> >>> >> -Djava.security.egd=file:/dev/./urandom
>>>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
>>>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
>>>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3) XNIO Version
>>>>> >>> >> 3.0.14.GA-redhat-1
>>>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO
>>>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
>>>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server] (Controller Boot Thread)
>>>>> >>> >> JBAS015888:
>>>>> >>> >> Creating http management service using socket-binding
>>>>> >>> >> (management-http)
>>>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss
>>>>> >>> >> Remoting version 3.3.5.Final-redhat-1
>>>>> >>> >> 
>>>>> >>> >> $ oc logs -f heapster-x2gj2
>>>>> >>> >> Endpoint Check in effect. Checking
>>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
>>>>> >>> >> Could not connect to
>>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> >>> >> Curl exit code: 6. Status Code 000
>>>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>>> >>> >> accessible
>>>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> >>> >> Could not connect to
>>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> >>> >> Curl exit code: 6. Status Code 000
>>>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>>> >>> >> accessible
>>>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> >>> >> Could not connect to
>>>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> >>> >> Curl exit code: 6. Status Code 000
>>>>> >>> >> 
>>>>> >>> >> 
>>>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
>>>>> >>> >> 
>>>>> >>> >> ++ oc create -f -
>>>>> >>> >> serviceaccount "heapster" created
>>>>> >>> >> service "heapster" created
>>>>> >>> >> replicationcontroller "heapster" created
>>>>> >>> >> + echo 'Success!'
>>>>> >>> >> Success!
>>>>> >>> >> 
>>>>> >>> >> --
>>>>> >>> >> Srinivas Kotaru
>>>>> >>> >> 
>>>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com> wrote:
>>>>> >>> >> 
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >----- Original Message -----
>>>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
>>>>> >>> >> >> To: users lists openshift redhat com
>>>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
>>>>> >>> >> >> Subject: Metrics deployment
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> Hi
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> Am trying to configure metrics in our newly installed clusters. Am
>>>>> >>> >> >> seeing
>>>>> >>> >> >> below errors once metrics-deploy script was successful. I used our
>>>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and MASTER_URL
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
>>>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
>>>>> >>> >> >> (
>>>>> >>> >> >> hard coded as values)
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> template "hawkular-heapster" created
>>>>> >>> >> >> 
>>>>> >>> >> >> Deploying the Heapster component
>>>>> >>> >> >> 
>>>>> >>> >> >> ++ echo 'Deploying the Heapster component'
>>>>> >>> >> >> 
>>>>> >>> >> >> ++ '[' -n '' ']'
>>>>> >>> >> >> 
>>>>> >>> >> >> ++ oc create -f -
>>>>> >>> >> >> 
>>>>> >>> >> >> ++ oc process hawkular-heapster -v
>>>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
>>>>> >>> >> >> 
>>>>> >>> >> >> serviceaccount "heapster" created
>>>>> >>> >> >> 
>>>>> >>> >> >> service "heapster" created
>>>>> >>> >> >> 
>>>>> >>> >> >> replicationcontroller "heapster" created
>>>>> >>> >> >> 
>>>>> >>> >> >> + echo 'Success!'
>>>>> >>> >> >> 
>>>>> >>> >> >> Success!
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> # oc get pods
>>>>> >>> >> >> 
>>>>> >>> >> >> NAME READY STATUS RESTARTS AGE
>>>>> >>> >> >> 
>>>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
>>>>> >>> >> >> 
>>>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
>>>>> >>> >> >> 
>>>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
>>>>> >>> >> >> 
>>>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
>>>>> >>> >> >> 
>>>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
>>>>> >>> >> >> 
>>>>> >>> >> >
>>>>> >>> >> >It looks like none of your containers are fully up and running yet.
>>>>> >>> >> >
>>>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, and
>>>>> >>> >> >Heapster
>>>>> >>> >> >will
>>>>> >>> >> >wait until Hawkular Metrics is fully running.
>>>>> >>> >> >
>>>>> >>> >> >Do you see anything in the Cassandra logs? The first step will be to
>>>>> >>> >> >get
>>>>> >>> >> >Cassandra running properly.
>>>>> >>> >> >
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> $ oc logs -f heapster-e8gbu
>>>>> >>> >> >> 
>>>>> >>> >> >> Endpoint Check in effect. Checking
>>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
>>>>> >>> >> >> 
>>>>> >>> >> >> Could not connect to
>>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> >>> >> >> Curl exit code: 6. Status Code 000
>>>>> >>> >> >> 
>>>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
>>>>> >>> >> >> accessible
>>>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
>>>>> >>> >> >> 
>>>>> >>> >> >> Could not connect to
>>>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
>>>>> >>> >> >> Curl exit code: 6. Status Code 000
>>>>> >>> >> >
>>>>> >>> >> >Heapster waits until Hawkular Metrics is started before trying to
>>>>> >>> >> >push
>>>>> >>> >> >metrics to it. The issue that you are seeing is because Heapster
>>>>> >>> >> >could
>>>>> >>> >> >not
>>>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular Metrics
>>>>> >>> >> >service
>>>>> >>> >> >is
>>>>> >>> >> >fully up, Heapster will not be able to connect to it.
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics rather
>>>>> >>> >> >> full
>>>>> >>> >> >> routing
>>>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
>>>>> >>> >> >
>>>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal hostname
>>>>> >>> >> >used
>>>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the external
>>>>> >>> >> >hostname (eg what is configured via HAWKULAR_METRICS_HOSTNAME). The
>>>>> >>> >> >OpenShift dns server will resolve hostnames to the name of services,
>>>>> >>> >> >which
>>>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
>>>>> >>> >> >
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> --
>>>>> >>> >> >> 
>>>>> >>> >> >> 
>>>>> >>> >> >> Srinivas Kotaru
>>>>> >>> >> >> 
>>>>> >>> >> >> _______________________________________________
>>>>> >>> >> >> users mailing list
>>>>> >>> >> >> users lists openshift redhat com
>>>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>>> >>> >> >> 
>>>>> >>> >> 
>>>>> >>> >> 
>>>>> >>> 
>>>>> >>> 
>>>>> >
>>>>> 
>>>>> 
>>>
>>
>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]