[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment




----- Original Message -----
> From: "Luke Meyer" <lmeyer redhat com>
> To: "Matt Wringe" <mwringe redhat com>
> Cc: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>, "users" <users lists openshift redhat com>
> Sent: Tuesday, June 14, 2016 5:10:05 PM
> Subject: Re: Metrics deployment
> 
> The readiness probe status seems like an important indicator to me:
> 
> Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or
> directory
> 
> What could cause that failure? Or is that a red herring...

red herring. Its on the readiness probe and happens when the readiness probe should be failing for Cassandra. It looks like we have a bug in our readiness probe script, but its benign and can be ignored.

> 
> On Tue, Jun 14, 2016 at 1:53 PM, Matt Wringe <mwringe redhat com> wrote:
> 
> > ----- Original Message -----
> > > From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> > > To: "Matt Wringe" <mwringe redhat com>
> > > Cc: users lists openshift redhat com
> > > Sent: Tuesday, June 14, 2016 1:37:01 PM
> > > Subject: Re: Metrics deployment
> > >
> > > I removed readiness probes from both hawkular-cassandra-1 &
> > hawkular-metrics
> > > as both status shows probes failed.
> >
> > You should not have to remove the probes, this indicates that something is
> > wrong with your installation.
> >
> > >
> > > It looks good now. Both containers looks and running
> > > (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
> > heapster-7yl34
> > > logs still shows Could not connect to
> > > https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6.
> > > Status Code 000.
> > >
> > > Are we good or still had issues?
> > >
> > >
> > > # oc get pods
> > > NAME                         READY     STATUS    RESTARTS   AGE
> > > hawkular-cassandra-1-kr8ka   1/1       Running   0          6m
> > > hawkular-metrics-vhe3u       1/1       Running   2          5m
> > > heapster-7yl34               0/1       Running   2          5m
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Srinivas Kotaru
> > >
> > > On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" <
> > skotaru cisco com>
> > > wrote:
> > >
> > > >Matt
> > > >
> > > >Just want to share more info by running describe pod.
> > > >
> > > >It seems to be health probe failing. Do you think it is the issue?
> > > >
> > > >
> > > >
> > > ># oc describe pod hawkular-cassandra-1-it5uh
> > > >Name:                hawkular-cassandra-1-it5uh
> > > >Namespace:   openshift-infra
> > > >Node:                l3inpn-id2-003.cisco.com/173.36.96.16
> > > >Start Time:  Tue, 14 Jun 2016 16:36:21 +0000
> > > >Labels:
> > > >
> >  metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
> > > >Status:              Running
> > > >IP:          10.1.9.2
> > > >Controllers: ReplicationController/hawkular-cassandra-1
> > > >Containers:
> > > >  hawkular-cassandra-1:
> > > >    Container ID:
> > > >
> >  docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
> > > >    Image:
> > registry.access.redhat.com/openshift3/metrics-cassandra:latest
> > > >    Image ID:
> > > >
> >  docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
> > > >    Ports:           9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
> > > >    Command:
> > > >      /opt/apache-cassandra/bin/cassandra-docker.sh
> > > >      --cluster_name=hawkular-metrics
> > > >      --data_volume=/cassandra_data
> > > >      --internode_encryption=all
> > > >      --require_node_auth=true
> > > >      --enable_client_encryption=true
> > > >      --require_client_auth=true
> > > >      --keystore_file=/secret/cassandra.keystore
> > > >      --keystore_password_file=/secret/cassandra.keystore.password
> > > >      --truststore_file=/secret/cassandra.truststore
> > > >      --truststore_password_file=/secret/cassandra.truststore.password
> > > >      --cassandra_pem_file=/secret/cassandra.pem
> > > >    QoS Tier:
> > > >      cpu:           BestEffort
> > > >      memory:                BestEffort
> > > >    State:           Running
> > > >      Started:               Tue, 14 Jun 2016 16:37:01 +0000
> > > >    Ready:           True
> > > >    Restart Count:   0
> > > >    Readiness:               exec
> > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
> > > >    delay=0s timeout=1s period=10s #success=1 #failure=3
> > > >    Environment Variables:
> > > >      CASSANDRA_MASTER:      true
> > > >      POD_NAMESPACE: openshift-infra (v1:metadata.namespace)
> > > >Conditions:
> > > >  Type               Status
> > > >  Ready      True
> > > >Volumes:
> > > >  cassandra-data:
> > > >    Type:    PersistentVolumeClaim (a reference to a
> > PersistentVolumeClaim in
> > > >    the same namespace)
> > > >    ClaimName:       metrics-cassandra-1
> > > >    ReadOnly:        false
> > > >  hawkular-cassandra-secrets:
> > > >    Type:    Secret (a volume populated by a Secret)
> > > >    SecretName:      hawkular-cassandra-secrets
> > > >  cassandra-token-4urfd:
> > > >    Type:    Secret (a volume populated by a Secret)
> > > >    SecretName:      cassandra-token-4urfd
> > > >Events:
> > > >  FirstSeen  LastSeen        Count   From
> >       SubobjectPath                           Type            Reason
> >   Message
> > > >  ---------  --------        -----   ----
> >       -------------                           --------        ------
> > > >             -------
> > > >  27m                27m             1       {default-scheduler }
> >                                                       Normal
> > Scheduled       Successfully
> > > >  assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com
> > > >  27m                27m             1       {kubelet
> > l3inpn-id2-003.cisco.com}
> > > >     spec.containers{hawkular-cassandra-1}   Normal          Pulling
> >      pulling image
> > > >  "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> > > >  27m                27m             1       {kubelet
> > l3inpn-id2-003.cisco.com}
> > > >     spec.containers{hawkular-cassandra-1}   Normal          Pulled
> >       Successfully pulled
> > > >  image "registry.access.redhat.com/openshift3/metrics-cassandra:latest
> > "
> > > >  27m                27m             1       {kubelet
> > l3inpn-id2-003.cisco.com}
> > > >     spec.containers{hawkular-cassandra-1}   Normal          Created
> >      Created container
> > > >  with docker id 17a9575eb655
> > > >  27m                27m             1       {kubelet
> > l3inpn-id2-003.cisco.com}
> > > >     spec.containers{hawkular-cassandra-1}   Normal          Started
> >      Started container
> > > >  with docker id 17a9575eb655
> > > >  27m                26m             3       {kubelet
> > l3inpn-id2-003.cisco.com}
> > > >     spec.containers{hawkular-cassandra-1}   Warning         Unhealthy
> >      Readiness probe
> > > >  failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> > > >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> > > >'Connection refused'.
> > > >Cassandra not in the up and normal state. Current state is
> > > >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
> > unary
> > > >operator expected
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >--
> > > >Srinivas Kotaru
> > > >
> > > >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)" <
> > skotaru cisco com>
> > > >wrote:
> > > >
> > > >>Matt
> > > >>
> > > >>DNS service is working
> > > >>
> > > >>
> > > >>~ dig +short @master-01 kubernetes.default.svc.cluster.local
> > > >>172.30.0.1
> > > >>~  dig +short @master-01  jenkins.alln-test.svc.cluster.local
> > > >>172.30.85.148
> > > >>~  dig +short @master-01 cakephp-example.alln-test.svc.cluster.local
> > > >>172.30.31.6
> > > >>
> > > >>I captured hawkular-metrics, it shows the problem. It seems to be
> > unable to
> > > >>connect Cassandra cluster
> > > >>
> > > >># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log
> > > >>
> > > >>12:40:58,848 INFO  [org.jboss.as.clustering.infinispan] (ServerService
> > > >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
> > > >>12:40:58,898 INFO  [org.jboss.as.webservices] (ServerService Thread
> > Pool --
> > > >>48) JBAS015537: Activating WebServices Extension
> > > >>12:40:58,901 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 46)
> > > >>JBAS010153: Node identifier property is set to the default value.
> > Please
> > > >>make sure it is unique.
> > > >>12:40:58,902 INFO  [org.jboss.as.security] (ServerService Thread Pool
> > --
> > > >>44) JBAS013371: Activating Security Subsystem
> > > >>12:40:58,912 INFO  [org.jboss.as.naming] (ServerService Thread Pool --
> > 39)
> > > >>JBAS011800: Activating Naming Subsystem
> > > >>12:40:58,928 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 37)
> > > >>JBAS012615: Activated the following JSF Implementations: [main, 1.2]
> > > >>12:40:58,939 INFO  [org.jboss.as.connector.logging] (MSC service thread
> > > >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
> > > >>1.0.32.Final-redhat-1)
> > > >>12:40:58,981 INFO  [org.jboss.as.security] (MSC service thread 1-2)
> > > >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
> > > >>12:40:58,961 INFO  [org.jboss.as.connector.subsystems.datasources]
> > > >>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant
> > > >>driver class org.h2.Driver (version 1.3)
> > > >>12:40:59,039 INFO  [org.jboss.as.naming] (MSC service thread 1-2)
> > > >>JBAS011802: Starting Naming Service
> > > >>12:40:59,052 INFO  [org.jboss.as.mail.extension] (MSC service thread
> > 1-4)
> > > >>JBAS015400: Bound mail session [java:jboss/mail/Default]
> > > >>12:40:59,614 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
> > service
> > > >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
> > > >>http-/0.0.0.0:8444
> > > >>12:40:59,620 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
> > service
> > > >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
> > > >>http-/0.0.0.0:8080
> > > >>12:40:59,621 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
> > service
> > > >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/
> > 0.0.0.0:8444
> > > >>12:40:59,627 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC
> > service
> > > >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/
> > 0.0.0.0:8080
> > > >>12:40:59,654 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC
> > > >>service thread 1-3) JBAS010400: Bound data source
> > > >>[java:jboss/datasources/ExampleDS]
> > > >>12:40:59,985 INFO  [org.jboss.as.server.deployment] (MSC service thread
> > > >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
> > (runtime-name:
> > > >>"activemq-rar.rar")
> > > >>12:40:59,986 INFO  [org.jboss.as.server.deployment] (MSC service thread
> > > >>1-1) JBAS015876: Starting deployment of
> > "hawkular-metrics-api-jaxrs.war"
> > > >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
> > > >>12:40:59,987 INFO  [org.jboss.as.server.deployment.scanner] (MSC
> > service
> > > >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for
> > directory
> > > >>/opt/eap/standalone/deployments
> > > >>12:41:00,011 INFO  [org.jboss.as.remoting] (MSC service thread 1-2)
> > > >>JBAS017100: Listening on 127.0.0.1:9999
> > > >>12:41:00,030 WARN  [org.jboss.as.domain.http.api] (MSC service thread
> > 1-3)
> > > >>JBAS015102: Unable to load console module for slot eap, disabling
> > console
> > > >>12:41:00,042 INFO  [org.jboss.ws.common.management] (MSC service thread
> > > >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server
> > > >>4.3.5.Final-redhat-3
> > > >>12:41:00,045 INFO  [org.jboss.as.remoting] (MSC service thread 1-4)
> > > >>JBAS017100: Listening on 0.0.0.0:4447
> > > >>12:41:01,361 INFO  [org.jboss.as.connector.deployers.RADeployer] (MSC
> > > >>service thread 1-3) IJ020001: Required license terms for
> > >
> > >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
> > > >>12:41:01,784 WARN  [org.jboss.as.dependency.private] (MSC service
> > thread
> > > >>1-1) JBAS015867: Deployment
> > "deployment.hawkular-metrics-api-jaxrs.war" is
> > > >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
> > > >>which may be changed or removed in future versions without notice.
> > > >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
> > thread
> > > >>1-1) JBAS015867: Deployment
> > "deployment.hawkular-metrics-api-jaxrs.war" is
> > > >>using a private module ("org.codehaus.jackson.jackson-core-asl:main")
> > > >>which may be changed or removed in future versions without notice.
> > > >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
> > thread
> > > >>1-1) JBAS015867: Deployment
> > "deployment.hawkular-metrics-api-jaxrs.war" is
> > > >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
> > > >>which may be changed or removed in future versions without notice.
> > > >>12:41:01,785 WARN  [org.jboss.as.dependency.private] (MSC service
> > thread
> > > >>1-1) JBAS015867: Deployment
> > "deployment.hawkular-metrics-api-jaxrs.war" is
> > > >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main")
> > > >>which may be changed or removed in future versions without notice.
> > > >>12:41:01,843 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1)
> > > >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war
> > > >>12:41:01,941 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1)
> > > >>JBAS016005: Starting Services for CDI deployment:
> > > >>hawkular-metrics-api-jaxrs.war
> > > >>12:41:01,986 INFO  [org.jboss.weld.Version] (MSC service thread 1-1)
> > > >>WELD-000900 1.1.31 (redhat)
> > > >>12:41:02,013 INFO  [org.jboss.weld.deployer] (MSC service thread 1-3)
> > > >>JBAS016008: Starting weld service for deployment
> > > >>hawkular-metrics-api-jaxrs.war
> > > >>12:41:02,820 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:02,835 INFO  [org.jboss.web] (ServerService Thread Pool -- 51)
> > > >>JBAS018210: Register web context: /hawkular/metrics
> > > >>12:41:02,958 INFO  [org.jboss.as.server] (ServerService Thread Pool --
> > 28)
> > > >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name :
> > > >>"hawkular-metrics-api-jaxrs.war")
> > > >>12:41:02,959 INFO  [org.jboss.as.server] (ServerService Thread Pool --
> > 28)
> > > >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
> > > >>"activemq-rar.rar")
> > > >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread)
> > JBAS015961: Http
> > > >>management interface listening on http://127.0.0.1:9990/management
> > > >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951:
> > > >>Admin console listening on http://127.0.0.1:9990
> > > >>12:41:02,994 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874:
> > > >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms -
> > Started
> > > >>267 of 303 services (62 services are lazy, passive or on-demand)
> > > >>12:41:02,999 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> > unknown
> > > >>error
> > > >>12:41:02,999 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying
> > > >>connecting to Cassandra cluster in [1]s...
> > > >>12:41:04,000 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:04,001 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:04,001 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying
> > > >>connecting to Cassandra cluster in [2]s...
> > > >>12:41:06,002 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:06,002 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:06,003 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying
> > > >>connecting to Cassandra cluster in [3]s...
> > > >>12:41:08,203 INFO  [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080
> > -2)
> > > >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
> > > >>12:41:08,299 INFO  [org.hibernate.validator.internal.util.Version]
> > > >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
> > 4.3.2.Final-redhat-2
> > > >>12:41:08,418 INFO
> > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > > >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
> > > >>12:41:08,421 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n]
> > > >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
> > > >>javax.ws.rs.core.Application: class
> > >
> > >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
> > > >>12:41:08,421 INFO
> > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > > >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting
> > > >>12:41:08,422 INFO
> > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > > >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> > > >>12:41:08,422 INFO
> > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > > >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> > > >>12:41:09,003 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:09,004 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:09,004 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying
> > > >>connecting to Cassandra cluster in [4]s...
> > > >>12:41:13,004 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:13,123 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> > unknown
> > > >>error
> > > >>12:41:13,124 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
> > > >>connecting to Cassandra cluster in [1]s...
> > > >>12:41:14,124 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:14,124 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:14,125 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
> > > >>connecting to Cassandra cluster in [2]s...
> > > >>12:41:16,125 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:16,126 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:16,126 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
> > > >>connecting to Cassandra cluster in [3]s...
> > > >>12:41:19,126 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:19,127 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:19,127 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
> > > >>connecting to Cassandra cluster in [4]s...
> > > >>12:41:23,128 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:23,249 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> > unknown
> > > >>error
> > > >>12:41:23,249 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
> > > >>connecting to Cassandra cluster in [1]s...
> > > >>12:41:24,250 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:24,250 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:24,250 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
> > > >>connecting to Cassandra cluster in [2]s...
> > > >>12:41:26,251 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:26,251 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:26,251 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying
> > > >>connecting to Cassandra cluster in [3]s...
> > > >>12:41:29,252 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:29,252 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:29,253 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying
> > > >>connecting to Cassandra cluster in [4]s...
> > > >>12:41:33,253 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:33,376 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> > unknown
> > > >>error
> > > >>12:41:33,377 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying
> > > >>connecting to Cassandra cluster in [1]s...
> > > >>12:41:34,377 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:34,378 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:34,378 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying
> > > >>connecting to Cassandra cluster in [2]s...
> > > >>12:41:36,378 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:36,379 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:36,379 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying
> > > >>connecting to Cassandra cluster in [3]s...
> > > >>12:41:39,379 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:39,380 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:39,380 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying
> > > >>connecting to Cassandra cluster in [4]s...
> > > >>12:41:43,380 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:43,503 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra:
> > unknown
> > > >>error
> > > >>12:41:43,504 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying
> > > >>connecting to Cassandra cluster in [1]s...
> > > >>12:41:44,504 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing
> > metrics
> > > >>service
> > > >>12:41:44,505 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect
> > to
> > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > > >>12:41:44,505 WARN
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying
> > > >>connecting to Cassandra cluster in [2]s...
> > > >>
> > > >>--
> > > >>Srinivas Kotaru
> > > >>
> > > >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwringe redhat com> wrote:
> > > >>
> > > >>>----- Original Message -----
> > > >>>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> > > >>>> To: "Matt Wringe" <mwringe redhat com>
> > > >>>> Cc: users lists openshift redhat com
> > > >>>> Sent: Monday, June 13, 2016 7:26:06 PM
> > > >>>> Subject: Re: Metrics deployment
> > > >>>>
> > > >>>> Matt
> > > >>>>
> > > >>>> PV issue resolved. Was able to to see PV successfully bounded and
> > > >>>> Casandra
> > > >>>> container has been running. However, it seems puzzle not fully yet
> > > >>>> solved.
> > > >>>
> > > >>>Are you sure the OpenShift DNS server is running?
> > > >>>
> > > >>>If you are running OSE 3.1, can you please follow this
> > > >>>https://access.redhat.com/solutions/2329131 and see if you are now
> > seeing
> > > >>>errors in the Hawkular Metrics logs (essentially just run `oc exec
> > > >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`)
> > > >>>
> > > >>>>
> > > >>>> I could see other container(heapster) not coming up, and seeing
> > below
> > > >>>> errors
> > > >>>>
> > > >>>> [skotaru l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc
> > > >>>> Endpoint Check in effect. Checking
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status
> > > >>>> Could not connect to
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> Curl exit code: 6. Status Code 000
> > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> > accessible
> > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> Could not connect to
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> Curl exit code: 6. Status Code 000
> > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> > accessible
> > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>>
> > > >>>>
> > > >>>> # oc get pv
> > > >>>> pv-5gb-0011   5Gi        RWO           Bound
> > > >>>> openshift-infra/metrics-cassandra-1             22m
> > > >>>>
> > > >>>>
> > > >>>> $ oc get pods
> > > >>>> NAME                         READY     STATUS      RESTARTS   AGE
> > > >>>> hawkular-cassandra-1-2pzd7   1/1       Running     0          20m
> > > >>>> hawkular-metrics-mf5qf       0/1       Running     7          20m
> > > >>>> heapster-fnkdc               0/1       Error       6          20m
> > > >>>> metrics-deployer-cvep0       0/1       Completed   0          21m
> > > >>>>
> > > >>>> # oc logs -f hawkular-metrics-mf5qf
> > > >>>>
> > > >>>> 19:20:00,819 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version
> > > >>>> 3.0.14.GA-redhat-1
> > > >>>> 19:20:00,831 INFO  [org.jboss.as.server] (Controller Boot Thread)
> > > >>>> JBAS015888:
> > > >>>> Creating http management service using socket-binding
> > (management-http)
> > > >>>> 19:20:00,834 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO
> > > >>>> Implementation Version 3.0.14.GA-redhat-1
> > > >>>> 19:20:00,844 INFO  [org.jboss.remoting] (MSC service thread 1-2)
> > JBoss
> > > >>>> Remoting version 3.3.5.Final-redhat-1
> > > >>>>
> > > >>>> $ oc logs -f heapster-fnkdc
> > > >>>> Endpoint Check in effect. Checking
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status
> > > >>>> Could not connect to
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> Curl exit code: 6. Status Code 000
> > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> > accessible
> > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> Could not connect to
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> Curl exit code: 6. Status Code 000
> > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> > accessible
> > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> Could not connect to
> > > >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> Curl exit code: 6. Status Code 000
> > > >>>>
> > > >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
> > > >>>> INFO  23:00:24 Starting listening for CQL clients on
> > > >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
> > > >>>> INFO  23:00:24 Binding thrift service to
> > > >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
> > > >>>> INFO  23:00:24 enabling encrypted thrift connections between client
> > and
> > > >>>> server
> > > >>>> INFO  23:00:24 Listening for thrift clients...
> > > >>>> INFO  23:00:26 Created default superuser role 'cassandra'
> > > >>>>
> > > >>>> # oc get svc
> > > >>>> NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)
> > > >>>> AGE
> > > >>>> hawkular-cassandra         172.30.2.13      <none>
> > > >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
> > > >>>> hawkular-cassandra-nodes   None             <none>
> > > >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP   25m
> > > >>>> hawkular-metrics           172.30.117.176   <none>        443/TCP
> > > >>>> 25m
> > > >>>> heapster                   172.30.107.135   <none>        80/TCP
> > > >>>> 25m
> > > >>>>
> > > >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
> > > >>>>
> > > >>>> HTTP/1.1 504 Gateway Timeout
> > > >>>> Mime-Version: 1.0
> > > >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
> > > >>>> Content-Type: text/html
> > > >>>> Connection: keep-alive
> > > >>>> Proxy-Connection: keep-alive
> > > >>>> Content-Length: 1572
> > > >>>>
> > > >>>> --
> > > >>>> Srinivas Kotaru
> > > >>>>
> > > >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
> > > >>>> <skotaru cisco com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> >Matt
> > > >>>> >
> > > >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and
> > > >>>> >working
> > > >>>> >
> > > >>>> >I adjusted PV to 5Gi and reran. Will update progress.
> > > >>>> >
> > > >>>> >Thanks you for your help so far.
> > > >>>> >
> > > >>>> >--
> > > >>>> >Srinivas Kotaru
> > > >>>> >
> > > >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwringe redhat com> wrote:
> > > >>>> >
> > > >>>> >>
> > > >>>> >>
> > > >>>> >>----- Original Message -----
> > > >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> > > >>>> >>> To: "Matt Wringe" <mwringe redhat com>
> > > >>>> >>> Cc: users lists openshift redhat com
> > > >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
> > > >>>> >>> Subject: Re: Metrics deployment
> > > >>>> >>>
> > > >>>> >>> Oh ok
> > > >>>> >>>
> > > >>>> >>> Am using PV for metrics
> > > >>>> >>>
> > > >>>> >>> description: "The persistent volume size for each of the
> > Cassandra
> > > >>>> >>> nodes"
> > > >>>> >>>   name: CASSANDRA_PV_SIZE
> > > >>>> >>>   value: "10Gi"
> > > >>>> >>>
> > > >>>> >>> oc get pv
> > > >>>> >>> NAME          CAPACITY   ACCESSMODES   STATUS      CLAIM
> > > >>>> >>> REASON
> > > >>>> >>> AGE
> > > >>>> >>> pv-1gb-001    1Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-1gb-002    1Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-1gb-003    1Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-1gb-004    1Gi        RWO           Bound
> >  thlatt/mongodb
> > > >>>> >>> 4d
> > > >>>> >>> pv-1gb-005    1Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-2gb-0010   2Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-2gb-006    2Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-2gb-007    2Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-2gb-008    2Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-2gb-009    2Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-5gb-0011   5Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-5gb-0012   5Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-5gb-0013   5Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-5gb-0014   5Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>> pv-5gb-0015   5Gi        RWO           Available
> > > >>>> >>> 4d
> > > >>>> >>>
> > > >>>> >>> am running with below command
> > > >>>> >>>
> > > >>>> >>> $ oc new-app -f metrics-deployer.yaml  ( hardcoded HOSTNAME,
> > > >>>> >>> MASTER_API
> > > >>>> >>> and
> > > >>>> >>> PV info so not passing any parameters)
> > > >>>> >>>
> > > >>>> >>
> > > >>>> >>I would suspect that Cassandra is blocked because its waiting for
> > 10Gi
> > > >>>> >>PV
> > > >>>> >>to become available, and none of the PV listed above are big
> > enough.
> > > >>>> >>
> > > >>>> >>>
> > > >>>> >>> --
> > > >>>> >>> Srinivas Kotaru
> > > >>>> >>>
> > > >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwringe redhat com> wrote:
> > > >>>> >>>
> > > >>>> >>> >----- Original Message -----
> > > >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> > > >>>> >>> >> To: "Matt Wringe" <mwringe redhat com>
> > > >>>> >>> >> Cc: users lists openshift redhat com
> > > >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
> > > >>>> >>> >> Subject: Re: Metrics deployment
> > > >>>> >>> >>
> > > >>>> >>> >> Matt
> > > >>>> >>> >>
> > > >>>> >>> >> Thanks for looking into. I rerun the setup, but had the same
> > > >>>> >>> >> issue
> > > >>>> >>> >>
> > > >>>> >>> >> # oc get pods
> > > >>>> >>> >> NAME                         READY     STATUS
> > > >>>> >>> >> RESTARTS
> > > >>>> >>> >> AGE
> > > >>>> >>> >> hawkular-cassandra-1-y2egy   0/1       ContainerCreating   0
> > > >>>> >>> >> 5m
> > > >>>> >>> >> hawkular-metrics-4b16f       0/1       Running             1
> > > >>>> >>> >> 4m
> > > >>>> >>> >> heapster-x2gj2               0/1       Running             2
> > > >>>> >>> >> 4m
> > > >>>> >>> >> metrics-deployer-9v7vc       0/1       Completed           0
> > > >>>> >>> >> 6m
> > > >>>> >>> >>
> > > >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
> > > >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod
> > > >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
> > > >>>> >>> >> ContainerCreating
> > > >>>> >>> >
> > > >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod
> > from
> > > >>>> >>> >starting.
> > > >>>> >>> >
> > > >>>> >>> >If you are using persistent storage, Cassandra will not start
> > until
> > > >>>> >>> >the
> > > >>>> >>> >PV
> > > >>>> >>> >is available. There may be some more information about
> > Cassandra in
> > > >>>> >>> >the
> > > >>>> >>> >pod
> > > >>>> >>> >section of the console under events.
> > > >>>> >>> >
> > > >>>> >>> >What command did you use when deploying the deployer?
> > > >>>> >>> >
> > > >>>> >>> >>
> > > >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
> > > >>>> >>> >>
> > > >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread
> > 1-4)
> > > >>>> >>> >> VM
> > > >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss
> > > >>>> >>> >> -D[Standalone]
> > > >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
> > > >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
> > > >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > >>>> >>> >> -XX:+UseGCLogFileRotation
> > > >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
> > > >>>> >>> >> -XX:-TraceClassUnloading
> > > >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
> > > >>>> >>> >> -Djava.net.preferIPv4Stack=true
> > > >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
> > > >>>> >>> >> -Djava.awt.headless=true
> > > >>>> >>> >> -Djboss.modules.policy-permissions=true
> > > >>>> >>> >>
> > -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
> > > >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
> > > >>>> >>> >>
> > -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/
> > kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
> > > >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
> > > >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
> > > >>>> >>> >>
> > -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
> > > >>>> >>> >> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3) XNIO
> > > >>>> >>> >> Version
> > > >>>> >>> >> 3.0.14.GA-redhat-1
> > > >>>> >>> >> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread 1-3)
> > XNIO
> > > >>>> >>> >> NIO
> > > >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
> > > >>>> >>> >> 16:54:27,101 INFO  [org.jboss.as.server] (Controller Boot
> > Thread)
> > > >>>> >>> >> JBAS015888:
> > > >>>> >>> >> Creating http management service using socket-binding
> > > >>>> >>> >> (management-http)
> > > >>>> >>> >> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service thread
> > 1-3)
> > > >>>> >>> >> JBoss
> > > >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
> > > >>>> >>> >>
> > > >>>> >>> >> $ oc logs -f heapster-x2gj2
> > > >>>> >>> >> Endpoint Check in effect. Checking
> > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
> > > >>>> >>> >> Could not connect to
> > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> >>> >> Curl exit code: 6. Status Code 000
> > > >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
> > not
> > > >>>> >>> >> accessible
> > > >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> >>> >> Could not connect to
> > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> >>> >> Curl exit code: 6. Status Code 000
> > > >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is
> > not
> > > >>>> >>> >> accessible
> > > >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> >>> >> Could not connect to
> > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> >>> >> Curl exit code: 6. Status Code 000
> > > >>>> >>> >>
> > > >>>> >>> >>
> > > >>>> >>> >>  $ oc logs -f metrics-deployer-9v7vc
> > > >>>> >>> >>
> > > >>>> >>> >> ++ oc create -f -
> > > >>>> >>> >> serviceaccount "heapster" created
> > > >>>> >>> >> service "heapster" created
> > > >>>> >>> >> replicationcontroller "heapster" created
> > > >>>> >>> >> + echo 'Success!'
> > > >>>> >>> >> Success!
> > > >>>> >>> >>
> > > >>>> >>> >> --
> > > >>>> >>> >> Srinivas Kotaru
> > > >>>> >>> >>
> > > >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com>
> > wrote:
> > > >>>> >>> >>
> > > >>>> >>> >> >
> > > >>>> >>> >> >
> > > >>>> >>> >> >----- Original Message -----
> > > >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com
> > >
> > > >>>> >>> >> >> To: users lists openshift redhat com
> > > >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
> > > >>>> >>> >> >> Subject: Metrics deployment
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Hi
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Am trying to configure metrics in our newly installed
> > > >>>> >>> >> >> clusters. Am
> > > >>>> >>> >> >> seeing
> > > >>>> >>> >> >> below errors once metrics-deploy script was successful. I
> > used
> > > >>>> >>> >> >> our
> > > >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and
> > MASTER_URL
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and
> > > >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
> > > >>>> >>> >> >> (
> > > >>>> >>> >> >> hard coded as values)
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> template "hawkular-heapster" created
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Deploying the Heapster component
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> ++ '[' -n '' ']'
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> ++ oc create -f -
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> ++ oc process hawkular-heapster -v
> > > >>>> >>> >> >> IMAGE_PREFIX=
> > registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> serviceaccount "heapster" created
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> service "heapster" created
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> replicationcontroller "heapster" created
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> + echo 'Success!'
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Success!
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> # oc get pods
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
> > > >>>> >>> >> >>
> > > >>>> >>> >> >
> > > >>>> >>> >> >It looks like none of your containers are fully up and
> > running
> > > >>>> >>> >> >yet.
> > > >>>> >>> >> >
> > > >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run,
> > and
> > > >>>> >>> >> >Heapster
> > > >>>> >>> >> >will
> > > >>>> >>> >> >wait until Hawkular Metrics is fully running.
> > > >>>> >>> >> >
> > > >>>> >>> >> >Do you see anything in the Cassandra logs? The first step
> > will
> > > >>>> >>> >> >be to
> > > >>>> >>> >> >get
> > > >>>> >>> >> >Cassandra running properly.
> > > >>>> >>> >> >
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> $ oc logs -f heapster-e8gbu
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Endpoint Check in effect. Checking
> > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Could not connect to
> > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> >>> >> >> Curl exit code: 6. Status Code 000
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > is not
> > > >>>> >>> >> >> accessible
> > > >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Could not connect to
> > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > > >>>> >>> >> >> Curl exit code: 6. Status Code 000
> > > >>>> >>> >> >
> > > >>>> >>> >> >Heapster waits until Hawkular Metrics is started before
> > trying
> > > >>>> >>> >> >to
> > > >>>> >>> >> >push
> > > >>>> >>> >> >metrics to it. The issue that you are seeing is because
> > Heapster
> > > >>>> >>> >> >could
> > > >>>> >>> >> >not
> > > >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular
> > Metrics
> > > >>>> >>> >> >service
> > > >>>> >>> >> >is
> > > >>>> >>> >> >fully up, Heapster will not be able to connect to it.
> > > >>>> >>> >> >
> > > >>>> >>> >> >
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics
> > > >>>> >>> >> >> rather
> > > >>>> >>> >> >> full
> > > >>>> >>> >> >> routing
> > > >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
> > > >>>> >>> >> >
> > > >>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal
> > > >>>> >>> >> >hostname
> > > >>>> >>> >> >used
> > > >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the
> > > >>>> >>> >> >external
> > > >>>> >>> >> >hostname (eg what is configured via
> > HAWKULAR_METRICS_HOSTNAME).
> > > >>>> >>> >> >The
> > > >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of
> > > >>>> >>> >> >services,
> > > >>>> >>> >> >which
> > > >>>> >>> >> >is where the internal 'hawkular-metrics' comes from.
> > > >>>> >>> >> >
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> --
> > > >>>> >>> >> >>
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> Srinivas Kotaru
> > > >>>> >>> >> >>
> > > >>>> >>> >> >> _______________________________________________
> > > >>>> >>> >> >> users mailing list
> > > >>>> >>> >> >> users lists openshift redhat com
> > > >>>> >>> >> >>
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> > > >>>> >>> >> >>
> > > >>>> >>> >>
> > > >>>> >>> >>
> > > >>>> >>>
> > > >>>> >>>
> > > >>>> >
> > > >>>>
> > > >>>>
> > > >>
> > > >
> > >
> > >
> >
> > _______________________________________________
> > users mailing list
> > users lists openshift redhat com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]