[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics deployment



----- Original Message -----
> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> To: "Matt Wringe" <mwringe redhat com>
> Cc: users lists openshift redhat com
> Sent: Monday, June 13, 2016 4:55:55 PM
> Subject: Re: Metrics deployment
> 
> Matt
> 
> Thanks for looking into. I rerun the setup, but had the same issue
> 
> # oc get pods
> NAME                         READY     STATUS              RESTARTS   AGE
> hawkular-cassandra-1-y2egy   0/1       ContainerCreating   0          5m
> hawkular-metrics-4b16f       0/1       Running             1          4m
> heapster-x2gj2               0/1       Running             2          4m
> metrics-deployer-9v7vc       0/1       Completed           0          6m
> 
> $ oc logs -f hawkular-cassandra-1-y2egy
> Error from server: container "hawkular-cassandra-1" in pod
> "hawkular-cassandra-1-y2egy" is waiting to start: ContainerCreating

Ok, so it looks like something is blocking the Cassandra pod from starting.

If you are using persistent storage, Cassandra will not start until the PV is available. There may be some more information about Cassandra in the pod section of the console under events.

What command did you use when deploying the deployer?

> 
> $ oc logs -f hawkular-metrics-4b16f
> 
> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread 1-4) VM
> Arguments: -Duser.home=/home/jboss -Duser.name=jboss -D[Standalone]
> -XX:+UseCompressedOops -verbose:gc -Xloggc:/opt/eap/standalone/log/gc.log
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading
> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true
> -Djboss.modules.system.pkgs=org.jboss.logmanager -Djava.awt.headless=true
> -Djboss.modules.policy-permissions=true
> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
> -Djava.security.egd=file:/dev/./urandom
> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
> 16:54:27,079 INFO  [org.xnio] (MSC service thread 1-3) XNIO Version
> 3.0.14.GA-redhat-1
> 16:54:27,083 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO
> Implementation Version 3.0.14.GA-redhat-1
> 16:54:27,101 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888:
> Creating http management service using socket-binding (management-http)
> 16:54:27,104 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss
> Remoting version 3.3.5.Final-redhat-1
> 
> $ oc logs -f heapster-x2gj2
> Endpoint Check in effect. Checking
> https://hawkular-metrics:443/hawkular/metrics/status
> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> Curl exit code: 6. Status Code 000
> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
> [HTTP status code: 000. Curl exit code 6]. Retrying.
> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> Curl exit code: 6. Status Code 000
> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
> [HTTP status code: 000. Curl exit code 6]. Retrying.
> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> Curl exit code: 6. Status Code 000
> 
> 
>  $ oc logs -f metrics-deployer-9v7vc
> 
> ++ oc create -f -
> serviceaccount "heapster" created
> service "heapster" created
> replicationcontroller "heapster" created
> + echo 'Success!'
> Success!
> 
> --
> Srinivas Kotaru
> 
> On 6/13/16, 1:49 PM, "Matt Wringe" <mwringe redhat com> wrote:
> 
> >
> >
> >----- Original Message -----
> >> From: "Srinivas Naga Kotaru (skotaru)" <skotaru cisco com>
> >> To: users lists openshift redhat com
> >> Sent: Monday, June 13, 2016 3:58:12 PM
> >> Subject: Metrics deployment
> >> 
> >> 
> >> 
> >> Hi
> >> 
> >> 
> >> 
> >> Am trying to configure metrics in our newly installed clusters. Am seeing
> >> below errors once metrics-deploy script was successful. I used our
> >> environment specific HAWKULAR_METRICS_HOSTNAME and MASTER_URL
> >> 
> >> 
> >> 
> >> # oc new-app -f metrics-deployer.yaml
> >> 
> >> 
> >> 
> >> Note: customized, CASSANDARA PV, MASTER_URL, and HAWKULAR_METRICS_HOSTNAME
> >> (
> >> hard coded as values)
> >> 
> >> 
> >> 
> >> template "hawkular-heapster" created
> >> 
> >> Deploying the Heapster component
> >> 
> >> ++ echo 'Deploying the Heapster component'
> >> 
> >> ++ '[' -n '' ']'
> >> 
> >> ++ oc create -f -
> >> 
> >> ++ oc process hawkular-heapster -v
> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
> >> 
> >> serviceaccount "heapster" created
> >> 
> >> service "heapster" created
> >> 
> >> replicationcontroller "heapster" created
> >> 
> >> + echo 'Success!'
> >> 
> >> Success!
> >> 
> >> 
> >> 
> >> # oc get pods
> >> 
> >> NAME READY STATUS RESTARTS AGE
> >> 
> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m
> >> 
> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
> >> 
> >> heapster-e8gbu 0/1 Running 2 4m
> >> 
> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s
> >> 
> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
> >> 
> >
> >It looks like none of your containers are fully up and running yet.
> >
> >Without Cassandra running, Hawkular Metrics will not run, and Heapster will
> >wait until Hawkular Metrics is fully running.
> >
> >Do you see anything in the Cassandra logs? The first step will be to get
> >Cassandra running properly.
> >
> >> 
> >> 
> >> 
> >> $ oc logs -f heapster-e8gbu
> >> 
> >> Endpoint Check in effect. Checking
> >> https://hawkular-metrics:443/hawkular/metrics/status
> >> 
> >> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> >> Curl exit code: 6. Status Code 000
> >> 
> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible
> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> >> 
> >> Could not connect to https://hawkular-metrics:443/hawkular/metrics/status.
> >> Curl exit code: 6. Status Code 000
> >
> >Heapster waits until Hawkular Metrics is started before trying to push
> >metrics to it. The issue that you are seeing is because Heapster could not
> >properly connect to Hawkular Metrics. Until the Hawkular Metrics service is
> >fully up, Heapster will not be able to connect to it.
> >
> >
> >> 
> >> 
> >> 
> >> 
> >> What is the wrong? Why it checking just hawkular-metrics rather full
> >> routing
> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME
> >
> >The Hawkular Metrics service has two hostnames: the internal hostname used
> >by the internal components (eg 'hawkular-metrics') and the external
> >hostname (eg what is configured via HAWKULAR_METRICS_HOSTNAME). The
> >OpenShift dns server will resolve hostnames to the name of services, which
> >is where the internal 'hawkular-metrics' comes from.
> >
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> --
> >> 
> >> 
> >> Srinivas Kotaru
> >> 
> >> _______________________________________________
> >> users mailing list
> >> users lists openshift redhat com
> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >> 
> 
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]