[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Metrics not accessible



Hi Tim.

on Mittwoch, 06. September 2017 at 13:43 was written:

> On 06/09/2017 12:33, Aleksandar Lazic wrote:
>> Hi Tim.
>>
>> A dump question but do you have any proxy setuped?
> No, just a vanilla ansible install running on a machine on EC2. The 
> inventory file was posted earlier.
>>
>> on Mittwoch, 06. September 2017 at 12:49 was written:
>>
>>> No joy.
>>> The cassandra pod starts fine but the hawkular on fails to start with
>>> what looks like the same errors as I described before.
>> One of the interesting par is that hawkular can connect to cas
>>
>> ###
>> 2017-09-05 14:54:48,123 INFO  [com.datastax.driver.core.Cluster] (ServerService Thread Pool -- 64) New Cassandra host hawkular-cassandra/172.30.151.137:9042 added
>> ...
>> 2017-09-05 14:54:48,276 INFO  [org.cassalog.core.CassalogImpl] (metricsservice-lifecycle-thread) Applying ChangeSet
>> -- version: set-keyspace
>> USE hawkular_metrics
>> ...
>> ####
>>
>> and then got you a NullPointerException
>>
>> ###
>> 2017-09-05 14:54:49,163 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200006: An error occurred trying to connect to the Cassandra cluster: java.lang.NullPointerException
>>          at org.hawkular.metrics.core.dropwizard.HawkularObjectNameFactory.createName(HawkularObjectNameFactory.java:54)
>>          at com.codahale.metrics.JmxReporter$JmxListener.createName(JmxReporter.java:656)
>>          at com.codahale.metrics.JmxReporter$JmxListener.onTimerAdded(JmxReporter.java:633)
>>          at com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:356)
>>          at com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:191)
>>          at com.codahale.metrics.JmxReporter.start(JmxReporter.java:715)
>>          at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:474)
>>          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)
>> ###
>>
>> Was the Cassandra up when the hawkular started?
> Yes.
>>
>> are you able to curl Cassandra from hawkular pod?
>>
>> oc debug rc/hawkular-metrics
>>
>> curl -v telnet://hawkular-cassandra:9042/
> Yes:

> $ curl -v telnet://hawkular-cassandra:9042/
> * About to connect() to hawkular-cassandra port 9042 (#0)
> *   Trying 172.30.78.190...
> * Connected to hawkular-cassandra (172.30.78.190) port 9042 (#0)

> ^C

that's strange.

Please can you try a previous image version of hawkular?

> Tim
>>
>>> Tim
>>
>>> On 06/09/2017 10:34, Aleksandar Lazic wrote:
>>>> Hi Tim.
>>>>
>>>> on Dienstag, 05. September 2017 at 17:10 was written:
>>>>
>>>>> Still no joy with this.
>>>>> I retried with the latest code and still hitting the same problem.
>>>>> Metrics does not seem to be working with a new Ansible install.
>>>>> I'm using a minimal setup with an inventory like this:
>>>>>> [OSEv3:children]
>>>> [snipp]
>>>>
>>>>> When the install completes the openshift-infra project pods ends up like
>>>>> this:
>>>>>> NAME                         READY STATUS             RESTARTS   AGE
>>>>>> hawkular-cassandra-1-4m7lq   1/1       Running 0          16m
>>>>>> hawkular-metrics-0nl1q       0/1       CrashLoopBackOff 7          16m
>>>>>> heapster-cgw0b               0/1       Running 1          16m
>>>>> The hawkular-metrics pods is failing, and it looks like its because it
>>>>> can't connect to the cassandra pod.
>>>>> The full log of the hawkular-metrics pod is here:
>>>>> https://gist.github.com/tdudgeon/f3099911eed441817369ee03635aad7d
>>>>> Any help resolving this would be appreciated.
>>>> I normally do the following.
>>>>
>>>> oc scale rc/hawkular-metrics --replicas=0
>>>> oc scale rc/heapster --replicas=0
>>>> oc scale rc/hawkular-cassandra --replicas=0
>>>>
>>>> after all pods are down start the new instaces.
>>>>
>>>> Sometimes I must send a
>>>>
>>>> oc delete pod <HANGING_POD> --grace-period=0
>>>>
>>>> when a pod is hanging.
>>>>
>>>> Now start cas
>>>>
>>>> oc scale rc/hawkular-cassandra --replicas=1
>>>>
>>>> Wait until cas is up without errors!
>>>>
>>>> oc logs -f <NEW_POD>
>>>>
>>>> Now run hawk
>>>>
>>>> oc scale rc/hawkular-metrics --replicas=1
>>>>
>>>> Wait until hawk is up without errors!
>>>>
>>>> oc logs -f <NEW_POD>
>>>>
>>>> Now run heapster
>>>>
>>>> oc scale rc/heapster --replicas=1
>>>>
>>>> oc logs -f <NEW_POD>
>>>>
>>>> Does this help?
>>>>
>>>>> Tim


-- 
Best Regards
Aleks

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]