[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Kibana Logs Empty



Hi All, 

I followed through the instructions on https://docs.openshift.org/latest/install_config/aggregate_logging.html and have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes. 

I am getting kibana logs on the logging project, but all my other projects do not have any logs; kibana shows "No results found", with occasional errors reading "Discover: An error occurred with your request. Reset your inputs and try again."

Probing the requests made by kibana, some calls to /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471245075265 are failing from time to time. 

Looking into the ES logs for all 3 cluster pods, I don't see much errors to be concerned, with the last error of 2 nodes similar to the following which seems to be a known issue with Openshift's setup (https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html) and possibly explains the failed requests made by kibana on auto-refresh, but that's a problem for another day: 

[2016-08-15 06:53:49,130][INFO ][cluster.service          ] [Gremlin] added {[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],}, reason: zen-disco-receive(from master [[One Above All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300]]])
[2016-08-15 06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all for action indices:admin/mappings/fields/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all

Looking into fluentd logs, one of my nodes is complaining of a "getaddrinfo" error: 

2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo: Name or service not known"
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `initialize'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `open'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `block in connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in `timeout'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in `connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in `do_start'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in `transmit'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in `get'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in `block in api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in `handle_exception'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in `api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in `api_valid?'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in `add_filter'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `load'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `<main>'
2016-08-15 03:45:18 -0400 [error]: fluentd main process died unexpectedly. restarting.

Other fluentd pods are fine: 

2016-08-15 02:55:58 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"

However, I'm not getting any logs on pods running on nodes that have fluentd deployed properly. 

Any thoughts will be much appreciated :)

Frank

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]