[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Kibana Logs Empty





On Mon, Aug 15, 2016 at 3:54 AM, Frank Liauw <frank vsee com> wrote:
Hi All, 

I followed through the instructions on https://docs.openshift.org/latest/install_config/aggregate_logging.html and have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes. 

I am getting kibana logs on the logging project, but all my other projects do not have any logs; kibana shows "No results found", with occasional errors reading "Discover: An error occurred with your request. Reset your inputs and try again."

Just to make sure... the default time period in Kibana is to look only 15 minutes in the past - are you sure your projects had logs in the last 15 minutes?
That wouldn't have anything to do with the errors you're seeing though.
 

Probing the requests made by kibana, some calls to /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471245075265 are failing from time to time. 

That certainly shouldn't be happening. Do you have any more details on how they're failing? Do they fail to connect, or just get back an error response code? Not sure if you can tell...
 

Looking into the ES logs for all 3 cluster pods, I don't see much errors to be concerned, with the last error of 2 nodes similar to the following which seems to be a known issue with Openshift's setup (https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html) and possibly explains the failed requests made by kibana on auto-refresh, but that's a problem for another day: 

[2016-08-15 06:53:49,130][INFO ][cluster.service          ] [Gremlin] added {[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],}, reason: zen-disco-receive(from master [[One Above All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300]]])

This is good, means your cluster is forming...
 
[2016-08-15 06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all for action indices:admin/mappings/fields/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all

Unfortunate SearchGuard behavior while the cluster is starting, but nothing to be concerned about as long as it doesn't continue.
 

Looking into fluentd logs, one of my nodes is complaining of a "getaddrinfo" error: 

2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo: Name or service not known"
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `initialize'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `open'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `block in connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in `timeout'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in `connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in `do_start'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in `transmit'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in `get'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in `block in api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in `handle_exception'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in `api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in `api_valid?'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in `add_filter'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `load'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `<main>'
2016-08-15 03:45:18 -0400 [error]: fluentd main process died unexpectedly. restarting.

Too bad ruby doesn't tell us what address it's trying to resolve when it gets this. It could be either the master or ES, but either way the answer should come from SkyDNS. I wonder if there is a networking issue or you have multiple masters and SkyDNS isn't responding perfectly reliably?
 

Other fluentd pods are fine: 

2016-08-15 02:55:58 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"

However, I'm not getting any logs on pods running on nodes that have fluentd deployed properly. 

Any thoughts will be much appreciated :)

Sorry I don't have a smoking gun to point at, nor great ideas where to look, just some context that might help get to the bottom of things.
 

Frank

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]