[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Kibana Logs Empty



Realized I never replied-all... Re-adding users_list

On Mon, Aug 15, 2016 at 10:58 AM, Eric Wolinetz <ewolinet redhat com> wrote:
Fluentd tries to connect to both "logging-es" | "logging-es-ops" in the logging namespace (if you're using the ops deployment) and "kubernetes" in the default namespace.  I think in this case it is having trouble connecting to the kubernetes service to look up metadata for your containers.


On Mon, Aug 15, 2016 at 10:54 AM, Frank Liauw <frank vsee com> wrote:
Oh stupid me; I was confused by my own namespaces; was looking at the wrong namespace, thinking that's the one with pods that have an active log stream. The logs are ingested fine, thanks for your assistance! :)

On the possible DNS issue of fluentd on one of my nodes, what hostname is fluentd trying to reach when starting up? We did perform some network changes to this particular node to aid public routing, but as far as the routing table is concerned, it should not have made a difference for local traffic. 

Normal functioning node without public routing changes

[root node1 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.0.5       0.0.0.0         UG    100    0        0 ens160
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tun0
10.10.0.0       0.0.0.0         255.255.0.0     U     100    0        0 ens160
172.30.0.0      0.0.0.0         255.255.0.0     U     0      0        0 tun0

Malfunctioning node with public routing changes

[root node2 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         199.27.105.1    0.0.0.0         UG    100    0        0 ens192
10.0.0.0        10.10.0.5       255.0.0.0       UG    100    0        0 ens160
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tun0
10.10.0.0       0.0.0.0         255.255.0.0     U     100    0        0 ens160
172.30.0.0      0.0.0.0         255.255.0.0     U     0      0        0 tun0
199.27.105.0    0.0.0.0         255.255.255.128 U     100    0        0 ens192

Frank 
Systems Engineer

On Mon, Aug 15, 2016 at 11:23 PM, Eric Wolinetz <ewolinet redhat com> wrote:
Correct, the way Fluentd pulls in the logs for your other containers is the same pipeline used for collecting logs for the below shown Kibana pod.

Going back to your ES logs, can you verify the date portion of a microsvc index line? 
We can then update time range in the upper-right corner of Kibana to change from the last hour to something like the last month (something that would encompass the date for the index).


On Mon, Aug 15, 2016 at 10:15 AM, Frank Liauw <frank vsee com> wrote:
Screencap is as follows: 


The query is as simple as it gets, *. I see my namespaces / projects as indexes. 

I see logs for logging project just fine: 



Fluentd is not ingesting the logs for pods in my namespaces. I'm yet to pull apart how fluentd does that, though there's no reason why logs for my other pods aren't getting indexed whereas kibana logs are if they are both ingested by fluentd, assuming that kibana logs use the same pipeline as all other pod logs. 

Frank 
Systems Engineer

On Mon, Aug 15, 2016 at 10:59 PM, Eric Wolinetz <ewolinet redhat com> wrote:
Can you either send a screencap of your Kibana console? Or describe how you are accessing Kibana and what you are seeing? (e.g. your query string, the index you're querying on, the time range for fetched responses)

On Mon, Aug 15, 2016 at 9:55 AM, Frank Liauw <frank vsee com> wrote:
I can see indexes of my namespaces, but nothing going on in actual logs in kibana though. 

Frank 
Systems Engineer

On Mon, Aug 15, 2016 at 10:37 PM, Eric Wolinetz <ewolinet redhat com> wrote:
True, we should be able to.  You should be able to see entries in the master ES node's logs that indices were created.  Based on your log snippet it should be "One Above All" in this pod: logging-es-0w45va6n-2-8m85p

If we don't see anything like "[logging.iasdf9123j-asdf.2016.02.19] creating index" then there might be something else going on...  Do you see any entries like that for your projects?

On Mon, Aug 15, 2016 at 9:05 AM, Frank Liauw <frank vsee com> wrote:
I reckoned the same, and already tried doing that, but still got the same message. That node have the same configuration as all my other nodes, so I don't see why that's the case. 

Even if it is having DNS issues, I should still be seeing logs from all other nodes that didn't report that error? 

Frank 
Systems Engineer

On Mon, Aug 15, 2016 at 9:49 PM, Eric Wolinetz <ewolinet redhat com> wrote:
It looks like that one Fluentd pod is having DNS issues making a connection to the Kubernetes service.  Can you try deleting that pod to have a new one redeployed and see if that is in a better state?

On Mon, Aug 15, 2016 at 2:54 AM, Frank Liauw <frank vsee com> wrote:
Hi All, 

I followed through the instructions on https://docs.openshift.org/latest/install_config/aggregate_logging.html and have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes. 

I am getting kibana logs on the logging project, but all my other projects do not have any logs; kibana shows "No results found", with occasional errors reading "Discover: An error occurred with your request. Reset your inputs and try again."

Probing the requests made by kibana, some calls to /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471245075265 are failing from time to time. 

Looking into the ES logs for all 3 cluster pods, I don't see much errors to be concerned, with the last error of 2 nodes similar to the following which seems to be a known issue with Openshift's setup (https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html) and possibly explains the failed requests made by kibana on auto-refresh, but that's a problem for another day: 

[2016-08-15 06:53:49,130][INFO ][cluster.service          ] [Gremlin] added {[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],}, reason: zen-disco-receive(from master [[One Above All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300]]])
[2016-08-15 06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all for action indices:admin/mappings/fields/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all

Looking into fluentd logs, one of my nodes is complaining of a "getaddrinfo" error: 

2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo: Name or service not known"
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `initialize'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `open'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in `block in connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in `timeout'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in `connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in `do_start'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in `transmit'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in `execute'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in `get'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in `block in api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in `handle_exception'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in `api'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in `api_valid?'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in `add_filter'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `block in configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `load'
  2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/bin/fluentd:23:in `<main>'
2016-08-15 03:45:18 -0400 [error]: fluentd main process died unexpectedly. restarting.

Other fluentd pods are fine: 

2016-08-15 02:55:58 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"

However, I'm not getting any logs on pods running on nodes that have fluentd deployed properly. 

Any thoughts will be much appreciated :)

Frank

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users









[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]