[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Logging seems to be working, but no logs are collected

Very strange.  It would appear that fluentd was not able to keep up with the log rate to the journal for such an extent that the fluentd current cursor position was rotated away . . .

You can "reset" fluentd by shutting it down, then removing that cursor file.  That will tell fluentd to start reading from the tail of the journal.  but NOTE - THAT WILL LOSE ALL RECORDS CURRENTLY IN THE JOURNAL.  If you want to try to recover everything in the journal, then oc set env ds/logging-fluentd JOURNAL_READ_FROM_HEAD=true - but note that this may take several hours until you have recent records in Elasticsearch, depending on what is the log rate to the journal and how fast fluentd can keep up.

If you go the JOURNAL_READ_FROM_HEAD=true route, setting the env should trigger a redeployment of fluentd, so you should not have to restart/relabel.

oc label node --all --overwrite logging-infra-fluentd-
... wait for oc pods to report no logging-fluentd pods ...
rm -f /var/log/journal.pos
oc label node --all --overwrite logging-infra-fluentd=true

Then, monitor fluentd like this:


and monitor the journald log rate (number of logs/minute) like this:


On 10/31/2017 11:57 AM, Tim Dudgeon wrote:
$ sudo docker info | grep -i log
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Logging Driver: journald
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

$ journalctl -r -n 1 --show-cursor
-- Logs begin at Sun 2017-10-29 03:04:42 UTC, end at Tue 2017-10-31 17:54:37 UTC. -- Oct 31 17:54:37 worker-1.openstacklocal dockerd-current[6135]: {"type":"response","@timestamp":"2017-10-31T17:54:37Z","tags":[],"pid":8," -- cursor: s=f746c7090d724f5ab0ece0d13683fc53;i=a54f2;b=93b6daa912044dd9ae9f05521c603efc;m=55116ad995;t=55cdb72d7c92d;x=5a16032caedc4423

On 31/10/2017 17:31, Rich Megginson wrote:

# docker info | grep -i log

# journalctl -r -n 1 --show-cursor

On 10/31/2017 11:12 AM, Tim Dudgeon wrote:

Thanks. Those links are useful.

It looks to me like its a problem at the fluentd level. This is what I see on on of the fluentd pods:

sh-4.2# cat /var/log/es-containers.log.pos
cat: /var/log/es-containers.log.pos: No such file or directory
sh-4.2# cat /var/log/journal.pos
sh-4.2# journalctl -c `cat /var/log/journal.pos`
No journal files were found.
-- No entries --

Which might sort of explain why everything is running but no logs are being processed.

This is based on a centos7 image with only the necessary openshift packages installed and then openshift installed using ansible. The logging setup in the inventory file is this:

openshift_hosted_logging_storage_labels={'storage': 'logging'}


On 31/10/2017 16:37, Jeff Cantrill wrote:
Please provide additional information, logs, etc or post the output of [1] someplace for review. Additionally, consider reviewing [2].

[1] https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh [2] https://github.com/openshift/origin-aggregated-logging/blob/master/docs/checking-efk-health.md

On Tue, Oct 31, 2017 at 11:47 AM, Tim Dudgeon <tdudgeon ml gmail com <mailto:tdudgeon ml gmail com>> wrote:

    Hi All,

    I've deployed logging using the ansible installer (v3.6.0) for a
    fairly simple openshift setup and everything appears to running:

    logging-curator-1-gvh73              1/1 Running 24         3d
    logging-es-data-master-xz0e7a0c-1-deploy   0/1 Error 0          3d
    logging-es-data-master-xz0e7a0c-4-deploy   0/1 Error 0          3d
    logging-es-data-master-xz0e7a0c-5-deploy   0/1 Error 0          3d
    logging-es-data-master-xz0e7a0c-7-t4xpf    1/1 Running 0          3d
    logging-fluentd-4rm2w              1/1 Running 0 3d
    logging-fluentd-8h944              1/1 Running 0 3d
    logging-fluentd-n00bn              1/1 Running 0 3d
    logging-fluentd-vt8hh              1/1 Running 0 3d
    logging-kibana-1-g7l4z              2/2 Running 0 3d

    (the failed pods were related to getting elasticsearch running,
    but that was resolved).

    The problem is that I don't see any logs in Kibana. When I look
    in the fluentd pod logs I see lots of stuff like this:

    2017-10-31 13:53:15 +0000 [warn]: no patterns matched
    2017-10-31 13:58:02 +0000 [warn]: no patterns matched
    2017-10-31 14:02:18 +0000 [warn]: no patterns matched
    2017-10-31 14:07:15 +0000 [warn]: no patterns matched
    2017-10-31 14:11:20 +0000 [warn]: no patterns matched
    2017-10-31 14:15:16 +0000 [warn]: no patterns matched
    2017-10-31 14:19:58 +0000 [warn]: no patterns matched

    Is this the cause, and if so what is wrong?
    If not how to debug this?


    users mailing list
    users lists openshift redhat com
    <mailto:users lists openshift redhat com>

Jeff Cantrill
Senior Software Engineer, Red Hat Engineering
OpenShift Integration Services
Red Hat, Inc.
*Office*: 703-748-4420 | 866-546-8970 ext. 8162420
jcantril redhat com <mailto:jcantril redhat com>

users mailing list
users lists openshift redhat com

users mailing list
users lists openshift redhat com

users mailing list
users lists openshift redhat com

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]