[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: split long log records



On 06/19/2017 10:48 AM, Peter Portante wrote:
Hi Andre,

This is a hard-coded Docker size.  For background see:

  * https://bugzilla.redhat.com/show_bug.cgi?id=1422008, "[RFE] Fluentd
handling of long log lines (> 16KB) split by Docker and indexed into
several ES documents"
    * And the reason for the original 16 KB limit:
https://bugzilla.redhat.com/show_bug.cgi?id=1335951, "heavy logging
leads to Docker daemon OOM-ing"

Not only that, but a PR to give the ability to docker to configure the size limit [1] was rejected due to $reasons . . .


The processor that reads the json-file documents for sending to
graylog needs to be endowed with the smarts to handle reconstruction
of those log lines, most likley, obviously with some other upper bound
(as a container is not required to emit newlines in stdout or stderr.

Regards,

-peter

[1] For the 90% of cases where this problem could have been easily solved by simply bumping the limit. Of course there will be times when no appropriate limit can be found. Of course a client that consumes docker logs must be able to handle partial messages, by reconstructing them in the logging collector, or further on down the chain.


On Mon, Jun 19, 2017 at 11:43 AM, Andre Esser
<andre esser voidbridge com> wrote:
We use Graylog for log visualisation. However that's not the culprit it
turns out. Log entries in the pod's log file are already split into chunks
of 16KB like this:

{"log":"The quick brown[...]jumps ov","stream":"stdout",\
"time":"2017-06-19T15:27:33.130524954Z"}

{"log":"er the lazy dog.\n","stream":"stdout",\
"time":"2017-06-19T15:27:33.130636562Z"}

So, to cut a long story short, is there any way to increase the size limit
before a log record gets split into two JSON records?




On 2017-06-19 16:21, Peter Portante wrote:
Who setup Graylog for openshift?

-peter

On Mon, Jun 19, 2017 at 11:18 AM, Andre Esser
<andre esser voidbridge com> wrote:
I meant the limit in Graylog. Although I just noticed that it is actually
16384 (16KB). The line split after 2048 characters only applies on the
web
UI.

Is this a Graylog limitation and can it be extended?


On 2017-06-19 14:21, Jessica Forrester wrote:

Are you asking about logs in the web console, the `oc logs` command, or
in
Kibana?

On Mon, Jun 19, 2017 at 8:29 AM, Andre Esser <andre esser voidbridge com
<mailto:andre esser voidbridge com>> wrote:

      Hi,

      In Origin 1.4.1 all log records longer than 2048 characters are
      split over two lines (longer than 4096 characters over three lines
      and so on).

      Is there any way to increase this limit?


      Thanks,

      Andre


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]