[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: split long log records



Hi Andre,

This is a hard-coded Docker size.  For background see:

 * https://bugzilla.redhat.com/show_bug.cgi?id=1422008, "[RFE] Fluentd
handling of long log lines (> 16KB) split by Docker and indexed into
several ES documents"
   * And the reason for the original 16 KB limit:
https://bugzilla.redhat.com/show_bug.cgi?id=1335951, "heavy logging
leads to Docker daemon OOM-ing"

The processor that reads the json-file documents for sending to
graylog needs to be endowed with the smarts to handle reconstruction
of those log lines, most likley, obviously with some other upper bound
(as a container is not required to emit newlines in stdout or stderr.

Regards,

-peter

On Mon, Jun 19, 2017 at 11:43 AM, Andre Esser
<andre esser voidbridge com> wrote:
> We use Graylog for log visualisation. However that's not the culprit it
> turns out. Log entries in the pod's log file are already split into chunks
> of 16KB like this:
>
> {"log":"The quick brown[...]jumps ov","stream":"stdout",\
> "time":"2017-06-19T15:27:33.130524954Z"}
>
> {"log":"er the lazy dog.\n","stream":"stdout",\
> "time":"2017-06-19T15:27:33.130636562Z"}
>
> So, to cut a long story short, is there any way to increase the size limit
> before a log record gets split into two JSON records?
>
>
>
>
> On 2017-06-19 16:21, Peter Portante wrote:
>>
>> Who setup Graylog for openshift?
>>
>> -peter
>>
>> On Mon, Jun 19, 2017 at 11:18 AM, Andre Esser
>> <andre esser voidbridge com> wrote:
>>>
>>> I meant the limit in Graylog. Although I just noticed that it is actually
>>> 16384 (16KB). The line split after 2048 characters only applies on the
>>> web
>>> UI.
>>>
>>> Is this a Graylog limitation and can it be extended?
>>>
>>>
>>> On 2017-06-19 14:21, Jessica Forrester wrote:
>>>>
>>>>
>>>> Are you asking about logs in the web console, the `oc logs` command, or
>>>> in
>>>> Kibana?
>>>>
>>>> On Mon, Jun 19, 2017 at 8:29 AM, Andre Esser <andre esser voidbridge com
>>>> <mailto:andre esser voidbridge com>> wrote:
>>>>
>>>>      Hi,
>>>>
>>>>      In Origin 1.4.1 all log records longer than 2048 characters are
>>>>      split over two lines (longer than 4096 characters over three lines
>>>>      and so on).
>>>>
>>>>      Is there any way to increase this limit?
>>>>
>>>>
>>>>      Thanks,
>>>>
>>>>      Andre
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users lists openshift redhat com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]