[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Logging] What component forward log entries to fluentd input service?



On Tue, Jul 11, 2017 at 9:00 AM, Alex Wauck <alexwauck exosite com> wrote:
> Last I checked (OpenShift Origin 1.2), fluentd was just slurping up the log
> files produced by Docker.  It can do that because the pods it runs in have
> access to the host filesystem.
>
> On Tue, Jul 11, 2017 at 6:12 AM, Stéphane Klein
> <contact stephane-klein info> wrote:
>>
>> Hi,
>>
>> I see here
>> https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2
>> that fluentd logging system use secure_forward input system.
>>
>> My question: what component forward log entries to fluentd input service ?

The "mux" service is a concentrator of sorts.

Without the mux service, each fluentd pod runs on a host in an
OpenShift cluster collecting logs and sending them to Elasticsearch
directly.  The collectors also have the responsibility of enhancing
the logs collected with the metadata that describes which
pod/container they came from.  This requires connections to the API
server to get that information.

So in a large cluster, 200+ nodes, maybe less, maybe more, the API
servers are overwhelmed by requests from all the fluentd pods.

With the mux service, all the fluentd collections pods only talk to
the mux service and DO NOT talk to the API server; they simply send
the logs they collect to the mux fluentd instance.

The mux fluentd instance in turns talks to the API service to enrich
the logs with the pod/container metadata and then send along to
Elasticsearch.

This scales much better.

-peter


>>
>> Best regards,
>> Stéphane
>> --
>> Stéphane Klein <contact stephane-klein info>
>> blog: http://stephane-klein.info
>> cv : http://cv.stephane-klein.info
>> Twitter: http://twitter.com/klein_stephane
>>
>> _______________________________________________
>> users mailing list
>> users lists openshift redhat com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>
>
>
> --
>
> Alex Wauck // Senior DevOps Engineer
>
> E X O S I T E
> www.exosite.com
>
> Making Machines More Human.
>
>
> _______________________________________________
> users mailing list
> users lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]