[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Design questions around Container logs, EFK & OCP



Thanks Aleks for the feedback.

This looks promising. 

We're using Enterprise OCP. Does that make a difference at that level of discussion ? 

For the External Elasticsearch instance configs you referred to , is it possible to co-exist both ? Some Worker nodes sending logs to the internal ES, and some other Worker nodes sending logs to the external one ? 


Opensource origin:
https://docs.openshift.org/latest/install_config/aggregate_logging.html#sending-logs-to-an-external-elasticsearch-instance
Enterprise:
https://docs.openshift.com/container-platform/3.7/install_config/aggregate_logging.html#sending-logs-to-an-external-elasticsearch-instance


Many Thanks, 
/Mo


On 7 March 2018 at 23:27, Aleksandar Lazic <openshift-dev me2digital com> wrote:
Hi.

Am 07.03.2018 um 23:47 schrieb Mohamed A. Shahat:
> Hi All, 
>
> My first question here, so i am hoping at least for some
> acknowledgement ! 
>
> _Background_
>
>   * OCP v3.7
>
Do you use the enterprise version or the opensource one?
>
>   * Several Worker Nodes
>   * Few Workload types 
>   * One Workload, let's call it WorkloadA is planned to have dedicated
>     Worker Nodes.
>
> _Objective_
>
>   * for WorkloadA , I'd like to send/route the Container Logs to an
>     External EFK / ELK stack other than the one that does get setup
>     with OCP
>
> _Motivation_ 
>
>   * For Workload A, an ES cluster does already exist, we would like to
>     reuse it. 
>   * There is an impression that the ES cluster that comes with OCP
>     might not necessarily scale if the team operating OCP does not
>     size it well
>
> _Inquiries_
>
>  1. Has this done before ? Yes / No ? Any comments ?
>
Yes.
As you may know is "handle logs in a proper way" not a easy task.
There are some serious questions like the following.

* How long should the logs be preserved
* How much logs are written
* How fast are the logs written
* What's the limit of the network
* What's the limit of the remote es
* and many many more questions

>  1. Is there anyway with the fluentd pods or else to route specific
>     Workload / Pods Container logs to an external ES cluster ? 
>  2. If not, i'm willing to deploy my own fluentd pods , what do i lose
>     by excluding the WorkloadA Worker Nodes to not have the OCP
>     fluentd pods ? for example i don't want to lose any Operations /
>     OCP related / Worker Nodes related logs going to the embedded ES
>     cluster, all i need is to have the Container Logs of WorkloadA to
>     another ES cluster.
>
Have you looked at the following doc part?

Opensource origin:
https://docs.openshift.org/latest/install_config/aggregate_logging.html#sending-logs-to-an-external-elasticsearch-instance

Enterprise:
https://docs.openshift.com/container-platform/3.7/install_config/aggregate_logging.html#sending-logs-to-an-external-elasticsearch-instance

As in the doc described you can send the collected fluentd logs to a
external es cluster.

You can find the source of the openshift logging solution in this repo.
https://github.com/openshift/origin-aggregated-logging

> Looking forward to hearing from you, 
>
> Thanks, 
Hth
Aleks


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]