Just to share an update
Am able to successfully install and configure metrics and logging using real certs, node classification and separate apps and OPS ES clusters.
Am happy that this setup finally working and live. Want to share feedback from operation side. There is some scope to improve and make it easy to setup logging, metrics, routers and registry components. They seem to be little difficult and with so many manual steps. Scope to improve documentation too.
Thanks for your help, as usual your co operation and willing to help is always on top. I also used RedHat global support extensively to bring all these services up and running in prod grade environment. It was great help from them too.
OK thanks. I deleted whole stack and let me run the deployed again by enabling true.
On Wed, Jun 15, 2016 at 4:01 PM, Srinivas Naga Kotaru (skotaru) <skotaru cisco com> wrote:
If you do not need to have physical separation of your operations logs and your application logs you can leave it with ENABLE_OPS_CLUSTER as false. Setting that to true don't add any extra logs, it just creates a second Elasticsearch cluster (the ops cluster) an Ops Kibana instance to serve up the logs within the Elasticsearch ops cluster and tells Fluentd that the operations logs that it is processing go to this new cluster instead.
To be honest, I would recommend reinstalling with ENABLE_OPS_CLUSTER=true and tricking Fluentd to reprocess all your logs as if it were a new installation. You are missing the ops templates for the different components which will come in handy especially when you want to later scale up the number of ES nodes for a cluster.
Also you have the added benefit that some of your operations logs aren't in the same ES cluster as your application logs (the main benefit for using this deployment option)
You can trick Fluentd into reprocessing logs on its node by
1. Stop Fluentd on that node
2. Delete the "/var/log/es-containers.log.pos" and "/var/log/node.log.pos" files on that node
3. Start Fluentd on that node again, it will act as if it had not processed any log files yet