[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: logging-es errors: shards failed

Our logging deployment is fairly standard.  It's just one node right now, and we set everything up using the built-in template in the usual way.  The current logging setup is not the original one; that one died a horrible death due to filesystem corruption likely brought on by horrible overloading in the cluster's early days when we didn't know what we were doing.  We nuked it entirely and created the new one from scratch.  The Elasticsearch persistent volume is a 500GB AWS EBS volume with 1500 provisioned IOPS.  It is in the same AZ as the node that logging-es is running on.

We have 6 compute nodes, two infrastructure nodes (they really only run the routers now), a node reserved for logging, and a node reserved for ops logging.  We have 50 projects with a total of 109 services.  The last 1000 lines of ES logs are attached.  (How do I get more than that?  Kibana isn't very friendly when you want to copy and paste raw log data.)  I suspect the NoShardAvailableActionException that I keep seeing there is related, but I still don't know how to track down why those shards aren't available.

On Fri, Jul 15, 2016 at 9:08 AM, Eric Wolinetz <ewolinet redhat com> wrote:
Adding Lukas and Peter

On Fri, Jul 15, 2016 at 8:07 AM, Luke Meyer <lmeyer redhat com> wrote:
Can you give a little more information about your logging deployment? Have you deployed multiple ES nodes for redundancy, and what are you using for storage? Could you attach full ES logs? How many OpenShift nodes and projects do you have? Any history of events that might have resulted in lost data? 
Alex Wauck // DevOps Engineer

E X O S I T E 

Making Machines More Human.

Attachment: logging-es.log
Description: Binary data

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]