[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: aggregate logging: kibana error



Hi Eric,
Thanks for your help. Attached the requested logs.


ES

[2016-04-20 06:57:08,912][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:09,683][WARN ][io.fabric8.kubernetes.api.KubernetesHelper] Lookup logging-es-cluster result: host not found
[2016-04-20 06:57:09,912][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Error checking ACL when seeding
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:09,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:10,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Error checking ACL when seeding
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:10,913][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:74)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:145)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:188)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.loadAcl(DynamicACLFilter.java:245)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.seedInitialACL(DynamicACLFilter.java:297)
at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.onSearchGuardACLActionRequest(DynamicACLFilter.java:118)
at io.fabric8.elasticsearch.plugin.acl.DefaultACLNotifierService$1.run(DefaultACLNotifierService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-20 06:57:11,186][WARN ][io.fabric8.kubernetes.api.KubernetesHelper] Lookup logging-es-cluster result: host not found
[2016-04-20 06:57:11,197][INFO ][cluster.service          ] [Baphomet] new_master [Baphomet][TSHXtJg3T864x7lBLTVqvA][logging-es-0g0sypsu-1-tw59i][inet[/x.x.x.x:9300]], reason: zen-disco-join (elected_as_master)
[2016-04-20 06:57:11,214][INFO ][http                     ] [Baphomet] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/x.x.x.x:9200]}
[2016-04-20 06:57:11,214][INFO ][node                     ] [Baphomet] started
[2016-04-20 06:57:11,227][INFO ][gateway                  ] [Baphomet] recovered [0] indices into cluster_state
[2016-04-20 06:57:12,254][INFO ][cluster.metadata         ] [Baphomet] [.searchguard.logging-es-0g0sypsu-1-tw59i] creating index, cause [auto(index api)], templates [], shards [5]/[1], mappings [ac]
[2016-04-20 06:57:12,537][INFO ][cluster.metadata         ] [Baphomet] [.searchguard.logging-es-0g0sypsu-1-tw59i] update_mapping [ac] (dynamic)
[2016-04-20 06:58:21,001][INFO ][cluster.metadata         ] [Baphomet] [logging.2016.04.20] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 06:58:21,120][INFO ][cluster.metadata         ] [Baphomet] [logging.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 06:59:39,673][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.07] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 06:59:39,748][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.20] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 06:59:40,003][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 06:59:42,509][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.07] update_mapping [fluentd] (dynamic)
[2016-04-20 07:00:32,865][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-20 09:19:32,061][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.08] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 09:19:32,916][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.08] update_mapping [fluentd] (dynamic)
[2016-04-20 10:07:26,159][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.09] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 10:07:26,432][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.09] update_mapping [fluentd] (dynamic)
[2016-04-20 10:22:17,801][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.10] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 10:22:18,235][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.10] update_mapping [fluentd] (dynamic)
[2016-04-20 10:40:59,387][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.11] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 10:40:59,695][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.11] update_mapping [fluentd] (dynamic)
[2016-04-20 10:53:44,022][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.12] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 10:53:44,218][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.12] update_mapping [fluentd] (dynamic)
[2016-04-20 11:13:14,707][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.13] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 11:13:14,934][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.13] update_mapping [fluentd] (dynamic)
[2016-04-20 11:29:00,212][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.14] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 11:29:00,665][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.14] update_mapping [fluentd] (dynamic)
[2016-04-20 11:44:03,826][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.15] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 11:44:04,612][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.15] update_mapping [fluentd] (dynamic)
[2016-04-20 11:55:34,170][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.16] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 11:55:34,338][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.16] update_mapping [fluentd] (dynamic)
[2016-04-20 12:13:41,236][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.17] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 12:13:41,920][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.17] update_mapping [fluentd] (dynamic)
[2016-04-20 12:27:59,827][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.18] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 12:28:00,589][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.18] update_mapping [fluentd] (dynamic)
[2016-04-20 12:36:39,258][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.19] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-20 12:36:39,640][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.19] update_mapping [fluentd] (dynamic)
[2016-04-20 12:45:37,104][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.20] update_mapping [fluentd] (dynamic)
[2016-04-21 00:00:18,301][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.21] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-21 00:00:18,393][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:01:43,323][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:03:18,723][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:06:34,760][INFO ][cluster.metadata         ] [Baphomet] [.operations.2016.04.21] update_mapping [fluentd] (dynamic)
[2016-04-21 00:07:21,827][INFO ][cluster.metadata         ] [Baphomet] [logging.2016.04.21] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [fluentd]
[2016-04-21 00:07:21,911][INFO ][cluster.metadata         ] [Baphomet] [logging.2016.04.21] update_mapping [fluentd] (dynamic)

Fluentd

Name: logging-fluentd-6c6bt
Namespace: logging
Node: xxxxx/x.x.x.x
Start Time: Wed, 20 Apr 2016 06:37:13 +0000
Labels: component=fluentd,provider=openshift
Status: Running
IP:  x.x.x.x
Controllers: DaemonSet/logging-fluentd
Containers:
  fluentd-elasticsearch:
    Container ID: docker://2cbe9d5813c3d771c4fc10c92be61136bb43e11f85fc9c117c64ca5e42060627
    Image: docker.io/openshift/origin-logging-fluentd:latest
    Image ID: docker://f841fe531e980e970d69b353bf49ec8f69dd13f88dccfdaff578d91f0fa58e63
    Port:
    QoS Tier:
      cpu: Guaranteed
      memory: BestEffort
    Limits:
      cpu: 100m
    Requests:
      cpu: 100m
    State: Running
      Started: Wed, 20 Apr 2016 07:43:35 +0000
    Ready: True
    Restart Count: 0
    Environment Variables:
      K8S_HOST_URL: https://kubernetes.default.svc.cluster.local
      ES_HOST: logging-es
      ES_PORT: 9200
      ES_CLIENT_CERT: /etc/fluent/keys/cert
      ES_CLIENT_KEY: /etc/fluent/keys/key
      ES_CA: /etc/fluent/keys/ca
      OPS_HOST: logging-es
      OPS_PORT: 9200
      OPS_CLIENT_CERT: /etc/fluent/keys/cert
      OPS_CLIENT_KEY: /etc/fluent/keys/key
      OPS_CA: /etc/fluent/keys/ca
      ES_COPY: false
      ES_COPY_HOST:
      ES_COPY_PORT:
      ES_COPY_SCHEME: https
      ES_COPY_CLIENT_CERT:
      ES_COPY_CLIENT_KEY:
      ES_COPY_CA:
      ES_COPY_USERNAME:
      ES_COPY_PASSWORD:
      OPS_COPY_HOST:
      OPS_COPY_PORT:
      OPS_COPY_SCHEME: https
      OPS_COPY_CLIENT_CERT:
      OPS_COPY_CLIENT_KEY:
      OPS_COPY_CA:
      OPS_COPY_USERNAME:
      OPS_COPY_PASSWORD:
Conditions:
  Type Status
  Ready  True 
Volumes:
  varlog:
    Type: HostPath (bare host directory volume)
    Path: /var/log
  varlibdockercontainers:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/docker/containers
  certs:
    Type: Secret (a volume populated by a Secret)
    SecretName: logging-fluentd
  dockerhostname:
    Type: HostPath (bare host directory volume)
    Path: /etc/hostname
  aggregated-logging-fluentd-token-f7ysx:
    Type: Secret (a volume populated by a Secret)
    SecretName: aggregated-logging-fluentd-token-f7ysx
No events.



On 20 Apr 2016, at 9:54 PM, Eric Wolinetz <ewolinet redhat com> wrote:

Hi Sebastian,

Your Elasticsearch instance does not seem to have started up completely within the pod you showed logs for.  Kibana will fail to start up if it is unable to reach its Elasticsearch instance after a certain period of time.

Can you send some more of your Elasticsearch logs?  It looks like its currently recovering/initializing.  Do you see any different ERROR messages within there?

Your Fluentd errors looks to be something else. What does the following look like?
    $ oc describe pod -l component=fluentd

On Wed, Apr 20, 2016 at 2:54 AM, Sebastian Wieseler <sebastian myrepublic com sg> wrote:
Dear community,

NAME                          READY     STATUS    RESTARTS   AGE
logging-kibana-1-uwob1        1/2       Error     12         43m


$ oc logs logging-kibana-1-uwob1  -c kibana
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":50,"err":{"message":"Request Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after 5000ms\n    at null.<anonymous> (/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)"},"msg":"","time":"2016-04-20T07:16:15.760Z","v":0}
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":60,"err":{"message":"Request Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after 5000ms\n    at null.<anonymous> (/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)"},"msg":"","time":"2016-04-20T07:16:15.762Z","v":0}
[root MRNZ-TS8-OC-MASTER-01 glusterfs]# oc logs logging-kibana-1-uwob1  -c kibana
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":50,"err":{"message":"Request Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after 5000ms\n    at null.<anonymous> (/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)"},"msg":"","time":"2016-04-20T07:38:40.789Z","v":0}
{"name":"Kibana","hostname":"logging-kibana-1-uwob1","pid":7,"level":60,"err":{"message":"Request Timeout after 5000ms","name":"Error","stack":"Error: Request Timeout after 5000ms\n    at null.<anonymous> (/opt/app-root/src/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)"},"msg":"","time":"2016-04-20T07:38:40.790Z","v":0}


Elastic search pod is running, but the log shows:
[2016-04-20 06:57:03,910][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Baphomet] Exception encountered when seeding initial ACL
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)

Fluent pod is running too, but the log shows:
2016-04-20 07:47:18 +0000 [error]: fluentd main process died unexpectedly. restarting.
2016-04-20 07:47:48 +0000 [error]: unexpected error error="getaddrinfo: Name or service not known"
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in `initialize'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in `open'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:878:in `block in connect'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/timeout.rb:52:in `timeout'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:877:in `connect'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:862:in `do_start'
  2016-04-20 07:47:48 +0000 [error]: /usr/share/ruby/net/http.rb:851:in `start'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/rest-client-1.8.0/lib/restclient/resource.rb:51:in `get'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:310:in `block in api'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:51:in `handle_exception'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:309:in `api'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/kubeclient-1.1.2/lib/kubeclient/common.rb:304:in `api_valid?'
  2016-04-20 07:47:48 +0000 [error]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:134:in `configure’
2016-04-20 07:50:42 +0000 [warn]: emit transaction failed: error_class=Fluent::ConfigError error="Exception encountered fetching metadata from Kubernetes API endpoint: getaddrinfo: Name or service not known" tag="kubernetes.var.log.containers.docker-registry-2-9tehj_default_registry-957340d6e4686b63bdcdccf18f1b0b4054d1faab9724097928f8102c3190f312.log"
  2016-04-20 07:50:42 +0000 [warn]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:232:in `rescue in start_watch'
  2016-04-20 07:50:42 +0000 [warn]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:228:in `start_watch'
  2016-04-20 07:50:42 +0000 [warn]: /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.18.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:140:in `block in configure'
2016-04-20 07:51:11 +0000 [warn]: emit transaction failed: error_class=SocketError error="getaddrinfo: Name or service not known" tag="kubernetes.var.log.containers.router-1-94cv3_default_router-3e8bc8d6d4ef8a52f8f46a7e83b5a510a4f1c8d63658cc49b3ce506822e84811.log”



Any idea why kibana is not starting properly?

Greetings,
   Sebastian



_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]