[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: hawkular-metrics Error on Origin 1.1



Version: oc v1.1.1.1

kubernetes v1.1.0-origin-1107-g4c8e6f4


I log in:

oc login (as admin which is a user with cluster-admin rights)

 

This are my commands, I just performed them in $ (no sudo)

 

oc project openshift-infra

 

oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-deployer
secrets:
- name: metrics-deployer
API
 
oadm policy add-role-to-user edit system:serviceaccount:openshift-infra:metrics-deployer
 
oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:heapster
 

 

oc secrets new metrics-deployer nothing=/dev/null

 

Than I go into the webconsole and deploy the metrics-template (with my hostname and without using persistant storage)

 

After the deployment everything seems to work fine. My hawkular-metrics container shows:

A time series metrics engine based on Cassandra
0.10.0-SNAPSHOT
(Git SHA1 - 324215665ceffc67080204414b604743b0da28be)
Metrics Service :STARTED


This is what I get after performing nodetools status

nodetool status                                                         

cat: /etc/ld.so.conf.d/*.conf: No such file or directory                        

Datacenter: datacenter1                                                         

=======================                                                         

Status=Up/Down                                                                  

|/ State=Normal/Leaving/Joining/Moving                                          

--  Address   Load       Tokens       Owns    Host ID                           

    Rack                                                                        

UN  10.1.1.7  96.5 KB    256          ?       b1802211-3154-40aa-b54d-d773450482

ac  rack1                                                                       

                                                                                

Note: Non-system keyspaces don't have the same replication settings, effective o

wnership information is meaningless


What's very strange is that after setting up this cluster-metrics a lot of stuff is broken. I can't save the configuration of my Jenkins, I don't get logs anymore in my kibana, I can't push new images to my secure registry. Very strange.

After I've deleted everything out of my openshift-infra project and I wait some minutes or perform a reboot. Everything works normal again.








> Date: Thu, 4 Feb 2016 11:22:11 -0500
> From: mwringe redhat com
> To: dencowboy hotmail com
> CC: users lists openshift redhat com
> Subject: Re: hawkular-metrics Error on Origin 1.1
>
> ----- Original Message -----
> > From: "Den Cowboy" <dencowboy hotmail com>
> > To: "Matt Wringe" <mwringe redhat com>
> > Cc: users lists openshift redhat com
> > Sent: Thursday, February 4, 2016 9:45:52 AM
> > Subject: RE: hawkular-metrics Error on Origin 1.1
> >
> > Yes it's running:
> >
> >
> > Do I need to open ports on my server? I would think that they can work
> > together by their container-ports.
> >
>
> The container-ports should work fine with this.
>
> For your Cassandra logs you posted, was that the total logs? It doesn't look like it fully started yet (the last message was "WARN 14:37:29 UnknownHostException for service 'hawkular-cassandra-nodes'. It may not be up yet. Trying again" which means it was still trying to start up).
>
> If you run 'nodetool status' what does this return (you can do it from the 'terminal' tab under the Cassandra pod)
>
> >
> > I will also add a picture so you can see everything is running and the
> > mappings.
> >
> >
> >
> >
> >
> > This are the logs of my cassandra:
> >
> >
> >
> >
> >
> >
> >
> > cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> >
> >
> >
> >
> > CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo
> > (Lorg/apache/cassandra/db/composites/Composite;)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned
> >
> > (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
> >
> >
> >
> >
> >
> > CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
> >
> >
> >
> >
> > CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds
> > (JJ)V
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary
> > (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan
> > (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
> >
> >
> >
> >
> > CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
> > (Ljava/nio/ByteBuffer;[B)I
> >
> >
> >
> >
> > CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
> > ([BLjava/nio/ByteBuffer;)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
> > (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
> > (Ljava/lang/Object;JILjava/lang/Object;JI)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
> > (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
> >
> >
> >
> >
> > CompilerOracle: inline
> > org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
> > (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> >
> >
> >
> >
> > INFO 14:36:57 Loading settings from
> > file:/opt/apache-cassandra-2.2.1/conf/cassandra.yaml
> >
> >
> >
> >
> > INFO 14:36:57 Node
> > configuration:[authenticator=AllowAllAuthenticator;
> > authorizer=AllowAllAuthorizer; auto_snapshot=true;
> > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> > cas_contention_timeout_in_ms=1000;
> > client_encryption_options=<REDACTED>;
> > cluster_name=hawkular-metrics; column_index_size_in_kb=64;
> > commit_failure_policy=stop;
> > commitlog_directory=/cassandra_data/commitlog;
> > commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> > commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> > concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
> > counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> > counter_write_request_timeout_in_ms=5000; cross_node_timeout=false;
> > data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop;
> > dynamic_snitch_badness_threshold=0.1;
> > dynamic_snitch_reset_interval_in_ms=600000;
> > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch;
> > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> > incremental_backups=false; index_summary_capacity_in_mb=null;
> > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> > internode_compression=all; key_cache_save_period=14400;
> > key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-roghn;
> > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> > memtable_allocation_type=heap_buffers; native_transport_port=9042;
> > num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> > permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000;
> > read_request_timeout_in_ms=5000;
> > request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> > request_timeout_in_ms=10000; row_cache_save_period=0;
> > row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-roghn;
> > rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync;
> > seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider,
> > parameters=[{seeds=hawkular-cassandra-1-roghn}]}];
> > server_encryption_options=<REDACTED>;
> > snapshot_before_compaction=false; ssl_storage_po
> >
> >
> >
> >
> > rt=7001; sstable_preemptive_open_interval_in_mb=50;
> > start_native_transport=true; start_rpc=true; storage_port=7000;
> > thrift_framed_transport_size_in_mb=15;
> > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
> > trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
> > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
> >
> >
> >
> >
> > INFO 14:36:57 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is
> > mmap
> >
> >
> >
> >
> > INFO 14:36:57 Global memtable on-heap threshold is enabled at 125MB
> >
> >
> >
> >
> > INFO 14:36:57 Global memtable off-heap threshold is enabled at 125MB
> >
> >
> >
> >
> > WARN 14:36:57 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:36:59 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:01 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:03 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > INFO 14:37:03 Loading settings from
> > file:/opt/apache-cassandra-2.2.1/conf/cassandra.yaml
> >
> >
> >
> >
> > INFO 14:37:03 Node
> > configuration:[authenticator=AllowAllAuthenticator;
> > authorizer=AllowAllAuthorizer; auto_snapshot=true;
> > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> > cas_contention_timeout_in_ms=1000;
> > client_encryption_options=<REDACTED>;
> > cluster_name=hawkular-metrics; column_index_size_in_kb=64;
> > commit_failure_policy=stop;
> > commitlog_directory=/cassandra_data/commitlog;
> > commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> > commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> > concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
> > counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> > counter_write_request_timeout_in_ms=5000; cross_node_timeout=false;
> > data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop;
> > dynamic_snitch_badness_threshold=0.1;
> > dynamic_snitch_reset_interval_in_ms=600000;
> > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch;
> > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> > incremental_backups=false; index_summary_capacity_in_mb=null;
> > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> > internode_compression=all; key_cache_save_period=14400;
> > key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-roghn;
> > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> > memtable_allocation_type=heap_buffers; native_transport_port=9042;
> > num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> > permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000;
> > read_request_timeout_in_ms=5000;
> > request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> > request_timeout_in_ms=10000; row_cache_save_period=0;
> > row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-roghn;
> > rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync;
> > seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider,
> > parameters=[{seeds=hawkular-cassandra-1-roghn}]}];
> > server_encryption_options=<REDACTED>;
> > snapshot_before_compaction=false; ssl_storage_po
> >
> >
> >
> >
> > rt=7001; sstable_preemptive_open_interval_in_mb=50;
> > start_native_transport=true; start_rpc=true; storage_port=7000;
> > thrift_framed_transport_size_in_mb=15;
> > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
> > trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
> > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
> >
> >
> >
> >
> > INFO 14:37:03 Hostname: hawkular-cassandra-1-roghn
> >
> >
> >
> >
> > INFO 14:37:03 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_65
> >
> >
> >
> >
> > INFO 14:37:03 Heap size: 526385152/526385152
> >
> >
> >
> >
> > INFO 14:37:03 Code Cache Non-heap memory: init = 2555904(2496K) used
> > = 3391040(3311K) committed = 3407872(3328K) max = 251658240(245760K)
> >
> >
> >
> >
> > INFO 14:37:03 Metaspace Non-heap memory: init = 0(0K) used = 15219576(14862K)
> > committed = 15466496(15104K) max = -1(-1K)
> >
> >
> >
> >
> > INFO 14:37:03 Compressed Class Space Non-heap memory: init = 0(0K)
> > used = 1756952(1715K) committed = 1835008(1792K) max =
> > 1073741824(1048576K)
> >
> >
> >
> >
> > INFO 14:37:03 Par Eden Space Heap memory: init = 83886080(81920K)
> > used = 55547680(54245K) committed = 83886080(81920K) max =
> > 83886080(81920K)
> >
> >
> >
> >
> > INFO 14:37:03 Par Survivor Space Heap memory: init =
> > 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max =
> > 10485760(10240K)
> >
> >
> >
> >
> > INFO 14:37:03 CMS Old Gen Heap memory: init = 432013312(421888K)
> > used = 0(0K) committed = 432013312(421888K) max = 432013312(421888K)
> >
> >
> >
> >
> > INFO 14:37:03 Classpath:
> > /opt/apache-cassandra/conf:/opt/apache-cassandra/build/classes/main:/opt/apache-cassandra/build/classes/thrift:/opt/apache-cassandra/lib/ST4-4.0.8.jar:/opt/apache-cassandra/lib/airline-0.6.jar:/opt/apache-cassandra/lib/antlr-runtime-3.5.2.jar:/opt/apache-cassandra/lib/apache-cassandra-2.2.1.jar:/opt/apache-cassandra/lib/apache-cassandra-clientutil-2.2.1.jar:/opt/apache-cassandra/lib/apache-cassandra-thrift-2.2.1.jar:/opt/apache-cassandra/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/opt/apache-cassandra/lib/cassandra-seed-provider.jar:/opt/apache-cassandra/lib/commons-cli-1.1.jar:/opt/apache-cassandra/lib/commons-codec-1.2.jar:/opt/apache-cassandra/lib/commons-lang3-3.1.jar:/opt/apache-cassandra/lib/commons-math3-3.2.jar:/opt/apache-cassandra/lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/apache-cassandra/lib/crc32ex-0.1.1.jar:/opt/apache-cassandra/lib/disruptor-3.0.1.jar:/opt/apache-cassandra/lib/ecj-4.4.2.jar:/opt/apache-cassandra/lib/guava-16.0.jar:/opt/apache-cassandra/lib/high-scale-lib-1.0.6.jar:/opt/apache-cassandra/lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra/lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar:/opt/apache-cassandra/lib/javax.inject.jar:/opt/apache-cassandra/lib/jbcrypt-0.3m.jar:/opt/apache-cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/jna-4.0.0.jar:/opt/apache-cassandra/lib/joda-time-2.4.jar:/opt/apache-cassandra/lib/json-simple-1.1.jar:/opt/apache-cassandra/lib/libthrift-0.9.2.jar:/opt/apache-cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/logback-classic-1.1.3.jar:/opt/apache-cassandra/lib/logback-core-1.1.3.jar:/opt/apache-cassandra/lib/lz4-1.3.0.jar:/opt/apache-cassandra/lib/metrics-core-3.1.0.jar:/opt/apache-cassandra/lib/metrics-logback-3.1.0.jar:/opt/apache-cassandra/lib/netty-all-4.0.23.Final.jar:/opt/apache-cassandra/lib/ohc-core-0.3.4.jar:/opt/apache-cassandra/lib/ohc-core-j8-0.3.4.jar:/opt/apache-cassandra/lib/reporter-
> >
> >
> >
> >
> > config-base-3.0.0.jar:/opt/apache-cassandra/lib/reporter-config3-3.0.0.jar:/opt/apache-cassandra/lib/sigar-1.6.4.jar:/opt/apache-cassandra/lib/slf4j-api-1.7.7.jar:/opt/apache-cassandra/lib/snakeyaml-1.11.jar:/opt/apache-cassandra/lib/snappy-java-1.1.1.7.jar:/opt/apache-cassandra/lib/stream-2.5.2.jar:/opt/apache-cassandra/lib/super-csv-2.1.0.jar:/opt/apache-cassandra/lib/thrift-server-0.3.7.jar:/opt/apache-cassandra/lib/jsr223/*/*.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar
> >
> >
> >
> >
> >
> > WARN 14:37:04 Unable to lock JVM memory (ENOMEM). This can result in
> > part of the JVM being swapped out, especially with mmapped I/O enabled.
> > Increase RLIMIT_MEMLOCK or run Cassandra as root.
> >
> >
> >
> >
> > WARN 14:37:04 JMX is not enabled to receive remote connections. Please see
> > cassandra-env.sh for more info.
> >
> >
> >
> >
> > WARN 14:37:04 OpenJDK is not recommended. Please upgrade to the newest Oracle
> > Java release
> >
> >
> >
> >
> > INFO 14:37:04 Initializing SIGAR library
> >
> >
> >
> >
> > INFO 14:37:04 Checked OS settings and found them configured for optimal
> > performance.
> >
> >
> >
> >
> > WARN 14:37:04 Directory /cassandra_data/data doesn't exist
> >
> >
> >
> >
> > WARN 14:37:04 Directory /cassandra_data/commitlog doesn't exist
> >
> >
> >
> >
> > WARN 14:37:04 Directory /opt/apache-cassandra/data/saved_caches doesn't exist
> >
> >
> >
> >
> > INFO 14:37:05 Initializing system.sstable_activity
> >
> >
> >
> >
> > INFO 14:37:07 Initializing key cache with capacity of 25 MBs.
> >
> >
> >
> >
> > INFO 14:37:07 Initializing row cache with capacity of 0 MBs
> >
> >
> >
> >
> > INFO 14:37:07 Initializing counter cache with capacity of 12 MBs
> >
> >
> >
> >
> > INFO 14:37:07 Scheduling counter cache save to every 7200 seconds (going to
> > save all keys).
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.hints
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.compaction_history
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.peers
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_columnfamilies
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_functions
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.IndexInfo
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_columns
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_triggers
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.local
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_usertypes
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.batchlog
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.available_ranges
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.schema_aggregates
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.paxos
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.peer_events
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.size_estimates
> >
> >
> >
> >
> > INFO 14:37:07 Initializing system.compactions_in_progress
> >
> >
> >
> >
> > INFO 14:37:08 Initializing system.schema_keyspaces
> >
> >
> >
> >
> > INFO 14:37:08 Initializing system.range_xfers
> >
> >
> >
> >
> > INFO 14:37:08 Enqueuing flush of local: 653 (0%) on-heap, 0 (0%) off-heap
> >
> >
> >
> >
> > INFO 14:37:08 Writing Memtable-local 1629331832(0.107KiB serialized bytes, 3
> > ops, 0%/0% of on/off-heap limit)
> >
> >
> >
> >
> > INFO 14:37:08 Completed flushing
> > /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/tmp-la-1-big-Data.db
> > (0.000KiB) for commitlog position
> > ReplayPosition(segmentId=1454596627682, position=283)
> >
> >
> >
> >
> > INFO 14:37:09 No commitlog files found; skipping replay
> >
> >
> >
> >
> > INFO 14:37:10 Cassandra version: 2.2.1
> >
> >
> >
> >
> > INFO 14:37:10 Thrift API version: 20.1.0
> >
> >
> >
> >
> > INFO 14:37:10 CQL supported versions: 3.3.0 (default: 3.3.0)
> >
> >
> >
> >
> > INFO 14:37:10 Initializing index summary manager with a memory pool size of
> > 25 MB and a resize interval of 60 minutes
> >
> >
> >
> >
> > INFO 14:37:10 Loading persisted ring state
> >
> >
> >
> >
> > WARN 14:37:10 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:12 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:14 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:16 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > INFO 14:37:16 Loading settings from
> > file:/opt/apache-cassandra-2.2.1/conf/cassandra.yaml
> >
> >
> >
> >
> > INFO 14:37:16 Node
> > configuration:[authenticator=AllowAllAuthenticator;
> > authorizer=AllowAllAuthorizer; auto_snapshot=true;
> > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> > cas_contention_timeout_in_ms=1000;
> > client_encryption_options=<REDACTED>;
> > cluster_name=hawkular-metrics; column_index_size_in_kb=64;
> > commit_failure_policy=stop;
> > commitlog_directory=/cassandra_data/commitlog;
> > commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> > commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> > concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
> > counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> > counter_write_request_timeout_in_ms=5000; cross_node_timeout=false;
> > data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop;
> > dynamic_snitch_badness_threshold=0.1;
> > dynamic_snitch_reset_interval_in_ms=600000;
> > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch;
> > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> > incremental_backups=false; index_summary_capacity_in_mb=null;
> > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> > internode_compression=all; key_cache_save_period=14400;
> > key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-roghn;
> > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> > memtable_allocation_type=heap_buffers; native_transport_port=9042;
> > num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> > permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000;
> > read_request_timeout_in_ms=5000;
> > request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> > request_timeout_in_ms=10000; row_cache_save_period=0;
> > row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-roghn;
> > rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync;
> > seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider,
> > parameters=[{seeds=hawkular-cassandra-1-roghn}]}];
> > server_encryption_options=<REDACTED>;
> > snapshot_before_compaction=false; ssl_storage_po
> >
> >
> >
> >
> > rt=7001; sstable_preemptive_open_interval_in_mb=50;
> > start_native_transport=true; start_rpc=true; storage_port=7000;
> > thrift_framed_transport_size_in_mb=15;
> > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
> > trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
> > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
> >
> >
> >
> >
> > WARN 14:37:16 No host ID found, created 83525f23-b8a5-4a7c-819f-664740cec837
> > (Note: This should happen exactly once per node).
> >
> >
> >
> >
> > INFO 14:37:16 Starting up server gossip
> >
> >
> >
> >
> > INFO 14:37:16 Enqueuing flush of local: 2125 (0%) on-heap, 0 (0%) off-heap
> >
> >
> >
> >
> > INFO 14:37:16 Writing Memtable-local 667834293(0.401KiB serialized bytes, 16
> > ops, 0%/0% of on/off-heap limit)
> >
> >
> >
> >
> > INFO 14:37:16 Completed flushing
> > /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/tmp-la-2-big-Data.db
> > (0.000KiB) for commitlog position
> > ReplayPosition(segmentId=1454596627682, position=103794)
> >
> >
> >
> >
> > WARN 14:37:16 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:18 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:20 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:22 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > INFO 14:37:22 Loading settings from
> > file:/opt/apache-cassandra-2.2.1/conf/cassandra.yaml
> >
> >
> >
> >
> > INFO 14:37:22 Node
> > configuration:[authenticator=AllowAllAuthenticator;
> > authorizer=AllowAllAuthorizer; auto_snapshot=true;
> > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> > cas_contention_timeout_in_ms=1000;
> > client_encryption_options=<REDACTED>;
> > cluster_name=hawkular-metrics; column_index_size_in_kb=64;
> > commit_failure_policy=stop;
> > commitlog_directory=/cassandra_data/commitlog;
> > commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> > commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> > concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
> > counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> > counter_write_request_timeout_in_ms=5000; cross_node_timeout=false;
> > data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop;
> > dynamic_snitch_badness_threshold=0.1;
> > dynamic_snitch_reset_interval_in_ms=600000;
> > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch;
> > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> > incremental_backups=false; index_summary_capacity_in_mb=null;
> > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> > internode_compression=all; key_cache_save_period=14400;
> > key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-roghn;
> > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> > memtable_allocation_type=heap_buffers; native_transport_port=9042;
> > num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> > permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000;
> > read_request_timeout_in_ms=5000;
> > request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> > request_timeout_in_ms=10000; row_cache_save_period=0;
> > row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-roghn;
> > rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync;
> > seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider,
> > parameters=[{seeds=hawkular-cassandra-1-roghn}]}];
> > server_encryption_options=<REDACTED>;
> > snapshot_before_compaction=false; ssl_storage_po
> >
> >
> >
> >
> > rt=7001; sstable_preemptive_open_interval_in_mb=50;
> > start_native_transport=true; start_rpc=true; storage_port=7000;
> > thrift_framed_transport_size_in_mb=15;
> > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
> > trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
> > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
> >
> >
> >
> >
> > INFO 14:37:23 Starting Encrypted Messaging Service on SSL port 7001
> >
> >
> >
> >
> > WARN 14:37:23 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:25 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:27 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > WARN 14:37:29 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> > INFO 14:37:29 Loading settings from
> > file:/opt/apache-cassandra-2.2.1/conf/cassandra.yaml
> >
> >
> >
> >
> > INFO 14:37:29 Node
> > configuration:[authenticator=AllowAllAuthenticator;
> > authorizer=AllowAllAuthorizer; auto_snapshot=true;
> > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> > cas_contention_timeout_in_ms=1000;
> > client_encryption_options=<REDACTED>;
> > cluster_name=hawkular-metrics; column_index_size_in_kb=64;
> > commit_failure_policy=stop;
> > commitlog_directory=/cassandra_data/commitlog;
> > commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> > commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> > concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
> > counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> > counter_write_request_timeout_in_ms=5000; cross_node_timeout=false;
> > data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop;
> > dynamic_snitch_badness_threshold=0.1;
> > dynamic_snitch_reset_interval_in_ms=600000;
> > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch;
> > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> > incremental_backups=false; index_summary_capacity_in_mb=null;
> > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> > internode_compression=all; key_cache_save_period=14400;
> > key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-roghn;
> > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> > memtable_allocation_type=heap_buffers; native_transport_port=9042;
> > num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> > permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000;
> > read_request_timeout_in_ms=5000;
> > request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> > request_timeout_in_ms=10000; row_cache_save_period=0;
> > row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-roghn;
> > rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync;
> > seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider,
> > parameters=[{seeds=hawkular-cassandra-1-roghn}]}];
> > server_encryption_options=<REDACTED>;
> > snapshot_before_compaction=false; ssl_storage_po
> >
> >
> >
> >
> > rt=7001; sstable_preemptive_open_interval_in_mb=50;
> > start_native_transport=true; start_rpc=true; storage_port=7000;
> > thrift_framed_transport_size_in_mb=15;
> > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
> > trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
> > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
> >
> >
> >
> >
> > INFO 14:37:29 This node will not auto bootstrap because it is configured to
> > be a seed node.
> >
> >
> > WARN 14:37:29 UnknownHostException for service 'hawkular-cassandra-nodes'. It
> > may not be up yet. Trying again
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Extra info:
> >
> >
> > I'm not using persistent storage for it, and I'm not using a certificate:
> >
> >
> > oc secrets new metrics-deployer nothing=/dev/null
> >
> > > Date: Thu, 4 Feb 2016 09:32:51 -0500
> > > From: mwringe redhat com
> > > To: dencowboy hotmail com
> > > CC: users lists openshift redhat com
> > > Subject: Re: hawkular-metrics Error on Origin 1.1
> > >
> > > Is your Cassandra instance up and running? The errors appear to be that
> > > Hawkular Metrics can no longer connect to the Cassandra instance or is
> > > having problems trying to connect to it.
> > >
> > > ----- Original Message -----
> > > > From: "Den Cowboy" <dencowboy hotmail com>
> > > > To: users lists openshift redhat com
> > > > Sent: Thursday, February 4, 2016 5:42:07 AM
> > > > Subject: RE: hawkular-metrics Error on Origin 1.1
> > > >
> > > > My heapster shows the whole time:
> > > > E0204 10:41:18.378835 1 driver.go:234] Could not update tags: Put
> > > > https://hawkular-metrics:443/hawkular/metrics/counters/heapster%2Fd8af2e68-cb28-11e5-9109-021fa5f1deed%2Fmemory%2Fpage_faults/tags:
> > > > net/http: request canceled while waiting for connection
> > > >
> > > >
> > > > From: dencowboy hotmail com
> > > > To: users lists openshift redhat com
> > > > Subject: hawkular-metrics Error on Origin 1.1
> > > > Date: Thu, 4 Feb 2016 10:30:00 +0000
> > > >
> > > > I've set up my Hawkul cluster metrics. The configuration went fine and
> > > > all
> > > > the pods are running.
> > > > But when I'm taking a watch into the metrics-tab of some apps I see:
> > > > Error fetching cpu/usage for container test. Failed to perform operation
> > > > due
> > > > to an error: All host(s) tried for query failed (no host was tried)
> > > >
> > > > For more information I took a look into the logs of my hawkular-metrics -
> > > > pod
> > > > which shows:
> > > >
> > > > [0m[31m10:24:33,210 ERROR [com.datastax.driver.core.ControlConnection]
> > > > (cluster9-reconnection-0) [Control connection] Cannot connect to any
> > > > host,
> > > > scheduling retry in 64000 milliseconds
> > > > [0m[31m10:25:37,211 ERROR [com.datastax.driver.core.ControlConnection]
> > > > (cluster9-reconnection-0) [Control connection] Cannot connect to any
> > > > host,
> > > > scheduling retry in 128000 milliseconds
> > > > [0m[31m10:27:45,211 ERROR [com.datastax.driver.core.ControlConnection]
> > > > (cluster9-reconnection-0) [Control connection] Cannot connect to any
> > > > host,
> > > > scheduling retry in 256000 milliseconds
> > > > [0m[31m10:28:06,554 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils]
> > > > (RxComputationThreadPool-2) HAWKMETRICS200010: Failed to process request:
> > > > java.util.concurrent.ExecutionException:
> > > > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> > > > tried for query failed (no host was tried)
> > > > at
> > > > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> > > > at
> > > > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> > > > at
> > > > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> > > > at org.hawkular.rx.cassandra.driver.RxUtil$2$1.run(RxUtil.java:82)
> > > > at org.hawkular.rx.cassandra.driver.RxUtil$1$1.call(RxUtil.java:59)
> > > > at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
> > > > at
> > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> > > > All
> > > > host(s) tried for query failed (no host was tried)
> > > > at
> > > > com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:219)
> > > > at
> > > > com.datastax.driver.core.RequestHandler.access$900(RequestHandler.java:45)
> > > > at
> > > > com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:279)
> > > > at
> > > > com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:118)
> > > > at
> > > > com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:94)
> > > > at
> > > > com.datastax.driver.core.SessionManager.execute(SessionManager.java:565)
> > > > at
> > > > com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:602)
> > > > at
> > > > com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:98)
> > > > at
> > > > org.hawkular.rx.cassandra.driver.RxSessionImpl.execute(RxSessionImpl.java:80)
> > > > at
> > > > org.hawkular.metrics.core.impl.DataAccessImpl.findGaugeData(DataAccessImpl.java:438)
> > > > at
> > > > org.hawkular.metrics.core.impl.DataAccessImpl.findGaugeData(DataAccessImpl.java:409)
> > > > at
> > > > org.hawkular.metrics.core.impl.MetricsServiceImpl.lambda$startUp$20(MetricsServiceImpl.java:223)
> > > > at
> > > > org.hawkular.metrics.core.impl.MetricsServiceImpl.lambda$findDataPoints$55(MetricsServiceImpl.java:644)
> > > > at com.codahale.metrics.Timer.time(Timer.java:99)
> > > > at
> > > > org.hawkular.metrics.core.impl.MetricsServiceImpl.time(MetricsServiceImpl.java:907)
> > > > at
> > > > org.hawkular.metrics.core.impl.MetricsServiceImpl.findDataPoints(MetricsServiceImpl.java:644)
> > > > at
> > > > org.hawkular.metrics.core.impl.MetricsServiceImpl.findGaugeStats(MetricsServiceImpl.java:713)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:497)
> > > > at
> > > > org.jboss.weld.bean.proxy.AbstractBeanInstance.invoke(AbstractBeanInstance.java:38)
> > > > at
> > > > org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
> > > > at
> > > > org.jboss.weld.proxies.MetricsService$962346502$Proxy$_$$_WeldClientProxy.findGaugeStats(Unknown
> > > > Source)
> > > > at
> > > > org.hawkular.metrics.api.jaxrs.handler.GaugeHandler.findGaugeData(GaugeHandler.java:305)
> > > > at
> > > > org.hawkular.metrics.api.jaxrs.handler.GaugeHandler$Proxy$_$$_WeldClientProxy.findGaugeData(Unknown
> > > > Source)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:497)
> > > > at
> > > > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
> > > > at
> > > > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
> > > > at
> > > > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
> > > > at
> > > > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
> > > > at
> > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
> > > > at
> > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
> > > > at
> > > > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
> > > > at
> > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
> > > > at
> > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
> > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> > > >
> > > > Someone known with this error?
> > > >
> > > > _______________________________________________ users mailing list
> > > > users lists openshift redhat com
> > > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> > > >
> > > > _______________________________________________
> > > > users mailing list
> > > > users lists openshift redhat com
> > > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> > > >
> >

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]