[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Origin 4 problem to add nodes to district



On Wed, Jan 14, 2015 at 06:25:21PM +0100, Massimiliano Dessì wrote:
2015-01-14 17:55 GMT+01:00 Miciah Dashiel Butler Masters <
mmasters redhat com>:

On Wed, Jan 14, 2015 at 01:55:29PM +0000, Massimiliano Dessě wrote:

HI all,
the oo-mco ping show  the two nodes,
but when I try to add with
oo-admin-ctl-district -c add-node -n small_district -i <node>
I receive the message
"ERROR OUTPUT: Cannot connect to node."
[...]
Is /opt/rh/ruby193/root/etc/mcollective/facts.yaml being updated? The
/etc/cron.minutely/openshift-facts cronjob should update facts.yaml
once every minute.  If facts.yaml isn't being updated, try running
`oo-exec-ruby /opt/rh/ruby193/root/usr/libexec/mcollective/update_yaml.rb`
manually to see whether that fails or whether cron is misconfigured.


yes is updated every minute

And the contents look OK? Do facts such as kernel, node_profile, and
public_hostname look correct?

 The oo-mco facts node-profile -v


That should be 'node_profile', with an underscore.



[root broker mcollective]# oo-mco facts node_profile -v
Discovering hosts using the mc method for 2 second(s) .... 2
Report for fact: node_profile


---- rpc stats ----
          Nodes: 2 / 0
    Pass / Fail: 0 / 0
     Start Time: 2015-01-14 09:15:00 -0800
 Discovery Time: 2050.83ms
     Agent Time: 12005.06ms
     Total Time: 14055.89ms


No response from:

  node01.<mydomain>  node02.<mydomain>



 show a message no response from the two nodes.

Any hints to fix ?


What does `oo-mco facts kernel` show? Should be 'Linux' (same as the
output of `uname -s`).


[root broker ~]# uname -s
Linux
[root broker mcollective]# oo-mco facts kernel
Report for fact: kernel


Finished processing 0 / 2 hosts in 12004.92 ms


No response from:

  node01.<mydomain>  node02.<mydomain>





How about `oo-mco inventory <node>`?



[root broker mcollective]# oo-mco inventory node01.<mydomain>
Did not receive any results from node node01.<mydomain>
[...]

Is oo-mco ping consistently showing responses from both nodes? If there
are intermittent failures, maybe the hosts' clocks are out of synch, or
there could be other problems with the ActiveMQ connections.  It looks
like MCollective is authenticating with ActiveMQ fine though and has the
correct collective configured in
/opt/rh/ruby193/root/etc/mcollective/server.cfg and
/opt/rh/ruby193/root/etc/mcollective/client.cfg seeing that oo-mco ping
is working.

Do you have direct_addressing = 1 in server.cfg and direct_addressing
= 0 in client.cfg? If not, try with those settings (you'll need to
restart ruby193-mcollective on the nodes to test this, and you'll need
to restart openshift-broker on the broker host if testing shows that
setting to be the issue).

Could you tailf /var/log/openshift/node/ruby193-mcollective.log on the
nodes as you run the oo-mco inventory or oo-mco facts commands? That
will let us determine whether the node is ignoring the request, on for
what reason.  I'm attaching some example log output in case that might
help you understand what I am asking you to look for.

--
Miciah Dashiel Butler Masters <mmasters redhat com>
Output in /var/log/openshift/node/ruby193-mcollective.log on node01 when
running `oo-mco inventory node02.hosts.example.com` on the broker:

D, [2015-01-14T16:33:19.271521 #22109] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
D, [2015-01-14T16:33:19.274167 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.274615 #22109] DEBUG -- : runnerstats.rb:38:in `validated' Incrementing validated stat
D, [2015-01-14T16:33:19.274801 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.275009 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.275220 #22109] DEBUG -- : base.rb:117:in `block (2 levels) in validate_filter?' Passing based on agent rpcutil
D, [2015-01-14T16:33:19.275522 #22109] DEBUG -- : base.rb:145:in `block in validate_filter?' Failed based on identity
D, [2015-01-14T16:33:19.275692 #22109] DEBUG -- : base.rb:159:in `validate_filter?' Message failed the filter checks
D, [2015-01-14T16:33:19.275842 #22109] DEBUG -- : runnerstats.rb:32:in `filtered' Incrementing filtered stat
D, [2015-01-14T16:33:19.276049 #22109] DEBUG -- : runner.rb:75:in `rescue in block in run' Message does not pass filters, ignoring
D, [2015-01-14T16:33:19.276198 #22109] DEBUG -- : activemq.rb:340:in `receive' Waiting for a message from ActiveMQ
D, [2015-01-14T16:33:19.308233 #22109] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
D, [2015-01-14T16:33:19.308830 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.309742 #22109] DEBUG -- : runnerstats.rb:38:in `validated' Incrementing validated stat
D, [2015-01-14T16:33:19.310198 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.310938 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:19.313839 #22109] DEBUG -- : base.rb:117:in `block (2 levels) in validate_filter?' Passing based on agent rpcutil
D, [2015-01-14T16:33:19.314807 #22109] DEBUG -- : base.rb:145:in `block in validate_filter?' Failed based on identity
D, [2015-01-14T16:33:19.315195 #22109] DEBUG -- : base.rb:159:in `validate_filter?' Message failed the filter checks
D, [2015-01-14T16:33:19.317216 #22109] DEBUG -- : runnerstats.rb:32:in `filtered' Incrementing filtered stat
D, [2015-01-14T16:33:19.318206 #22109] DEBUG -- : runner.rb:75:in `rescue in block in run' Message does not pass filters, ignoring
D, [2015-01-14T16:33:19.318692 #22109] DEBUG -- : activemq.rb:340:in `receive' Waiting for a message from ActiveMQ

Output in /var/log/openshift/node/ruby193-mcollective.log on node01 when
running `oo-mco inventory node01.hosts.example.com` on the broker:

D, [2015-01-14T16:33:48.979025 #22109] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
D, [2015-01-14T16:33:48.980225 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:48.980640 #22109] DEBUG -- : runnerstats.rb:38:in `validated' Incrementing validated stat
D, [2015-01-14T16:33:48.980823 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:48.981106 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:48.981330 #22109] DEBUG -- : base.rb:117:in `block (2 levels) in validate_filter?' Passing based on agent rpcutil
D, [2015-01-14T16:33:48.981623 #22109] DEBUG -- : base.rb:142:in `block in validate_filter?' Passing based on identity
D, [2015-01-14T16:33:48.981771 #22109] DEBUG -- : base.rb:153:in `validate_filter?' Message passed the filter checks
D, [2015-01-14T16:33:48.981910 #22109] DEBUG -- : runnerstats.rb:26:in `passed' Incrementing passed stat
D, [2015-01-14T16:33:48.982052 #22109] DEBUG -- : runner.rb:94:in `agentmsg' Handling message for agent 'rpcutil' on collective 'mcollective'
D, [2015-01-14T16:33:48.982192 #22109] DEBUG -- : agents.rb:119:in `dispatch' Dispatching a message to agent rpcutil
D, [2015-01-14T16:33:48.982515 #22109] DEBUG -- : activemq.rb:340:in `receive' Waiting for a message from ActiveMQ
D, [2015-01-14T16:33:48.982693 #22109] DEBUG -- : pluginmanager.rb:88:in `[]' Returning new plugin rpcutil_agent with class MCollective::Agent::Rpcutil
D, [2015-01-14T16:33:48.982961 #22109] DEBUG -- : cache.rb:105:in `read' Cache hit on 'ddl' key 'agent/rpcutil'
D, [2015-01-14T16:33:48.983481 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin global_stats with class MCollective::RunnerStats
D, [2015-01-14T16:33:48.983863 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:48.983995 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:48.984234 #22109] DEBUG -- : base.rb:168:in `create_reply' Encoded a message for request 0456882baaa75bddae9154d080e04e9f
D, [2015-01-14T16:33:48.984574 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Activemq
D, [2015-01-14T16:33:48.984785 #22109] DEBUG -- : activemq.rb:373:in `publish' Sending a broadcast message to ActiveMQ target '/queue/mcollective.reply.broker01.hosts.example.com_18759' with headers '{"timestamp"=>"1421271228000", "expires"=>"1421271298000"}'
D, [2015-01-14T16:33:48.985479 #22109] DEBUG -- : runnerstats.rb:56:in `block in sent' Incrementing replies stat
D, [2015-01-14T16:33:49.011836 #22109] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
D, [2015-01-14T16:33:49.012026 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:49.012232 #22109] DEBUG -- : runnerstats.rb:38:in `validated' Incrementing validated stat
D, [2015-01-14T16:33:49.012350 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:49.012528 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:49.012680 #22109] DEBUG -- : base.rb:117:in `block (2 levels) in validate_filter?' Passing based on agent rpcutil
D, [2015-01-14T16:33:49.012877 #22109] DEBUG -- : base.rb:142:in `block in validate_filter?' Passing based on identity
D, [2015-01-14T16:33:49.012977 #22109] DEBUG -- : base.rb:153:in `validate_filter?' Message passed the filter checks
D, [2015-01-14T16:33:49.013066 #22109] DEBUG -- : runnerstats.rb:26:in `passed' Incrementing passed stat
D, [2015-01-14T16:33:49.013157 #22109] DEBUG -- : runner.rb:94:in `agentmsg' Handling message for agent 'rpcutil' on collective 'mcollective'
D, [2015-01-14T16:33:49.013351 #22109] DEBUG -- : agents.rb:119:in `dispatch' Dispatching a message to agent rpcutil
D, [2015-01-14T16:33:49.013650 #22109] DEBUG -- : activemq.rb:340:in `receive' Waiting for a message from ActiveMQ
D, [2015-01-14T16:33:49.013856 #22109] DEBUG -- : pluginmanager.rb:88:in `[]' Returning new plugin rpcutil_agent with class MCollective::Agent::Rpcutil
D, [2015-01-14T16:33:49.014117 #22109] DEBUG -- : cache.rb:105:in `read' Cache hit on 'ddl' key 'agent/rpcutil'
D, [2015-01-14T16:33:49.014667 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin facts_plugin with class MCollective::Facts::Yaml_facts
D, [2015-01-14T16:33:49.015926 #22109] DEBUG -- : yaml_facts.rb:51:in `block in force_reload?' Forcing fact reload due to age of /opt/rh/ruby193/root/etc/mcollective/facts.yaml
D, [2015-01-14T16:33:49.016139 #22109] DEBUG -- : base.rb:30:in `block in get_fact' Resetting facter cache, now: 1421271229 last-known-good: 1421270988
D, [2015-01-14T16:33:49.022268 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:49.022553 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2015-01-14T16:33:49.023038 #22109] DEBUG -- : base.rb:168:in `create_reply' Encoded a message for request c37df6e4e6b45eeca3ae46e66bc995f2
D, [2015-01-14T16:33:49.023231 #22109] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Activemq
D, [2015-01-14T16:33:49.023564 #22109] DEBUG -- : activemq.rb:373:in `publish' Sending a broadcast message to ActiveMQ target '/queue/mcollective.reply.broker01.hosts.example.com_18759' with headers '{"timestamp"=>"1421271229000", "expires"=>"1421271299000"}'
D, [2015-01-14T16:33:49.024148 #22109] DEBUG -- : runnerstats.rb:56:in `block in sent' Incrementing replies stat

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]