[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: OpenshiftOrigin DNS Problem



So, I worked a bit more on this today, focusing my attention on the script that is used to perform the updates.  Running a packet capture I was able to see what was being tossed in for an update:

 

cname=nodejs9-demo.apps.domain.com&fqdn=osb.apps.domain.com&key_name=osb.apps.domain.com&key_value=apps.domain.com

 

That _seems_ legit but I guess it depends on what the app is going to do.  I peeled open the source code and saw something what I _think_ is awkward looking:

https://github.com/openshift/origin-server/blob/master/extras/avahi-cname-manager/bin/avahi-cname-manager#L84-85

 

cname isn’t empty, or nil, and I think it’s odd that we are demanding the record have the word local in it.  That simply doesn’t make sense to me as I’m never going to browse an application to something such as nodejs9-demo.local.  I validated that this works by sending a POST request myself using nodejs9-demo.local as an example, and it works fine.  But again, I don’t understand this.

 

Side note: I see this app was built with a bit of security in mind, but despite running POST requests from a remote machine, my request was still successful.  The script might need some more work with regards to this particular feature:

 

Mar 26 14:35:38 osb avahi-cname-manager[1115]: W, [2014-03-26T14:35:38.671014 #1115]  WARN -- : attack prevented by Rack::Protection::HttpOrigin

Mar 26 14:35:38 osb avahi-cname-manager[1115]: CNAME: nodejs9-demo.apps.local; FQDN: osb.apps.local

Mar 26 14:35:38 osb avahi-cname-manager[1115]: Adding alias nodejs9-demo.apps.local -> osb.apps.local

Mar 26 14:35:38 osb avahi-cname-manager[1115]: Clearing all aliases for reload

Mar 26 14:35:38 osb avahi-cname-manager[1115]: ...succesful

 

 

John Skarbek, System Administrator 

 

 

 

Hey, thanks for the reply,

 

So as suggested, I rocked out some nsupdate which appeared to work very well:

 

[root osb named]# nsupdate -v -y "apps.domain.com:scramblethis=="

> server 192.168.1.1

> zone apps.nextcentury.com

> update add nodejs7-jskarbek.apps.nextcentury.com 3600 A 192.168.1.1

> send

> quit

[root osb named]# nslookup nodejs7-jskarbek.apps.domain.com

Server:         192.168.1.1

Address:        192.168.1.1#53

 

Name:   nodejs7-jskarbek.apps.domain.com

Address: 192.168.1.1

 

I turned up logging on named and here was that little bit of output:

client 192.168.1.1#25866/key apps.domain.com: signer "apps.domain.com" approved

client 192.168.1.1#25866/key apps.domain.com: updating zone 'apps.domain.com/IN': adding an RR at 'nodejs7-jskarbek.apps.domain.com' A

zone_needdump: zone apps.domain.com/IN: enter

zone_settimer: zone apps.domain.com/IN: enter

zone_settimer: zone apps.domain.com/IN: enter

zone_timer: zone apps.domain.com/IN: enter

zone_maintenance: zone apps.domain.com/IN: enter

zone_settimer: zone apps.domain.com/IN: enter

 

However, when going through the web interface or using rhc, I still find this in the messages file ( trying to create a nodejs8-jskarbek app):

Mar 19 08:28:48 osb avahi-cname-manager[859]: 192.168.1.1 - - [19/Mar/2014 08:28:48] "POST /add_alias HTTP/1.1" 400 53 0.0010

Mar 19 08:28:48 osb avahi-cname-manager[859]: osb.hq.domain.com - - [19/Mar/2014:08:28:48 EDT] "POST /add_alias HTTP/1.1" 400 53

Mar 19 08:28:48 osb avahi-cname-manager[859]: - -> /add_alias

Mar 19 08:28:48 osb avahi-cname-manager[859]: 192.168.1.1 - - [19/Mar/2014 08:28:48] "POST /remove_alias HTTP/1.1" 400 53 0.0007

Mar 19 08:28:48 osb avahi-cname-manager[859]: osb.hq.domain.com - - [19/Mar/2014:08:28:48 EDT] "POST /remove_alias HTTP/1.1" 400 53

Mar 19 08:28:48 osb avahi-cname-manager[859]: - -> /remove_alias

 

And then in the log for named, nothing.  I ponder if there’s a problem with the avahi-cname-manager.  Beyond changing the debug statement for named and a minor change to the db for this domain, named is largely untouched.

 

John Skarbek, System Administrator 

 

From: Mfawa Alfred Onen [mailto:muffycompoqm gmail com]
Sent: Tuesday, March 18, 2014 5:38 PM
To: users lists openshift redhat com; John Skarbek
Subject: Re: OpenshiftOrigin DNS Problem

 

You need to ensure that bind was properly setup with all the dynamic DNS bits laid out. You can manually us nsupdate to add or remove CNAME records and see if that works. As for the SELinux bits, ensure that you have ruby193-mcollective installed or started properly. I will suggest you checkout http://openshift.github.io/documentation/oo_deployment_guide_puppet.html for your Puppet needs.

 

Regards!

 

On Tue, Mar 18, 2014 at 3:23 PM, <users-request lists openshift redhat com> wrote:

Send users mailing list submissions to
        users lists openshift redhat com

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.openshift.redhat.com/openshiftmm/listinfo/users
or, via email, send a message with subject or body 'help' to
        users-request lists openshift redhat com

You can reach the person managing the list at
        users-owner lists openshift redhat com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of users digest..."

Today's Topics:

   1. OpenshiftOrigin DNS Problem (John Skarbek)


---------- Forwarded message ----------
From: John Skarbek <john skarbek nextcentury com>
To: "users lists openshift redhat com" <users lists openshift redhat com>
Cc: 
Date: Tue, 18 Mar 2014 14:23:44 +0000
Subject: OpenshiftOrigin DNS Problem

Good Morning!

 

This is a long email…. I can’t get openshift to update DNS properly so apps create, but then you can’t reach them, and you can’t remove them easily.

 

I’m trying out this openshift origin and I used puppet to configure everything on a single host.  This including, dns, node, broker, etc…  When deploying an application I run into a problem, the web interface reports we are undergoing maintenance…  Looking at the broker/production.log:

 

2014-03-18 10:07:21.738 [INFO ] Started POST "/broker/rest/domain/jskarbek/applications.json" for 127.0.0.1 at 2014-03-18 10:07:21 -0400 (pid:2598)

2014-03-18 10:07:21.808 [INFO ] Processing by ApplicationsController#create as JSON (pid:2598)

2014-03-18 10:07:21.812 [INFO ] Parameters: {"cartridges"=>["nodejs-0.10"], "domain_id"=>"jskarbek", "gear_profile"=>"small", "initial_git_url"=>"", "name"=>"nodejs2", "scale"=>"false"} (pid:2598)

2014-03-18 10:07:29.429 [ERROR] Invlaid CNAME 'nodejs2-jskarbek.apps.domain.com' (pid:2598)

<SNIP>

2014-03-18 10:07:29.653 [ERROR] Reference ID: a0bb10a1f4d8aed5e83cf90313b8bc18 - Invlaid CNAME 'nodejs2-jskarbek.apps.domain.com'

<SNIP>

  /usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>' (pid:2598)

2014-03-18 10:07:29.658 [INFO ] Completed 503 Service Unavailable in 8576.9ms (Views: 1.0ms) (pid:2598)

 

Which sucks, cuz the app will create, dns will fail as an entry is not added, and then you can’t delete the app using the web interface because it can’t resolve dns.  But here’s what’s in the messages file:

 

Mar 18 10:07:29 osb avahi-cname-manager[7118]: 192.168.1.1 - - [18/Mar/2014 10:07:29] "POST /add_alias HTTP/1.1" 400 53 0.0014

Mar 18 10:07:29 osb avahi-cname-manager[7118]: osb.domain.com - - [18/Mar/2014:10:07:29 EDT] "POST /add_alias HTTP/1.1" 400 53

Mar 18 10:07:29 osb avahi-cname-manager[7118]: - -> /add_alias

Mar 18 10:07:29 osb avahi-cname-manager[7118]: 192.168.1.1  - - [18/Mar/2014 10:07:29] "POST /remove_alias HTTP/1.1" 400 53 0.0009

Mar 18 10:07:29 osb avahi-cname-manager[7118]: osb.domain.com - - [18/Mar/2014:10:07:29 EDT] "POST /remove_alias HTTP/1.1" 400 53

Mar 18 10:07:29 osb avahi-cname-manager[7118]: - -> /remove_alias

 

Those 400 returns are probably the reason why, but I have no clue how to troubleshoot this because I don’t know how this DNS is all rolled together.

 

oo-diag reports a couple of things that are awkward:

 

[root osb ~]# oo-diagnostics

/usr/share/gems/gems/psych-2.0.0/lib/psych.rb:98: warning: already initialized constant Psych::VERSION

/usr/share/ruby/vendor_ruby/psych.rb:98: warning: previous definition of VERSION was here

/usr/share/gems/gems/psych-2.0.0/lib/psych.rb:101: warning: already initialized constant Psych::LIBYAML_VERSION

/usr/share/ruby/vendor_ruby/psych.rb:101: warning: previous definition of LIBYAML_VERSION was here

WARN: test_node_profiles_districts_from_broker

        No districts are defined. Districts should be used in any production installation.

        Please consult the Administration Guide.

 

FAIL: run_script

oo-accept-node had errors:

--BEGIN OUTPUT--

FAIL: Could not get SELinux context for mcollective

FAIL: Could not get SELinux context for oddjobd

FAIL: tc htb qdisc not configured

FAIL: directory 5328531986e5798ff3000002 doesn't have a cartridge directory

FAIL: directory 532851bf86e5791d6d000045 doesn't have a cartridge directory

FAIL: directory 5328518e86e5791d6d00002f doesn't have a cartridge directory

6 ERRORS

 

--END oo-accept-node OUTPUT--

sed: can't read /var/log/mcollective.log: No such file or directory

WARN: test_broker_certificate

Using a self-signed certificate for the broker

grep: /etc/httpd/conf.d/openshift: Is a directory

WARN: block (2 levels) in test_broker_certificate

            /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf

            defines ServerName as osb.apps.domain.com.  This does not match the certificate common name of

            osb.hq.domain.com.

            This can cause errors when client tools try to connect to the broker.

 

3 WARNINGS

1 ERRORS

 

And the below is my puppet config that deployed this single host:

 

class { 'openshift_origin':

  domain                     => 'apps.domain.com',

  roles                      => [

    'broker',

    'node',

    'activemq',

    'datastore',

    'named',

  ],

  broker_hostname            => 'osb.apps.domain.com',

  node_hostname              => 'osb.apps.domain.com',

  named_hostname             => 'osb.apps.domain.com',

  activemq_hostname          => 'osb.apps.domain.com',

  datastore_hostname         => 'osb.apps.domain.com',

  named_ip_addr              => $ipaddress,

  broker_ip_addr             => $ipaddress,

  node_ip_addr               => $ipaddress,

  conf_named_upstream_dns    => [

    ‘192.168.1.2’,

    ‘192.168.1.3’,

    $ipaddress,

  ],

  node_unmanaged_users       => [ 'root', ],

  development_mode           => true,

  conf_node_external_eth_dev => 'eth0',

  install_method             => 'yum',

  install_login_shell        => true,

  register_host_with_named   => true,

  broker_auth_plugin         => 'htpasswd',

  broker_dns_plugin          => 'avahi',

  jenkins_repo_base          => 'http://pkg.jenkins-ci.org/redhat',

  bind_key                   => "Abagu+ub1SDFS9n5cig==",

  activemq_admin_password    => 'scrambled',

  install_cartridges         => [

    'cron',

    'jenkins',

    'nodejs',

    'jbossas',

  ],

}

 

Any advice is appreciated!

 

John Skarbek, System Administrator 

 


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]