[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Custom plugin for DNS question



Sorry, Luke, I missed "reply-all"  on one of my comments.

Kyle, this highlights two things:

I think the TTL issue is separate from your observations about the Rackspace response to your "create record" query.  It's important to be aware of, but I don't think it's the current problem... directly.

I'm looking now to see if I can find the broker code which polls the DNS to verify that a new app CNAME is available and to see if the query host is configurable.

- Mark


On Fri, Feb 14, 2014 at 1:11 PM, Kyle Crumpton (kcrumpto) <kcrumpto cisco com> wrote:
From Rackspace: 

"From the time a change is made in the customer portals or internal
tools until it propagates to all of our nameservers globally is almost
always under 60 seconds.  However, the TTL of a record is very
important.  If you have a 24 hour TTL and you make a change, regardless
of whether you drop the TTL at the exact same time, you will have to
wait up to 24 hours for it to propagate.  For a 5 minute TTL, you will
wait 5 minutes, and so on."

From: Mark Lamourine <markllama gmail com>
Date: Friday, February 14, 2014 10:43 AM

To: Kyle Crumpton <kcrumpto cisco com>
Subject: Re: Custom plugin for DNS question




On Fri, Feb 14, 2014 at 11:29 AM, Kyle Crumpton (kcrumpto) <kcrumpto cisco com> wrote:
That is pretty interesting. So to make sure I understand the issue; 
The 300sec TTL window is leaving much too large a time frame.. Then trying to do things in that same timeframe? AKA

testapp1-ns1.example.com CNAME IN node.example.com propogates to RackspaceNS1 with a TTL of 300, then immediately tries to propagate to broker.example.com, and fails the latter?

More like this:

Broker sends "create record" request to Rackspace: please add testapp1-ns1.example.com CNAME node.example.com (with incidental TTL 300).

Rackspace has to stick that in the database for the nameservers listed as the NS records for example.com.  Adding records changes the SOA serial number for example.com (indicating that there has been a change)  Downstream (caching) nameservers will only check for changes when the TTL for a record expires, so there's some latency in any changes.

The record doesn't "propagate to the broker".  The broker makes active checks of the DNS service by issuing a sequence of requests for testapp1-ns1.example.com to *some* nameserver (generally one in /etc/resolv.conf on the broker, but I *think* configurable).  The right nameserver to check is one of the servers listed in the NS records for example.com.  These are the primary servers and will get zone updates immediately with no TTL timeout latency either for the SOA or CNAME records.  If Rackspace is updating the primary servers immediately, the broker queries should detect the record presence as soon as Rackspace claims to have completed the create request (you may have to poll Rackspace to know when IT thinks it's completed the transaction)

That's really the best the broker can do. The client may still see record caching latency as it is most likely querying a nearby caching DNS server rather than the primary servers for the app zone.

 In short, there are a number of moving parts each of which can be the cause of a propagation delay (or the appearance of one). 

I haven't had these issues with Route53.  Online has had some issues around this with dynect, but I think the code is in to manage them. I don't think the dynect plugin has been published.

I'll try to get a min today to look at the config and the code to see if there's a config value available to set the DNS server to be used for propagation test queries by the OpenShift broker during app create/delete.

- Mark

From: Mark Lamourine <markllama gmail com>
Date: Friday, February 14, 2014 9:50 AM

To: Kyle Crumpton <kcrumpto cisco com>
Subject: Re: Custom plugin for DNS question

I'm pretty sure there are ways to manage this.  I'm fairly certain that the rhc CLI app has a switch to let the app creation run async.

As far as the broker ensuring that DNS is propagated before marking the app as successfully created, you're seeing the repeated DNS tests.  It works best if the DNS tests are running CNAME queries against the primary NS hosts for the app zone.  I'd have to look if there's a config option to indicate what the NS hosts are for that test.

The TTL really only becomes a problem if you try to create/remove/recreate an app with the same name in time frames near or below the TTL value.  Cycling a particular app in under 15 min with a 300sec TTL will be a problem.

One think that's not clear is how Rackspace responds to a record delete request submitted before the create request has propagated.

- Mark


On Fri, Feb 14, 2014 at 10:43 AM, Kyle Crumpton (kcrumpto) <kcrumpto cisco com> wrote:



From: Mark Lamourine <markllama gmail com>
Date: Friday, February 14, 2014 8:37 AM
To: Kyle Crumpton <kcrumpto cisco com>
Subject: Re: Custom plugin for DNS question

Kyle: one interesting thing is that you were seeing (in another email thread) that Rackspace wasn't accepting TTL less than 300 seconds.

That could be a problem for openshift if Rackspace doesn't promise to update and propagate new records (or record deletions) in a timely fashion.  Openshift really needs a propagation response (at least for new apps) of under 30 seconds in general.  It might make sense to ask Rackspace what their update characteristics will be and if they're tunable.

Now, the TTL requirement and the change propagation (at least to Rackspace's primary DNS servers) may not be related.  It may also be necessary to add a config to OpenShift DNS plugin which allows OpenShift to test a specific (set of) DNS servers (the service primary servers) so that it can eliminate zone transfer propagation and negative cached records.  I *think* there may already be an argument in the openshift config for that, though I'm not sure if it's global or if it was part of a specific plugin that I don't remember.

Luke's comment is also to the point, the error your getting seems to come either from the Fog library or from Rackspace itself in response to the change request.  In either of those cases the only place you can affect it is in the logic of the DNS plugin.  The trick now is to see if you can isolate it more and reproduce it in the plugin test code.

Sorry to not be much help.

- Mark


On Thu, Feb 13, 2014 at 5:07 PM, Kyle Crumpton (kcrumpto) <kcrumpto cisco com> wrote:
Hi all

Working on a plugin for DNS. I am running into an error when I try to create an application with it added:

[root broker kcrumpto]# rhc app-create test cisco-dsx-0.1
Application Options
-------------------
  Domain:     dsx
  Cartridges: cisco-dsx-0.1
  Gear Size:  default
  Scaling:    no

Creating application 'test' ... 
Unable to complete the requested operation due to: Wait on job e0729da9-517d-4d91-9729-fb157537d693 took too long.
Reference ID: 845e15cdc80272f83f2e35991ace2e0b


The plugin is designed with the intent of propagating appropriate CNAME entry to Rackspace's DNS records when an application is created.
The problem now is that when I try to create an application I receive the above error. My code for this plugin can be found here: https://github.com/kcrumpto/origin-server/blob/master/plugins/dns/rackspace/lib/openshift/rackspace_plugin.rb

Any insight appreciated..

Thank you and BR

Kyle 


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users




--
----
Mark Lamourine <markllama gmail com>
Dad, Hubbie, Software Developer, System Administrator, Road Cyclist



--
----
Mark Lamourine <markllama gmail com>
Dad, Hubbie, Software Developer, System Administrator, Road Cyclist



--
----
Mark Lamourine <markllama gmail com>
Dad, Hubbie, Software Developer, System Administrator, Road Cyclist



--
----
Mark Lamourine <markllama gmail com>
Dad, Hubbie, Software Developer, System Administrator, Road Cyclist

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]