[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: DNS support added to OpenShift

On 03/09/2015 11:16 AM, Clayton Coleman wrote:
> Yes - it's also to remove the need for you to use environment variables at all.  If you have a DB service you're expecting, just reference "db.local" in your config files, or set your own custom env var.  Ordering is a big one - you can launch a pod that refers to "db.local" before the "db" service is created, and it will eventually resolve.  After all, DNS is the original service-discovery.
> Eventually, this will support things like headless services, so you can run clustered software - imagine a Replication controller that gives each pod a unique label "member=<number>" and ensures that there's always a 1, 2, 3, etc.  Then imagine a service called "mongodb" that is enabled to support this.  The name:
>    mongodb.mynamespace.local
> would resolve to multiple A records - one for each pod.  But the service would also expose a DNS name for each named member:
>    1.mongodb.mynamespace.local
>    2.mongodb.mynamespace.local
>    3.mongodb.mynamespace.local
> The DNS value for each member would resolve to the pod directly - if a pod gets deleted, and recreated, then eventually the DNS name (assuming you have a properly configured resolve.conf that honors TTL) would pick up that 1.mongodb.mynamespace.local changed from to  This isn't a final implementation of course.

That sounds rather dangerous. What happens when a container looks for a
service endpoint via DNS that no longer exists? What is the TTL for
these skydns-based entries?

This sounds like we're going down the same path of "dns for ha" which
does not work because of TTL and/or performance of lookups.

> Also, there's the potential with DNSSEC to distribute certificates for services to other components - using the trusted local domain to distribute the public key for lots of services means you can do less configuration.  And for things like SRV requests and TXT records we can also conceivably expose other info.
> ----- Original Message -----
>> +++ Luke Meyer [09/03/15 11:03 -0400]:
>>> It probably seems obvious to you, but... in addition to the "what" and "how"
>>> could you talk a bit about "why"? What are the main use cases where this
>>> comes into play?
>> I'm not sure if this is the main motivation but I could see this
>> keeping pods from having to restart when services come and go.
>> eg, Today with the sample app
>> (https://github.com/openshift/origin/tree/master/examples/sample-app)
>> if you create the frontend first and then later create the database
>> you have to restart (or recreate) the frontend pod to pick up the
>> service addition.  That's because the only interface for injecting the
>> service is an environment variable ultimately set when Docker launches
>> the container.
>> --Brenton
>>> ----- Original Message -----
>>> From: "Clayton Coleman" <ccoleman redhat com>
>>> To: dev lists openshift redhat com
>>> Sent: Sunday, March 8, 2015 9:17:05 PM
>>> Subject: DNS support added to OpenShift
>>> Pull https://github.com/openshift/origin/pull/1254 adds split-horizon DNS
>>> inside the cluster by running a SkyDNS server on each master that can
>>> answer DNS queries for services.
>>> By default, when the master starts it will start listening on port 53.  It
>>> will bind to the same addresses the master is configured for, and in an
>>> all-in-one the Kubelet will use that address to query for requests.  When
>>> the node is started in a real cluster (openshift start node) the Kubelet
>>> will look up the "kubernetes" service and assume the first endpoint for
>>> that service can be used to talk DNS.  If you are running OpenShift on top
>>> of Kube you'll need to change the kubelet --cluster-dns parameters to point
>>> to the OpenShift master.
>>> On startup of a node the presence of this message indicates the Kubelet
>>> correctly resolved the master:
>>> 0308 19:51:03.118430    4484 node.go:197] Started Kubelet for node
>>> openshiftdev.local, server at
>>> I0308 19:51:03.118459    4484 node.go:199]   Kubelet is setting as
>>> a DNS nameserver for domain "local"
>>> If you don't see the second message, the "kubernetes" service may not be
>>> available.
>>> NOTE: If you are not running as root when you start OpenShift, you can't
>>> bind to port 53. The master will instead use 8053, but because libc name
>>> resolution doesn't have a way to use custom ports you can only query
>>> directly against the master with tools like dig, and cluster DNS won't
>>> work.  Unless you're on Mac, which allows custom ports in resolve.conf, but
>>> Macs can't run Docker anyway, so the rabbit hole stops there.
>>> How does cluster DNS work?
>>> --------------------------
>>> When nodes are properly configured, each Docker container gets an additional
>>> nameserver (the master) added to the front of its nameserver list, and the
>>> default search domain for the container will be ".<pod_namespace>.local".
>>> The container will then direct nameserver queries to the master first,
>>> before using any other nameservers configured on the node (the Docker
>>> default behavior).
>>> The master will answer queries on the ".local" domain that have the
>>> following form:
>>>    <namespace_name>.local
>>>    <service_name>.<namespace_name>.local
>>> SkyDNS will respond with a single A record for any service you ask for that
>>> contains the service Portal IP.  Since that IP is resolvable within your
>>> containers, you'll get automatic DNS for services.  For example, if you're
>>> in namespace "foo" and you have a service "bar", any pods in your namespace
>>> will be able to:
>>>    $ dig foo.bar.local. A
>>>    # ....
>>>    foo.bar.local.	30	IN	A
>>> You can also get SRV, CNAME, and do wildcard lookups. Reverse lookups for a
>>> service portal IP will resolve to the CNAME of the service
>>> ("foo.bar.local"). Also, because the search domain is set, you can lookup
>>> "foo" or "foo.local" and see similar results.
>>> SkyDNS is set up to forward any queries outside of .local to the nameservers
>>> set in your master (so configure /etc/resolv.conf normally).  You can
>>> populate the etcd schema for SkyDNS (/skydns/*) and it should also match
>>> those values, so you can set up custom queries with the full power of
>>> SkyDNS (weights, priority, etc).
>>> Future work may be to implement headless services (service will respond with
>>> multiple A records, one for each pod), allow routes to resolve internally,
>>> allow an external delegated domain to resolve to OpenShift, and to enable
>>> DNSSEC (which SkyDNS fully supports but I have not yet enabled).
>>> There's probably some rough edges still, so folks with deeper DNS experience
>>> than I please get involved and try out the support.
>>> _______________________________________________
>>> dev mailing list
>>> dev lists openshift redhat com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>>> _______________________________________________
>>> dev mailing list
>>> dev lists openshift redhat com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]