[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: HA applications - and docs

Awesome write up! Thanks for clearing a lot of the confusion.

On Fri, Jul 25, 2014 at 12:25 AM, Luke Meyer <lmeyer redhat com> wrote:
Sorry for the delay, but I did finally write up that blog post:

Questions/comments there or here are welcome :)

----- Original Message -----
From: "Andrew Lau" <andrew andrewklau com>
To: "Luke Meyer" <lmeyer redhat com>
Cc: "Openshift Dev" <dev lists openshift redhat com>
Sent: Monday, July 14, 2014 9:50:03 AM
Subject: Re: HA applications - and docs

The answer of silence!

I was curious in understanding how the in built HA option works, based
on one of the previous (now 404) PEP's it mentioned components such as
moving the loadbalancer outside of gears and onto the nodes, also
using activemq routing and something like Nginx (ie. [1] and [2])

They are cool and all, but no 'out of the box' solution yet. Based on
what you were saying before, even just enabling HA opens it to a whole
new world of things like multiple HAProxy gears. So has the concept of
removing the haproxy gear no longer part of the HA capabilities?

I'm trying to get a better picture of where the point of failures are,
and how the 'enable ha' option changes that. ie. if you have multiple
haproxy gears, are you then relying on DNS to loadbalance between the
gears, what happens if one drops etc. So now that HA has been
'implemented' into things like ActiveMQ, mongo etc as part of the
broker support services. Can we tolerate a node failure?

Sorry if I started to ramble a bit, I just think HA is just such an
interesting space..

[1] https://github.com/worldline/openshift-nginx-routing
[2] https://github.com/openshift/openshift-extras/tree/enterprise-2.1/admin/routing-listeners

On Mon, Jul 14, 2014 at 10:16 PM, Luke Meyer <lmeyer redhat com> wrote:
> Well, silence is a kind of answer :)
> I filed a bug to remind us to update Enterprise docs at least. I may kick the process off with a blog post at some point soon, time permitting. I know a fair bit now - have any specific questions?
> ----- Original Message -----
> From: "Andrew Lau" <andrew andrewklau com>
> To: "Luke Meyer" <lmeyer redhat com>
> Cc: "Openshift Dev" <dev lists openshift redhat com>
> Sent: Sunday, July 13, 2014 8:59:33 PM
> Subject: Re: HA applications - and docs
> Hi Luke,
> Curious, did you get any answers for your questions?
> On Tue, Jun 24, 2014 at 1:14 AM, Luke Meyer <lmeyer redhat com> wrote:
>> There seems to be a dearth of documentation on highly available applications. I hope I'm just missing it.
>> I had a couple specific questions and so far I've just had to dig around and experiment. I'm guessing in all cases looking at the code would probably clarify things, but is that what we expect customers to do?
>> 1. How do I determine which gears become load balancers? If I make an existing scaled app HA, how do I add LB gears?
>> What appears to happen is that haproxy cartridges are added to gears as the framework cartridge scales. So if I scale up an app to say 4 gears (which of course has only one haproxy gear), and then make it HA... nothing happens until I add the next gear. Nothing ever re-provisions an existing gear. Also if you then scaled back down... gears are removed in reverse order, so you'd lose the second haproxy gear.
>> New gears are made haproxy gears in accordance with the HA multiplier (default 3), except that if you scale from 1 to 2 gears, the second gear gets haproxy. No way to control this that I can see.
>> 2. What happens with traffic as the app scales? Are framework cartridges disabled on haproxy gears?
>> As the PEP indicates, all haproxy instances apparently route to all framework gears; except that the gears that house haproxy instances themselves are rotated out at some point. This doesn't result in the framework instances actually shutting down - they are just removed from rotation via haproxy conf and get no traffic. Interestingly, with a standard HA app, this results in a counter-intuitive scale *down* at one point where scaling "up".
>> If you start with an ordinary scaled app, gear 1 is rotated out after bringing up gear 3, meaning only gears 2 and 3 are actually getting traffic (so scaling from 2 to 3 is a "horizontal scale").
>> If you start with a HA app, gears 1 and 2 get haproxy instances. Gears 3 and 4 start framework instances as usual (so at gear 4, there are 4 framework gears in rotation). After gear 5 is started, gears 1 and 2 are taken out of rotation, leaving only 3 - an anti-scale-up. This seems... surprising.
>> ==============
>> The docs for the HA app use case are... pretty sparse. I'm just wondering how we expect customers to find out about all this.
>> There's the HA PEP, which is mostly interesting for historical curiosity at this point, as there's no way to tell which parts were actually implemented and how, other than deep diving in Trello:
>> http://openshift.github.io/documentation/openshift-pep-005-available-web.html
>> Enabling users and broker to do HA apps is described in the Deploy guide and (redundantly) in the Admin guide:
>> https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Deployment_Guide/index.html#sect-Using_an_External_Load_Balancer_for_High-Availability_Applications
>> https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Administration_Guide/index.html#Enabling_Support_for_High-Availability_Applications
>> The only mention of how to actually make an app HA is buried in the API guide:
>> https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/REST_API_Guide/index.html#Make_an_Application_Highly_Available_HA
>> I thought you could just scale up the haproxy cartridge explicitly, but you can't. "rhc cartridge scale" reports that it is not scalable.
>> Changing the HAproxy multiplier on an app is, bizarrely, an administrative action not available to the user, and mentioned obscurely without explanation in the admin guide entry for oo-admin-ctl-app (while the other functionality for this command is not described at all):
>> https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Administration_Guide/index.html#oo-admin-ctl-app
>> Do we have any docs describing what the "gear" command does and use cases (or that it even exists)?
>> Are any of the above documented in Origin docs or alongside the code where I've just missed it?
>> _______________________________________________
>> dev mailing list
>> dev lists openshift redhat com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]