[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Help with Scaled Cartridges Racing (mariadb galera)


*insert victory song here*

My mariadb galera cluster is functional and alive!! The cartridge still needs some work on logic and code clean up, but for now I'll be going to sleep happy. 

After my code clean up and testing (ie dynamic passwords), my next item to bug you guys with will be haproxy =)

Thanks everyone for your help.

On Thu, Feb 27, 2014 at 11:18 PM, Andrew Lau <andrew andrewklau com> wrote:
Ignore that, I've got it working! 

Still some minor issues, code cleaning and logic to fix up.

Thanks for the help.

On Thu, Feb 27, 2014 at 7:57 AM, Andrew Lau <andrew andrewklau com> wrote:
I think I may have hit a roadblock..

The galera cluster config will only allow me to specify a ipaddress/host to bind to, but that is also the host which seems to announce it self as. With OpenShift's architecture, I need it to bind to the loopback address but broadcast the public address. Typically the case could be to use hosts files, but that presents two issues:

- No way to specify the NAT port changes.. although it may work without that
- HOSTALIASES will only work from fakehost->realhost, I can't specify fakehost->loopback-address

Has anyone got any suggestions?


On Wed, Feb 26, 2014 at 7:09 PM, Andrew Lau <andrew andrewklau com> wrote:
On Wed, Feb 26, 2014 at 7:04 PM, Romain <filirom1 gmail com> wrote:

2014-02-26 2:15 GMT+01:00 Andrew Lau <andrew andrewklau com>:

On Wed, Feb 26, 2014 at 11:18 AM, Clayton Coleman <ccoleman redhat com> wrote:

On Feb 25, 2014, at 6:42 PM, Andrew Lau <andrew andrewklau com> wrote:

I've got a quick question in regards to the logic in this..

So we start up 3 gears, they will all go through and run their:
- setup
- install
- post_install
- publish_hooks
- subscribe_hooks

At this stage, we already lost the ability to post a client_result?

Subscribe hooks can't publish client results, but start can.

However, I'm thinking it would be better off to provide a dynamic ENV variable anyway to accommodate for the dynamic gears.

Then subscribe_hooks we will check if there's at least 3 gears who have announced their presence. Once the minimum 3 has been met, we will do a conversion to determine the lowest ID of the three, and determine that as the "master". That "master" will then do the initial DB setup (ie. create table, drop test user etc.) and then start the cluster as the first host. 

So then we mark that this DB step has been initialized, and let the other gears know that they can join the cluster. Is it possible to then execute another publish hook to let the other gears know "hey, I'm ready for you to join me?"

There is not.  All the gears can reach the same conclusion, and on start try to connect to the master if it's ready.  You can also just make master election depend on an app env var value and have the start step check for it.  The subscribe hook could also do a restart and wait for the master to be available - that has other issues obviously.  

Thanks for the tip. Turns out there's a lot more logic to be thought through as well proper understanding of the underlying architecture.
Few more pestering question:

- When a new gear get added, this new gear should receive the publish_hooks from the other gears?
- Per discussions with John (warthog9), is anyone able to comment on gear idling? The current fear is possible data corruption with gears idling and the cluster having issues. To my understanding the web cartridge will idle after 2 days of no activity on the webproxy but what is the situation with database cartridges. Are we able to schedule pings to keep it alive or something?

Some gears may have a marker file ~/.disable_stale. These gears are supporting other gears and should never be idled

Thanks for the link. Would one just need to "touch" that file during the install process?

Quick Update:
I've followed through with the logic that was discussed on IRC earlier today or last night for some of you. I've successfully been able to spin up the min-3 instances and determine a "master" to provision the systems. Running into an issue with connecting the cluster, but that's more of a galera config issue I'm trying to debug now.


On Wed, Feb 26, 2014 at 7:40 AM, Ben Parees <bparees redhat com> wrote:
Seems like Clayton's suggestion is the right way to go.  Assuming you know the initial number of gears (3 in your case), you can wait until you have publications from 3 unique gears, then sort them by id and elect a master by lowest id.  You'll then want to publish that master so additional gears (scale out) know the master.

If you don't know the initial number of gears, things certainly get trickier.  I'm not sure if openshift exposes any information to the cartridge itself, about how many gears exist for that cartridge.

Ben Parees | OpenShift

----- Original Message -----
> From: "Andrew Lau" <andrew andrewklau com>
> To: "John Hawley" <jhawley redhat com>
> Cc: dev lists openshift redhat com
> Sent: Tuesday, February 25, 2014 3:07:30 PM
> Subject: Re: Help with Scaled Cartridges Racing (mariadb galera)
> On Wed, Feb 26, 2014 at 5:29 AM, John Hawley < jhawley redhat com > wrote:
> On 02/25/2014 01:47 AM, Andrew Lau wrote:
> > Hi guys,
> >
> > I've been working on bringing MariaDB Galera [1] to OpenShift and so far
> > I've quite successful with the help from many different people. I'm
> > stuck on a few things though which I'm hoping someone could help me with.
> Just a general word of warning, I found Galera clustering to have some
> rather nasty rough edges, particularly with the possibility of some
> transactions never actually making it out to the other nodes. I
> confirmed that pathology with Monty, and I don't know if there's been a
> resolution to it yet. I'm not trying to deter, just trying to make it
> known that Galera is not a solve-all solution.
> Thanks for the words of warning :)
> I've been using MariaDB Galera clusters for quite some time and haven't run
> into any issues, as of yet. *knock on wood*
> If you think the out of sync masters will be a common issue, we can look into
> some sort of logic to force sync the SST procedure every so often (possibly
> with cron?)
> > *- Determining a master gear for the database initialization*
> > With the galera cluster, we need to have one gear create the database,
> > drop test user. etc. all the normal stuff, however only one needs to do
> > this. The problem is, starting the cartridge with min 3 gears, they will
> > all race to start, this makes using a hook method not possible because
> > there'll be too many inconsistencies. I'd ideally not want to be
> > creating two separate cartridges just for this.
> You are probably going to have to solve this similarly to the way I
> solved it with the original master/slave stuff and have an explicit
> "init" cartridge that has that logic in it, and then instantiate a your
> "slaves". The init node obviously shouldn't be scalable, but you can
> make the "slaves" scale at will at that point. If you make a dependency
> for the slaves to require the init that should prevent anything too
> weird, and prevent a race for the "slaves" to come up before the init
> has completed and settled.
> Not ideal, obviously, but I don't know of anyway to prevent that race
> condition you are describing, and it's not a dissimilar one to what I
> was seeing with the master/slave stuff.
> I was originally following your master-slave style cartridge, and that's what
> gave me some of the initial inspiration. However, I wasn't a fan of having
> to use two cartridges and the manual intervention required to get it going.
> I want this cartridge to be as simple as deploying any other official
> openshift cartridge.
> - John 'Warthog9' Hawley
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

dev mailing list
dev lists openshift redhat com

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]