[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Help with Scaled Cartridges Racing (mariadb galera)



On Wed, Feb 26, 2014 at 9:42 AM, John Hawley <jhawley redhat com> wrote:
What would almost make more sense (and assuming a small cartridge
change) is some mechanism to declare a pre-scale set of events for a
cartridge, let that cartridge do some things and then once that's
completed any other cartridges would then be allowed to spawn.  Kind
alike an rpm pre section.  A thought anyway.

That'd be a good concept to consider, another issue I ran into was the timeout of cartridge building so adding these kind of waits may need some sort of allowance with the custom cartridge timeout.
 

The election system might work, even for less than 3 nodes but in this
case you only need to elect a master once, ever.  I'm also not sure how
having an explicit mater / slave type cartridge is any less easy to use,
and has the advantages of being explicit that setup has been completed
and you can guarantee the cluster is in a state that should work.

Well, galera likes to have at least 3 to avoid the split brain scenario so I'm really leaning towards that min 3 election style setup. The reason why I want to avoid a master/slave style is later on when we spin up our origin deployment into production, I'd want to package this up as a RPM and not have to bother about maintaining multiple cartridges etc. But if the first method fails, it's good to know I can fall back to another workable method.
 

- John

On 02/25/2014 12:52 PM, Andrew Lau wrote:
> Thanks for the sum up, makes a lot of sense now. I'm leaning towards
> that method as the cluster would require atleast 3 gears.
>
> On Wed, Feb 26, 2014 at 7:40 AM, Ben Parees <bparees redhat com
> <mailto:bparees redhat com>> wrote:
>
>     Seems like Clayton's suggestion is the right way to go.  Assuming
>     you know the initial number of gears (3 in your case), you can wait
>     until you have publications from 3 unique gears, then sort them by
>     id and elect a master by lowest id.  You'll then want to publish
>     that master so additional gears (scale out) know the master.
>
>     If you don't know the initial number of gears, things certainly get
>     trickier.  I'm not sure if openshift exposes any information to the
>     cartridge itself, about how many gears exist for that cartridge.
>
>
>     Ben Parees | OpenShift
>
>     ----- Original Message -----
>     > From: "Andrew Lau" <andrew andrewklau com
>     <mailto:andrew andrewklau com>>
>     > To: "John Hawley" <jhawley redhat com <mailto:jhawley redhat com>>
>     > Cc: dev lists openshift redhat com
>     <mailto:dev lists openshift redhat com>
>     > Sent: Tuesday, February 25, 2014 3:07:30 PM
>     > Subject: Re: Help with Scaled Cartridges Racing (mariadb galera)
>     >
>     > On Wed, Feb 26, 2014 at 5:29 AM, John Hawley < jhawley redhat com
>     <mailto:jhawley redhat com> > wrote:
>     >
>     >
>     >
>     > On 02/25/2014 01:47 AM, Andrew Lau wrote:
>     > > Hi guys,
>     > >
>     > > I've been working on bringing MariaDB Galera [1] to OpenShift
>     and so far
>     > > I've quite successful with the help from many different people. I'm
>     > > stuck on a few things though which I'm hoping someone could help
>     me with.
>     >
>     > Just a general word of warning, I found Galera clustering to have some
>     > rather nasty rough edges, particularly with the possibility of some
>     > transactions never actually making it out to the other nodes. I
>     > confirmed that pathology with Monty, and I don't know if there's
>     been a
>     > resolution to it yet. I'm not trying to deter, just trying to make it
>     > known that Galera is not a solve-all solution.
>     >
>     >
>     > Thanks for the words of warning :)
>     >
>     > I've been using MariaDB Galera clusters for quite some time and
>     haven't run
>     > into any issues, as of yet. *knock on wood*
>     > If you think the out of sync masters will be a common issue, we
>     can look into
>     > some sort of logic to force sync the SST procedure every so often
>     (possibly
>     > with cron?)
>     >
>     >
>     > > *- Determining a master gear for the database initialization*
>     > > With the galera cluster, we need to have one gear create the
>     database,
>     > > drop test user. etc. all the normal stuff, however only one
>     needs to do
>     > > this. The problem is, starting the cartridge with min 3 gears,
>     they will
>     > > all race to start, this makes using a hook method not possible
>     because
>     > > there'll be too many inconsistencies. I'd ideally not want to be
>     > > creating two separate cartridges just for this.
>     >
>     > You are probably going to have to solve this similarly to the way I
>     > solved it with the original master/slave stuff and have an explicit
>     > "init" cartridge that has that logic in it, and then instantiate a
>     your
>     > "slaves". The init node obviously shouldn't be scalable, but you can
>     > make the "slaves" scale at will at that point. If you make a
>     dependency
>     > for the slaves to require the init that should prevent anything too
>     > weird, and prevent a race for the "slaves" to come up before the init
>     > has completed and settled.
>     >
>     > Not ideal, obviously, but I don't know of anyway to prevent that race
>     > condition you are describing, and it's not a dissimilar one to what I
>     > was seeing with the master/slave stuff.
>     >
>     > I was originally following your master-slave style cartridge, and
>     that's what
>     > gave me some of the initial inspiration. However, I wasn't a fan
>     of having
>     > to use two cartridges and the manual intervention required to get
>     it going.
>     > I want this cartridge to be as simple as deploying any other official
>     > openshift cartridge.
>     >
>     >
>     >
>     >
>     > - John 'Warthog9' Hawley
>     >
>     >
>     > _______________________________________________
>     > dev mailing list
>     > dev lists openshift redhat com <mailto:dev lists openshift redhat com>
>     > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>     >
>
>
>
>
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]