[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Custom Cartridge Help (MariaDB Galera Cluster)



Hi,

I finally had some time to get around to looking at this again, I've somewhat got some success with this!

I'm happy to say, I've got mariadb working on OpenShift Online (as a single gear) however I don't have enough free gears to fully try the galera functionality. I still need to clean up the code a lot, and I don't think some of the variables/hooks are working properly but in theory with a bit of tinkering this could be the start of a mariadb galera cluster on openshift!
I'm having problems with the VM (laptop issues) and our own Origin deployment is behaving a bit funky which I'll need to debug a little later. 

However - if anyone has got some free time could they take it for a whirl [1]. 

Couple of things:
 - It needs to be behind a front end, for now... ie. PHP (I wonder if it's possible to expose the mysql port proxied through port 80, then apply an ACL so one galera cluster can serve multiple applications)
 - I'm not sure how to properly do the minimum 3 gear quorum as deploying 3 at the same time would definitely time out but to have a proper galera cluster it enforces 3 gears/servers. Please let me know if the ENV variables and hooks get updated correctly and (conf/my.cnf) gets the correct variables.
- I use `nc` within my script, which I just realized is not available in the origin VM or online nodes, however telnet seems to be. Am I safe to assume telnet all online nodes have telnet available?

What works: mariadb on OpenShift Online (single gear), I haven't been able to test multiple gears due to limitations mentioned above.

Thanks again for all your help.

[1]  https://github.com/andrewklau/openshift-galera-mariadb


On Thu, Jan 30, 2014 at 12:26 AM, Clayton Coleman <ccoleman redhat com> wrote:
If you host your binaries on S3 or inside another app you should get much faster downloads... How big are they?

On Jan 29, 2014, at 8:22 AM, Andrew Lau <andrew andrewklau com> wrote:

On Thu, Jan 30, 2014 at 12:20 AM, Clayton Coleman <ccoleman redhat com> wrote:


On Jan 29, 2014, at 3:40 AM, Andrew Lau <andrew andrewklau com> wrote:

Hi Mateus,

Thanks for all your help. I've somewhat nearly got there, but having problems testing it because of the binary file being quite large and openshift only having a finite execution time for third party cartridges.

Is most of the creation time taken up by downloading the cart binaries ?  Or by compilation / other?


Downloading the binaries.

If anyone's got any suggestions on a workaround for that limit would be much appreciated

My source code is here:

Cheers,
Andrew


On Mon, Jan 27, 2014 at 10:15 PM, Mateus Caruccio <mateus caruccio getupcloud com> wrote:

Mateus Caruccio
Master of Puppets
+55 (51) 8298.0026
gtalk: mateus caruccio getupcloud com
twitter: @MateusCaruccio

This message and any attachment are solely for the intended
recipient and may contain confidential or privileged information
and it can not be forwarded or shared without permission.
Thank you!


On Mon, Jan 27, 2014 at 9:06 AM, Andrew Lau <andrew andrewklau com> wrote:

On Fri, Jan 24, 2014 at 11:16 PM, Andrew Lau <andrew andrewklau com> wrote:
On Fri, Jan 24, 2014 at 11:09 PM, Mateus Caruccio <mateus caruccio getupcloud com> wrote:

On Fri, Jan 24, 2014 at 6:09 AM, Andrew Lau <andrew andrewklau com> wrote:

On Fri, Jan 24, 2014 at 12:45 AM, Mateus Caruccio <mateus caruccio getupcloud com> wrote:
Hi.


On Thu, Jan 23, 2014 at 4:13 AM, Andrew Lau <andrew andrewklau com> wrote:
On Thu, Jan 23, 2014 at 4:53 PM, Clayton Coleman <ccoleman redhat com> wrote:


On Jan 22, 2014, at 11:03 PM, Andrew Lau <andrew andrewklau com> wrote:

Hi,

I was dissecting my possibility of running a MariaDB Galera cluster within OpenShift a few days ago, but I got stuck on a few things I was hoping someone could help me with:

I started here if anyone is interested in helping https://github.com/andrewklau/openshift-galera-mariadb

- Is there a way to have an environment variable of some sort which could list all the gears running? I would like to create the whole cartridge similar to a web-app style so that it can scale infinitely. >From what I understand, environment variables set are global across all the gears upon first run, but does that mean they can be dynamically updated? I need this to write some logic to prevent a split brain. This is because the cluster requires the first "gear" to be brought up with the slightly different command. 

Currently the way to get this information is to use a connection hook (which gives you variables from all of the gears) and then do the same configuration on each gear.  If you look at the redis cart you can see an example of the parsing of the data, although we'd like to make this simpler in the future by piping JSON to the stdin of the hook.  For now, the example below demonstrates breaking the info up and then using it to assign a master (the gear with the lowest alphabetical ordered uuid the first time the cluster starts up).



Thanks for the reply. I have no clue where to start with ruby, tbh none of it actually makes sense to me :( Would I just need to modify a few variables to grab just the output? With the reddis cartridge, what happens if all the nodes crash and are brought up all at the same time or the master crashes and the other nodes are running. I haven't used reddis, but with mariadb this would cause issues which would need some sort of logic or manual intervention.

Take a look at these cartridge hooks:

- Runs on every gear when a new gear is added. Everything printed to stdout is temporally stored by openshift to feed all subscribers of this hook.

- Runs on every gear with stdin from publishers' stdout (hat publish-es-node printed). Stdin on subscriber hooks are as follow:
[app-name] [namespace] [local-gear-uuid] [stdout-from-publisher]
Note there are only 4 parameters.
Stdout from publisher is a single parameter. Its format is "[gear-uui]=[publisher-stdout-from-that-gear] [gear-uui]=[publisher-stdout-from-that-gear] ..."
I suggest you to use a non-space separator on publisher. That way is more simple to parse it on subscriber (see set-es-node:11)
 
In fact, hooks can written in any langague, since it is available on the node level.

Thanks a lot for that example, mind if I reuse it?


Absolutely! This is why it's on github for :)


Awesome - cheers!

 
 
Two questions, how are you handling scaling down? And are you able to handle a crashed node? 

Scale down is just like scale out. Both events trigger publishing hooks. This cartridge doesn't need to handle scale down since elasticsearch does that already.
 

Ah ok, I'm going to have to do some thinking on the logic for mariadb.
 

- Can anyone explain a little further on how the inter-gear networking works? Looking at this post on Master-Slave it appears there doesn't seem to be anything special required which is really confusing me about the whole multi-tenancy concept https://www.openshift.com/blogs/cloudy-with-a-chance-of-data-master-slave-database-replication-on-openshift 

- Is it possible to use the haproxy routing to loadbalance the requests to the cluster? I'm not so familiar with haproxy, but I'm assuming as it can run like a tcp loadbalancer similar to LVS (which I currently use for my external cluster) I don't see why it couldn't work. It'd obviously need a custom port?

Scalable apps are load-balanced by default, but for HTTP only. There is a second haproxy on the node routing tcp connections inter-gears. The entry point is $OPENSHIFT_GEAR_DNS.
I don't think it is possible to accept external connections other than http[s] on app's haproxy.


This may be interesting, is $OPENSHIFT_GEAR_DNS unique per scaled gear? When you say external connections does that include other apps within the openshift environment? eg. App1 mariadb and app2 php

Yes, It's unique since has gear UUID on it.
External connections are any connection external to gear itself: another app or somewhere outside openshift


Hmm... this makes it hard as I was planning on building the cluster to be shared with multiple PHP applications (within openshift).

 
 
 
- I think I read somewhere, that by default the gears will try to do negative affinity by default when scaling out? eg. be hosted on a separate node. Is that the default behavior, or am I thinking of something else?

AFAIK, broker chooses the least full node. Could someone please confirm this?
- Am I wrong to assume this won't work with Online as it requires some external dependencies from the MariaDB repo (not SCL)?

You could deploy mariadb binary along with your cartridge.


Where would I need to save these binary files? It also a few dependencies would I need to package those up too?

Create a versions/<version-number>/ and install mariadb there, with any mariadb necessary libs (some may be installed on host already). Config file and/or startup script may be edited so mariadb prefix dir point to this directory.


I'll give that a try.

Do you know if there's a link lurking around somewhere which explains this procedure? I'm a little confused about how to do this.

 
 
When done, would you mind sharing it here?


My repo is here, but it's very much just learning and planning stages still https://github.com/andrewklau/openshift-galera-mariadb

 
 





_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]