[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: git push workflow and node sync up



I would imagine a HA setup for broker /MQ in production with multiple instances.
However, if the approach as you described is taken then both options auto and admin managed is desirable.
As I commented in my earlier mail given the current decoupling of deployment with the system, it might be easier to migrate puppet based deployments currently being worked on in my company. We have not looked in depth as to whether each gear needs a puppet agent in it or one puppet agent per node. 
But for me an equally dependable system from Openshift does not force us to look at puppet immediately but later. 

Sent from my iPhone

On Jan 30, 2013, at 7:48 AM, Luke Meyer <lmeyer redhat com> wrote:

Nodes are deliberately passive with respect to the broker (with very few exceptions: Jenkins and scale up/down, AFAIK). On the whole, the more we can decentralize application interactions, the better - that's the design decision I believe has been made. As it is, even if your broker/DB/messaging is completely dead in the water, application availability and application re-deploys just continue working, and I don't think we want to complicate that.

Regarding a resync after a node outage, I don't think this is a case for storing state in the broker. One option to consider is to have the primary gear track rsync failures and have a cron job that retries failed ones until they succeed. Another option would be an administrator tool to trigger a re-sync on all scaled apps after a known node outage (or just any apps with gears on the specified node).

----- Original Message -----
From: "Meghdoot" <meghdoot_b yahoo com>
To: "Clayton Coleman" <ccoleman redhat com>
Cc: dev lists openshift redhat com
Sent: Tuesday, January 29, 2013 2:58:16 PM
Subject: Re: git push workflow and node sync up

In this scenario I meant down nodes joining later missing the git push window, that you covered below. In a large installation it's painful to figure out bad nodes with wrong versions and then manually fix it. So, yes broker orchestration would be the ideal solution.
Thx 


Sent from my iPhone

On Jan 29, 2013, at 9:48 AM, Clayton Coleman <ccoleman redhat com> wrote:



----- Original Message -----

Thx for the replies. If gear resync is on the roadmap, I would like
to know if there are any proposed designs that you can share. Our
release engg is moving towards puppet based deployments that fixes
the sync issue and hence we need to solve the gear resync for both
temp failures or nodes joining later.

Nodes joining later shouldn't have an effect - or at least, in the current model adding new nodes simply allow new gears to be allocated.  There is no automatic reallocation to new nodes, just manual moves.  Can you elaborate on what you're expecting?

The first stage of a resync design is simply a task run on the node after it's brought up from downtime (to handle the scenario "node X was down for 10 minutes"), or a manual action on a specific app executed via the broker (distributed to the head gear) that triggers a resync.  Any design more complicated than that would involve transactional broker state and the integration of git pushes back to the broker (at least the notification thereof) - basically, a git push would trigger a rolling upgrade in the broker, which would schedule and orchestrate stop/resync and pull/start on each node.  We don't have a concrete timeframe for when we'd deliver but it's certainly a priority for us.

Other alternative I suppose is to replace the current openshift
deployment model with the puppet based one since the deployment
looks pretty decoupled from the system currently.
Thx

Sent from my iPhone

On Jan 29, 2013, at 6:25 AM, Clayton Coleman < ccoleman redhat com >
wrote:








On Jan 29, 2013, at 3:06 AM, Krishna Raman < kraman gmail com >
wrote:




Some answers inline.


--kr



On Jan 28, 2013, at 6:20 PM, Meghdoot < meghdoot_b yahoo com > wrote:




Can somebody reply on this if possible? Don't want to bug Brenton
since he is probably jet lagged now.
Otherwise I will wait for him.
Thx

Sent from my iPhone

On Jan 23, 2013, at 11:21 PM, meghdoot bhattacharya <
meghdoot_b yahoo com > wrote:







Further additions to my comments inline and couple of new questions.
Question of unreachable gears and auto syncing of them is important.
Our largest web app runs in thousands of boxes :)








From: Meghdoot < meghdoot_b yahoo com >
To: Brenton Leanhardt < bleanhar redhat com >
Cc: dev lists openshift redhat com
Sent: Wednesday, January 23, 2013 11:17 AM
Subject: Re: git push workflow and node sync up

My comments/questions inline.

On Jan 22, 2013, at 11:54 PM, Brenton Leanhardt < bleanhar redhat com
wrote:

+++ meghdoot bhattacharya [22/01/13 17:47 -0800]:
Hi,
I had couple of questions on the workflow.

1. When we git push to the master, how does the nodes get the their
git repo updated? Does the git hooks in the master send a message to
the broker and that in turn uses mcollective to broadcast the
message to the nodes and the gears in turn do a git pull from the
master? Or do the Haproxy distribute the code/artifacts directly? I
am imagining the nodes have to do a pull from the master.

If you look in a scaled gear's git repo you will see a post receive
hook called 'post-receive'. The chain of events looks like this:

1 post-receive
2 post_receive_app.sh
3 post_start_app function in the abstract cartridge 4 deploy.sh in
the haproxy cartridge
5 sync_gears.sh in the haproxy cartridge (rsync)











So am I correct in stating the git operations only matter between
local dev repo and master haproxy node. Between the master node
and node gears we are just syncing content without using as such
git operations.



Yes, this is correct. The git push only happens between your local
repo and the repo on the haproxy gear. After that the code is
rsync'd to the other gears.










But the git hooks in node gears come in use or not?



Only on the haproxy gear










Looks like I am wrong. There is no git repo in the individual gear
nodes and looks like there is just a repo/content directory. And
the sync_gears script runs the pre scripts, installs content and
runs the post scripts in a ssh loop. Am I right?



Yes, Sync gear calls the pre, and post scripts. But these are not git
hooks.










Also sync_gears.sh in step 5 comes from the primary cartridge, so I
am guessing instead of haproxy it will be nodejs sync_gears.sh for
the scaled app.



The sync_gears.sh comes from the abstract cartridge which is
inherited by the other cartridges. Its the same script no matter
which cartridge you call it from.










Hmm, in that case my 2nd question becomes all the more relevant
because there may be rsync failures on some of the nodes and how do
we deal with it.
And why are we not using mcollective notification and git pull from
individual nodes for this?


Brenton? Mike?


Orchestrated git, push, and gear resynchronization are on the roadmap
as part of rolling restarts and upgrades. Rsync is the simplest
thing that actually works, so that's where we started. Also, git
scaled pushes will continue to work even in the presence of an
mcollective outage. If there isn't a resynchronization mechanism
though there should be.
















And the other question is does the mongo db keep track of any of
these code pushes so that it may be used for the source of truth for
deployment related queries.



Git push and sync are purely on the nodes. They do not interact with
the broker so mongo is not notified of git pushes at the moment.











2. If few nodes get offline and come back up and had missed the new
git push, how do they sync up automatically?



AFAIK, currently the node will not sync up. You will need to issue
another git push for the sync to take place.











I don't know this off the top of my head so let's see if someone else
chimes in. If not I'll dig into this for you
Thx. This is important feature for us.


Thx.

_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev




_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]