[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: git push workflow and node sync up




----- Original Message -----
> 
> Thx for the replies. If gear resync is on the roadmap, I would like
> to know if there are any proposed designs that you can share. Our
> release engg is moving towards puppet based deployments that fixes
> the sync issue and hence we need to solve the gear resync for both
> temp failures or nodes joining later.

Nodes joining later shouldn't have an effect - or at least, in the current model adding new nodes simply allow new gears to be allocated.  There is no automatic reallocation to new nodes, just manual moves.  Can you elaborate on what you're expecting?

The first stage of a resync design is simply a task run on the node after it's brought up from downtime (to handle the scenario "node X was down for 10 minutes"), or a manual action on a specific app executed via the broker (distributed to the head gear) that triggers a resync.  Any design more complicated than that would involve transactional broker state and the integration of git pushes back to the broker (at least the notification thereof) - basically, a git push would trigger a rolling upgrade in the broker, which would schedule and orchestrate stop/resync and pull/start on each node.  We don't have a concrete timeframe for when we'd deliver but it's certainly a priority for us.

> Other alternative I suppose is to replace the current openshift
> deployment model with the puppet based one since the deployment
> looks pretty decoupled from the system currently.
> Thx
> 
> Sent from my iPhone
> 
> On Jan 29, 2013, at 6:25 AM, Clayton Coleman < ccoleman redhat com >
> wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> On Jan 29, 2013, at 3:06 AM, Krishna Raman < kraman gmail com >
> wrote:
> 
> 
> 
> 
> Some answers inline.
> 
> 
> --kr
> 
> 
> 
> On Jan 28, 2013, at 6:20 PM, Meghdoot < meghdoot_b yahoo com > wrote:
> 
> 
> 
> 
> Can somebody reply on this if possible? Don't want to bug Brenton
> since he is probably jet lagged now.
> Otherwise I will wait for him.
> Thx
> 
> Sent from my iPhone
> 
> On Jan 23, 2013, at 11:21 PM, meghdoot bhattacharya <
> meghdoot_b yahoo com > wrote:
> 
> 
> 
> 
> 
> 
> 
> Further additions to my comments inline and couple of new questions.
> Question of unreachable gears and auto syncing of them is important.
> Our largest web app runs in thousands of boxes :)
> 
> 
> 
> 
> 
> 
> 
> 
> From: Meghdoot < meghdoot_b yahoo com >
> To: Brenton Leanhardt < bleanhar redhat com >
> Cc: dev lists openshift redhat com
> Sent: Wednesday, January 23, 2013 11:17 AM
> Subject: Re: git push workflow and node sync up
> 
> My comments/questions inline.
> 
> On Jan 22, 2013, at 11:54 PM, Brenton Leanhardt < bleanhar redhat com
> > wrote:
> 
> +++ meghdoot bhattacharya [22/01/13 17:47 -0800]:
> Hi,
> I had couple of questions on the workflow.
> 
> 1. When we git push to the master, how does the nodes get the their
> git repo updated? Does the git hooks in the master send a message to
> the broker and that in turn uses mcollective to broadcast the
> message to the nodes and the gears in turn do a git pull from the
> master? Or do the Haproxy distribute the code/artifacts directly? I
> am imagining the nodes have to do a pull from the master.
> 
> If you look in a scaled gear's git repo you will see a post receive
> hook called 'post-receive'. The chain of events looks like this:
> 
> 1 post-receive
> 2 post_receive_app.sh
> 3 post_start_app function in the abstract cartridge 4 deploy.sh in
> the haproxy cartridge
> 5 sync_gears.sh in the haproxy cartridge (rsync)
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> >> So am I correct in stating the git operations only matter between
> >> local dev repo and master haproxy node. Between the master node
> >> and node gears we are just syncing content without using as such
> >> git operations.
> 
> 
> 
> Yes, this is correct. The git push only happens between your local
> repo and the repo on the haproxy gear. After that the code is
> rsync'd to the other gears.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> But the git hooks in node gears come in use or not?
> 
> 
> 
> Only on the haproxy gear
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> >> Looks like I am wrong. There is no git repo in the individual gear
> >> nodes and looks like there is just a repo/content directory. And
> >> the sync_gears script runs the pre scripts, installs content and
> >> runs the post scripts in a ssh loop. Am I right?
> 
> 
> 
> Yes, Sync gear calls the pre, and post scripts. But these are not git
> hooks.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Also sync_gears.sh in step 5 comes from the primary cartridge, so I
> am guessing instead of haproxy it will be nodejs sync_gears.sh for
> the scaled app.
> 
> 
> 
> The sync_gears.sh comes from the abstract cartridge which is
> inherited by the other cartridges. Its the same script no matter
> which cartridge you call it from.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Hmm, in that case my 2nd question becomes all the more relevant
> because there may be rsync failures on some of the nodes and how do
> we deal with it.
> And why are we not using mcollective notification and git pull from
> individual nodes for this?
> 
> 
> Brenton? Mike?
> 
> 
> Orchestrated git, push, and gear resynchronization are on the roadmap
> as part of rolling restarts and upgrades. Rsync is the simplest
> thing that actually works, so that's where we started. Also, git
> scaled pushes will continue to work even in the presence of an
> mcollective outage. If there isn't a resynchronization mechanism
> though there should be.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> And the other question is does the mongo db keep track of any of
> these code pushes so that it may be used for the source of truth for
> deployment related queries.
> 
> 
> 
> Git push and sync are purely on the nodes. They do not interact with
> the broker so mongo is not notified of git pushes at the moment.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 2. If few nodes get offline and come back up and had missed the new
> git push, how do they sync up automatically?
> 
> 
> 
> AFAIK, currently the node will not sync up. You will need to issue
> another git push for the sync to take place.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> I don't know this off the top of my head so let's see if someone else
> chimes in. If not I'll dig into this for you
> >> Thx. This is important feature for us.
> 
> 
> Thx.
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 
> 
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 
> 
> 
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]