[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Socket activation




----- Original Message -----
> 
> On Sep 4, 2013, at 1:54 PM, David Strauss <david davidstrauss net> wrote:
> 
> > On Wed, Sep 4, 2013 at 12:22 PM, Krishna Raman <kraman gmail com> wrote:
> >> Could make this a configuration/setup option. If the router/switch
> >> misbehaves with it gets SYN failures then use a haproxy,
> >> else live with 1 SYN failure per gear un-idle.
> >> 
> >> What do you think?
> > 
> > I'm just wary of relying on failure-retry mechanisms for normal
> > operations. It takes network monitoring from "no SYN-ACK failures
> > should occur under normal conditions" to "there is some amount
> > expected, and expectations might even be exceeded under normal
> > conditions " Noise like that makes the ops job hard without getting
> > either false positives (on widespread activation) or false negatives
> > (on network failure). Even if you can tolerate it, it's not a fun
> > model to work with in practice.
> > 
> > It might be possible to map gear activations to SYN-ACK failures to
> > discover network failure, but network and daemon monitoring are rarely
> > integrated. If you're at a colo facility, they're not even managed by
> > the same parties.
> > 
> > Using haproxy to mask the SYN-ACK failure is interesting. But, unlike
> > the socat shim *behind* socket activation, the haproxy would have to
> > persistently run for every container/gear on the host. That would
> > really hurt density. What would you think of haproxy *behind* socket
> > activation as the shim? I don't think it supports it at the moment,
> > but it would not be hard to implement.
> 
> We already run a haproxy on the node to do port-forwarding and it seems to
> perform fine under load and at density. Its not as fast as iptables but at
> least it
> will hide the SYN-ACK problem.

Yeah but that's going away very, very soon.  Could we simply drop packets on node startup until a reasonable number of gears are reactivated (via iptables)?  Then we are reduced back to - "what's the best way to queue packets while a relatively short container activation is executed"?

> 
> Running it inside the gear is still not suitable because of the extra memory
> consumption.
> 
> > 
> > I'd also like to note that the dropped-SYN activation model won't work
> > for most UDP protocols. Some will retry. Some will just have data go
> > missing. If you're using statsd or GELF, that would be pretty
> > annoying.
> 
> Perhaps the node level haproxy is best in this case. It might be able to
> handle
> this better as well.
> 
> > 
> > --
> > David Strauss
> >   | david davidstrauss net
> >   | +1 512 577 5827 [mobile]
> 
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]