> Date: Fri, 27 Mar 2015 08:21:19 -0400
> From: lmeyer redhat com
> To: michael mcconachie hotmail com
> CC: dev lists openshift redhat com
> Subject: Re: UID TC issue?
> ----- Original Message -----
> > From: "Michael McConachie" <michael mcconachie hotmail com>
> > To: dev lists openshift redhat com
> > Sent: Thursday, March 26, 2015 4:05:36 PM
> > Subject: UID TC issue?
> > Hi All,
> > We've been running into issues concerning app create failures on
> > small districts (where most of our users 'live'). This is a v3
> > system, where we are currently migrating to an M4 system. I am
> > trying to determine if we'll run into the same problem later on M4.
> > (We also have OSE 2.2.)
> > The problem we've been experiencing is that OSO is generating TC
> > class IDs outside of the 16-bit range that is accepted by TC. The
> > exact cause for this behavior is not completely clear, but may have
> > to do with UID limits set on those nodes. The node UID limits are
> > currently set to 1000 - 6500. We have approximately 900 gears in
> > that district, being spread over 5 nodes - and have been
> > experiencing consistent failures when trying to build applications
> > in that district. We tried increasing the GEAR_MAX_UID to 10000 on
> > every node in the district, to no avail.
> District/node UID parameters are poorly represented in the config files. There was a massive change in early 2013 that actually started using some new parameters while leaving the old ones as vestigial entries in the config files. I'm not sure exactly what your M3 codebase is looking at without digging deeper. Districts ignore these anyway when giving out UIDs (the district creates the UID pool), but the node uses them to calculate which resources should be owned by each gear. I don't see where we officially documented anywhere how this works; I think we only give directions for configuring these in a private (OSE) kbase article.
Hi Luke, this is good info, thanks for taking the time to respond. (I'll go dig for the kbase on RHN.) Thank you the clarification.
> Don't try to change node UID parameters after the node is districted; at best it will do nothing, or at worst your gears will end up with overlapping port ranges/IPs and the like. The range on nodes should remain the default 1000-6999 unless you have really carefully coordinated the change between district definition and node and are sure the conf settings are being used like you think they are. Also note gears don't get reconfigured when you change these settings, so existing gears will keep using their previously-configured port ranges/IPs.
Agreed; this was an initial concern and we immediately revoked the change after noticing the inability to create small gears persisted. Hopefully we'll be OK since we haven't been able to create new small gears anyway.
> This doesn't directly address what you're talking about with TC class IDs but all gear resources are calculated based on UID and expected UID range so it's probably related. Maybe you could indicate more about what kind of failure exactly you're seeing?
I'll be able to provide more info in a follow-up to this thread; I need to do some fact finding to give you useful info.
> > I understand that OpenShift wraps the qdiscs to 64K, but what about
> > the class IDs that are being fed to TC? It appears that those
> > generated class ID's aren't wrapping, causing failures. I'd be very
> > curious to know how you're doing it w/ online. Unfortunately, we
> > don't have the same user-base/load in our OSE environment, so I
> > can't confirm the issue there.
> > A cursory search led me to Brenton's article:
> > https://brenton-leanhardt.rhcloud.com/understanding-how-uids-are-used-in-openshift-origin/
> > We'd appreciate any thoughts you might have.
> > Thanks,
> > Michael J. McConachie | keys.fedoraproject.org | PubKey: 0x7BCD88F8
> > _______________________________________________
> > dev mailing list
> > dev lists openshift redhat com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev