[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: OSE 3.X upper limits



There is a particularly inefficient operation in the Docker API that we use from the Kubelet to get container status.  The more containers you have the worse Docker behaves, so there is work ongoing to get container status in a different way.  In addition, there are a few known inefficiencies inside the Kubelet that make the problem worse.

Fixing it a high priority for Kube in the 1.2 release and for Origin in 1.2

On Jan 6, 2016, at 3:06 AM, Amit Gupta <amitkgupta84 gmail com> wrote:

Thanks for the info Clayton.  You mentioned something about recommending 100 containers/node as an upper bound, and pointed at an issue on kubernetes.  Didn't see anything specific in that issue about this though.  Is this a docker thing or kubelet thing?

On Tue, Jan 5, 2016 at 6:11 AM, Clayton Coleman <ccoleman redhat com> wrote:
Kubelet coordination is unlikely to be a significant source of load -
having less nodes will be more efficient for scheduling, but right now
density of nodes is going to have an upper bound.

In those tests, there were 30-40 pods running per node.  The tests
were not necessarily production style apps, but the load generated on
the kubelet and app server is roughly the same for any container.

On Tue, Jan 5, 2016 at 4:58 AM, Amit Gupta <amitkgupta84 gmail com> wrote:
> Is a denser deployment more or less likely to be robust at scale? Running
> ~10-30k containers on ~1k nodes vs running it on ~500nodes? I would think
> ~500 nodes would be better because the scheduler has less kubelets to
> coordinate, and I get more efficient utilization of my nodes.
>
> When you say Kubernetes and OpenShift have tested up to 1k+ nodes, what
> exactly was being run on those nodes? Was it a simulation of a real life
> production workload?
>
> On Mon, Jan 4, 2016 at 1:46 PM, Clayton Coleman <ccoleman redhat com> wrote:
>>
>> Kubernetes supports 150 nodes per cluster at a very specific SLA (all
>> calls <2s).  That is not the largest cluster you can run.  In
>> practice, Kubernetes and OpenShift have tested clusters up to 1k+
>> nodes and our concerns are optimizations in particular parts of the
>> code, not the maximum scale of the cluster.
>>
>> On Mon, Jan 4, 2016 at 4:11 PM, Srinivas Naga Kotaru (skotaru)
>> <skotaru cisco com> wrote:
>> > Clayton
>> >
>> > This is good info to keep in mind while designing. What is the practical
>> > limit of nodes per cluster? We are thinking to use multiple clusters for
>> > non-prod/prod and internal/External apps.
>> >
>> > We heard Kubernates 1.1 support ~150 nodes per cluster.
>> >
>> >
>> > --
>> > Srinivas Kotaru
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On 1/4/16, 12:53 PM, "Clayton Coleman" <ccoleman redhat com> wrote:
>> >
>> >>We recommend no more than 100 containers per node at this time, while
>> >>performance issues in Docker and Kubernetes are being sorted out.
>> >>There is work on going to increase that to several hundred
>> >>(https://github.com/kubernetes/kubernetes/issues/19094)
>> >>
>> >>You can run many containers in one pod - you should not see a
>> >>significant difference between 1 container per pod and 20.  I would
>> >>recommend not going over 20 containers per pod, if only because we
>> >>haven't extensively tested it.
>> >>
>> >>If you have suitably powerful masters, you can expect to run ~10-30k
>> >>pods on ~1k nodes.  You may need to configure your network properly to
>> >>ensure that openvswitch can scale to those levels, or choose a static
>> >>/ non-openshift-sdn configuration like static route assignment.
>> >>Masters use a fair amount of memory at scale - you may need 16-32GB
>> >>per master.
>> >>
>> >>If you are running thousands of projects per cluster, you will also
>> >>need more memory than that.  Exact numbers are hard to say but ~1GB
>> >>per 1k of projects is not unreasonable today (can vary a lot).
>> >>
>> >>
>> >>On Mon, Jan 4, 2016 at 3:45 PM, Srinivas Naga Kotaru (skotaru)
>> >><skotaru cisco com> wrote:
>> >>> What are the upper limits of
>> >>> - How many nodes in a single cluster
>> >>>        -  How many pods in  a single node
>> >>>       -   How many containers in a single POD
>> >>>
>> >>> Looking for  practical values which can be used as a baseline for
>> >>> cluster
>> >>> design.
>> >>>
>> >>> --
>> >>> Srinivas Kotaru
>> >>>
>> >>> _______________________________________________
>> >>> dev mailing list
>> >>> dev lists openshift redhat com
>> >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>> >>>
>>
>> _______________________________________________
>> dev mailing list
>> dev lists openshift redhat com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
>


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]