[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Multi-tenant security

Hey Mark!

On Mon, Apr 10, 2017 at 12:16 PM, Mark Nuttall-Smith <marknuttallsmith gmail com> wrote:

I'm toying with the idea of building a Python-only micro-PaaS (at least that's what I'm calling it ;)) on top of OpenShift - something similar to PythonAnywhere, but for a specific business niche. 

Cool, I look forward to seeing what you build!

I've been thinking through the security aspects of the project. It would be a multitenant application so one of the big questions is whether to isolate customers with a hypervisor, or if the container isolation is sufficient. I would prefer the latter, of course, given the reduced complexity and increased efficiency.

This is a judgment call that will vary based on who you're talking to, your risk tolerance, threat model, etc. Here are some facts:

1. OpenShift Online is a public, multi-tenant instance of OpenShift that relies on container security mechanisms to isolate tenants
2. All running containers that happen to be scheduled on the same node will share a kernel, and it is possible (though not easy) for a series of unpatched/unknown bugs to lead to a container breakout
3. Virtualization adds isolation using specialized instructions (VT-x) and is more mature, so it is generally considered to be more secure. But hypervisor breakouts have been known to happen as well.
4. It's good practice to run an operating system with strong security support (such as, ahem, RHEL)
5. Red Hat practices a defense-in-depth approach to security; our SELinux policy is non-intrusive (it Just Works for containers) and helps to protect against breakouts: http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/

In the Adrian Mouat's ebook Docker Security he writes, "If you have a multitenancy setup, running containers for multiple users (whether these are internal users in your organization or external customers), ensure that each user is placed on a separate Docker host". This advice is now two years old, so I'm wondering if the improvements in Docker security since then (eg. user-namepsaces, seccomp), together with the security features that OpenShift brings/integrates with (multi-tenant networking, SELinux, s2i builds etc) mean that this advice is still current?

Again, currency depends on your own risk assessment.

Our own Dan Walsh has said early on that "Containers don't contain" - but as a company we've been doing a lot of work to improve isolation, and believe that the multi-tenant security features are strong enough that we have confidence running them on a production cloud service (namely, OpenShift Online). This also requires some constraints - for example, applications run as a random unprivileged uid inside the container, and Docker builds are not permitted. There are many other defaults OpenShift has chosen to be "secure by default" in contrast to other orchestrators.

I heard Brendan Burns discussing Kubernetes security on the ArchiTECHt podcast. He says that unless you trust two users to have a login to the same machine then you shouldn't allow those users to run containers on the machine because any kernel level vulnerability is going to be exploitable, and you'll be able to jump out of one container and jump into the other. He goes on to mention hyperd. If you have authenticated (paying) users, up to date kernel patches and the security features mentioned above how likely do you think an exploit like this is?

I don't personally consider this to be a huge risk, but it also depends on the system that you're building - no system can defend itself against a well-funded, determined attacker indefinitely. The name of the game in security isn't to have an inpenetrable fortress for a house, but to be better defended than your neighbors; in other words, you don't have to outrun the tiger, you have to outrun the guy next to you.
Also, has anyone given any thought to integrating hyperd and OpenShift?

We're working on CRI-O [0], and have partners at Intel working on Clear Containers to provide virtualization-level isolation for OpenShift. While this gives you an isolated kernel and pretty solid security (via VT-x), hypervisor breakouts are not completely impossible.

[0] http://www.projectatomic.io/blog/2017/02/crio-runtimes/

Jonathan Yu / Software Engineer, OpenShift by Red Hat / @jawnsy

“There are a million ways to get rich. But there’s only one way to stay rich: Humility, often to the point of paranoia. The irony is that few things squash humility like getting rich in the first place.” — Morgan Housel, Getting Rich vs. Staying Rich

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]