I created a template to deploy a Prometheus instance inside of projects. The idea is to have a small instance that does the service discovery and scraping, but only stores data for a short time (2 hours) in an EmptyDir. You can then federate that instance with another Prometheus outside of Openshift for proper long-term storage, alerting, etc. This way it's super easy to deploy. No configuration necessary. It also deploys blackbox-exporter + kube-state-metrics in the same pod, so that it can also do probing and record the project state.https://github.com/adnovum/adnovum-openshift-templates/tree/master/prometheus-ephemeral
Unfortunately, per-container system metrics (CPU/memory) are not
available. If anybody knows how to gather this without privileges, I'd
be very interested to know.
Let me know what you think of it. Feedback welcome.