We have application as follows where every one of our pods exposes a Prometheus http end point to scrape. We had to build our own docker image of Prometheus with static tagerts pointed to service end points. This has worked ok more or less with one exception where if you increase instance of a pod to 2 or more you only still see metrics for one pod because of load balancing. So the solution there is to get it to scrape not services but individual pods. We don't know how to do that, because pods could restart and number of pods can increase per service.
What is the recommended, best approach to get Prometheus to scrape pods? What tools does OKD 3.11 provide with built in Prometheus and Grafana to help us do this, if any?