[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Changing Prometheus rules



On Wed, Nov 20, 2019 at 10:23 AM Mateus Caruccio
<mateus caruccio getupcloud com> wrote:
>
> Isn't cluster version operator something from okd 4.x?

hmm, I might be confusing 3.x and 4.x indeed...

>
> Em Qua, 20 de nov de 2019 05:37, Simon Pasquier <spasquie redhat com> escreveu:
>>
>> On Tue, Nov 19, 2019 at 6:33 PM Mateus Caruccio
>> <mateus caruccio getupcloud com> wrote:
>> >
>> > You must disable cluster-monitoring-operator since it will try to reconcile the whole monitoring stack.
>> >
>> > $ oc scale --replicas=0 deploy/cluster-monitoring-operator
>>
>> You'd need to disable the cluster version operator too IIRC and this
>> has a bigger impact.
>>
>> >
>> > Muting alerts using inhibit rules may have an unexpected side-effect as noted by [1]. The recommended approach is to send alerts for a "blackhole" receiver (rationale and example in the link)
>> >
>> > [1] https://medium.com/@wrossmann/suppressing-informational-alerts-with-prometheus-and-alertmanager-4237feab7ce9
>>
>> What I've described should work because source and target labels won't
>> match the same alerts. Agreed that blackholing the notification is
>> also a good solution.
>>
>> >
>> > --
>> > Mateus Caruccio / Master of Puppets
>> > GetupCloud.com
>> > We make the infrastructure invisible
>> > Gartner Cool Vendor 2017
>> >
>> >
>> > Em ter., 19 de nov. de 2019 às 13:27, Tim Dudgeon <tdudgeon ml gmail com> escreveu:
>> >>
>> >> No joy with that approach. I tried editing the ConfigMap and the CRD but both got reset when the cluster-monitoring-operator was restarted.
>> >>
>> >> Looks like I'll have to live with silencing the alert.
>> >>
>> >> On 19/11/2019 07:56, Vladimir REMENAR wrote:
>> >>
>> >> Hi Tim,
>> >>
>> >> You need to stop cluster-monitoring-operator than and then edit configmap. If cluster-monitoring-operator is running while editing configmap it will always revert it to default.
>> >>
>> >>
>> >> Uz pozdrav,
>> >> Vladimir Remenar
>> >>
>> >>
>> >>
>> >> From:        Tim Dudgeon <tdudgeon ml gmail com>
>> >> To:        Simon Pasquier <spasquie redhat com>
>> >> Cc:        users <users lists openshift redhat com>
>> >> Date:        18.11.2019 17:46
>> >> Subject:        Re: Changing Prometheus rules
>> >> Sent by:        users-bounces lists openshift redhat com
>> >> ________________________________
>> >>
>> >>
>> >>
>> >> The KubeAPILatencyHigh alert fires several times a day for us (on 2
>> >> different OKD clusters).
>> >>
>> >> On 18/11/2019 15:17, Simon Pasquier wrote:
>> >> > The Prometheus instances deployed by the cluster monitoring operator
>> >> > are read-only and can't be customized.
>> >> > https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#alerting-rules_prometheus-cluster-monitoring
>> >> >
>> >> > Can you provide more details about which alerts are noisy?
>> >> >
>> >> > On Mon, Nov 18, 2019 at 2:43 PM Tim Dudgeon <tdudgeon ml gmail com> wrote:
>> >> >> What is the "right" way to edit Prometheus rules that are deployed by
>> >> >> default on OKD 3.11?
>> >> >> I have alerts that are annoyingly noisy, and want to silence them forever!
>> >> >>
>> >> >> I tried editing the definition of the PrometheusRule CRD and/or the
>> >> >> prometheus-k8s-rulefiles-0 ConfigMap in the openshift-monitoring project
>> >> >> but my changes keep getting reverted back to the original.
>> >> >>
>> >> >> _______________________________________________
>> >> >> users mailing list
>> >> >> users lists openshift redhat com
>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >> >>
>> >>
>> >> _______________________________________________
>> >> users mailing list
>> >> users lists openshift redhat com
>> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> users mailing list
>> >> users lists openshift redhat com
>> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >
>> > _______________________________________________
>> > users mailing list
>> > users lists openshift redhat com
>> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]