[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Changing Prometheus rules



On Tue, Nov 19, 2019 at 6:33 PM Mateus Caruccio
<mateus caruccio getupcloud com> wrote:
>
> You must disable cluster-monitoring-operator since it will try to reconcile the whole monitoring stack.
>
> $ oc scale --replicas=0 deploy/cluster-monitoring-operator

You'd need to disable the cluster version operator too IIRC and this
has a bigger impact.

>
> Muting alerts using inhibit rules may have an unexpected side-effect as noted by [1]. The recommended approach is to send alerts for a "blackhole" receiver (rationale and example in the link)
>
> [1] https://medium.com/@wrossmann/suppressing-informational-alerts-with-prometheus-and-alertmanager-4237feab7ce9

What I've described should work because source and target labels won't
match the same alerts. Agreed that blackholing the notification is
also a good solution.

>
> --
> Mateus Caruccio / Master of Puppets
> GetupCloud.com
> We make the infrastructure invisible
> Gartner Cool Vendor 2017
>
>
> Em ter., 19 de nov. de 2019 às 13:27, Tim Dudgeon <tdudgeon ml gmail com> escreveu:
>>
>> No joy with that approach. I tried editing the ConfigMap and the CRD but both got reset when the cluster-monitoring-operator was restarted.
>>
>> Looks like I'll have to live with silencing the alert.
>>
>> On 19/11/2019 07:56, Vladimir REMENAR wrote:
>>
>> Hi Tim,
>>
>> You need to stop cluster-monitoring-operator than and then edit configmap. If cluster-monitoring-operator is running while editing configmap it will always revert it to default.
>>
>>
>> Uz pozdrav,
>> Vladimir Remenar
>>
>>
>>
>> From:        Tim Dudgeon <tdudgeon ml gmail com>
>> To:        Simon Pasquier <spasquie redhat com>
>> Cc:        users <users lists openshift redhat com>
>> Date:        18.11.2019 17:46
>> Subject:        Re: Changing Prometheus rules
>> Sent by:        users-bounces lists openshift redhat com
>> ________________________________
>>
>>
>>
>> The KubeAPILatencyHigh alert fires several times a day for us (on 2
>> different OKD clusters).
>>
>> On 18/11/2019 15:17, Simon Pasquier wrote:
>> > The Prometheus instances deployed by the cluster monitoring operator
>> > are read-only and can't be customized.
>> > https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#alerting-rules_prometheus-cluster-monitoring
>> >
>> > Can you provide more details about which alerts are noisy?
>> >
>> > On Mon, Nov 18, 2019 at 2:43 PM Tim Dudgeon <tdudgeon ml gmail com> wrote:
>> >> What is the "right" way to edit Prometheus rules that are deployed by
>> >> default on OKD 3.11?
>> >> I have alerts that are annoyingly noisy, and want to silence them forever!
>> >>
>> >> I tried editing the definition of the PrometheusRule CRD and/or the
>> >> prometheus-k8s-rulefiles-0 ConfigMap in the openshift-monitoring project
>> >> but my changes keep getting reverted back to the original.
>> >>
>> >> _______________________________________________
>> >> users mailing list
>> >> users lists openshift redhat com
>> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> >>
>>
>> _______________________________________________
>> users mailing list
>> users lists openshift redhat com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users lists openshift redhat com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
> _______________________________________________
> users mailing list
> users lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]