[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: horizontal autoscaler does not get cpu utilization



2 things:

1) you need to provide authentication when you make the curl request

2) you need a : in between https and heapster (so httpsheapster: becomes https:heapster:)

On Thu, Feb 25, 2016 at 3:24 AM, Den Cowboy <dencowboy hotmail com> wrote:
I got this from the curl (as system:admin)
curl -k https://ip-xx-xx-xx-xx.xx-xx-1.compute.internal:8443/api/v1/proxy/namespaces/openshift-infra/services/httpsheapster:/validate
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "User \"system:anonymous\" cannot \"proxy\" \"services\" with name \"httpsheapster:\" in project \"openshift-infra\"",
  "reason": "Forbidden",
  "details": {
    "name": "httpsheapster:",
    "kind": "services"
  },
  "code": 403



From: dencowboy hotmail com
To: mwringe redhat com
Subject: RE: horizontal autoscaler does not get cpu utilization
Date: Thu, 25 Feb 2016 07:59:11 +0000
CC: users lists openshift redhat com


Some logs are showing:

Failed to reconcile test-scaler: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/test/test: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value
Feb 25 07:48:57 ip-172-31-xx-xx origin-master: E0225 07:48:57.079028    2242 event.go:192] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"test-scaler.14361ecd543d4608", GenerateName:"", Namespace:"test", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"test", Name:"test-scaler", UID:"f7bac384-db00-11e5-ac6e-06b94d3c6589", APIVersion:"extensions", ResourceVersion:"13501", FieldPath:""}, Reason:"FailedGetMetrics", Message:"failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value",

I used this configuration:
$ oc secrets new metrics-deployer nothing=/dev/null



> Date: Wed, 24 Feb 2016 13:29:21 -0500
> From: mwringe redhat com
> To: dencowboy hotmail com
> CC: users lists openshift redhat com
> Subject: Re: horizontal autoscaler does not get cpu utilization
>
>
>
> ----- Original Message -----
> > From: "Den Cowboy" <dencowboy hotmail com>
> > To: users lists openshift redhat com
> > Sent: Wednesday, February 24, 2016 9:35:34 AM
> > Subject: RE: horizontal autoscaler does not get cpu utilization
> >
> > I don't know if this is maybe the issue?
> > In my browser https://hawkular-metrics.xx.xx.com/hawkular/metrics/status
> > {"MetricsService":"STARTED","Implementation-Version":"0.13.0-SNAPSHOT","Built-From-Git-SHA1":"7dee24acfcfb3beac356e2c4d83b7b1704ebf82x"}
> > curl on my master or nodes:
> > curl -X GET https://hawkular-metrics.xx.xx.com/hawkular/metrics/status
> > curl: (6) Could not resolve host: hawkular-metrics.xx.xx.com; Name or service
> > not known
> >
> > I'm just describing the IP of the node where my router is in my local
> > /etc/hosts
> > like this: xx.xx.xx.xx hawkular-metrics.xx.xx.com
>
> The router configuration is not used for the HPA and so not being able to resolve the hostname from within the node or container should not be an issue.
>
> What the HPA does use is the API proxy.
>
> You can check if Heapster is accessible via the API proxy through the following command:
>
> curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXX" \
> -X GET https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/
>
> Are there any other errors in the OpenShift logs [not the container logs for Hawkular-Metrics, Cassandra or Heapster, those appear to be working since you can see metrics in the browser]
>
> >
> >
> > From: dencowboy hotmail com
> > To: users lists openshift redhat com
> > Subject: horizontal autoscaler does not get cpu utilization
> > Date: Wed, 24 Feb 2016 13:56:24 +0000
> >
> > I'm on Origin 1.1.3
> > I've confgured the cluster-metrics (its in the openshift-infra project!). I'm
> > able to see all the metrics (memory & cpu) on my metrics-tab.
> > Now I try to create a simple autoscaler:
> > oc autoscale dc/test --min 2 --max 15 --cpu-percent=70
> >
> > I've edited the dc of my container so now it's using resources requests and
> > limits.
> > In my webconsole I see the 2 cirkles and the MiB and millicores used.
> >
> > But
> > oc get hpa
> > NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
> > test DeploymentConfig/test/scale 70% <waiting> 2 15 21m
> >
> > Target CPU utilization: 70%
> > Current CPU utilization: <not available>
> >
> > I read it was normal that the current state was waiting in the beginning. But
> > it's already on <waiting> for 21 minutes.
> > How can I check what's going wrong?
> >
> > The logs of the pod where I've created the autoscaler doesn't show anything
> > different than normal:
> > AH00558: httpd: Could not reliably determine the server's fully qualified
> > domain name, using 10.1.1.14. Set the 'ServerName' directive globally to
> > suppress this message
> >
> > The logs of my metrics (but they seem to work fine):
> > casandra: INFO 12:59:51 Initializing hawkular_metrics.leases
> >
> > heapster:
> > I0224 13:28:35.108008 1 manager.go:152] ScrapeMetrics: time: 107.715184ms
> > size: 160
> > I0224 13:29:05.000203 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:28:30 +0000 UTC, end: 2016-02-24 13:29:00 +0000 UTC
> > I0224 13:29:05.118256 1 manager.go:152] ScrapeMetrics: time: 117.97843ms
> > size: 160
> > I0224 13:29:35.000209 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:29:00 +0000 UTC, end: 2016-02-24 13:29:30 +0000 UTC
> > I0224 13:29:35.106305 1 manager.go:152] ScrapeMetrics: time: 106.009207ms
> > size: 160
> > I0224 13:30:05.000266 1 manager.go:79] Scraping metrics start: 2016-02-24
> > 13:29:30 +0000 UTC, end: 2016-02-24 13:30:00 +0000 UTC
> > I0224 13:30:05.113913 1 manager.go:152] ScrapeMetrics: time: 113.477259ms
> > size: 160
> >
> > metrics deployer:
> > ++ '[' -n '' ']'
> > Deploying the Heapster component
> > ++ oc process hawkular-heapster -v
> > IMAGE_PREFIX=docker.io/openshift/origin-,IMAGE_VERSION=latest,MASTER_URL=https://kubernetes.default.svc:443
> > ++ oc create -f -
> > serviceaccount "heapster" created
> > service "heapster" created
> > replicationcontroller "heapster" created
> > + echo 'Success!'
> > Success!
> >
> > Hawkular-metrics:
> > [0m[0m12:59:48,619 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051:
> > Admin console listening on http://127.0.0.1:9990
> > [0m[0m12:59:48,620 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025:
> > WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 16058ms -
> > Started 374 of 664 services (385 services are lazy, passive or on-demand)
> > [0m[0m12:59:52,446 INFO
> > [org.hawkular.metrics.core.service.MetricsServiceImpl]
> > (metricsservice-lifecycle-thread) HAWKMETRICS100001: Using a key space of
> > 'hawkular_metrics'
> > [0m[0m12:59:52,561 INFO
> > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200005: Metrics service started
> >
> > _______________________________________________ users mailing list
> > users lists openshift redhat com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >
> > _______________________________________________
> > users mailing list
> > users lists openshift redhat com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >

_______________________________________________ users mailing list users lists openshift redhat com http://lists.openshift.redhat.com/openshiftmm/listinfo/users

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]