[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: HAProxy gear logs



Hi Igor,
    From the logs you have only 11 sessions, so 3 gears should be enough. 

    But from your haproxy-status, one of those gears is down - which could be an issue - can you please check the haproxy.cfg for that "down" gear 
    and see if you can get to that gear's server? curl  $gear_d7f86b8193_ip:$gear_d7f86b8193_exposed_port    -  
    and also ssh into that gear and see what the logs say. 
   
    Let's see what that error is first.  On a related note, looks like there might be an issue w/ the auto-scaler "counting" that gear even though its down. 

    Thanks, 

Ram//
main(O,s){s=--O;10<putchar(3^O?97-(15&7183>>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}

On Jan 12, 2013, at 10:24 AM, Igor Laskovy wrote:

Thanks Mrunal, Mike!

Well, at my case I just create simple php app (only index.html and couple img on it) with -s and then try to load it with two simultaneously ab. 
At the beginning lab had figure out my load and add additional gear, but after some moments something goes wrong and all scaling feature (up and down) nod working at all. 

What I can see:

From haproxy-status page:

stats
QueueSession rate SessionsBytesDeniedErrors WarningsServer
CurMaxLimit CurMaxLimitCur MaxLimitTotalLbTot InOutReqResp ReqConnRespRetr RedisStatusLastChkWght ActBckChkDwn DwntmeThrtle
Frontend 1 2 -1 13000 45 42977 5795870 00 OPEN
Backend 00 00 003000 0 042977 579587 00 00 00 18m20s UP 0 00  0 

express
QueueSession rateSessions BytesDeniedErrors WarningsServer
CurMaxLimitCur MaxLimitCurMax LimitTotalLbTotIn OutReqRespReq ConnRespRetrRedis StatusLastChkWghtAct BckChkDwnDwntme Thrtle
Frontend 0 16 -6767 3000 95 74541403 903578060 00 0 OPEN
filler 00 -00 00 -0 00 00 0 0 00 1- Y-
gear-27b9d03820-nspace 00 -122131 29 30- 117210 11720537267033 451714235 0 0 0 0018m20s UP L7OK/200 in 249ms 1Y- 0 00s-
gear-cb5ac386ba-nspace 00 -122131 29 30- 117244 11724437274370 451863825 0 0 0 0018m20s UP L7OK/200 in 228ms 1Y- 0 00s-
gear-d7f86b8193-nspace 00 -04 03 -12 40 00 0 0 8418m19s DOWN L4CON in 0ms 1Y -0 118m19s -
local-gear 00 -00 00 -0 00 00 0 0 00 0Y --
Backend 0 0 244261 58 603000 234454 234453 74541403 903578060 0 0 0 0 84 18m20s UP  22 1  00s

From haproxy-1.4/logs/scale_events.log for that app:

# Logfile created on Sat Jan 12 19:35:15 +0200 2013 by logger.rb/1.2.6
I, [2013-01-12T19:35:21.708074 #17464]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:35:21.711129 #17464] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-01-12T19:43:00.607716 #21946]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:43:00.610906 #21946] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-01-12T19:45:40.098578 #21946]  INFO -- : GEAR_UP - capacity: 90.0% gear_count: 1 sessions: 9 up_thresh: 90.0%
I, [2013-01-12T19:46:44.906404 #23436]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:46:44.920943 #23436] DEBUG -- : GEAR_INFO - capacity: 40.0% gear_count: 1 sessions: 4 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-01-12T19:46:45.006991 #23436]  INFO -- : GEAR_UP - capacity: 110.0% gear_count: 1 sessions: 11 up_thresh: 90.0%
I, [2013-01-12T19:47:09.326502 #23662]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:47:09.447347 #23662] DEBUG -- : GEAR_INFO - capacity: 50.0% gear_count: 2 sessions: 10 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-01-12T19:47:45.916311 #24023]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:47:46.014610 #24023] DEBUG -- : GEAR_INFO - capacity: 50.0% gear_count: 2 sessions: 11 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-01-12T19:48:01.015038 #24165]  INFO -- : Starting haproxy_ctld
D, [2013-01-12T19:48:01.019328 #24165] DEBUG -- : GEAR_INFO - capacity: 30.0% gear_count: 3 sessions: 11 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20

Last two messages at /var/log/mcollective.log at that node where haproxy gear for this app reside:

I, [2013-01-12T19:48:01.145103 #1410]  INFO -- : openshift.rb:612:in `reap_output' cartridge_do_action (0)
------
Sat Jan 12 19:47:48 EET 2013: Conditionally reloading HAProxy service

------)
I, [2013-01-12T19:48:01.147980 #1410]  INFO -- : openshift.rb:572:in `execute_parallel_action' execute_parallel_action call - 10 taggear91153799a9db4afe8babf9a37559a619jobargs--gear-uuid91153799a9db4afe8babf9a37559a619--input-argsm4 nspace 91153799a9db4afe8babf9a37559a619 \'27b9d03820f4451085fa0aa470062fcc\'\=\'27b9d03820-nspace.oshift.labspace.studiogrizzly.com\|192.168.2.151:35541'
'\'\ \'cb5ac386ba0343e6b99ae3d12095db63\'\=\'cb5ac386ba-nspace.oshift.labspace.studiogrizzly.com\|192.168.2.151:35551'
'\'\ \'d7f86b8193004976b330ec21728015d8\'\=\'d7f86b8193-nspace.oshift.labspace.studiogrizzly.com\|192.168.2.153:35541'
'\'--hook-nameset-proxy--cart-namehaproxy-1.4actionconnector-executecartridgeopenshift-origin-noderesult_exit_code0result_stdoutSat Jan 12 19:47:48 EET 2013: Conditionally reloading HAProxy service
result_stderr


On Fri, Jan 11, 2013 at 3:06 AM, Mike McGrath <mmcgrath redhat com> wrote:
On Thu, 10 Jan 2013, Igor Laskovy wrote:

> Hi All!
> Kindly ask clarify what logs should I look when troubleshooting scaling behavior. 
>
> I have test load by pointing two simultaneously "ab -c 11" to my test app on my local openshift lab and on haproxy-status page I can see:
>

Let us know what's working, we've got plenty of people using scaled apps
today but not a lot of feedback on it when it does/doesn't work.  Any
details about what you're doing and what you're seeing would be greatly
appreciated.

        -Mike

> stats
> Queue
> Session rate
> Sessions
> Bytes
> Denied
> Errors
> Warnings
> Server
> Cur
> Max
> Limit
> Cur
> Max
> Limit
> Cur
> Max
> Limit
> Total
> LbTot
> In
> Out
> Req
> Resp
> Req
> Conn
> Resp
> Retr
> Redis
> Status
> LastChk
> Wght
> Act
> Bck
> Chk
> Dwn
> Dwntme
> Thrtle
> Frontend
> 1
> 2
> -
> 1
> 1
> 3000
> 24
> 17766
> 263810
> 0
> 0
> 0
> OPEN
> Backend
> 0
> 0
> 0
> 0
> 0
> 0
> 3000
> 0
> 0
> 17766
> 263810
> 0
> 0
> 0
> 0
> 0
> 0
> 16m58s UP
>  
> 0
> 0
> 0
>  
> 0
>  
>
> express
> Queue
> Session rate
> Sessions
> Bytes
> Denied
> Errors
> Warnings
> Server
> Cur
> Max
> Limit
> Cur
> Max
> Limit
> Cur
> Max
> Limit
> Total
> LbTot
> In
> Out
> Req
> Resp
> Req
> Conn
> Resp
> Retr
> Redis
> Status
> LastChk
> Wght
> Act
> Bck
> Chk
> Dwn
> Dwntme
> Thrtle
> Frontend
> 0
> 8
> -
> 29
> 29
> 3000
> 57
> 45660283
> 772931274
> 0
> 0
> 0
> OPEN
> filler
> 0
> 0
> -
> 0
> 0
> 0
> 0
> -
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 1
> -
> Y
> -
> gear-d6b6cbae5e-nspace
> 0
> 0
> -
> 89
> 100
> 20
> 21
> -
> 76855
> 76832
> 24443323
> 413644074
> 0
> 0
> 0
> 0
> 0
> 16m58s UP
> L7OK/200 in 226ms
> 1
> Y
> -
> 0
> 0
> 0s
> -
> local-gear
> 0
> 0
> -
> 80
> 82
> 2
> 2
> 2
> 66722
> 66722
> 21216960
> 359287200
> 0
> 0
> 0
> 0
> 0
> 16m58s UP
> L7OK/200 in 33ms
> 1
> Y
> -
> 0
> 0
> 0s
> -
> Backend
> 0
> 0
> 169
> 180
> 22
> 23
> 3000
> 143577
> 143554
> 45660283
> 772931274
> 0
> 0
> 0
> 0
> 0
> 0
> 16m58s UP
>  
> 2
> 2
> 1
>  
> 0
> 0s
>
> --
> Igor Laskovy
> facebook.com/igor.laskovy
> Kiev, Ukraine
>
>



--
Igor Laskovy
facebook.com/igor.laskovy
Kiev, Ukraine
_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]