[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: a weird problem: app create fails, but app is really created and accessible



Title: RE: a weird problem: app create fails, but app is really created and accessible
I solved this issue after I changed the Timeout config option in /etc/httpd/conf/httpd.conf. Don't need to change the ProxyTimeout mentioned in last email.

This operation( POST https://localhost.localdomain/broker/rest/domains/mypaas/applications ) still takes longer than ‎60 seconds. I don't understand why so slow for an all-in-one instance.
Btw, this VM is cloned without change from another VM which performs normally fast. Even more strange...why cloned vm behaves differently in this regard

Shoumeng

-----Original Message-----
From: Yan, Shoumeng
Sent: Thursday, June 13, 2013 11:18 AM Malay Peninsula Standard Time
To: Yan, Shoumeng; Luke Meyer
Cc: dev lists openshift redhat com; Yan, Shoumeng
Subject: RE: a weird problem: app create fails, but app is really created and accessible

I found the option in /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf. The value specified is already 300. I changed it to 1200, but still got the same error. And, I noticed the code 502 still associated with 60xxx ms. Seems the change I made did not take effect at all. Maybe the config file I was using is not the one you meant?

And, what is the potential reasons for being so slow to get response from node to broker? In my case, node and broker are located on one single VM.

[root localhost syan3]# rhc app create mydiy7 diy -d
DEBUG: Connecting to https://localhost.localdomain/broker/rest/api
DEBUG: Getting all cartridges
DEBUG: Request GET https://localhost.localdomain/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 502 61225 ms
DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET
DEBUG: Request GET https://localhost.localdomain/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200 14909 ms
DEBUG: Request GET https://localhost.localdomain/broker/rest/cartridges
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200  518 ms
Using diy-0.1 (Do-It-Yourself) for 'diy'
DEBUG: Getting all domains
DEBUG: Request GET https://localhost.localdomain/broker/rest/domains
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200 1185 ms
DEBUG: Getting all domains
DEBUG: Finding domain mypaas
DEBUG: Getting all domains

Application Options
-------------------
DEBUG: Getting all domains
  Namespace:  mypaas
  Cartridges: diy-0.1
  Gear Size:  default
  Scaling:    no

Creating application 'mydiy7' ... DEBUG: Creating application 'mydiy7' with these options - {:cartridges=>["diy-0.1"]}
DEBUG: Adding application mydiy7 to domain mypaas
DEBUG: Request POST https://localhost.localdomain/broker/rest/domains/mypaas/applications
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 502 60346 ms
DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET
An error occurred while communicating with the server. This problem may only be
temporary. Check that you have correctly specified your proxy server
'http://proxy-sc.intel.com:912' as well as your OpenShift server
'https://localhost.localdomain/broker/rest/domains/mypaas/applications'.

Thanks,
Shoumeng


-----Original Message-----
From: Yan, Shoumeng
Sent: Thursday, June 13, 2013 10:34 AM
To: Luke Meyer
Cc: dev lists openshift redhat com; Yan, Shoumeng
Subject: RE: a weird problem: app create fails, but app is really created and accessible

Hi Luke,

Which config file specifies the ProxyTimeout option?

Thanks,
Shoumeng


-----Original Message-----
From: Luke Meyer [mailto:lmeyer redhat com]
Sent: Monday, June 10, 2013 9:32 AM
To: Yan, Shoumeng
Cc: dev lists openshift redhat com
Subject: Re: a weird problem: app create fails, but app is really created and accessible

A 502 error is basically the proxy reporting that something went wrong with the response (i.e. it didn't get one).

On the broker host there is an httpd proxy externally-facing on the host which proxies back to the broker webapp (which binds to localhost). Here it looks like the app create took about 70s. Perhaps the ProxyTimeout on the host httpd is set to 60 or some such? It really should be set to 240 or higher; the broker is often sitting around waiting for the node to do something. The only real error is that the success of the operation didn't make it back to the client.

----- Original Message -----
From: "Shoumeng Yan" <shoumeng yan intel com>
To: dev lists openshift redhat com
Sent: Saturday, June 8, 2013 11:27:43 PM
Subject: a weird problem: app create fails, but app is really created and       accessible





Hi there,



My openshift origin installation met a weird issue. When I create app, it reports some error (code 502). But the app is actually created (rhc apps show the app) and also accessible using git and ssh.



I attached below my related logs where I did not see an obvious issue. Hope someone could provide some hints on further investigation. Thanks.



Thanks,

Shoumeng



[root localhost ~]# rhc app create mydiy7 diy -d

DEBUG: Connecting to https://localhost.localdomain/broker/rest/api

DEBUG: Getting all cartridges

DEBUG: Request GET https://localhost.localdomain/broker/rest/api

DEBUG: SSL Verification failed -- Using self signed cert

DEBUG: code 200 1382 ms

DEBUG: Request GET https://localhost.localdomain/broker/rest/cartridges

DEBUG: SSL Verification failed -- Using self signed cert

DEBUG: code 200 704 ms

Using diy-0.1 (Do-It-Yourself) for 'diy'

DEBUG: Getting all domains

DEBUG: Request GET https://localhost.localdomain/broker/rest/domains

DEBUG: SSL Verification failed -- Using self signed cert

DEBUG: code 200 793 ms

DEBUG: Getting all domains

DEBUG: Finding domain mypaas

DEBUG: Getting all domains



Application Options

-------------------

DEBUG: Getting all domains

Namespace: mypaas

Cartridges: diy-0.1

Gear Size: default

Scaling: no



Creating application 'mydiy7' ... DEBUG: Creating application 'mydiy7' with these options - {:cartridges=>["diy-0.1"]}

DEBUG: Adding application mydiy7 to domain mypaas

DEBUG: Request POST https://localhost.localdomain/broker/rest/domains/mypaas/applications

DEBUG: SSL Verification failed -- Using self signed cert

DEBUG: code 502 60483 ms

DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET

An error occurred while communicating with the server. This problem may only be

temporary. Check that you have correctly specified your proxy server

'http://proxy-sc.mycorp.com:1111' as well as your OpenShift server

'https://localhost.localdomain/broker/rest/domains/mypaas/applications'.



[root localhost broker]# tail /var/log/openshift/broker/user_action.log

SUCCESS DATE=2013-06-09 TIME=11:06:15 ACTION="" REQ_ID=6659e6c04d9a1ac51c5d08789c8a21ac USER_ID= LOGIN= List all cartridges

SUCCESS DATE=2013-06-09 TIME=11:06:16 ACTION="" REQ_ID=7b06fe00205d9854c9980e3ab50e6fe0 USER_ID=51ae13cd30050a3592000001 LOGIN=admin IP=127.0.0.1 SCOPES=session Authenticated

SUCCESS DATE=2013-06-09 TIME=11:06:17 ACTION="" REQ_ID=7b06fe00205d9854c9980e3ab50e6fe0 USER_ID=51ae13cd30050a3592000001 LOGIN=admin

SUCCESS DATE=2013-06-09 TIME=11:06:17 ACTION="" REQ_ID=90d9ed2251371ddc6755604de61019bf USER_ID=51ae13cd30050a3592000001 LOGIN=admin IP=127.0.0.1 SCOPES=session Authenticated

SUCCESS DATE=2013-06-09 TIME=11:07:27 ACTION="" REQ_ID=90d9ed2251371ddc6755604de61019bf USER_ID=51ae13cd30050a3592000001 LOGIN=admin APP=mydiy7 DOMAIN=mypaas APP_UUID=51b3f129e13823d5d300002c Application mydiy7 was created.



[root localhost broker]# tail /var/log/openshift/broker/development.log

MOPED: 127.0.0.1:27017 UPDATE database=openshift_broker_dev collection=applications selector={"_id"=>"51b3f129e13823d5d300002c"} update={"$set"=>{"pending_op_groups.0.pending_ops.11.state"=>:completed}} flags=[]

COMMAND database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (2.5373ms)

MOPED: 127.0.0.1:27017 UPDATE database=openshift_broker_dev collection=applications selector={"_id"=>"51b3f129e13823d5d300002c"} update={"$pull"=>{"pending_op_groups"=>{"_id"=>"51b3f129e13823d5d300002d"}}} flags=[]

COMMAND database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (4.5893ms)

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=applications selector={:_id=>"51b3f129e13823d5d300002c"} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.2468ms)

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=domains selector={"$query"=>{"_id"=>"51ae13da30050a3592000005"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (8.6658ms)

MOPED: 127.0.0.1:27017 COMMAND database=openshift_broker_dev command={:findAndModify=>"locks", :query=>{"user_id"=>"51ae13cd30050a3592000001", "locked"=>false, "app_ids.51b3f129e13823d5d300002c"=>{"$exists"=>true}}, :new=>true, :update=>{"$unset"=>{"app_ids.51b3f129e13823d5d300002c"=>""}}} (0.4041ms)

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=domains selector={:_id=>"51ae13da30050a3592000005"} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (2.2936ms)

SUCCESS ACTION="" USER_ID=51ae13cd30050a3592000001 LOGIN=admin APP=mydiy7 DOMAIN=mypaas APP_UUID=51b3f129e13823d5d300002c Application mydiy7 was created. Application mydiy7 was created., 172.16.117.49, Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on Openshift.

Completed 201 Created in 70137ms (Views: 17.0ms)



[root localhost broker]# tail /var/log/openshift/broker/httpd/access_log

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:13 +0800] "GET /broker/rest/api HTTP/1.1" 200 3450 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))"

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:14 +0800] "GET /broker/rest/cartridges HTTP/1.1" 200 11414 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))"

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:16 +0800] "GET /broker/rest/domains HTTP/1.1" 200 2098 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))"

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:17 +0800] "POST /broker/rest/domains/mypaas/applications HTTP/1.1" 201 6858 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))"



[root localhost broker]# tail /var/log/openshift/node/platform.log

D, [2013-06-09T11:07:20.482391 #21265] DEBUG -- : Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-21265-9s7xba/new.db' ran. rc=0 out=

D, [2013-06-09T11:07:20.482580 #21265] DEBUG -- : httxt2dbm: /etc/httpd/conf.d/openshift/nodes: 0: stdout: stderr:

I, [2013-06-09T11:07:20.803428 #2123] INFO -- : (0)

------

Initialized empty Git repository in /var/lib/openshift/51b3f129e13823d5d300002c/git/mydiy7.git/

CLIENT_RESULT: Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on Openshift.



------)

D, [2013-06-09T11:07:21.426829 #2123] DEBUG -- : Creating v1 model for 51b3f129e13823d5d300002c: initialize

D, [2013-06-09T11:07:24.715672 #2123] DEBUG -- : Creating v1 model for 51b3f129e13823d5d300002c: initialize



[root localhost broker]# tail /var/log/openshift/node/platform-trace.log

D, [2013-06-09T10:56:03.485480 #15611] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-15611-1ff1c1i/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>}

D, [2013-06-09T11:07:19.795604 #21265] DEBUG -- : oo_spawn running service openshift-node-web-proxy reload: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>}

D, [2013-06-09T11:07:19.983377 #21265] DEBUG -- : oo_spawn buffer(9/) Reloading node-web-proxy:

D, [2013-06-09T11:07:19.997423 #21265] DEBUG -- : oo_spawn buffer(9/) [

D, [2013-06-09T11:07:19.997565 #21265] DEBUG -- : oo_spawn buffer(9/) OK

D, [2013-06-09T11:07:19.997671 #21265] DEBUG -- : oo_spawn buffer(9/) ]

D, [2013-06-09T11:07:20.008338 #21265] DEBUG -- : oo_spawn buffer(9/)

D, [2013-06-09T11:07:20.008596 #21265] DEBUG -- : oo_spawn buffer(9/)



D, [2013-06-09T11:07:20.106700 #21265] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-21265-9s7xba/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>} _______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]