[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: a weird problem: app create fails, but app is really created and accessible



Sorry for the delay in responding.

If the proxy timeout was already longer than 70 seconds, increasing it wouldn't make a difference. It sounds like the TimeOut option you found in the main config was the reason for the client error you saw. The default Timeout is 300. I think you want that to be as high as your ProxyTimeout.

We need to be clear about what is being slow here. It is unlikely that the network communication between your broker and node is slow, given they're on the same host (although possible with issues like /etc/resolv.conf having a broken first nameserver, but you'd probably notice that in other ways). It looks like the gear creation itself is simply taking 70 seconds, and everything is just waiting on that to happen (and timing out if it takes too long). Gear creation can take some time, to be sure, but I would expect it to be much faster than 70 seconds for a DIY cartridge (there isn't even any server to eait on starting). I would suggest looking through the various node logs more closely. No problems are evident from the snippets you've given, but you may want to look in /var/log/mcollective.log (may need to bump up the log level in /etc/mcollective/server.conf) to see if the node will tell you what's going on.

If your host was cloned from another that was working faster, perhaps you have not replaced all instances of the original host's IP. Perhaps a DNS lookup is having to time out, or something else related to a wrong IP. It is hard to tell without a lot of hands-on detective work. Look for nameserver and other timeouts in any logs. Also look in /var/log/openshift/broker/httpd/error_log - I'm actually not sure where a DDNS update failure would be recorded, but it might be here.

----- Original Message -----
From: "Shoumeng Yan" <shoumeng yan intel com>
To: "Shoumeng Yan" <shoumeng yan intel com>, "Luke Meyer" <lmeyer redhat com>
Cc: dev lists openshift redhat com, "Shoumeng Yan" <shoumeng yan intel com>
Sent: Wednesday, June 12, 2013 11:18:39 PM
Subject: RE: a weird problem: app create fails, but app is really created and	accessible

I found the option in /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf. The value specified is already 300. I changed it to 1200, but still got the same error. And, I noticed the code 502 still associated with 60xxx ms. Seems the change I made did not take effect at all. Maybe the config file I was using is not the one you meant?

And, what is the potential reasons for being so slow to get response from node to broker? In my case, node and broker are located on one single VM. 

[root localhost syan3]# rhc app create mydiy7 diy -d
DEBUG: Connecting to https://localhost.localdomain/broker/rest/api
DEBUG: Getting all cartridges
DEBUG: Request GET https://localhost.localdomain/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 502 61225 ms
DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET
DEBUG: Request GET https://localhost.localdomain/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200 14909 ms
DEBUG: Request GET https://localhost.localdomain/broker/rest/cartridges
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200  518 ms
Using diy-0.1 (Do-It-Yourself) for 'diy'
DEBUG: Getting all domains
DEBUG: Request GET https://localhost.localdomain/broker/rest/domains
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200 1185 ms
DEBUG: Getting all domains
DEBUG: Finding domain mypaas
DEBUG: Getting all domains

Application Options
-------------------
DEBUG: Getting all domains
  Namespace:  mypaas
  Cartridges: diy-0.1
  Gear Size:  default
  Scaling:    no

Creating application 'mydiy7' ... DEBUG: Creating application 'mydiy7' with these options - {:cartridges=>["diy-0.1"]}
DEBUG: Adding application mydiy7 to domain mypaas
DEBUG: Request POST https://localhost.localdomain/broker/rest/domains/mypaas/applications
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 502 60346 ms
DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET
An error occurred while communicating with the server. This problem may only be
temporary. Check that you have correctly specified your proxy server
'http://proxy-sc.intel.com:912' as well as your OpenShift server
'https://localhost.localdomain/broker/rest/domains/mypaas/applications'.

Thanks,
Shoumeng


-----Original Message-----
From: Yan, Shoumeng 
Sent: Thursday, June 13, 2013 10:34 AM
To: Luke Meyer
Cc: dev lists openshift redhat com; Yan, Shoumeng
Subject: RE: a weird problem: app create fails, but app is really created and accessible

Hi Luke, 

Which config file specifies the ProxyTimeout option?

Thanks,
Shoumeng


-----Original Message-----
From: Luke Meyer [mailto:lmeyer redhat com] 
Sent: Monday, June 10, 2013 9:32 AM
To: Yan, Shoumeng
Cc: dev lists openshift redhat com
Subject: Re: a weird problem: app create fails, but app is really created and accessible

A 502 error is basically the proxy reporting that something went wrong with the response (i.e. it didn't get one).

On the broker host there is an httpd proxy externally-facing on the host which proxies back to the broker webapp (which binds to localhost). Here it looks like the app create took about 70s. Perhaps the ProxyTimeout on the host httpd is set to 60 or some such? It really should be set to 240 or higher; the broker is often sitting around waiting for the node to do something. The only real error is that the success of the operation didn't make it back to the client.

----- Original Message -----
From: "Shoumeng Yan" <shoumeng yan intel com>
To: dev lists openshift redhat com
Sent: Saturday, June 8, 2013 11:27:43 PM
Subject: a weird problem: app create fails, but app is really created and	accessible





Hi there, 



My openshift origin installation met a weird issue. When I create app, it reports some error (code 502). But the app is actually created (rhc apps show the app) and also accessible using git and ssh. 



I attached below my related logs where I did not see an obvious issue. Hope someone could provide some hints on further investigation. Thanks. 



Thanks, 

Shoumeng 



[root localhost ~]# rhc app create mydiy7 diy -d 

DEBUG: Connecting to https://localhost.localdomain/broker/rest/api 

DEBUG: Getting all cartridges 

DEBUG: Request GET https://localhost.localdomain/broker/rest/api 

DEBUG: SSL Verification failed -- Using self signed cert 

DEBUG: code 200 1382 ms 

DEBUG: Request GET https://localhost.localdomain/broker/rest/cartridges 

DEBUG: SSL Verification failed -- Using self signed cert 

DEBUG: code 200 704 ms 

Using diy-0.1 (Do-It-Yourself) for 'diy' 

DEBUG: Getting all domains 

DEBUG: Request GET https://localhost.localdomain/broker/rest/domains 

DEBUG: SSL Verification failed -- Using self signed cert 

DEBUG: code 200 793 ms 

DEBUG: Getting all domains 

DEBUG: Finding domain mypaas 

DEBUG: Getting all domains 



Application Options 

------------------- 

DEBUG: Getting all domains 

Namespace: mypaas 

Cartridges: diy-0.1 

Gear Size: default 

Scaling: no 



Creating application 'mydiy7' ... DEBUG: Creating application 'mydiy7' with these options - {:cartridges=>["diy-0.1"]} 

DEBUG: Adding application mydiy7 to domain mypaas 

DEBUG: Request POST https://localhost.localdomain/broker/rest/domains/mypaas/applications 

DEBUG: SSL Verification failed -- Using self signed cert 

DEBUG: code 502 60483 ms 

DEBUG: ERROR: Received bad gateway from server, will retry once if this is a GET 

An error occurred while communicating with the server. This problem may only be 

temporary. Check that you have correctly specified your proxy server 

'http://proxy-sc.mycorp.com:1111' as well as your OpenShift server 

'https://localhost.localdomain/broker/rest/domains/mypaas/applications'. 



[root localhost broker]# tail /var/log/openshift/broker/user_action.log 

SUCCESS DATE=2013-06-09 TIME=11:06:15 ACTION=LIST_CARTRIDGES REQ_ID=6659e6c04d9a1ac51c5d08789c8a21ac USER_ID= LOGIN= List all cartridges 

SUCCESS DATE=2013-06-09 TIME=11:06:16 ACTION=AUTHENTICATE REQ_ID=7b06fe00205d9854c9980e3ab50e6fe0 USER_ID=51ae13cd30050a3592000001 LOGIN=admin IP=127.0.0.1 SCOPES=session Authenticated 

SUCCESS DATE=2013-06-09 TIME=11:06:17 ACTION=LIST_DOMAINS REQ_ID=7b06fe00205d9854c9980e3ab50e6fe0 USER_ID=51ae13cd30050a3592000001 LOGIN=admin 

SUCCESS DATE=2013-06-09 TIME=11:06:17 ACTION=AUTHENTICATE REQ_ID=90d9ed2251371ddc6755604de61019bf USER_ID=51ae13cd30050a3592000001 LOGIN=admin IP=127.0.0.1 SCOPES=session Authenticated 

SUCCESS DATE=2013-06-09 TIME=11:07:27 ACTION=ADD_APPLICATION REQ_ID=90d9ed2251371ddc6755604de61019bf USER_ID=51ae13cd30050a3592000001 LOGIN=admin APP=mydiy7 DOMAIN=mypaas APP_UUID=51b3f129e13823d5d300002c Application mydiy7 was created. 



[root localhost broker]# tail /var/log/openshift/broker/development.log 

MOPED: 127.0.0.1:27017 UPDATE database=openshift_broker_dev collection=applications selector={"_id"=>"51b3f129e13823d5d300002c"} update={"$set"=>{"pending_op_groups.0.pending_ops.11.state"=>:completed}} flags=[] 

COMMAND database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (2.5373ms) 

MOPED: 127.0.0.1:27017 UPDATE database=openshift_broker_dev collection=applications selector={"_id"=>"51b3f129e13823d5d300002c"} update={"$pull"=>{"pending_op_groups"=>{"_id"=>"51b3f129e13823d5d300002d"}}} flags=[] 

COMMAND database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (4.5893ms) 

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=applications selector={:_id=>"51b3f129e13823d5d300002c"} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.2468ms) 

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=domains selector={"$query"=>{"_id"=>"51ae13da30050a3592000005"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (8.6658ms) 

MOPED: 127.0.0.1:27017 COMMAND database=openshift_broker_dev command={:findAndModify=>"locks", :query=>{"user_id"=>"51ae13cd30050a3592000001", "locked"=>false, "app_ids.51b3f129e13823d5d300002c"=>{"$exists"=>true}}, :new=>true, :update=>{"$unset"=>{"app_ids.51b3f129e13823d5d300002c"=>""}}} (0.4041ms) 

MOPED: 127.0.0.1:27017 QUERY database=openshift_broker_dev collection=domains selector={:_id=>"51ae13da30050a3592000005"} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (2.2936ms) 

SUCCESS ACTION=ADD_APPLICATION USER_ID=51ae13cd30050a3592000001 LOGIN=admin APP=mydiy7 DOMAIN=mypaas APP_UUID=51b3f129e13823d5d300002c Application mydiy7 was created. Application mydiy7 was created., 172.16.117.49, Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on Openshift. 

Completed 201 Created in 70137ms (Views: 17.0ms) 



[root localhost broker]# tail /var/log/openshift/broker/httpd/access_log 

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:13 +0800] "GET /broker/rest/api HTTP/1.1" 200 3450 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))" 

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:14 +0800] "GET /broker/rest/cartridges HTTP/1.1" 200 11414 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))" 

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:16 +0800] "GET /broker/rest/domains HTTP/1.1" 200 2098 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))" 

127.0.0.1 localhost.localdomain - - [09/Jun/2013:11:06:17 +0800] "POST /broker/rest/domains/mypaas/applications HTTP/1.1" 201 6858 "-" "rhc/1.7.5 (ruby 1.9.3; x86_64-linux) (2.3.3, ruby 1.9.3 (2012-11-10))" 



[root localhost broker]# tail /var/log/openshift/node/platform.log 

D, [2013-06-09T11:07:20.482391 #21265] DEBUG -- : Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-21265-9s7xba/new.db' ran. rc=0 out= 

D, [2013-06-09T11:07:20.482580 #21265] DEBUG -- : httxt2dbm: /etc/httpd/conf.d/openshift/nodes: 0: stdout: stderr: 

I, [2013-06-09T11:07:20.803428 #2123] INFO -- : (0) 

------ 

Initialized empty Git repository in /var/lib/openshift/51b3f129e13823d5d300002c/git/mydiy7.git/ 

CLIENT_RESULT: Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on Openshift. 



------) 

D, [2013-06-09T11:07:21.426829 #2123] DEBUG -- : Creating v1 model for 51b3f129e13823d5d300002c: initialize 

D, [2013-06-09T11:07:24.715672 #2123] DEBUG -- : Creating v1 model for 51b3f129e13823d5d300002c: initialize 



[root localhost broker]# tail /var/log/openshift/node/platform-trace.log 

D, [2013-06-09T10:56:03.485480 #15611] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-15611-1ff1c1i/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>} 

D, [2013-06-09T11:07:19.795604 #21265] DEBUG -- : oo_spawn running service openshift-node-web-proxy reload: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>} 

D, [2013-06-09T11:07:19.983377 #21265] DEBUG -- : oo_spawn buffer(9/) Reloading node-web-proxy: 

D, [2013-06-09T11:07:19.997423 #21265] DEBUG -- : oo_spawn buffer(9/) [ 

D, [2013-06-09T11:07:19.997565 #21265] DEBUG -- : oo_spawn buffer(9/) OK 

D, [2013-06-09T11:07:19.997671 #21265] DEBUG -- : oo_spawn buffer(9/) ] 

D, [2013-06-09T11:07:20.008338 #21265] DEBUG -- : oo_spawn buffer(9/) 

D, [2013-06-09T11:07:20.008596 #21265] DEBUG -- : oo_spawn buffer(9/) 



D, [2013-06-09T11:07:20.106700 #21265] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130609-21265-9s7xba/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 10>, :err=>#<IO:fd 8>} _______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]