[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[solved] sample app deploy also failed Re: [v3] docker-registry deploy doesn't



Continuing with the same separate hosts and a functional registry now, I had problems at the next step of https://github.com/sosiouxme/origin/blob/master/examples/sample-app/README.md:

# osc get projects
NAME                      DISPLAY NAME        NAMESPACE
hello-openshift-project   Hello OpenShift     hello-openshift-project
# openshift cli process -f application-template-stibuild.json | openshift cli apply --namespace=hello-openshift-project -f -
I0116 15:40:27.982323   17242 apply.go:59] Creation succeeded for Service with name frontend
I0116 15:40:27.984247   17242 apply.go:59] Creation succeeded for ImageRepository with name origin-ruby-sample
I0116 15:40:27.984739   17242 apply.go:59] Creation succeeded for ImageRepository with name ruby-20-centos
I0116 15:40:27.985284   17242 apply.go:59] Creation succeeded for BuildConfig with name ruby-sample-build
I0116 15:40:27.985826   17242 apply.go:59] Creation succeeded for DeploymentConfig with name frontend
I0116 15:40:27.986403   17242 apply.go:59] Creation succeeded for Service with name database
I0116 15:40:27.986801   17242 apply.go:59] Creation succeeded for DeploymentConfig with name database
# curl -X POST http://localhost:8080/osapi/v1beta1/buildConfigHooks/ruby-sample-build/secret101/generic?namespace=hello-openshift-project


Sounds good, but when I look to see builds happening, pods being created, etc. - nothing new:
# osc get pods
POD                                    CONTAINER(S)         IMAGE(S)                    HOST                LABELS                                                                           STATUS
f18b3fb7-9d8f-11e4-a2d8-fa163e33787b   registry-container   openshift/docker-registry   172.16.4.201/       deployment=docker-registry-1,deploymentconfig=docker-registry,name=registrypod   Running
# osc get services
NAME                LABELS                                    SELECTOR            IP                  PORT
docker-registry     <none>                                    name=registrypod    172.30.17.3         5001
kubernetes          component=apiserver,provider=kubernetes   <none>              172.30.17.253       443
kubernetes-ro       component=apiserver,provider=kubernetes   <none>              172.30.17.251       80
# osc get imagerepository -o json origin-ruby-sample
F0119 12:43:35.998836   18892 get.go:75] imageRepository "origin-ruby-sample" not found
# osc get imagerepository -o json ruby-20-centos
F0119 12:43:46.869342   18900 get.go:75] imageRepository "ruby-20-centos" not found
# osc get buildconfig -o json ruby-sample-build
F0119 12:44:20.975768   18927 get.go:75] buildConfig "ruby-sample-build" not found
# osc get deploymentconfig -o json frontend 
F0119 12:44:51.961399   18943 get.go:75] deploymentConfig "frontend" not found
# osc get deploymentconfig -o json database
F0119 12:44:56.710531   18951 get.go:75] deploymentConfig "database" not found

The master logs showed things basically in a holding pattern without any real errors (http://fpaste.org/171445/). The problem was on the node, where I'd left openshift running a few days at this point and it had filled my tiny test filesystem with lots of exited docker containers.

There's still no "docker rm --exited" or the like (https://github.com/docker/docker/issues/928) but "docker rm -v $(docker ps -aq)" does slowly clean things out. At least, until I ran into https://github.com/docker/docker/issues/9644 where journalctl just reports:
Jan 19 14:46:35 lmeyer-0113-f21osv3-3da73419-06c1-45c5-a091-d18ccd14c5ee docker[9066]: time="2015-01-19T14:46:35Z" level="fatal" msg="Error loading device metadata file .tmp274926013"

If you're a docker n00b, it's not completely obvious that it's referring specifically to /var/lib/docker/devicemapper/metadata/.tmp274926013 and you can apparently just nuke that to allow docker to restart successfully. And then openshift/kubernetes picks up and does whatever's needed to deploy what the master is expecting to be there.

Well, sort of. The STI builder seems to be running repeatedly on the node but it's hard to tell what else is going on (still nothing showing up "osc get <type>" though I may just not know the right ones).

Anyway, I guess the lesson is, give docker some space, and clear out exited containers regularly.

----- Original Message -----
From: "Luke Meyer" <lmeyer redhat com>
To: "Clayton Coleman" <ccoleman redhat com>
Cc: "OpenShift Development" <dev lists openshift redhat com>
Sent: Thursday, January 15, 2015 2:16:22 PM
Subject: Re: [v3] docker-registry deploy doesn't

I killed the node server and added that to the startup, and I now have containers running on the node :)

I really didn't see anything amiss in the logs, though. "osc get minions" reported the node, I could see health checks coming in on the node, there was stuff like this in the master log:
I0115 18:04:11.167747   22569 factory.go:289] Attempting to bind e8434f6e-9ce0-11e4-b0ed-fa163e33787b to 172.16.4.201
... but no followup "hey, what's that? sorry, can't bind it there after all, actually I have no idea who this node is" unless I'm missing something. So, an opportunity for more troubleshooting info somewhere!

Thanks!

----- Original Message -----
From: "Clayton Coleman" <ccoleman redhat com>
To: "Luke Meyer" <lmeyer redhat com>
Cc: "OpenShift Development" <dev lists openshift redhat com>
Sent: Thursday, January 15, 2015 2:03:04 PM
Subject: Re: [v3] docker-registry deploy doesn't

Oh, I looked at what you wrote - add "--hostname 172.16.4.201" to your node startup.  The hostname of the node is the "identity" of the node, and it defaults to hostname -f

----- Original Message -----
> Not making it out of unknown almost certainly means that the master can't
> find the nodes.
> 
> Do "osc describe pod e8434f6e-9ce0-11e4-b0ed-fa163e33787b" and I bet you'll
> see a bunch of errors about being unable to schedule.
> 
> The master log should probably have health check warnings on your node.
> 
> ----- Original Message -----
> > I'm trying to follow the sample-app deploy at
> > https://github.com/openshift/origin/blob/master/examples/sample-app/README.md
> > and getting tripped up early on with deploying the docker-registry.
> > Pointers
> > would be appreciated as I have basically no idea how to find out what's
> > wrong.
> > 
> > The context is that I have compiled OpenShift from source and run it as
> > master on one host, node on another:
> > 
> > master# openshift start master --nodes=172.16.4.201 --loglevel=4
> > node# openshift start node --master=172.16.4.200 --loglevel=4
> > 
> > The hosts run Fedora 21, SELinux is permissive, firewalld isn't on, and
> > I've
> > added the --insecure-registry flag though it seems unlikely to be relevant
> > yet.
> > 
> > I do pullimages on the node to get docker warmed up. Then I try to apply
> > the
> > docker registry conf:
> > 
> > master [sample-app]# osc apply -f docker-registry-config.json
> > I0115 18:04:10.873990   22621 apply.go:59] Creation succeeded for Service
> > with name docker-registry
> > I0115 18:04:10.874574   22621 apply.go:59] Creation succeeded for
> > DeploymentConfig with name docker-registry
> > 
> > So this seems to be working fine, but the pod never makes it out of
> > "Unknown"
> > state, and the container never appears on the node according to docker ps
> > -a.
> > 
> > # osc get pods
> > POD                                    CONTAINER(S)        IMAGE(S)
> > HOST                LABELS              STATUS
> > e8434f6e-9ce0-11e4-b0ed-fa163e33787b   deployment
> > openshift/origin-deployer   172.16.4.201/       <none>
> > Unknown
> > 
> > # osc get services
> > NAME                LABELS                                    SELECTOR
> > IP                  PORT
> > kubernetes          component=apiserver,provider=kubernetes   <none>
> > 172.30.17.92        443
> > kubernetes-ro       component=apiserver,provider=kubernetes   <none>
> > 172.30.17.189       80
> > docker-registry     <none>
> > name=registrypod    172.30.17.3
> > 5001
> > 
> > # osc get deploymentconfig
> > NAME                TRIGGERS            LATEST VERSION
> > docker-registry     ConfigChange        1
> > 
> > # osc get pods
> > POD                                    CONTAINER(S)        IMAGE(S)
> > HOST                LABELS              STATUS
> > e8434f6e-9ce0-11e4-b0ed-fa163e33787b   deployment
> > openshift/origin-deployer   172.16.4.201/       <none>
> > Unknown
> > 
> > 
> > I'm attaching master and node logs... there's probably something indicative
> > in there but I can't decipher it at this point. More importantly, is there
> > any way to query openshift/kubernetes itself to see what is going wrong
> > with
> > this pod?
> > 
> > 
> > _______________________________________________
> > dev mailing list
> > dev lists openshift redhat com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> >

_______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]