[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [solved] sample app deploy also failed Re: [v3] docker-registry deploy doesn't




----- Original Message -----
> Continuing with the same separate hosts and a functional registry now, I had
> problems at the next step of
> https://github.com/sosiouxme/origin/blob/master/examples/sample-app/README.md:
> 
> # osc get projects
> NAME                      DISPLAY NAME        NAMESPACE
> hello-openshift-project   Hello OpenShift     hello-openshift-project
> # openshift cli process -f application-template-stibuild.json | openshift cli
> apply --namespace=hello-openshift-project -f -
> I0116 15:40:27.982323   17242 apply.go:59] Creation succeeded for Service
> with name frontend
> I0116 15:40:27.984247   17242 apply.go:59] Creation succeeded for
> ImageRepository with name origin-ruby-sample
> I0116 15:40:27.984739   17242 apply.go:59] Creation succeeded for
> ImageRepository with name ruby-20-centos
> I0116 15:40:27.985284   17242 apply.go:59] Creation succeeded for BuildConfig
> with name ruby-sample-build
> I0116 15:40:27.985826   17242 apply.go:59] Creation succeeded for
> DeploymentConfig with name frontend
> I0116 15:40:27.986403   17242 apply.go:59] Creation succeeded for Service
> with name database
> I0116 15:40:27.986801   17242 apply.go:59] Creation succeeded for
> DeploymentConfig with name database
> # curl -X POST
> http://localhost:8080/osapi/v1beta1/buildConfigHooks/ruby-sample-build/secret101/generic?namespace=hello-openshift-project
> 
> 
> Sounds good, but when I look to see builds happening, pods being created,
> etc. - nothing new:
> # osc get pods
> POD                                    CONTAINER(S)         IMAGE(S)
> HOST                LABELS
> STATUS
> f18b3fb7-9d8f-11e4-a2d8-fa163e33787b   registry-container
> openshift/docker-registry   172.16.4.201/
> deployment=docker-registry-1,deploymentconfig=docker-registry,name=registrypod
> Running
> # osc get services
> NAME                LABELS                                    SELECTOR
> IP                  PORT
> docker-registry     <none>
> name=registrypod    172.30.17.3
> 5001
> kubernetes          component=apiserver,provider=kubernetes   <none>
> 172.30.17.253       443
> kubernetes-ro       component=apiserver,provider=kubernetes   <none>
> 172.30.17.251       80
> # osc get imagerepository -o json origin-ruby-sample
> F0119 12:43:35.998836   18892 get.go:75] imageRepository "origin-ruby-sample"
> not found
> # osc get imagerepository -o json ruby-20-centos
> F0119 12:43:46.869342   18900 get.go:75] imageRepository "ruby-20-centos" not
> found
> # osc get buildconfig -o json ruby-sample-build
> F0119 12:44:20.975768   18927 get.go:75] buildConfig "ruby-sample-build" not
> found
> # osc get deploymentconfig -o json frontend
> F0119 12:44:51.961399   18943 get.go:75] deploymentConfig "frontend" not
> found
> # osc get deploymentconfig -o json database
> F0119 12:44:56.710531   18951 get.go:75] deploymentConfig "database" not
> found
> 
> The master logs showed things basically in a holding pattern without any real
> errors (http://fpaste.org/171445/). The problem was on the node, where I'd
> left openshift running a few days at this point and it had filled my tiny
> test filesystem with lots of exited docker containers.
> 
> There's still no "docker rm --exited" or the like
> (https://github.com/docker/docker/issues/928) but "docker rm -v $(docker ps
> -aq)" does slowly clean things out. At least, until I ran into
> https://github.com/docker/docker/issues/9644 where journalctl just reports:
> Jan 19 14:46:35 lmeyer-0113-f21osv3-3da73419-06c1-45c5-a091-d18ccd14c5ee
> docker[9066]: time="2015-01-19T14:46:35Z" level="fatal" msg="Error loading
> device metadata file .tmp274926013"

Garbage collection of old containers is in the next Kube rebase, slated to land in a few days.

> 
> If you're a docker n00b, it's not completely obvious that it's referring
> specifically to /var/lib/docker/devicemapper/metadata/.tmp274926013 and you
> can apparently just nuke that to allow docker to restart successfully. And
> then openshift/kubernetes picks up and does whatever's needed to deploy what
> the master is expecting to be there.
> 
> Well, sort of. The STI builder seems to be running repeatedly on the node but
> it's hard to tell what else is going on (still nothing showing up "osc get
> <type>" though I may just not know the right ones).
> 
> Anyway, I guess the lesson is, give docker some space, and clear out exited
> containers regularly.
> 
> ----- Original Message -----
> From: "Luke Meyer" <lmeyer redhat com>
> To: "Clayton Coleman" <ccoleman redhat com>
> Cc: "OpenShift Development" <dev lists openshift redhat com>
> Sent: Thursday, January 15, 2015 2:16:22 PM
> Subject: Re: [v3] docker-registry deploy doesn't
> 
> I killed the node server and added that to the startup, and I now have
> containers running on the node :)
> 
> I really didn't see anything amiss in the logs, though. "osc get minions"
> reported the node, I could see health checks coming in on the node, there
> was stuff like this in the master log:
> I0115 18:04:11.167747   22569 factory.go:289] Attempting to bind
> e8434f6e-9ce0-11e4-b0ed-fa163e33787b to 172.16.4.201
> ... but no followup "hey, what's that? sorry, can't bind it there after all,
> actually I have no idea who this node is" unless I'm missing something. So,
> an opportunity for more troubleshooting info somewhere!
> 
> Thanks!
> 
> ----- Original Message -----
> From: "Clayton Coleman" <ccoleman redhat com>
> To: "Luke Meyer" <lmeyer redhat com>
> Cc: "OpenShift Development" <dev lists openshift redhat com>
> Sent: Thursday, January 15, 2015 2:03:04 PM
> Subject: Re: [v3] docker-registry deploy doesn't
> 
> Oh, I looked at what you wrote - add "--hostname 172.16.4.201" to your node
> startup.  The hostname of the node is the "identity" of the node, and it
> defaults to hostname -f
> 
> ----- Original Message -----
> > Not making it out of unknown almost certainly means that the master can't
> > find the nodes.
> > 
> > Do "osc describe pod e8434f6e-9ce0-11e4-b0ed-fa163e33787b" and I bet you'll
> > see a bunch of errors about being unable to schedule.
> > 
> > The master log should probably have health check warnings on your node.
> > 
> > ----- Original Message -----
> > > I'm trying to follow the sample-app deploy at
> > > https://github.com/openshift/origin/blob/master/examples/sample-app/README.md
> > > and getting tripped up early on with deploying the docker-registry.
> > > Pointers
> > > would be appreciated as I have basically no idea how to find out what's
> > > wrong.
> > > 
> > > The context is that I have compiled OpenShift from source and run it as
> > > master on one host, node on another:
> > > 
> > > master# openshift start master --nodes=172.16.4.201 --loglevel=4
> > > node# openshift start node --master=172.16.4.200 --loglevel=4
> > > 
> > > The hosts run Fedora 21, SELinux is permissive, firewalld isn't on, and
> > > I've
> > > added the --insecure-registry flag though it seems unlikely to be
> > > relevant
> > > yet.
> > > 
> > > I do pullimages on the node to get docker warmed up. Then I try to apply
> > > the
> > > docker registry conf:
> > > 
> > > master [sample-app]# osc apply -f docker-registry-config.json
> > > I0115 18:04:10.873990   22621 apply.go:59] Creation succeeded for Service
> > > with name docker-registry
> > > I0115 18:04:10.874574   22621 apply.go:59] Creation succeeded for
> > > DeploymentConfig with name docker-registry
> > > 
> > > So this seems to be working fine, but the pod never makes it out of
> > > "Unknown"
> > > state, and the container never appears on the node according to docker ps
> > > -a.
> > > 
> > > # osc get pods
> > > POD                                    CONTAINER(S)        IMAGE(S)
> > > HOST                LABELS              STATUS
> > > e8434f6e-9ce0-11e4-b0ed-fa163e33787b   deployment
> > > openshift/origin-deployer   172.16.4.201/       <none>
> > > Unknown
> > > 
> > > # osc get services
> > > NAME                LABELS                                    SELECTOR
> > > IP                  PORT
> > > kubernetes          component=apiserver,provider=kubernetes   <none>
> > > 172.30.17.92        443
> > > kubernetes-ro       component=apiserver,provider=kubernetes   <none>
> > > 172.30.17.189       80
> > > docker-registry     <none>
> > > name=registrypod    172.30.17.3
> > > 5001
> > > 
> > > # osc get deploymentconfig
> > > NAME                TRIGGERS            LATEST VERSION
> > > docker-registry     ConfigChange        1
> > > 
> > > # osc get pods
> > > POD                                    CONTAINER(S)        IMAGE(S)
> > > HOST                LABELS              STATUS
> > > e8434f6e-9ce0-11e4-b0ed-fa163e33787b   deployment
> > > openshift/origin-deployer   172.16.4.201/       <none>
> > > Unknown
> > > 
> > > 
> > > I'm attaching master and node logs... there's probably something
> > > indicative
> > > in there but I can't decipher it at this point. More importantly, is
> > > there
> > > any way to query openshift/kubernetes itself to see what is going wrong
> > > with
> > > this pod?
> > > 
> > > 
> > > _______________________________________________
> > > dev mailing list
> > > dev lists openshift redhat com
> > > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> > >
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 
> _______________________________________________
> dev mailing list
> dev lists openshift redhat com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]