[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: 500 Error when pushing to internal registry



It does appear to be broken for every project we create on the cluster.  Containers that are from outside sources pull and run fine, but all fail when we try to push to the registry.

The ansible playbook is the standard openshift origin one found at https://github.com/openshift/openshift-ansible.

I am unable to login to the registry using "docker login -u "$(oc whoami)" -p "$(oc whoami -t)" 172.30.241.53:5000".  It spins a while and then returns a 
"Error response from daemon: Get https://172.30.241.53:5000/v1/users/: dial tcp 172.30.241.53:5000: i/o timeout".

I confirmed the IP of the docker-registry with "oc -n default get svc/docker-registry" as 172.30.241.53, which is correct.

There are 5 pods in the registry.  I checked the logs for the registry and all he shows are successful health checks as well.

The strange portion of this is that when I try to push to the registry, I can't see attempted connections in the docker-registry logs at all.  We did see once a permissions error, but we have looked at our storage and the permissions appear to be correct.  Is it possible that there are some holdover files from a previous install that are for some reason interfering with openshift as it tries to write?

On Tue, Sep 19, 2017 at 3:25 AM, Alexey Gladkov <agladkov redhat com> wrote:
On Mon, Sep 18, 2017 at 11:46:40PM -0400, Ben Parees wrote:
> On Mon, Sep 18, 2017 at 12:47 PM, Graigry Henrie <gwhenrie chem byu edu>
> wrote:
>
> > We recently installed openshift origin using an ansible playbook.  We had
> > used the same one before without issue, except that now when projects
> > attempt to push to the internal registry, it results in a 500 error.
> >
> > Attached is the log from attempting to build the Django Example project.
> > The build itself proceeds perfectly, but fails with a "500 Internal Server
> > Error" when attempting to push to the internal registry.  Checking the logs
> > of the registry itself only shows health checks, indicating that container
> > has come up properly.
> >
> > What makes it more confusing is that we have installed with this same file
> > multiple times and sometimes it works and sometime it doesn't.  We believe
> > that it is a permissions issue.  We have checked and rechecked our ansible
> > hosts file, but we cannot see what the problem is.  I have included it as
> > well.
> >
>
> can you try manually pushing to the registry using the service account's
> token?  (ie docker login to the registry w/ the token and then tag an
> image+push to it).
>
> There is a known issue that the registry returns 500 errors when it should
> return 403s, so your theory that it is a permission issue is plausible but
> it is strange that some of your clusters work.  When it is not working, it
> sounds like it is broken for all projects in the cluster, not just some or
> fails randomly?
>
> Adding our registry gurus to CC.

Can we see this ansible playbook ?

How many registry pods in the cluster (maybe the logs from the requests on
the other pod) ?

Is there a router in front of him and what's in his logs ?

--
Rgrds, agladkov



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]