[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Problem authenticating to private docker registry



No worries. Thanks everyone for the help so far. Let me know if there's any other helpful information I can provide.

I am able to pull the image down without any issues if I use the latest tag in case that helps:

[root os-node1 ~]# docker pull docker-lab.example.com:5000/testwebapp:latest
Trying to pull repository docker-lab.example.com:5000/testwebapp ...
latest: Pulling from docker-lab.example.com:5000/testwebapp
3d8673bd162a: Pull complete
855e002c7563: Pull complete
2c2e00e4aa2a: Pull complete
Digest: sha256:5216e79273fc6221f8f5896632e4de633eeaa66347d0500c39d9d0006912e42d
Status: Downloaded newer image for docker-lab.example.com:5000/testwebapp:latest

On Wed, Aug 10, 2016 at 2:21 PM, Andy Goldstein <agoldste redhat com> wrote:
Ok, thanks. I'm not really involved with the registry any more, so I'll have to defer to Maciej and Michal. We may need to try to reproduce to see what's going on. Sorry I couldn't be more helpful.

Andy

On Wed, Aug 10, 2016 at 2:18 PM, Tony Saxon <tony saxon gmail com> wrote:

On Wed, Aug 10, 2016 at 2:07 PM, Andy Goldstein <agoldste redhat com> wrote:
Tony, can you show the output when you try to manually 'docker pull'?

On Wed, Aug 10, 2016 at 2:04 PM, Cesar Wong <cewong redhat com> wrote:
Hmm, I didn't know the issue existed between 1.10 and 1.12 as well. 

Andy, what would you recommend?


On Aug 10, 2016, at 1:58 PM, Tony Saxon <tony saxon gmail com> wrote:

Ok, maybe that is the issue. I can not do the docker pull referencing the sha256 hash on the node.

The docker version running on the node is docker 1.10.3, and the docker version on the machine that pushed the image is 1.12.0. Is there a potential workaround for this, or do I need to get the docker version updated on the nodes? For reference, I installed the openshift platform using the ansible advanced installation referenced in the documentation.

On Wed, Aug 10, 2016 at 1:46 PM, Cesar Wong <cewong redhat com> wrote:
Tony,

The only other time that I've seen the manifest not found error was when there was a version mismatch between the Docker version that pushed the image vs the version that was consuming the image (ie. images pushed with Docker 1.9 and pulled with Docker 1.10). Are you able to pull the image spec directly from your node using the Docker cli?


On Aug 10, 2016, at 1:02 PM, Tony Saxon <tony saxon gmail com> wrote:

I'm not sure if this has anything to do with it, but I looked at the details of the imagestream that I imported and see that it has this as the docker image reference:

status:
  dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp
  tags:
  - items:
    - created: 2016-08-10T13:26:01Z
      dockerImageReference: docker-lab.example.com:5000/testwebapp sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
      generation: 1
      image: sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
    tag: latest

I also see these errors show up on the docker registry when I try to deploy the app:

time="2016-08-10T16:58:26Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="docker-lab.evolveip.net:5000" http.request.id=ecce6c57-6273-42d6-b7a9-441877c0338f http.request.method=GET http.request.remoteaddr="192.168.122.156:35858" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
192.168.122.156 - - [10/Aug/2016:16:58:26 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-10T16:58:26Z" level=error msg="response completed with error" auth.user.name=maven err.code="manifest unknown" err.detail="unknown manifest name=testwebapp revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" err.message="manifest unknown" go.version=go1.6.3 http.request.host="docker-lab.evolveip.net:5000" http.request.id=b994a477-6beb-4908-8589-c051b9048e87 http.request.method=GET http.request.remoteaddr="192.168.122.156:35860" http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.04215ms http.response.status=404 http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name=testwebapp vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" version=v2.5.0
192.168.122.156 - - [10/Aug/2016:16:58:26 +0000] "GET /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"

So it looks like the manifest isn't found, or am I misunderstanding that?

The imagestream was imported by simply:

[root os-master ~]# oc import-image testwebapp --confirm --from=docker-lab.example.com:5000/testwebapp:latest
The import completed successfully.
 
Name:                   testwebapp
Created:                Less than a second ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2016-08-10T17:01:46Z
Docker Pull Spec:       172.30.11.167:5000/testwebapp/testwebapp
 
Tag     Spec                                            Created         PullSpec                                                                Image
latest  docker-lab.example.com:5000/testwebapp:latest  1 seconds ago   docker-lab.example.com:5000/testwebapp sha256:9799a25cd6fd7f...        <same>
 

On Wed, Aug 10, 2016 at 11:39 AM, Tony Saxon <tony saxon gmail com> wrote:
Not a problem, can you point me in the right direction for how to get those logs?

For the first one, I know I can set the USER option in the dockerfile that builds the docker image. Does it matter what user I set that as? I'm assuming that the user needs to exist on the nodes...

On Wed, Aug 10, 2016 at 11:20 AM, Maciej Szulik <maszulik redhat com> wrote:
There are two problems here:
1. WARNING: Image "testwebapp" runs as the 'root' user which may not be permitted by your cluster administrator
    unless your user is allowed to run privileged containers you should fix that.
2. Error from server: container "testwebapp" in pod "testwebapp-1-1x7ex" is waiting to start: image can't be pulled

First it would be good to get rid of no 1. for the latter I'd need to see master logs with loglevel=8 if it's not a problem.



On Wed, Aug 10, 2016 at 5:14 PM, Tony Saxon <tony saxon gmail com> wrote:
It does not work:

[root os-master ~]# oc new-app testwebapp
--> Found image 59826fe (4 days old) in image stream testwebapp under tag "latest" for "testwebapp"
 
    * This image will be deployed in deployment config "testwebapp"
    * The image does not expose any ports - if you want to load balance or send traffic to this component
      you will need to create a service with 'expose dc/testwebapp --port=[port]' later
    * WARNING: Image "testwebapp" runs as the 'root' user which may not be permitted by your cluster administrator
 
--> Creating resources with label app=testwebapp ...
    deploymentconfig "testwebapp" created
--> Success
    Run 'oc status' to view your app.
[root os-master ~]# oc status
In project testwebapp on server https://os-master.libvirt:8443
 
dc/testwebapp deploys istag/testwebapp:latest
  deployment #1 pending 5 seconds ago
 
1 warning identified, use 'oc status -v' to see details.
[root os-master ~]# oc get pods
NAME                  READY     STATUS              RESTARTS   AGE
testwebapp-1-1x7ex    0/1       ContainerCreating   0          3s
testwebapp-1-deploy   1/1       Running             0          9s
[root os-master ~]# oc get pods
NAME                  READY     STATUS         RESTARTS   AGE
testwebapp-1-1x7ex    0/1       ErrImagePull   0          6s
testwebapp-1-deploy   1/1       Running        0          12s
[root os-master ~]# oc logs testwebapp-1-1x7ex
Error from server: container "testwebapp" in pod "testwebapp-1-1x7ex" is waiting to start: image can't be pulled
[root os-master ~]#

On Wed, Aug 10, 2016 at 10:58 AM, Maciej Szulik <maszulik redhat com> wrote:
Yeah, that looks ok, did that work? If not, what error did you get?



On Wed, Aug 10, 2016 at 4:19 PM, Tony Saxon <tony saxon gmail com> wrote:
I've done that:

[root os-master ~]# oc get secrets
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-7bjoo    kubernetes.io/dockercfg               1         1d
builder-token-gi9o9        kubernetes.io/service-account-token   3         1d
builder-token-wf31u        kubernetes.io/service-account-token   3         1d
default-dockercfg-pfota    kubernetes.io/dockercfg               1         1d
default-token-vbcmc        kubernetes.io/service-account-token   3         1d
default-token-xffu5        kubernetes.io/service-account-token   3         1d
deployer-dockercfg-lfiuw   kubernetes.io/dockercfg               1         1d
deployer-token-9euo2       kubernetes.io/service-account-token   3         1d
deployer-token-mq6vw       kubernetes.io/service-account-token   3         1d
docker-lab                 kubernetes.io/dockercfg               1         19h

[root os-master ~]# oc describe sa default
Name:           default
Namespace:      testwebapp
Labels:         <none>
 
Image pull secrets:     default-dockercfg-pfota
                        docker-lab
 
Mountable secrets:      default-token-xffu5
                        default-dockercfg-pfota
 
Tokens:                 default-token-vbcmc
                        default-token-xffu5


One thing to note is that the "link" command didn't work. I had to use 'oc secrets add default docker-lab --for="">

Can you open an issue it's not working? Either docs should be updated or code fixed.
 

 
Does it need to be added to a different service account? Here are the accounts that currently exist:

[root os-master ~]# oc get sa
NAME       SECRETS   AGE
builder    2         1d
default    2         1d
deployer   2         1d

On Wed, Aug 10, 2016 at 10:09 AM, Maciej Szulik <maszulik redhat com> wrote:


On Wed, Aug 10, 2016 at 3:59 PM, Cesar Wong <cewong redhat com> wrote:
Hi Tony, 

Sorry for over-complicating it then :)

Using the image stream name is the right thing to do. However, new-app won't include the pull secret in the deployment config that it generates. So you have to specify it manually by editing the generated dc 

Unless, (Maciej, you'd know better than me) there is some magic that the deployer uses to figure out the right secret to use.


 

On Aug 10, 2016, at 9:47 AM, Tony Saxon <tony saxon gmail com> wrote:

Thanks Cesar, but actually Maciej had it right. I'm trying to use a preexisting private docker registry as the source of the image. The preexisting registry is tied in with our current CI system, so my ultimate goal is to use an image from that repository as the source for an app in Openshift for a continuous deployment system.

I went and did the import and it imported successfully:

# oc import-image testwebapp --confirm --from=docker-lab.example.com:5000/testwebapp:latest
The import completed successfully.
 
Name:                   testwebapp
Created:                1 seconds ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2016-08-10T13:26:01Z
Docker Pull Spec:       172.30.11.167:5000/testwebapp/testwebapp
 
Tag     Spec                                            Created         PullSpec                                                                Image
latest  docker-lab.example.com:5000/testwebapp:latest  1 seconds ago   docker-lab.example.com:5000/testwebapp sha256:9799a25cd6fd7f...        <same>
 
# oc get is
NAME         DOCKER REPO                                TAGS      UPDATED
testwebapp   172.30.11.167:5000/testwebapp/testwebapp   latest    6 seconds ago

I'm a little confused as to how to then deploy this as a new app. I tried running the same 'oc new-app' command as I was running previously, but it gave an error about an imagestream 'testwebapp' already existing. I then tried to run just 'oc new-app testwebapp' and it still gives an error in the pod about not being able to pull the image. I also tried 'oc new-app 172.30.11.167:5000/testwebapp/testwebapp' which gave an error about no match.

On Wed, Aug 10, 2016 at 6:40 AM, Cesar Wong <cewong redhat com> wrote:
Maciej,

In this case, Tony is trying to connect to the OpenShift registry, so the secret should exist; the dockercfg secret for the project's default service account. 

Tony, two things that may be your issue:

1) You're using a route for your registry (docker-lab.example.net). The dockercfg secret will likely only have an entry for the ip address of the registry and not the route. (Maciej, maybe you know of a way to get the secrets to include an entry for the host of the route). Otherwise, you're better off specifying the service ip when invoking new-app.

You can check what hosts are included in the dockercfg secret by doing 'oc describe secret/default-dockercfg-XXXX' where XXXX is whatever suffix is used in your project.

2) The image ref that you're using in your new-app invocation doesn't include a namespace. All images on the OpenShift registry will have a namespace and name like:
 [registry-host]:[port]/projectname/testwebapp:latest. Make sure you have the full spec for the image (from 'oc get is').

On Aug 10, 2016, at 5:44 AM, Maciej Szulik <maszulik redhat com> wrote:

to setup the secret in the same project your ImageStream is created and then re-import the image.
During import proper secrets will be picked automatically based on the urls of the registry and your image metadata
should be downloaded to the server. This will handle the import part, now for actually using an image from private
registry you need to follow this: https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries

Hope that helps,
Maciej

On Tue, Aug 9, 2016 at 4:00 PM, Tony Saxon <tony saxon gmail com> wrote:
I'm not sure what I'm missing here. I have a private docker registry that is set up securely and uses authentication. I followed the docs at https://docs.openshift.org/latest/dev_guide/managing_images.html#using-image-pull-secrets to create the secret with the username and password to authenticate with the docker registry. I verified that I can manually login to the docker registry from the master and the nodes. However, when I go to deploy a new app based on an image from the docker registry it seem to be failing to authenticate. The command that I'm running to create the new app:

oc new-app docker-lab.example.net:5000/testwebapp:latest

It creates the imagestream and attempts to deploy the pod. I get the following in the logs on the pod:

# oc logs testwebapp-1-us1wu
Error from server: container "testwebapp" in pod "testwebapp-1-us1wu" is waiting to start: image can't be pulled

The logs on the docker registry show:

time="2016-08-09T13:54:45Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="docker-lab.example.net:5000" http.request.id=f5aeb8b9-ce4e-41b7-86a8-76e8c520bd22 http.request.method=GET http.request.remoteaddr="192.168.122.158:54436" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
192.168.122.158 - - [09/Aug/2016:13:54:45 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-09T13:54:45Z" level=error msg="response completed with error" auth.user.name=tsaxon err.code="manifest unknown" err.detail="unknown manifest name=testwebapp revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" err.message="manifest unknown" go.version=go1.6.3 http.request.host="docker-lab.example.net:5000" http.request.id=130a9014-7c19-48f7-bef3-2b8cfe0470a0 http.request.method=GET http.request.remoteaddr="192.168.122.158:54438" http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.174905ms http.response.status=404 http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name=testwebapp vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" version=v2.5.0
192.168.122.158 - - [09/Aug/2016:13:54:45 +0000] "GET /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-09T13:54:45Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="docker-lab.example.net:5000" http.request.id=0185e07b-f1c1-48e6-91ea-dede2339f087 http.request.method=GET http.request.remoteaddr="192.168.122.158:54440" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
192.168.122.158 - - [09/Aug/2016:13:54:45 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-09T13:54:46Z" level=error msg="response completed with error" auth.user.name=tsaxon err.code="manifest unknown" err.detail="unknown manifest name=testwebapp revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" err.message="manifest unknown" go.version=go1.6.3 http.request.host="docker-lab.example.net:5000" http.request.id=c1ab0cd7-42ac-4fef-b2c4-0f451976e302 http.request.method=GET http.request.remoteaddr="192.168.122.158:54442" http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.28913ms http.response.status=404 http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name=testwebapp vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" version=v2.5.0
192.168.122.158 - - [09/Aug/2016:13:54:46 +0000] "GET /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"

Here are the service accounts showing that they have the image pull secret added (docker-lab):

[root os-master ~]# oc get serviceaccounts
NAME       SECRETS   AGE
builder    3         21h
default    2         21h
deployer   3         21h
[root os-master ~]# oc describe serviceaccounts default
Name:           default
Namespace:      testwebapp
Labels:         <none>

Image pull secrets:     default-dockercfg-pfota
                        eip-docker
                        docker-lab

Mountable secrets:      default-token-xffu5
                        default-dockercfg-pfota

Tokens:                 default-token-vbcmc
                        default-token-xffu5



[root os-master ~]# oc describe serviceaccounts builder
Name:           builder
Namespace:      testwebapp
Labels:         <none>

Image pull secrets:     builder-dockercfg-7bjoo
                        docker-lab

Mountable secrets:      builder-token-wf31u
                        builder-dockercfg-7bjoo
                        eip-docker

Tokens:                 builder-token-gi9o9
                        builder-token-wf31u



[root os-master ~]# oc describe serviceaccounts deployer
Name:           deployer
Namespace:      testwebapp
Labels:         <none>

Image pull secrets:     deployer-dockercfg-lfiuw
                        docker-lab

Mountable secrets:      deployer-token-9euo2
                        deployer-dockercfg-lfiuw
                        eip-docker

Tokens:                 deployer-token-9euo2
                        deployer-token-mq6vw


Not sure what I could be missing.

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users


_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users










_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users








[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]