[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: 4.3 must-gather





On Sun, Dec 1, 2019 at 9:37 PM Jon Stanley <jonstanley gmail com> wrote:
I thought that I was having a problem with my 4.3 test cluster (turns
out to be operator error) and was gathering the must-gather and came
into a problem there. In a disconnected cluster, getting the image for
a must-gather doesn't take into account the icsp (looks like the image
it tries to get isn't covered by the installer icsp?)

The image it tries to use is part of the OCP payload(thus is mirrored to your mirror registry and covered by your ICSP) and, as of 4.3, the imagestream import does respect the ICSP, that's why you got the CA error.
 

Looking into it further, it seems more fundamentally that imagestreams
don't obey the cluster CA trust? I put an additionalTrustBundle in the
install-config.yaml and the cluster installed fine, and it looks like
it tries to pull the local registry into the imagestream but doesn't
because the CA isn't trusted?

There are two ways to ensure imagestream import trusts additional CAs:

1) If you define a proxy config with additional CAs, those CAs will be used during imagestream import (as well as consumed by many other components).  This is true even if you don't have a proxy, so you so can define a dummy proxy config that has no "http/httpsProxy" values but just has a reference to your additional CA bundle.  If you are doing it at install time, I think you have to provide a dummy "noProxy" value, this will trick the installer into setting up a proxyconfig that references the additionalTrustBundle you provided in the install-config. 

2) You can expliciltly setup trusts for imagestream import (you can't easily do this at install time though):
https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#image-configuration

"additionalTrustedCA: A reference to a ConfigMap containing additional CAs that should be trusted during ImageStream import, pod image pull, openshift-image-registry pullthrough, and builds. The namespace for this ConfigMap is openshift-config. The format of the ConfigMap is to use the registry hostname as the key, and the base64-encoded certificate as the value, for each additional registry CA to trust."

One you take one of those actions to setup the trust you'll need to reimport the imagestream.  Note that the openshift apiserver pods will have to cycle to pick up the change before the import will succeed.



 

jstanley localhost ocp-local]$ oc -n openshift describe is/must-gather
Name:                   must-gather
Namespace:              openshift
Created:                3 hours ago
Labels:                 <none>
Annotations:            <none>
Image Repository:       <none>
Image Lookup:           local=false
Unique Images:          0
Tags:                   1

latest
  updates automatically from registry
quay.io/openshift-release-dev/ocp-v4 0-art-dev sha256:b91dba68e6d6df556550855922e06fc52e185cc7ac0388e0c4dc3955c8601ca2

  ! error: Import failed (InternalError): Internal error occurred:
[registry.test:5000/ocp4/openshift4 sha256:b91dba68e6d6df556550855922e06fc52e185cc7ac0388e0c4dc3955c8601ca2:
Get https://registry.test:5000/v2/: x509: certificate signed by
unknown authority,
quay.io/openshift-release-dev/ocp-v4 0-art-dev sha256:b91dba68e6d6df556550855922e06fc52e185cc7ac0388e0c4dc3955c8601ca2:
Get https://quay.io/v2/: dial tcp 3.230.48.144:443: i/o timeout]
      3 hours ago


Since the imagestream was a no-go, it seems that it falls back on a
hardcoded image for the must-gather? You can override the image and it
works fine if you do (but then you have to know what the image is!!).

Not sure what if anything can be done here.

[jstanley localhost ocp-local]$ oc adm must-gather
[must-gather      ] OUT unable to resolve the imagestream tag
openshift/must-gather:latest
[must-gather      ] OUT
[must-gather      ] OUT Using must-gather plugin-in image:
quay.io/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-9t68j created
[must-gather      ] OUT
clusterrolebinding.rbac.authorization.k8s.io/must-gather-t5v5c created
[must-gather      ] OUT pod for plug-in image
quay.io/openshift/origin-must-gather:latest created
[must-gather-rsvxn] OUT gather did not start: unable to pull image:
ErrImagePull: rpc error: code = Unknown desc = error pinging docker
registry quay.io: Get https://quay.io/v2/: dial tcp 3.230.48.144:443:
connect: no route to host
[must-gather      ] OUT
clusterrolebinding.rbac.authorization.k8s.io/must-gather-t5v5c deleted
[must-gather      ] OUT namespace/openshift-must-gather-9t68j deleted
error: gather did not start for pod must-gather-rsvxn: unable to pull
image: ErrImagePull: rpc error: code = Unknown desc = error pinging
docker registry quay.io: Get https://quay.io/v2/: dial tcp
3.230.48.144:443: connect: no route to host
[jstanley localhost ocp-local]$

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users



--
Ben Parees | OpenShift


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]