[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Failing to bootstrap disconnected 4.2 cluster on metal





On Oct 28, 2019, at 8:07 PM, Joel Pearson <japearson agiledigital com au> wrote:

> Maybe must-gather could be included in the release manifest so that it's available in disconnected environments by default?
It is:
  $ oc adm release info --image-for="">quay.io/openshift-release-dev/ocp-release:4.2.0
  quay.io/openshift-release-dev/ocp-v4 0-art-dev sha256:34ff29512304f77b0ab70ea6850e7f8295a4d19e497ab690ea5102a7044ea993
If your 'oc adm must-gather' is reaching out to Quay, instead of
hitting your mirror, it may be because your samples operator has yet
to get the mirrored must-gather ImageStream set up. 
 
It looks like image streams don't honor the imageContentSources mirror, and try to reach out to the internet.

I had a look at the openshift/must-gather image stream and there was an error saying:

Internal error occurred: Get https://quay.io/v2: dial tcp: lookup quay.io on 172.30.0.10:53 server misbehaving


That (un)available typo should be fixed in master by [1], but looks
like that hasn't been backported to 4.2.z.  But look for the
machine-config daemon that is unready (possibly by listing Pods), and
see why it's not going ready.  

Turns out that all of the machine-config daemon's are ready (I can see 7 of them all marked as ready). But the machine-config operator just doesn't appear to be trying anymore.

It's listed as Available=False Progressing=False and Degraded=True.

I tried deleting the operator pod in the hope that it'd kickstart something, but it didn't seem to help.

I noticed a message right up the top saying:
event.go:247] Could not construct reference to: '&v1.ConfigMap...' Will not report event 'Normal' 'LeaderElection' 'machine-config-operator-5f47... become leader'

The pod that I deleted had that same message too, is this a red herring?

I have must-gather logs now, except that it will probably be complicated to get them off this air-gapped system.  Are there any pointers about where I should look to find out why it's no longer progressing? Can I make the operator try again somehow?

I also noticed that the dns operator is marked available, but there is a degraded status saying that "Not all desired DNS DaemonSets available" however, they are all available.

Almost always means a node is broken / blocked / unable to schedule pods, which prevents DNS from deploying.


On Tue, 29 Oct 2019 at 05:24, W. Trevor King <wking redhat com> wrote:
On Mon, Oct 28, 2019 at 4:05 AM Joel Pearson wrote:
> Maybe must-gather could be included in the release manifest so that it's available in disconnected environments by default?

It is:

  $ oc adm release info --image-for=""> quay.io/openshift-release-dev/ocp-release:4.2.0
  quay.io/openshift-release-dev/ocp-v4 0-art-dev sha256:34ff29512304f77b0ab70ea6850e7f8295a4d19e497ab690ea5102a7044ea993

If your 'oc adm must-gather' is reaching out to Quay, instead of
hitting your mirror, it may be because your samples operator has yet
to get the mirrored must-gather ImageStream set up.

>> Failed to resync 4.2.0 because: timed out waiting for the condition during waitForFaemonsetRollout: Daemonset machine-config-daemon is not ready. status (desired:7, updated 7, ready: 6, unavailable: 6)

That (un)available typo should be fixed in master by [1], but looks
like that hasn't been backported to 4.2.z.  But look for the
machine-config daemon that is unready (possibly by listing Pods), and
see why it's not going ready.

Cheers,
Trevor

[1]: https://github.com/openshift/machine-config-operator/commit/efb6a96a5bcb13cb3c0c0a0ac0c2e7b022b72665


--
Kind Regards,

Joel Pearson
Agile Digital | Senior Software Consultant

Love Your Software™ | ABN 98 106 361 273
p: 1300 858 277 | m: 0405 417 843 | w: agiledigital.com.au
_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]