[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Fwd: "Unauthorized" oauth tokens in 3.10 cluster



Hi!

I have a cluster that was initially installed with 3.7 (afaik) and
upgraded to 3.10.

It has been running fine for a many month now, but yesterday problems
occured.

 * ImagePullBackOff for newly scheduled pods

 * 403 from docker registry (error: build error: Failed to push
       image... HTTP 403 response body: unexpected end of JSON input: "")

I had that error (with origin 1.2 or even earlier) when it could be
fixed by deleting the dockercfg secrets.  I did that but that did not
fix the problem.

I tried deleting the builder serviceaccount and the dockerfg does not
get created anymore (the new builder sa and secret builder-token-XXXXX
do get created).


In the master-api-<node> logs I get the following log
  1 authentication.go:63] Unable to authenticate the request due to an
  error: [invalid bearer token, [invalid bearer token, [Token has been
  invalidated, oauthaccesstokens.oauth.openshift.io "XXXXXX" not found]]]

and many events about that problem (in fact, if I filter oauthaccesstoken.oauth.openshift.io and audits, I hardly get any logs; only etcd diagnostics).


In the master-controllers-<node> logs I get the following log entries

  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125:
  Failed to list <nil>: Unauthorized

  endpoints_controller.go:375] Error syncing endpoints for service "a_namespace/a_service": Unauthorized

Which most likely means that it just can't speak to the api server.


Anything I can do to fix this problem?

Thank you for your help,
 Tobias Florek

Attachment: signature.asc
Description: signature


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]