I hit this problem again, this time with the cassandra pod for Hawkular metrics.
This has been running without problem for some months, but now I found that the cassandra pod could not start because of file permissions writing to the /cassandra_data/data directory.
Looking at that directory the ownership was set to 1000040000.65534, but cassandra was running as user 313 so could not write to that directory. Manually changing permissions to 313.65534 (the 65534 group is nfsnobody, and the cassandra user is a member of that group) fixed the problem and allowed the cassandra pod to start.
Clearly the 1000040000 user is an openshift assigned user, but as the container is running as the cassandra user (313) I have no idea how this could have happened.
Can anyone explain what is going on here?
On 02/07/18 16:27, Tim Dudgeon wrote:
I've hit a strange problem with directory ownership for the docker registry a couple of times, and don't understand what is causing this.
The registry was working fine for some time. I'm using a Cinder volume for the registry storage, but don't know if that's relevant.
Then something happened that stopped pods pushing to the registry, with the problem being that the registry pod was getting "Permission denied" errors when it was trying to create directories under /registry/docker/registry/v2/r
Looking at the file system the directories were all owned by 1000000000.1000000000 which explains why the registry process (running as user 1001) could not write to these directories. e.g.
sh-4.2$ cd /registry/docker/registry/v2/
sh-4.2$ ls -al
drwxrwsr-x. 4 1000000000 1000000000 39 Apr 20 15:51 .
drwxrwsr-x. 3 1000000000 1000000000 16 Apr 20 15:51 ..
drwxrwsr-x. 3 1000000000 1000000000 20 Apr 20 15:51 blobs
drwxrwsr-x. 15 1000000000 1000000000 215 May 29 14:14 repositories
Doing a `docker -exec -u 0 <registry-pod> on the infra node and then a `chown -R 1001.0 /registry/docker/registry` to reset the permissions fixed the problem.
Anyone any idea what's going on here?
users mailing list
users lists openshift redhat c