[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Node fails to start after system reboot



Apparently we had run out of disk space on the node. 

I couldn't recover from that even after clearing space and deleting /var/lib/docker.  Had to throw away the node and start again. Wonder if there was a better way of handling this. 

On 10 October 2016 at 12:12, Lionel Orellana <lionelve gmail com> wrote:
Something broke docker.

# docker run hello-world
docker: Error response from daemon: devicemapper: Error running deviceCreate (createSnapDevice) dm_task_run failed.
See '/usr/bin/docker-current run --help'.

# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 283
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-0:38-148510880-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 38.19 GB
 Data Space Total: 107.4 GB
 Data Space Available: 16.89 GB
 Metadata Space Used: 45.18 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.102 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 3.703 GiB
Name: xxxxxx
ID: S7TX:CAUS:656Z:TLJN:IS2F:CICD:RISE:7N76:MRSA:HUBP:MINL:WU7M
Debug mode (server): true
 File Descriptors: 16
 Goroutines: 24
 System Time: 2016-10-10T12:11:00.561344233+11:00
 EventsListeners: 0
 Init SHA1: 1f87ef7fa7fd7401b9aa61ca3a3e096b4fd2228e
 Init Path: 
 Docker Root Dir: /var/lib/docker
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries: docker.io (secure)



On 10 October 2016 at 10:36, Lionel Orellana <lionelve gmail com> wrote:
I deleted all containers with 

docker rm -v $(docker ps -q -f status=exited)

The openvswitch service continous to fail claiming the name already exists. 

Oct 10 10:30:28 poc-docker02.aipo.gov.au openvswitch[17706]: docker: Error response from daemon: Conflict. The name "/openvswitch" is already in use by container b9a94ed6a5885c5b02f9fe456b27e409c4ce2732a3e7be1bce19d8d69f397d2d. You have to remove (or rename) that container to be able to reuse that name..
Oct 10 10:30:28 poc-docker02.aipo.gov.au openvswitch[17706]: See '/usr/bin/docker-current run --help'.
--

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES




On 10 October 2016 at 09:44, Lionel Orellana <lionelve gmail com> wrote:
Hi,

I'm getting this error on my nodes after the host has rebooted. 

Oct 10 09:22:13 poc-docker02.aipo.gov.au systemd[1]: Job origin-node.service/start failed with result 'dependency'.
Oct 10 09:22:23 poc-docker02.aipo.gov.au systemd[1]: Dependency failed for origin-node.service.
-- Subject: Unit origin-node.service has failed
-- Defined-By: systemd
-- 
-- Unit origin-node.service has failed.
-- 
-- The result is dependency.

No container is running. (i.e. docker ps returns nothing but -a does).

It seems to be failing to create openvswitch because it already exists? Should I delete old containers? Any advice on getting a clean system reboot? 

$journalct -xe

Oct 10 09:37:47 poc-docker02.aipo.gov.au docker[7343]: time="2016-10-10T09:37:47.353549643+11:00" level=debug msg="Calling POST /v1.22/containers/create"
Oct 10 09:37:47 poc-docker02.aipo.gov.au docker[7343]: time="2016-10-10T09:37:47.353603207+11:00" level=debug msg="POST /v1.22/containers/create?name=openvswitch"
Oct 10 09:37:47 poc-docker02.aipo.gov.au docker[7343]: time="2016-10-10T09:37:47.353898488+11:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entry
Oct 10 09:37:47 poc-docker02.aipo.gov.au docker[7343]: time="2016-10-10T09:37:47.354052936+11:00" level=info msg="{Action="" LoginUID=4294967295, PID=9878}"
Oct 10 09:37:47 poc-docker02.aipo.gov.au docker[7343]: time="2016-10-10T09:37:47.359950019+11:00" level=error msg="Handler for POST /v1.22/containers/create returned error: Conflict. The name \"/openvswitch\" is already in use by con
Oct 10 09:37:47 poc-docker02.aipo.gov.au openvswitch[9878]: docker: Error response from daemon: Conflict. The name "/openvswitch" is already in use by container 497532ea92a4e14782b96b15f2763d3883d2843d5fdf473f720170756e4de815. You ha
Oct 10 09:37:47 poc-docker02.aipo.gov.au openvswitch[9878]: See '/usr/bin/docker-current run --help'.
Oct 10 09:37:47 poc-docker02.aipo.gov.au systemd[1]: openvswitch.service: main process exited, code=exited, status=125/n/a
Oct 10 09:37:50 poc-docker02.aipo.gov.au polkitd[847]: Registered Authentication Agent for unix-process:9885:60099516 (system bus name :1.3589 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Au
Oct 10 09:37:50 poc-docker02.aipo.gov.au systemd[1]: Cannot add dependency job for unit origin-master.service, ignoring: Unit origin-master.service failed to load: No such file or directory.
Oct 10 09:37:50 poc-docker02.aipo.gov.au systemd[1]: Cannot add dependency job for unit origin-master.service, ignoring: Unit origin-master.service failed to load: No such file or directory.
Oct 10 09:37:50 poc-docker02.aipo.gov.au systemd[1]: Started origin-node-dep.service.


thanks


Lionel




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]