[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: Pod not passing deployment tests



Awesome ! it seems to solve the problem, indeed.
>From what I understand it means that until the issue is closed, we have to "link" the image build to a deployment through a trigger if we want to be able to deploy this image. Otherwise it just won't work ?


Thank you very much for your help

---------------------
François Villain
DevOps - AAIO




De : Michalis Kargakis <mkargaki redhat com>
Envoyé : jeudi 11 février 2016 16:27
À : François VILLAIN
Cc : Dan Mace; users lists openshift redhat com
Objet : Re: Pod not passing deployment tests
 
This is the same issue as https://github.com/openshift/origin/issues/6934
github.com
Failed to resolve ImageStreamTag in deployment #6934. Open mfojtik opened this Issue Feb 1, 2016 · 13 comments



I reproduced it and see in the kubelet logs

Error syncing pod c81fef8b-d0d0-11e5-a0c1-080027242396, skipping: failed to "StartContainer" for "sample-phpinfo" with ErrImagePull: "Error: image library/sample-phpinfo:latest not found"

If you want to workaround that issue, you will need to setup an ICT and have the deployment run automatically once the build finishes.

    triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
        - sample-phpinfo
        from:
          kind: ImageStreamTag
          name: sample-phpinfo:latest
      type: ImageChange

instead of

   triggers: []


On Thu, Feb 11, 2016 at 2:31 PM, François VILLAIN <f villain linkbynet com> wrote:

Hello Dan,


Sorry for the lack of response.. I went on vacations and a crash made us reinstall the whole cluster... 


Anyway , I'm just getting back to the subject and still getting the same results and error messages.

I recreated everything, so here are the definition files and various logs :


- Definition of template objects : http://pastebin.com/6srmPJAS 
- Log level (9) of deployment on CLI : http://pastebin.com/Z4m0YfB8
- Logs of the deploy pod : http://pastebin.com/6Uz3yhW1
- Describe of the deploy pod : http://pastebin.com/2cZN04zG
- Docker logs of containers : http://pastebin.com/ds9A4RgX 
- Deploy pod YAML output : http://pastebin.com/5BKtYbdK

I hope this can help you 


Thanks for you help

---------------------
François Villain
DevOps - AAIO




De : Dan Mace <dmace redhat com>
Envoyé : mercredi 6 janvier 2016 16:26
À : François VILLAIN
Cc : Michalis Kargakis; users lists openshift redhat com
Objet : Re: Pod not passing deployment tests
 


On Wed, Jan 6, 2016 at 10:00 AM, François VILLAIN <f villain linkbynet com> wrote:
> Here is all i got :
>
> CONTAINER ID        IMAGE                                                                                                                           COMMAND                  CREATED             STATUS                           PORTS               NAMES
> 9e332c821d82        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   31 minutes ago      Exited (0) 21 minutes ago                            k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd
> 55967e6c5c82        172.30.56.201:5000/jfr4/debian-80-lc-apache-wordpress sha256:53bbfa0e4b8aa0f6d1c3ee709107066169d3507307014912726c8bebe5e9e9f0   "/usr/sbin/runit_boot"   42 minutes ago      Up 42 minutes                                        k8s_test-apachewordpress.336aaeef_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_7d5b41e0
> 422e4332abab        openshift/origin-docker-registry:v1.1.0.1                                                                                       "/bin/sh -c 'REGISTRY"   58 minutes ago      Up 58 minutes                                        k8s_registry.5a0244b_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_e1f870e0
> aca25d6b48c4        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   58 minutes ago      Up 58 minutes                                        k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_4bcf1754
> 49e418f3c464        openshift/origin-haproxy-router:v1.1.0.1                                                                                        "/usr/bin/openshift-r"   58 minutes ago      Up 58 minutes                                        k8s_router.8fe4297d_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_5687da01
> 4c019f72a1b6        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   58 minutes ago      Created                                              k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_5fe0fd41
> 0e86cbfab7f0        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   58 minutes ago      Up 58 minutes                                        k8s_POD.bafffdb8_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_3af3542f
> 519829fd6951        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   58 minutes ago      Up 58 minutes                                        k8s_POD.425ddcfc_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_a85a5ead
> 3dcb164c7f8a        172.30.56.201:5000/jfr4/debian-80-lc-apache-wordpress sha256:53bbfa0e4b8aa0f6d1c3ee709107066169d3507307014912726c8bebe5e9e9f0   "/usr/sbin/runit_boot"   29 hours ago        Exited (111) About an hour ago                       k8s_test-apachewordpress.336aaeef_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_b65b30ed
> 01a3b9501f7f        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   29 hours ago        Exited (0) About an hour ago                         k8s_POD.bafffdb8_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_a61e7d00
> 6156365c321a        openshift/origin-docker-registry:v1.1.0.1                                                                                       "/bin/sh -c 'REGISTRY"   8 days ago          Exited (2) About an hour ago                         k8s_registry.5a0244b_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_221bd690
> c7fdabda7172        openshift/origin-haproxy-router:v1.1.0.1                                                                                        "/usr/bin/openshift-r"   8 days ago          Exited (2) About an hour ago                         k8s_router.8fe4297d_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_22b62a48
> e75a4750e79b        openshift/origin-pod:v1.1.0.1                                                                                                   "/pod"                   8 days ago          Exited (0) About an hour ago                         k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_196dba88
> 482fb0703f7e        openshift/origin-pod:v1.1.0.1                                                                                                   “/
pod”                   8 days ago          Exited (0) About an hour ago                         k8s_POD.425ddcfc_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_851a9591


François
,

Can you also show the output of `oc get pod -o yams 
test/phpfpm-app-1
`? If you have OpenShift master logs (preferably running at 
loglevel=4) that would also be helpful.


> ________________________________________
> De : Dan Mace <dmace redhat com>
> Envoyé : mercredi 6 janvier 2016 15:53
> À : François VILLAIN
> Cc : Michalis Kargakis; users lists openshift redhat com
> Objet : Re: Pod not passing deployment tests
>
> On Wed, Jan 6, 2016 at 9:35 AM, François VILLAIN
> <f villain linkbynet com> wrote:
>>
>> This is what i get (I just started another deploy manually):
>>
>> # sudo docker ps
>> CONTAINER ID        IMAGE                                                              COMMAND          CREATED                STATUS              PORTS            NAMES
>> 9e332c821d82        openshift/origin-pod:v1.1.0.1                     "/pod"                   4 minutes ago       Up 4 minutes                            k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd
>>
>> # sudo docker inspect 9e332c821d82
>> [
>> {
>>     "Id": "9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a",
>>     "Created": "2016-01-06T14:27:28.296408249Z",
>>     "Path": "/pod",
>>     "Args": [],
>>     "State": {
>>         "Running": true,
>>         "Paused": false,
>>         "Restarting": false,
>>         "OOMKilled": false,
>>         "Dead": false,
>>         "Pid": 7739,
>>         "ExitCode": 0,
>>         "Error": "",
>>         "StartedAt": "2016-01-06T14:27:29.339842008Z",
>>         "FinishedAt": "0001-01-01T00:00:00Z"
>>     },
>>     "Image": "04d505e60cd48657230be57fcaa8a490b3c48b0d0217a632f3a5bf7e3ffc4302",
>>     "NetworkSettings": {
>>         "Bridge": "",
>>         "EndpointID": "15137677eb35d24abf9840b60ab68aeb824aec91b90dd77b48d41bd78737131a",
>>         "Gateway": "10.1.1.1",
>>         "GlobalIPv6Address": "",
>>         "GlobalIPv6PrefixLen": 0,
>>         "HairpinMode": false,
>>         "IPAddress": "10.1.1.4",
>>         "IPPrefixLen": 24,
>>         "IPv6Gateway": "",
>>         "LinkLocalIPv6Address": "",
>>         "LinkLocalIPv6PrefixLen": 0,
>>         "MacAddress": "02:42:0a:01:01:04",
>>         "NetworkID": "777c1fc8f94a1b06445092cf2441d0423f5f2f1b51fe500a27c76f4121953369",
>>         "PortMapping": null,
>>         "Ports": {},
>>         "SandboxKey": "/var/run/docker/netns/9e332c821d82",
>>         "SecondaryIPAddresses": null,
>>         "SecondaryIPv6Addresses": null
>>     },
>>     "ResolvConfPath": "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/resolv.conf",
>>     "HostnamePath": "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/hostname",
>>     "HostsPath": "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/hosts",
>>     "LogPath": "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a-json.log",
>>     "Name": "/k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd",
>>     "RestartCount": 0,
>>     "Driver": "devicemapper",
>>     "ExecDriver": "native-0.2",
>>     "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c6,c0",
>>     "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c6,c0",
>>     "AppArmorProfile": "",
>>     "ExecIDs": null,
>>     "HostConfig": {
>>         "Binds": null,
>>         "ContainerIDFile": "",
>>         "LxcConf": null,
>>         "Memory": 0,
>>         "MemorySwap": -1,
>>         "CpuShares": 2,
>>         "CpuPeriod": 0,
>>         "CpusetCpus": "",
>>         "CpusetMems": "",
>>         "CpuQuota": 0,
>>         "BlkioWeight": 0,
>>         "OomKillDisable": false,
>>         "MemorySwappiness": null,
>>         "Privileged": false,
>>         "PortBindings": null,
>>         "Links": null,
>>         "PublishAllPorts": false,
>>         "Dns": [
>>             "10.100.134.27",
>>             "10.100.0.2"
>>         ],
>>         "DnsSearch": [
>>             "test.svc.cluster.local",
>>             "svc.cluster.local",
>>             "cluster.local",
>>             "eu-west-1.compute.internal"
>>         ],
>>         "ExtraHosts": null,
>>         "VolumesFrom": null,
>>         "Devices": null,
>>         "NetworkMode": "default",
>>         "IpcMode": "",
>>         "PidMode": "",
>>         "UTSMode": "",
>>         "CapAdd": null,
>>         "CapDrop": null,
>>         "GroupAdd": null,
>>         "RestartPolicy": {
>>             "Name": "",
>>             "MaximumRetryCount": 0
>>         },
>>         "SecurityOpt": [
>>             "label:level:s0:c6,c0"
>>         ],
>>         "ReadonlyRootfs": false,
>>         "Ulimits": null,
>>         "LogConfig": {
>>             "Type": "json-file",
>>             "Config": {}
>>         },
>>         "CgroupParent": "",
>>         "ConsoleSize": [
>>             0,
>>             0
>>         ]
>>     },
>>     "GraphDriver": {
>>         "Name": "devicemapper",
>>         "Data": {
>>             "DeviceId": "4090",
>>             "DeviceName": "docker-202:1-50348421-9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a",
>>             "DeviceSize": "107374182400"
>>         }
>>     },
>>     "Mounts": [],
>>     "Config": {
>>         "Hostname": "phpfpm-app-2-8g07w",
>>         "Domainname": "",
>>         "User": "",
>>         "AttachStdin": false,
>>         "AttachStdout": false,
>>         "AttachStderr": false,
>>         "ExposedPorts": null,
>>         "PublishService": "",
>>         "Tty": false,
>>         "OpenStdin": false,
>>         "StdinOnce": false,
>>         "Env": [
>>             "NGINX_APP_SERVICE_PORT=80",
>>             "PHPFPM_APP_PORT_9000_TCP=tcp://172.30.164.254:9000",
>>             "KUBERNETES_PORT_53_UDP_PROTO=udp",
>>             "KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53",
>>             "KUBERNETES_PORT_53_TCP_PROTO=tcp",
>>             "NGINX_APP_SERVICE_HOST=172.30.152.11",
>>             "NGINX_APP_PORT_80_TCP_ADDR=172.30.152.11",
>>             "KUBERNETES_SERVICE_HOST=172.30.0.1",
>>             "KUBERNETES_SERVICE_PORT_HTTPS=443",
>>             "KUBERNETES_PORT_443_TCP_PROTO=tcp",
>>             "PHPFPM_APP_PORT_9000_TCP_PORT=9000",
>>             "KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1",
>>             "NGINX_APP_PORT_80_TCP_PROTO=tcp",
>>             "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
>>             "KUBERNETES_PORT_443_TCP_PORT=443",
>>             "KUBERNETES_PORT_53_UDP_PORT=53",
>>             "NGINX_APP_PORT=tcp://172.30.152.11:80",
>>             "PHPFPM_APP_SERVICE_HOST=172.30.164.254",
>>             "KUBERNETES_SERVICE_PORT=443",
>>             "KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53",
>>             "KUBERNETES_PORT_53_TCP_PORT=53",
>>             "NGINX_APP_PORT_80_TCP_PORT=80",
>>             "PHPFPM_APP_PORT=tcp://172.30.164.254:9000",
>>             "PHPFPM_APP_PORT_9000_TCP_PROTO=tcp",
>>             "NGINX_APP_PORT_80_TCP=tcp://172.30.152.11:80",
>>             "PHPFPM_APP_SERVICE_PORT_PHPFPM=9000",
>>             "PHPFPM_APP_PORT_9000_TCP_ADDR=172.30.164.254",
>>             "KUBERNETES_SERVICE_PORT_DNS=53",
>>             "KUBERNETES_SERVICE_PORT_DNS_TCP=53",
>>             "NGINX_APP_SERVICE_PORT_HTTP=80",
>>             "PHPFPM_APP_SERVICE_PORT=9000",
>>             "KUBERNETES_PORT=tcp://172.30.0.1:443",
>>             "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
>>             "KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1"
>>         ],
>>         "Cmd": null,
>>         "Image": "openshift/origin-pod:v1.1.0.1",
>>         "Volumes": null,
>>         "VolumeDriver": "",
>>         "WorkingDir": "",
>>         "Entrypoint": [
>>             "/pod"
>>         ],
>>         "NetworkDisabled": false,
>>         "MacAddress": "",
>>         "OnBuild": null,
>>         "Labels": {
>>             "io.kubernetes.pod.name": "test/phpfpm-app-2-8g07w",
>>             "io.kubernetes.pod.terminationGracePeriod": "30"
>>         }
>>     }
>> }
>> ]
>>
>> ---------------------
>> François Villain
>> DevOps - AAIO
>
> François,
>
> Is this the only container you see? The one you are looking at is the
> Kube networking container which isn’t the same as the app container. I
> notice that you didn’t use “docker ps -a” which will also show dead
> containers. Could you try that and see if there’s another failed
> container?
>
>> ________________________________________
>> De : Dan Mace <dmace redhat com>
>> Envoyé : mercredi 6 janvier 2016 14:46
>> À : François VILLAIN
>> Cc : Michalis Kargakis; users lists openshift redhat com
>> Objet : Re: Pod not passing deployment tests
>>
>> François,
>>
>> Do you have access to the docker daemon in this environment? I’d be
>> interested to see the output of `docker ps -a` and then a `docker
>> inspect` of your app container which may have failed to start or be
>> created.
>>
>> Michalis, regarding the recreate strategy: even when using the rolling
>> strategy, the deployer will always start with recreate for the initial
>> deployment since the “legacy” Kubernetes rolling updater doesn’t
>> support initial rollouts (only transitions between two existing RCs).
>>
>> On Wed, Jan 6, 2016 at 6:15 AM, François VILLAIN
>> <f villain linkbynet com> wrote:
>>> I first tried without the probes , but wasn't any more successful
>>>
>>> I have another DC in my template indeed.
>>>
>>>
>>> Here is the full template file i created : http://pastebin.com/EZJaCeQj
>>>
>>>
>>>
>>> ---------------------
>>> François Villain
>>> DevOps - AAIO
>>>
>>>
>>>
>>> ________________________________
>>> De : Michalis Kargakis <mkargaki redhat com>
>>> Envoyé : mercredi 6 janvier 2016 12:10
>>> À : François VILLAIN
>>> Cc : users lists openshift redhat com
>>> Objet : Re: Pod not passing deployment tests
>>>
>>> Hi François,
>>>
>>> can you deploy the app if you remove the probes?
>>> Also it's really strange that you are specifying a Rolling strategy but in
>>> the logs it seems that Recreate is running.
>>> Do you have any other DCs in your template?
>>>
>>> On Wed, Jan 6, 2016 at 11:11 AM, François VILLAIN <f villain linkbynet com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>> I'm having trouble deploying a pod running a php-fpm daemon (on port
>>>> 9000).
>>>> I created a DeploymentConfig in my template for that:
>>>>
>>>>
>>>>   - kind: DeploymentConfig
>>>>     apiVersion: v1
>>>>     metadata:
>>>>       name: "phpfpm-${SERVICE_NAME}"
>>>>       annotations:
>>>>         description: How to deploy the PHP App
>>>>     spec:
>>>>       strategy:
>>>>         type: Rolling
>>>>       replicas: 1
>>>>       selector:
>>>>         name: "phpfpm-${SERVICE_NAME}"
>>>>       template:
>>>>         metadata:
>>>>           name: "phpfpm-${SERVICE_NAME}"
>>>>           labels:
>>>>             name: "phpfpm-${SERVICE_NAME}"
>>>>         spec:
>>>>           volumes: []
>>>>           containers:
>>>>             - name: "phpfpm-${SERVICE_NAME}"
>>>>               imagePullPolicy: 'Always'
>>>>               image: "test/sample-phpinfo:latest"
>>>>               ports:
>>>>                 - containerPort: 9000
>>>>               env: []
>>>>               readinessProbe:
>>>>                 timeoutSeconds: 1
>>>>                 initialDelaySeconds: 5
>>>>                 tcpSocket:
>>>>                   port: 9000
>>>>               livenessProbe:
>>>>                 timeoutSeconds: 1
>>>>                 initialDelaySeconds: 5
>>>>                 tcpSocket:
>>>>                   port: 9000
>>>>
>>>>
>>>> And here is what i get in the deployment step:
>>>>
>>>> 1 I0104 14:57:09.139422       1 deployer.go:196] Deploying
>>>> test/phpfpm-app-1 for the first time (replicas: 1)
>>>> 2 I0104 14:57:09.143539       1 recreate.go:105] Scaling test/phpfpm-app-1
>>>> to 1 before performing acceptance check
>>>> 3 I0104 14:57:11.194413       1 recreate.go:110] Performing acceptance
>>>> check of test/phpfpm-app-1
>>>> 4 I0104 14:57:11.195425       1 lifecycle.go:379] Waiting 600 seconds for
>>>> pods owned by deployment "test/phpfpm-app-1" to become ready (checking every
>>>> 1 seconds; 0 pods previously accepted)
>>>> 5 F0104 15:07:11.195704       1 deployer.go:65] update acceptor rejected
>>>> test/phpfpm-app-1: pods for deployment "test/phpfpm-app-1" took longer than
>>>> 600 seconds to become ready
>>>>
>>>> Ca someone please tell me what's wrong ?
>>>>
>>>>
>>>> I got a same pod with a nginx instance perfectly working btw, so i guess
>>>> something is bad but i can't figure ou what
>>>>
>>>>  exactly...
>>>>
>>>> ---------------------
>>>> François Villain
>>>> DevOps - AAIO
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users lists openshift redhat com
>>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users lists openshift redhat com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]