[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: wait for build triggered by new-app



Ben,

    That explains it the build wait failure - you can see from the output I included at the end of my email, that I'm at v4.3.12 . Does this explain my observations with the project delete not waiting correctly either?

Regards,
Marvin


On Thu, May 7, 2020 at 2:53 PM Ben Parees <bparees redhat com> wrote:
what version is your cluster?  conditions were only recently added to the build.status section to OCP 4.4. oc wait doesn't work no resources that don't have a status.conditions section.




On Thu, May 7, 2020 at 2:47 PM Just Marvin <marvin the cynical robot gmail com> wrote:
Ben,

     It looks like I have a general problem with "oc wait" or waits, in general, not working reliably on my cluster. Here is my code now:

#  Step 7: Wait for the build to start. For some reason, the wait command doesn't work, saying that the build doesn't exist even
#  though the new-app command has completed.
buildName=""
while true; do
        sleep 1
        buildName=`oc get build | sed '1d' | awk '/^teams[-]/{print $1; exit 0}!/^teams[-]/{exit 1}'`
        if [[ $? -eq 0 ]]; then
                echo "build detected. Waiting for it to complete..."
                break;
        fi
done
#  Step 8: wait for the build to complete. Sigh. Doesn't work.
oc wait build/${buildName} --for="" --timeout=240s

......which produces this output:

--> Success
    Build scheduled, use 'oc logs -f bc/teams' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/teams'
    Run 'oc status' to view your app.
build detected. Waiting for it to complete...
error: timed out waiting for the condition on builds/teams-1

.....which may seem ok, but is a problem because the build completes in less than 4 mins, so the timeout should never happen. Here is a portion of the build/teams-1 yaml:

  phase: Complete
  stages:
  - durationMilliseconds: 341
    name: FetchInputs
    startTime: "2020-05-07T18:35:37Z"
    steps:
    - durationMilliseconds: 341
      name: FetchGitSource
      startTime: "2020-05-07T18:35:37Z"
  - durationMilliseconds: 39822
    name: PullImages
    startTime: "2020-05-07T18:35:40Z"
    steps:
    - durationMilliseconds: 21704
      name: PullBaseImage
      startTime: "2020-05-07T18:35:40Z"
    - durationMilliseconds: 18117
      name: PullBaseImage
      startTime: "2020-05-07T18:36:02Z"
  - durationMilliseconds: 55408
    name: Build
    startTime: "2020-05-07T18:36:20Z"
    steps:
    - durationMilliseconds: 55408
      name: DockerBuild
      startTime: "2020-05-07T18:36:20Z"
  - durationMilliseconds: 2102
    name: PushImage
    startTime: "2020-05-07T18:37:15Z"
    steps:
    - durationMilliseconds: 2102
      name: PushDockerImage
      startTime: "2020-05-07T18:37:15Z"
  startTimestamp: "2020-05-07T18:35:35Z"

.....so the build actually completes in about 2 mins.

Other observations:

[zaphod oc3027208274 gatt]$ oc delete project demo-mongo-apps --wait=true;date
project.project.openshift.io "demo-mongo-apps" deleted
Thu May  7 14:11:46 EDT 2020
[zaphod oc3027208274 gatt]$ oc delete project demo-mongo-apps --wait=true;date
project.project.openshift.io "demo-mongo-apps" deleted
Thu May  7 14:11:48 EDT 2020
[zaphod oc3027208274 gatt]$ oc delete project demo-mongo-apps --wait=true;date
project.project.openshift.io "demo-mongo-apps" deleted
Thu May  7 14:11:50 EDT 2020
[zaphod oc3027208274 gatt]$ oc delete project demo-mongo-apps --wait=true;date
project.project.openshift.io "demo-mongo-apps" deleted
Thu May  7 14:11:51 EDT 2020
.....and it takes about 30 seconds or so for the delete to take effect.

    One unusual think I've noticed about my cluster (I didn't set it up) is that it is a three node cluster with all three nodes sharing the master and worker node roles. Could this be a cause of the symptoms I'm seeing?

[zaphod oc3027208274 gatt]$ oc version
Client Version: openshift-clients-4.3.0-201909231341
Server Version: 4.3.12
Kubernetes Version: v1.16.2
[zaphod oc3027208274 gatt]$

Regards,
Marvin







On Wed, May 6, 2020 at 1:20 PM Ben Parees <bparees redhat com> wrote:


On Wed, May 6, 2020 at 12:50 PM Just Marvin <marvin the cynical robot gmail com> wrote:
Ben,

   I appreciate the pointer, but this doesn't seem to do the right thing. Here is the relevant portion of my code:

oc new-app --name=teams git@<enterprise git repo> -e MONGODB_USER=uuuu -e MONGODB_PASSWORD=pppp -e MONGODB_HOST=mongodb -e MONGODB_PORT=27017 -e MONGODB_DATABASE=teamdb --source-secret=<enterprise git secret>
#  Adding waits for teams and teams-1 because the build resource seems to be teams-1
oc wait build/teams --for="">

this line won't do anything, there's no build named "teams", only "teams-1" (and later teams-2, etc)

 
oc wait build/teams-1 --for="">

this looks correct


   And here is the output:

warning: Cannot check if git requires authentication.
--> Found container image 1db9786 (6 months old) from docker.io for "docker.io/websphere-liberty:javaee8"

    * An image stream tag will be created as "websphere-liberty:javaee8" that will track the source image
    * A Docker build using source code from git@<enterprise git repo> will be created
      * The resulting image will be pushed to image stream tag "teams:latest"
      * Every time "websphere-liberty:javaee8" changes a new build will be triggered
      * WARNING: this source repository may require credentials.
                 Create a secret with your git credentials and use 'oc set build-secret' to assign it to the build config.
    * This image will be deployed in deployment config "teams"
    * Ports 9080/tcp, 9443/tcp will be load balanced by service "teams"
      * Other containers can access this service through the hostname "teams"

--> Creating resources ...
    imagestream.image.openshift.io "websphere-liberty" created
    imagestream.image.openshift.io "teams" created
    buildconfig.build.openshift.io "teams" created
    deploymentconfig.apps.openshift.io "teams" created
    service "teams" created
--> Success
    Build scheduled, use 'oc logs -f bc/teams' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/teams'
    Run 'oc status' to view your app.
Error from server (NotFound): builds.build.openshift.io "teams" not found

this error is because as noted above, build/teams isn't ever going to exist.
 
Error from server (NotFound): builds.build.openshift.io "teams-1" not found

this error doesn't make sense to me since new-app should have created the build already, however there is a bit of a timing window since new-app creates the buildconfig and then relies on the build controller to kick off the first build.  It's possible that new-app is finishing and your oc wait is starting, before the build controller sees your buildconfig and creates the first build (teams-1)

unfortunately I don't think oc wait can wait for a resource to exist, so you're stuck doing either:
1) a short sleep before you invoke oc wait, to give the build time to be created (easy, but potentially flaky)
2) an oc get loop that loops until the resource exists, and then you can start waiting on it.  (of course if you do that, you can also use a go template to extract the phase of the build once it exists and watch it yourself, instead of using oc wait)

 
deploymentconfig.apps.openshift.io/teams patched
deploymentconfig.apps.openshift.io/teams patched
error: timed out waiting for the condition on deploymentconfigs/teams
error: no matching resources found
route.route.openshift.io/teams exposed
route.route.openshift.io/teams patched
The teams api is now available at http://teams.<a domain name>

    At this point, I can see that the build resource has been created, even though the "wait" that I tried to put in place has not worked.

[zaphod oc3027208274 gatt]$ oc get builds
NAME      TYPE     FROM          STATUS    STARTED          DURATION
teams-1   Docker   Git b4dba21   Running   36 seconds ago  
[zaphod oc3027208274 gatt]$ 

     What am I doing wrong?

Regards,
Marvin

On Wed, May 6, 2020 at 11:52 AM Ben Parees <bparees redhat com> wrote:


On Wed, May 6, 2020 at 11:34 AM Just Marvin <marvin the cynical robot gmail com> wrote:
Hi,

    I've been trying to write a script that runs various commands following a new-app. These commands will operate on the dc created by new-app, but fail if the resources haven't reached the right stage, so I'd like to wait for the build to complete. Unfortunately, I can't seem to figure out the right condition to wait for using the "oc wait" command. Does anyone have a suggestion? Wait on the bc or the build or.... (and for what condition)?

oc wait build/buildresourcename --for="">

unfortunately that won't help if the build in question fails (Complete means success) so set your timeout appropriately.



 

Thanks,
Marvin
_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users


--
Ben Parees | OpenShift



--
Ben Parees | OpenShift



--
Ben Parees | OpenShift


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]