Jenkins Pipeline Timeout Handling for Queue Wait Time

[Zhenye-Na]

What feature do you want to see added?

Problem Description

When using lockable resources in Jenkins pipelines with a timeout set, there is a potential issue with pipelines timing out due to the queue wait time. Specifically:

• There is a single lockable resource, e.g., LOCK_0.
• Two pipelines, PIPELINE_A and PIPELINE_B, are competing to acquire LOCK_0.
• Both pipelines have a timeout set to 12 hours.
• PIPELINE_A acquires LOCK_0 and runs for 10 hours.
• After PIPELINE_A finishes, PIPELINE_B starts, but it has been waiting in the queue for 10 hours.
• PIPELINE_B needs more than 2 hours to complete, which exceeds the remaining timeout.

The issue here is that PIPELINE_B times out due to the queue wait time, even though its actual execution time would have been within the timeout period.

Proposed Solutions

1. Is there a way to exclude the queue wait time from the timeout calculation?
2. Are there any existing workarounds to handle this scenario?
3. If not, could this be considered as a feature request?

Thanks!

Upstream changes

No response

Are you interested in contributing this feature?

No response

[mPokornyETM]

What you describe is, how the timeout() step works.
You can use the timeout also inside lock like

lock('resourceA') {
  timeout(10) {
     ....
  }
}

is it what you need?

[Zhenye-Na]

pipeline {
...
...

    options {
        timeout(time: 5, unit: 'DAYS')
    }

...
...

    triggers {
        cron('H H * * *')
    }

...

...


        stage("Example") {
            steps {
                lock('resourceA') {
                    runCommand('command X')
                }
            }
        }

This pipeline has a potential issue with resource locking and timeouts:

1. The pipeline:

   • Runs daily (via cron('H H * * *'))
   • Has a 5-day timeout (timeout(time: 5, unit: 'DAYS'))
   • Uses a lockable resource 'resourceA'
   • Can potentially take more than 24 hours to complete

2. The problem:

   • If a pipeline run is still executing when the next daily trigger occurs
   • The new run will be queued, waiting to acquire the 'resourceA' lock
   • The 5-day timeout counter starts immediately when the pipeline is queued
   • This means the timeout includes 1\ the waiting time for the lock AND 2\ the actual execution time

In other words, if a pipeline run takes too long, subsequent runs might timeout while just waiting in the queue for the lock to be released, even before they get a chance to actually execute their steps.

I am wondering if there is a solution to not include the waiting time for the lock ?

[mPokornyETM]

pipeline {
...
...

    options {
        timeout(time: 5, unit: 'DAYS')
    }

...
...

    triggers {
        cron('H H * * *')
    }

...

...


        stage("Example") {
            steps {
                lock('resourceA') {
                    runCommand('command X')
                }
            }
        }

This pipeline has a potential issue with resource locking and timeouts:

1. The pipeline:

   • Runs daily (via cron('H H * * *'))
   • Has a 5-day timeout (timeout(time: 5, unit: 'DAYS'))
   • Uses a lockable resource 'resourceA'
   • Can potentially take more than 24 hours to complete

2. The problem:

   • If a pipeline run is still executing when the next daily trigger occurs
   • The new run will be queued, waiting to acquire the 'resourceA' lock
   • The 5-day timeout counter starts immediately when the pipeline is queued
   • This means the timeout includes 1\ the waiting time for the lock AND 2\ the actual execution time

In other words, if a pipeline run takes too long, subsequent runs might timeout while just waiting in the queue for the lock to be released, even before they get a chance to actually execute their steps.

I am wondering if there is a solution to not include the waiting time for the lock ?

I am not familiar with parametrized pipelines, but this shall works:

pipeline {

...
...

    triggers {
        cron('H H * * *')
    }

...

...


        stage("Example") {
            steps {
                lock('resourceA') {
                    timeout(time: 5, unit: 'DAYS') {
                      runCommand('command X')
                    }
                }
            }
        }

[mPokornyETM]

Looks, that my proposal works. It will be fine, to add this information somewhere in the docu

[Zhenye-Na]

Looks, that my proposal works. It will be fine, to add this information somewhere in the docu

That will surely help a lot

[mPokornyETM]

Looks, that my proposal works. It will be fine, to add this information somewhere in the docu

That will surely help a lot

you are welcome to contirbute ;-)