Add feature to configure minimum instances by gbhat618 · Pull Request #533 · jenkinsci/google-compute-engine-plugin

gbhat618 · 2026-03-27T13:31:27Z

Adds feature minimum number of instances and minimum number of spare instances - which can be active in a configurable time range.

This feature is implemented by referring to that existing in the ec2-plugin. The logic is almost exactly as that of ec2-plugin, adjustments are made such that code is decoupled for better testing (especially with the retention strategy, and code computation parts). Files related in ec2-plugin

retention strategy related - https://github.com/jenkinsci/ec2-plugin/blob/9c48b882174fe2131e7fe9611e3e03ecc68326f0/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java#L127-L141; however in here oneShot configuration needed to be considered and race condition possibilities.
https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/util/MinimumInstanceChecker.java - copied almost exactly
https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/util/MinimumNumberOfInstancesTimeRangeConfig.java - copied almost exactly

UI visualization

Description	screenshot
form with new fields

Testing done

Automated testing

6 new integration tests are added covering the integration that check all the features introduced in here.
3 new unit test files added (each with several tests) that exercise the core logic part of the computation parts of minimum instances, whether to keep the instance, time range logic.

Manual testing

CasC based environment manual testing is conducted to ensure that the VMs are provisioned, stopping the jenkins deletes the idle instances

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests that demonstrate the feature works or the issue is fixed

gbhat618 · 2026-03-30T14:36:19Z

+        initCredentials(j);
+        client = ClientUtil.getClientFactory(j.jenkins, PROJECT_ID).computeClient();
+        // CasC loaded min/spare values but credentials weren't available yet so fails; trigger provisioning now
+        SaveableListener.fireOnChange(j.jenkins, null);


Due to the credentials initialized here, the tests are throwing an exception, later this SaveableListener call makes the provisioning take place,

error log

0.586 [id=630] SEVERE c.g.j.p.c.ComputeEngineCloud#createClient: Exception when creating GCE client hudson.AbortException: Failed to initialize HTTP transport: hudson.AbortException: Could not retrieve credentials: tiger-team-testing at com.google.jenkins.plugins.computeengine.client.ClientUtil.getClientFactory(ClientUtil.java:59) at com.google.jenkins.plugins.computeengine.client.ClientUtil.getClientFactory(ClientUtil.java:73) at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.createClient(ComputeEngineCloud.java:190) at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.getClient(ComputeEngineCloud.java:208) at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.availableNodeCapacity(ComputeEngineCloud.java:387) at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.provisionSpares(ComputeEngineCloud.java:310) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.lambda$performCheck$0(MinimumInstanceChecker.java:280) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.lambda$performCheck$1(MinimumInstanceChecker.java:265) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.performCheck(MinimumInstanceChecker.java:265) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.checkForMinimumInstances(MinimumInstanceChecker.java:144) at com.google.jenkins.plugins.computeengine.InstanceConfiguration$OnSaveListener.onChange(InstanceConfiguration.java:1082) at hudson.model.listeners.SaveableListener.lambda$fireOnChange$0(SaveableListener.java:90) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners.notify(Listeners.java:70) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:90) at jenkins.model.Jenkins.save(Jenkins.java:3616) at hudson.BulkChange.commit(BulkChange.java:98) at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:278) at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$6(ConfigurationAsCode.java:858) at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:808) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:858) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:730) at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:699) at io.jenkins.plugins.casc.misc.JenkinsConfiguredWithCodeRule.before(JenkinsConfiguredWithCodeRule.java:46) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:653) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.lang.Thread.run(Thread.java:1583) 0.586 [id=630] WARNING jenkins.util.Listeners#lambda$notify$0 java.lang.NullPointerException: Cannot invoke "com.google.cloud.graphite.platforms.plugin.client.ComputeClient.listInstancesWithLabel(String, java.util.Map)" because the return value of "com.google.jenkins.plugins.computeengine.ComputeEngineCloud.getClient()" is null at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.availableNodeCapacity(ComputeEngineCloud.java:387) at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.provisionSpares(ComputeEngineCloud.java:310) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.lambda$performCheck$0(MinimumInstanceChecker.java:280) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.lambda$performCheck$1(MinimumInstanceChecker.java:265) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.performCheck(MinimumInstanceChecker.java:265) at com.google.jenkins.plugins.computeengine.MinimumInstanceChecker.checkForMinimumInstances(MinimumInstanceChecker.java:144) at com.google.jenkins.plugins.computeengine.InstanceConfiguration$OnSaveListener.onChange(InstanceConfiguration.java:1082) at hudson.model.listeners.SaveableListener.lambda$fireOnChange$0(SaveableListener.java:90) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners.notify(Listeners.java:70) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:90) at jenkins.model.Jenkins.save(Jenkins.java:3616) at hudson.BulkChange.commit(BulkChange.java:98) at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:278) at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$6(ConfigurationAsCode.java:858) at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:808) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:858) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:730) at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:699) at io.jenkins.plugins.casc.misc.JenkinsConfiguredWithCodeRule.before(JenkinsConfiguredWithCodeRule.java:46) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:653) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.lang.Thread.run(Thread.java:1583)

gbhat618 · 2026-03-30T15:23:43Z

/label enhancement

jglick · 2026-03-30T18:18:20Z

stopping the jenkins deletes the idle instances

You mean when #530 is enabled?

jglick

Looks OK from a quick read.

jglick · 2026-03-30T18:20:05Z

+     */
+    public List<PlannedNode> provisionSpares(InstanceConfiguration config, int numberToProvision) {
+        List<PlannedNode> result = new ArrayList<>();
+        if (Jenkins.get().isQuietingDown() || Jenkins.get().isTerminating()) {


Beware that isTerminating is false immediately after termination begins! It only switches to true sometime later. Longtime annoyance in core (CB-internal references upon request); could use a new API for this.

jglick · 2026-03-30T18:24:41Z

+
+    @Override
+    public long getRecurrencePeriod() {
+        return recurrencePeriod.toMillis();


(would be nice to improve core to allow this to be specified directly as a Duration)

jglick · 2026-03-30T18:29:32Z

+    public static final class OnSaveListener extends SaveableListener {
+        @Override
+        public void onChange(Saveable o, XmlFile file) {
+            if (o instanceof Jenkins) {


Oof, there is no more specific event for a particular Cloud?

looks like not. looked at the ec2-plugin, kubernetes plugin and also tried checking in the jenkins core if there is anything like ~~CloudListener~~ - none found.

https://github.com/jenkinsci/ec2-plugin/blob/9c48b882174fe2131e7fe9611e3e03ecc68326f0/src/main/java/hudson/plugins/ec2/SlaveTemplate.java#L3255-L3263

https://github.com/jenkinsci/kubernetes-plugin/blob/a82d78a0087e9eaa10ec2339b48891549345c1a4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L124-L128

perhaps we need to implement one in core maybe https://github.com/jenkinsci/jenkins/blob/b679ff1627f7f4be7c0f255399b0460b0d593e73/core/src/main/java/hudson/slaves/Cloud.java#L323, https://github.com/jenkinsci/jenkins/blob/b679ff1627f7f4be7c0f255399b0460b0d593e73/core/src/main/java/hudson/slaves/Cloud.java#L313 and maybe also https://github.com/jenkinsci/jenkins/blob/b679ff1627f7f4be7c0f255399b0460b0d593e73/core/src/main/java/jenkins/model/Jenkins.java#L4033 but only when clouds added or removed.

Well Clouds each have their own config page now so it is not even using that endpoint to submit changes I think.

gbhat618 · 2026-03-31T04:04:24Z

stopping the jenkins deletes the idle instances

You mean when #530 is enabled?

yes, right; using terminateIdleDuringShutdown: true in the config.

…taskCompletedWithProblems when oneShot

…atic field reference to avoid class loading and property getting set to default

…ountedAsSpare became flake after removing the casc file and setting the retention time in code - go back to casc file for deterministic test execution

gbhat618 · 2026-03-31T09:10:55Z

one of the test ComputeEngineCloudMinimumInstancesIT#testAgentTemporarilyOffline_notCountedAsSpare had started flaking after I tried reduce the CasC configuration test file and started to set the retention time via code - this wouldn't work because the agents are spawned immediately after jenkins is started. Now added the separate CasC file again for this particular test.

As of now all tests are passing. The PR looks to be final with code changes.

mikecirioli · 2026-03-31T12:06:34Z

Minor concurrency note on MinimumInstanceChecker:

shouldPreserve() calls incrementPendingTerminations() outside the checking AtomicBoolean guard, while checkForMinimumInstances() calls resetPendingTerminations() inside it. Since shouldPreserve is invoked from ComputeEngineRetentionStrategy.check() on its own retention timer, there's a small race window:

shouldPreserve → incrementPendingTerminations(config) (counter = 1)
checkForMinimumInstances → resetPendingTerminations() (counter wiped)
Another shouldPreserve call sees counter = 0, allows a second termination that should have been blocked

Impact is low — worst case an extra agent gets terminated and the next periodic check (≤10 min) replenishes it. Just flagging for awareness, not a blocker.

gbhat618 · 2026-03-31T13:33:09Z

Minor concurrency note on MinimumInstanceChecker [...]

I see. Thank you for noting it down. Agree on the race condition existence. Maybe the fix for race condition is to not clear the map, but decrement when agent is terminated, perhaps via NodeListener.onDeleted. I am thinking to leave it at here for now, given the periodic task will offset if a spare gets deleted due to the race condition.

gbhat618 · 2026-03-31T14:06:12Z

facing issue with CI jenkins-infra/helpdesk#5055

gbhat618 · 2026-03-31T14:07:35Z

retriggering build to see if it works.

gbhat618 · 2026-03-31T15:17:20Z

All looks good with this PR,

reran the integration tests — all 6 are passing
did a manual interactive test
- by not configuring minimum instances - no spares created.
- configuring min instances on Monday (today is Tuesday) - no spares created
- configure today - instance created
- remove today and keep yesterday - spares deleted.

Overall all looks good imo

gbhat618 · 2026-03-31T15:17:57Z

IMPORTANT: /label enhancement

Add option to configure minimum instances

7d518ad

github-advanced-security AI found potential problems Mar 27, 2026

View reviewed changes

gbhat618 added 2 commits March 30, 2026 17:35

code cleanup, and integration tests

ff2e1ec

use an Optional as return value of shouldPreserveAgent

72dd14e

gbhat618 commented Mar 30, 2026

View reviewed changes

Comment thread src/main/java/com/google/jenkins/plugins/computeengine/ComputeEngineMonitor.java Outdated

code fixes - from self code review

9b3e6a6

gbhat618 commented Mar 30, 2026

View reviewed changes

one less casc config file

2a7cdc1

gbhat618 changed the title ~~Add option to configure minimum instances~~ Add feature to configure minimum instances Mar 30, 2026

gbhat618 added 4 commits March 30, 2026 21:48

switch to ItemListener.onLoaded

7e66d03

comment fixes and access specifier fixes

ae40d4d

spotbugs after access specifier change

8c80b9d

spotless:fix

9441402

gbhat618 marked this pull request as ready for review March 30, 2026 17:18

jglick approved these changes Mar 30, 2026

View reviewed changes

gbhat618 added 6 commits March 31, 2026 09:57

unparseable instanceCapStr shouldn't be silently passed

dc9058f

minimumInstanceCheckPeriod property name be derived from the class name

072ad1c

count the availableCapacity() per iteration as it may have changed

bd6f9fc

document the need for minimum instance checking in taskCompleted and …

c8ed732

…taskCompletedWithProblems when oneShot

initialize the minimumCheckPeriod property by using string and not st…

c914f61

…atic field reference to avoid class loading and property getting set to default

ComputeEngineCloudMinimumInstancesIT#testAgentTemporarilyOffline_notC…

f9e4881

…ountedAsSpare became flake after removing the casc file and setting the retention time in code - go back to casc file for deterministic test execution

jglick approved these changes Mar 31, 2026

View reviewed changes

Comment thread src/main/java/com/google/jenkins/plugins/computeengine/ComputeEngineMonitor.java Outdated

Comment thread ...m/google/jenkins/plugins/computeengine/integration/ComputeEngineCloudMinimumInstancesIT.java Outdated

use FlagRule to configure system property in test

7193e02

Trigger CI

e9cad18

jtnord added the enhancement New feature or request label Mar 31, 2026

jtnord merged commit 3634530 into jenkinsci:develop Mar 31, 2026
17 checks passed

Uh oh!

Conversation

gbhat618 commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing done

Submitter checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbhat618 Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gbhat618 commented Mar 30, 2026

Uh oh!

jglick commented Mar 30, 2026

Uh oh!

jglick left a comment

Choose a reason for hiding this comment

Uh oh!

jglick Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jglick Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jglick Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

gbhat618 Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

jglick Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

Uh oh!

Uh oh!

mikecirioli commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

gbhat618 commented Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gbhat618 commented Mar 27, 2026 •

edited

Loading

gbhat618 Mar 30, 2026 •

edited

Loading