It’s new, so it must have been introduced in latest 2 released versions.
I am currently using 1.9.2.1522.
After a weekend, I have build hanging for 3 days with status that it aways for a lock.
Build page:
Hi Michal,
Can you send the timeline for the build #5133 and more details about the lock that was acquired e.g. stage or configuration, acquire all, acquire any, acquire specific or acquire expression, if expression, what was the expression? Also, let us know if there were any errors in the event log at the time the build #5133 was running.
We have extensive unit tests for shared resource locks so we need some more details to figure out what we’ve missed.
Ok, so the stopping status is actually because the hanging build was tried to be stopped by a user.
The lock is for queueing builds, so configuration and it’ an expression lock:
Also the hanging build first waited also for other build:
| State | When | Duration | Message |
|---|---|---|---|
| Build awaiting queue conditions | 3 days ago | 2 hours, 41 minutes | Build is waiting to acquire shared resource locks: |
| Cannot acquire any lock on ‘wrp-testauto05’ label of shared resource ‘Server.Master.GuiRegression’ as a write lock is reserved by build #5133 of config ‘Run GUI Regression Tests’. | |||
| Build awaiting queue conditions | 3 days ago | 12 minutes | Build is waiting to acquire shared resource locks: |
| Cannot acquire any lock on ‘wrp-testauto05’ label of shared resource ‘Server.Master.GuiRegression’ as a write lock is reserved by build #5132 of config ‘Run GUI Regression Tests’. |
I see no event logged at that time.
Timeline for #5133:
| State | When | Duration | Message |
|---|---|---|---|
| Build failed | 3 days ago | 0 milliseconds | Build failed |
| Build completing | 3 days ago | 46 milliseconds | Build failed. Completing failed build… |
| Stage executing | 3 days ago | 15 milliseconds | Completing stage. |
| Stage executing | 3 days ago | 46 milliseconds | Updating unit test totals |
| Stage executing | 3 days ago | 31 milliseconds | Loading build variables from stage |
| Stage executing | 3 days ago | 0 milliseconds | Registering artifacts |
| Stage executing | 3 days ago | 15 milliseconds | Stage done on agent ‘wrp-testauto05’. |
| Stage executing | 3 days ago | 218 milliseconds | Syncing workspace from agent ‘wrp-testauto05’ to server |
| Stage executing | 3 days ago | 0 milliseconds | Creating archive files in workspace of agent ‘wrp-testauto05’ ready to copy to server |
| Stage executing | 3 days ago | 2 hours, 44 minutes | Executing stage on agent ‘wrp-testauto05’ |
| Stage initialising | 3 days ago | 11 seconds | Initialising workspace on agent ‘wrp-testauto05’ |
| Stage initialising | 3 days ago | 62 milliseconds | Sending stage ‘Execute’ to agent ‘wrp-testauto05’ |
| Stage initialising | 3 days ago | 156 milliseconds | Initialising stage… |
| Stage ready | 3 days ago | 46 milliseconds | Agent ‘wrp-testauto05’ reserved and license allocated for stage ‘Execute’ |
| Stage awaiting agent | 3 days ago | 171 milliseconds | Reserving agent ‘wrp-testauto05’ for stage ‘Execute’ which is at position 1 in the stage queue. |
| Stage awaiting agent | 3 days ago | 650 milliseconds | Adding stage ‘Execute’ to stage queue. There are currently 0 stages on the queue. |
| Stage awaiting agent | 3 days ago | 15 milliseconds | Queuing stage ‘Select Agent’ |
| Stage completed | 3 days ago | 46 milliseconds | Stage completed. |
| Stage executing | 3 days ago | 0 milliseconds | Completing stage. |
| Stage executing | 3 days ago | 31 milliseconds | Updating unit test totals |
| Stage executing | 3 days ago | 31 milliseconds | Loading build variables from stage |
| Stage executing | 3 days ago | 0 milliseconds | Registering artifacts |
| Stage executing | 3 days ago | 15 milliseconds | Stage done on agent ‘wrpbuildce02’. |
| Stage executing | 3 days ago | 359 milliseconds | Creating archive files in workspace of agent ‘wrpbuildce02’ ready to copy to server |
| Stage executing | 3 days ago | 0 milliseconds | Syncing workspace from agent ‘wrpbuildce02’ to server |
| Stage executing | 3 days ago | 124 milliseconds | Executing stage on agent ‘wrpbuildce02’ |
| Stage initialising | 3 days ago | 21 seconds | Initialising workspace on agent ‘wrpbuildce02’ |
| Stage initialising | 3 days ago | 78 milliseconds | Sending stage ‘Select Agent’ to agent ‘wrpbuildce02’ |
| Stage initialising | 3 days ago | 234 milliseconds | Initialising stage… |
| Stage ready | 3 days ago | 46 milliseconds | Agent ‘wrpbuildce02’ reserved and license allocated for stage ‘Select Agent’ |
| Stage awaiting agent | 3 days ago | 156 milliseconds | Reserving agent ‘wrpbuildce02’ for stage ‘Select Agent’ which is at position 1 in the stage queue. |
| Stage awaiting agent | 3 days ago | 984 milliseconds | Adding stage ‘Select Agent’ to stage queue. There are currently 0 stages on the queue. |
| Stage awaiting agent | 3 days ago | 31 milliseconds | Queuing stage ‘Select Agent’ |
| Build started | 3 days ago | 31 milliseconds | Build started |
| Build starting | 3 days ago | 0 milliseconds | Preparing to start build. |
| Build starting | 3 days ago | 62 milliseconds | Associating build repository cache revisions |
| Build queued | 3 days ago | 31 milliseconds | Starting build. |
| Build awaiting queue conditions | 3 days ago | 15 milliseconds | Preparing to start build. |
| Build awaiting queue conditions | 3 days ago | 0 milliseconds | Acquiring shared resource locks. |
| Build awaiting queue conditions | 3 days ago | 0 milliseconds | Build has acquired the following shared resource locks: |
| a write lock on label wrp-testauto05 of server shared resource ‘Server.Master.GuiRegression’ with quota of 1 | |||
| Build awaiting queue conditions | 3 days ago | 296 milliseconds | Checking repository: GuiRegressionTestingScenarios |
| Build awaiting queue conditions | 3 days ago | 312 milliseconds | Checking repository: GuiRegressionTestingTool |
| Build awaiting queue conditions | 3 days ago | 322 milliseconds | Checking repository: CIBuildTools |
| Build awaiting queue conditions | 3 days ago | 16 milliseconds | Checking repositories are up to date |
| Build awaiting queue conditions | 3 days ago | 0 milliseconds | Checking 3 build repositories. |
| Build awaiting queue conditions | 3 days ago | 47 minutes | Build is waiting to acquire shared resource locks: |
| Cannot acquire any lock on ‘wrp-testauto05’ label of shared resource ‘Server.Master.GuiRegression’ as a write lock is reserved by build #5132 of config ‘Run GUI Regression Tests’. | |||
| Build awaiting queue conditions | 3 days ago | 2 hours, 25 minutes | Build is waiting to acquire shared resource locks: |
| Cannot acquire any lock on ‘wrp-testauto05’ label of shared resource ‘Server.Master.GuiRegression’ as a write lock is reserved by build #5131 of config ‘Run GUI Regression Tests’. | |||
| Build awaiting queue conditions | 3 days ago | 46 milliseconds | Checking configuration conditions. |
| Build awaiting queue conditions | 3 days ago | 0 milliseconds | No configuration conditions to check. |
| Acquiring any shared resource locks. | |||
| Build awaiting queue conditions | 3 days ago | 15 milliseconds | Adding to build queue |
| Build awaiting queue conditions | 3 days ago | 0 milliseconds | Added to build queue. There are now 8 builds on the queue. |
| Build awaiting queue conditions | 3 days ago | 46 milliseconds | Serialising build variables to workspace |
| Build initialising | 3 days ago | 0 milliseconds | Queuing build |
| Build initialising | 3 days ago | 0 milliseconds | Preparing to queue build. |
| Build initialising | 3 days ago | 15 milliseconds | Creating build stages. |
| Build initialising | 3 days ago | 0 milliseconds | Build created. |
| Build initialising | 3 days ago | 15 milliseconds | Creating build log file. |
| Build initialising | 3 days ago | 15 milliseconds | Initialising build workspace on server. |
| Build initialising | 3 days ago | 62 milliseconds | Waiting for 0 repositories in total to be checked and updated. |
| Build initialising | 3 days ago | 0 milliseconds | Associating latest changesets with build. |
| Build initialising | 3 days ago | 234 milliseconds | Finished checking repository ‘GuiRegressionTestingTool’. Last check started at 08.08.2025 03:01:27 and finished at 08.08.2025 03:01:35. Build started at 08.08.2025 03:01:27 before last finished repository check started. So moving on to next repository in queue. |
| Build initialising | 3 days ago | 0 milliseconds | Finished checking repository ‘GuiRegressionTestingScenarios’. Last check started at 08.08.2025 03:01:27 and finished at 08.08.2025 03:01:34. Build started at 08.08.2025 03:01:27 before last finished repository check started. So moving on to next repository in queue. |
| Build initialising | 3 days ago | 0 milliseconds | Finished checking repository ‘CIBuildTools’. Last check started at 08.08.2025 03:01:27 and finished at 08.08.2025 03:01:34. Build started at 08.08.2025 03:01:27 before last finished repository check started. So moving on to next repository in queue. |
| Build initialising | 3 days ago | 7 seconds | Waiting for 3 repositories in total to be checked and updated. |
| Build initialising | 3 days ago | 218 milliseconds | Waiting for repository ‘GuiRegressionTestingScenarios’ to be checked and updated. |
| Build initialising | 3 days ago | 15 milliseconds | Waiting for repository ‘GuiRegressionTestingTool’ to be checked and updated. |
| Build initialising | 3 days ago | 0 milliseconds | Waiting for repository ‘CIBuildTools’ to be checked and updated. |
| Build initialising | 3 days ago | 0 milliseconds | Waiting for repositories to be checked |
| Build initialising | 3 days ago | 15 milliseconds | Waiting for repositories to be initialised |
So, it looks like the issue is that the build did not stop correctly when the Stop button was clicked while it was waiting for a shared resource lock? Does it stop if you click the Stop button now?
The hanging build was “stopped” 2 minutes before 5133 finished.
When I clicked stop again, it stopped fine.
We’ve seen this type of issue where a build gets stuck in “Build Stopping” status before but have never been able to reliably reproduce and diagnose the cause. We’ll put some more time into investigating this.
