Hi, I have a subversion trigger for one of my builds that has been functioning correctly for a while now. Recently it started to hang a lot:
This trigger has been running for over 57 minutes. It may have hung. Would you like to terminate it?
Is there a way to dig further into what may be causing this? The error gets hidden and the only way I know it’s happening is one of my developers tells me builds are not automatically starting.
Hi Nelson,
You can force FinalBuilder Server to log all messages to a text file by opening ‘FinalBuilderServer.exe.config’ located within the installation directory and adding:
< add key=“TraceOutput” value=“true” / >
to the ‘appSettings’ node. Once the FinalBuilder Server build service has been restarted a new text file will be created in the same directory called ‘TraceOutput.txt’. This should contain all the output from the Subversion process, and hopefully you’ll be able to determine why subversion is taking so long. Be sure to disable TraceOutput when you have found a solution as it can take up a considerable amount of space if left enabled.
Regards,
Paul.
I have enabled this and the error happened again. Unfortunately I’m not sure where in the log I should look because there is no timestamping and I wasn’t able to look at the log immediately.
Hi Nelson,
If you have a look through the log you should be able to locate where FinalBuilder Server executes the subversion client and directly under that should be output from Subversion, does the output give any indication about what is causing the Subversion client to take so long? Could you try executing the same command using the windows command prompt and seeing whether it takes the same length of time, you should be able to copy/paste from the log to determine which command arguments we’re using. If you still can’t find a cause, send in the log to support [at] finalbuilder.com and I’ll see whether we can work out whats going on.
Regards,
Paul.
Ok I finally caught the FinalBuilder Server reporting the trigger has taken more than 39 minutes, do I want to terminate. Here is the tail of the trace log. I’ve attached the tail en of the trace log (removing company specific info). I did drop to a command prompt on the server and execute the command and it returns almost immediately. Any ideas?trace.txt (1.626 KB)
Hi Nelson,
From the output that you’ve sent in it’s clear that the Subversion process is hanging (or taking an extremely long time), you can see that we’re starting the process, the process starts, then nothing, the process never finishes. I’m not sure what could be causing the behavior as you can see that FinalBuilder Server executed the exact same command previously and it worked without a problem, could you check whether you are running the most recent version of Subversion. Also, do you know whether any other actions were being performed on the subversion repository at roughly the same time?
Regards,
Paul.
Subversion version is 1.5.1 on the build server. This machine is dedicated to a single build and the build queue is set to only allow 1 build at a time. Although currently there is only one project configured anyway. It is certainly possible commands are running concurrently against the svn repository from developers machines, but it’s definitely only running 1 instance from the build box. Perhaps a timeout on the subversion trigger is necessary? If it times out, it should behave the same way as clicking terminate and re-activating the trigger.
One thing of note is the subversion repository is hosted offsite by Collabnet. It is possible a momentary loss in network connetivity could cause the hang. The problem is definitely intermitant and I’ve seen no pattern.
Hi Nelson,
Thanks for the update, I’m in the process of updating the version control triggers to contain a timeout option, I’ll let you know as soon as a new build is available. Also, it may be worth adjusting how often the Subversion repository is polled, you can do this by opening the ‘FinalBuilderServer.exe.config’ file located in the installation directory and modify the ‘PollingInterval’ value (value is in milliseconds). The new value will take effect once the build service is restarted.
Regards,
Paul.
Hi Paul,
Any progress on this item? I did adjust the polling and that helped, but it still occasionally hangs.
Thanks!
Hi Nelson,
Sorry for the delay, here’s the new build of FinalBuilder Server which includes an option to Timeout a trigger process after a set number of minutes. You can download the build from:
URL: https://www.finalbuilder.com/downloads/fbserver6/FBServer_6.2.0.651.exe
Regards,
Paul.
Paul,
Is this ability to set a timeout on a trigger available in version 6.3.0.670 of finalbuilder server?
Pancho
Yes, it’s the Process Timeout field, only available on Triggers that actually run processes.