Continua build hanging in Finalbuilder

We have had at least 5 builds over the last 4 days seemingly hang when we trigger the build of a finalbuilder project through continua.  This doesn’t happen all the time, but is happening at a concerning rate. 

The build via continua is a single stage that triggers a Finalbuilder project that does a lot of things:
- gets source
- copies libraries
- builds 32 bit (using Incredibuild)
- zips output
- builds 64 bit (using Incredibuild) ***hang usually occurs somewhere in here 
- zips output
- builds 2nd 64 bit configuration (if applicable) (using Incredibuild)***hang usually occurs somewhere in here 
- zips output
- builds 3rd 64 bit configuration (if applicable) (using Incredibuild)***hang usually occurs somewhere in here 
- zips output
-copies output to server location
- builds 64 bit installer
-copies output to server location
- builds 64 bit 2nd configuration installer
-copies output to server location
- builds 64 bit 3rd configuration installer
-copies output to server location

I’m not sure if some kind of timeout is occurring between agent and server or if something else is happening.  The only errors in the event longs are from us stopping the build and the agent being unable to clean up the temp file(s) because Finalbuilder still had them locked.

Any ideas on things we could/should try?

Thanks

Jeremy

Hi Jeremy

My thought is that this has nothing to do with Continua CI, and more to do with Incredibuild. If I had to guess I’d say it is prompting for input or waiting for a user response after an error (product licensing issues are a common cause of this sort of thing).

If you have a look on the agent machine, is incredibuild still running when it Finalbuilder appears to be hung? Also, try running the FinalBuilder project on the agent machine using the same user as the agent service is running under. I’d recommend setting the cleanup policy on for the agent workspace so that it doesn’t cleanup the agent workspace when a user stops the build. When it hangs, stop the build go to the agent workspace folder and run the FinalBuilder project from there, so it is using the same files etc as when run from Continua CI. That will at least allow you to see if it hangs outside of Continua CI.

Some additional information from our builds:

We continue to see the Continua server hang during the second or third Incredibuild step described above. Looking at the Incredibuild logs, these builds are finishing, but this information is not being communicated back to either Finalbuilder or Continua. The Incredibuild output that we see in Continua just stops reporting information.

We’re running Finalbuilder v8.0.0.1029 (looks like we need to update) and Continua 1.7.0.108 (we can update to the beta version if that’s advisable.)

thanks,
Jeffrey Dinkler
Release Engineer

Hi Jeffrey,

We’ll need some more information to troubleshoot this. Firstly, can you run the same FinalBuilder command line independently of Continua so we can ascertain whether this has anything to do with Continua. Either way can you send your FinalBuilder project file and the log file to support@finalbuilder.comIt may also help if you send the Incredibuild logs.

Can you also check, using the task manager, if the Incredibuild process is still running after the FinalBuilder process finishes. it is possible that is still stays running after logging that it has completed.

If, for some strange reason, the issue only happens while running in Continua, can you enable debug logging for both the Continua server and agent? After updating the configuration files, restart both services and then run the builds which hang. Once the issue has been reproduced (and Incredibuild has finished running), send the debug logs to support@finalbuilder.com