Hi,
we would like some advice on how to best use multi core CPUs in FinalBuilder.
We want to efficiently distribute the work of our build script across multiple cores. This workload consists mainly of building about 2000 delphi projects (give or take a few hundred). These projects can be built in parallel.
Currently we split these projects according to organisational units that exist within our product. We then build the all the units in parallel by using an async action group.
This approach has the obvious flaw that not all of these units are equally complex so some take significantly longer to finish than others. This has the downside that for a large portion of the runtime we are actually only using one CPU core to build a number of projects sequentially.
Our new approach is to instead ignore the organisational units and just split the list of all projects into four equally sized blocks and build those in parallel. This looks good in our first proof of concept. However we currently have to hardcode the number of parallel blocks. This is not a problem right now because all of our build servers have the exactly four cores. But this is likely to change in the future.
I would like to know if our approach is reasonable or if there are better alternatives. Any tips for most efficiently using multi core cpus - especially with heterogeneous build servers - would be appreciated.
Regards,
Christoph
Hi Christoph
This is a very difficult question to answer.
In general terms, you should limit the use of threads where they will be competing for the same resources, for example I/O is something that doesn’t scale very far on windows. Compilers can be cpu bound, or i/o bound, or both. I would certainly limit the number of threads running to no more than 1 per core. Threading does have overhead, and that applies in FinalBuilder as well, we do try to avoid locking but that’s not always possible. A few years back I improved a customer’s build process by 1000% just by limiting the number of threads they were using (they had around 200 going, I set it to the number of cores!).
You might want to take a look at the Queue actions, I have added an example which shows using multiple threads to process a queue :
https://github.com/VSoftTechnologie…allelQueue
The advantage of using a queue like this is you keep the threads busy and avoid some of the overhead of constantly spinning up new threads and you keep the threads busy, even if you have one that is performing a long running task, the others keep plugging through the list of projects.
Hope that helps.
Thanks for the reply.
Limiting the number of threads is also one of the reasons we want to rework the script.
Using a queue looks like a good approach. I’ll give that a try.