VSoft Technologies BlogsVSoft Technologies Blogs - posts about our products and software development.https://www.finalbuilder.com/resources/blogsContinuous Integration Server performancehttps://www.finalbuilder.com/resources/blogs/postid/754/continuous-integration-server-performance.NET,Continua CI,Delphi,FinalBuilder,Git,Mercurial,Web DevelopmentMon, 11 Sep 2017 14:31:58 GMT<p>Continuous Integration Servers are often underspecified when it comes to hardware. In the early days of Automated Builds, the build server was quite often that old pc in the corner of the office, or an old server in the data center that no one else wanted. Developers weren't doing many builds per day, so it worked, it was probably slow but that didn't seem to matter much.</p> <p>Fast forward 20 years, and the Continuous Integration Server is now a critical service. The volume and frequency of builds has increased dramatically and a slow CI server can be a real problem in an environment where we want fast feedback on that code we just committed (even though it "worked on my machine"). Continuous Deployment only adds to the workload of the CI server.&nbsp; </p> <p>In this post, I'm going to cover off some ideas to hopefully improve the performance of your CI server. I'm not going to cover compilation, unit tests etc. (which can be where a lot of the time is spent). Instead, I'll focus on the environment, machine configuration and some settings on your Continua CI configurations.</p> <h2>Hardware Requirements</h2> <p>It's impossible to provide hard and fast specs for hardware or virtual machines, as it varies greatly depending on the expected load.</p> <p>There are a bunch of things you can tweak that may improve performance. I will touch on some key points for virtual hosts, but I'm not going to go too deep into tuning virtual hosts, that's not my area of expertise. Of course, dedicated physical machines would be the ideal, but these days, even if you do get dedicated hardware for CI/CD, it's most likely going to be as a virtual host (hyper-v or vmware) rather than an OS installed on bare metal (do companies still provision a single os on bare metal servers these days?). Virtualisation brings in a whole bunch of benefits, but it also brings with it some limitations that cannot be ignored.</p> <p>Continuous Integration environments are mostly I/O bound and Continua CI is no different in that regard. So let's look at the various resources used by CI/CD.</p> <h3>CPU</h3> <p>It's unlikely that CPU will be a limiting factor in the performance of your CI server, unless you are running other CPU intensive tasks on your server. If that's the case, then move your CI server to dedicated hardware, or at least a dedicated virtual host. </p> <p>At a minimum you should have at least 2 cores on the server. On our production server, which is a virtual machine (on Hyper-V 2012R2) with 4 virtual cores and dynamic RAM, the Windows resource monitor shows that average CPU usage usually sits around 2% when idle (no running builds, measured on the guest OS using resource monitor on the Continua Server Service). With 10 concurrent builds running, the Continua CI server service was using around 6% cpu.</p> <p>Adding another 4 cores made very little difference. The Hyper-V host machine, which is also running a bunch of agent VM's, has plenty of CPU capacity, with the average CPU usage round 5-7%. Cutting down the number of cores to 2 did make a slight difference, with the VM showing slightly higher CPU usage, however no discernible difference in build times.</p> <p> This is obviously not very scientific, but it did demonstrate (well to me at least) that CPU is not the limiting factor. I set the server VM back to 4 cores and left it at that. Our Hyper-V host machines are a few years old now, and have 7200 rpm SAS hard drives (in Raid 10) rather than SSD's (they were still too expensive when we bought the machines).</p> <p>On a Continua CI Agent, we recommend at least 2 cpu cores, and limit the concurrent builds running on the agent to 1 per core. This isn't a hard and fast rule, just a convention we adhere to here (based on some performance testing). You may want to add extra cores depending on what compilers or tools you are running during your build process. The only way to know if this is needed is to monitor cpu on the agent machine while a build is running.</p> <h3>I/O</h3> <p>The most used resources are disk read/write and network read/write. Poor I/O performance will really slow down your builds.</p> <h4>Disk</h4> <p>It goes without saying, but use the fastest disks you have available to you. If you can afford it, new generation nvme/pcie SSD's are the way to go. They are still quite expensive for larger capacities though. At the very least, use a separate disk for the operating system and software installation, and another disk for your Continua CI Server's share folder (or the agents workspace folder on agent machines). This is where most of the I/O happens during builds. This recommendation applies whether running on dedicated hardware or in a virtual machine.</p> <p> If you are running the server and agent machines on the same virtual host (as we do for our production environment) then this is very important to get right. Poor I/O performance in virtualised environments is not uncommon - having agents and the server fighting for a slice of the same I/O pie is not a good idea.<br /> <br /> On the agent machines, good disk performance is critical. When a build is started on the agent, the first thing it does is create a workspace folder. It then exports the source code from the repository cache(s) (Mercurial repo which was cloned from the server) to that folder, using the repository rules (more on this later). This workspace initialisation phase can be very slow if you have poor I/O performance.</p> <h4>Network</h4> <p>Continua CI uses networking to transfer files, repository changes etc between the server and the agents. Poor network performance will impact on build initialisation times (updating the agents repo cache, build workspace) and on build completion times (transferring workspace changes back to the server). Logging between the agent and the server will also be impacted by poor network performance.</p> <p>By default, Continua CI uses SMB to transfer files, source code (repository caches) between the server and the agents. When the server's share folder is not accessible by SMB, Continua CI will try to use SSH/SFTP (Continua CI installs it own specialised SSH service). In high latency networks (for example if the agent is remote from the server), SSH/SFTP may perform better than SMB.</p> <p>You can force an agent to use SSH/SFTP by setting the agents ServerFileTransport.ForceSSH property to true.</p> <h3>Database</h3> <p>Continua CI supports PostgreSQL (the default) or Microsoft SQL Server. If you chose to use MSSQL, we recommend running it on a separate well specified machine. MSSQL is quite heavy in it's use of RAM and disk I/O - it's best run on a machine that has been tuned to run it properly. I'm not going to go into that here, that's a whole other topic on an area that I'm definitely not an expert.</p> <p>The PostgreSQL database server that is installed by default (unless you select otherwise) with Continua CI is much more more frugal when it comes to resources. On our main Continua CI server, PostgreSQL typically using around 60MB of ram. Contrast that with SQL Server running on my dev machine, not used or touched for weeks and it's using 800MB! PostgreSQL can also be tuned, we have tried to provision it with sensible defaults that strike a balance between performance and resource usage. If you need to tune PostgreSQL, then we recommend installing your own PostgreSQL instance and pointing Continua CI at it.</p> <p>Currently the Continua CI installer doesn't provide any options for the database install location (C:\ProgramData\VSoft\ContinuaCI\PostgreSQLDB ), this is something we are looking at for a future release, that will make it possible to put the database on it's own drive. For now, it's possible to move the database to another location by using a symlink, we have a few customers who have done this successfully. Contact support if you need help with this.</p> <h2>Virtualisation Tips</h2> <h3>Virtual CPU Cores</h3> <p>In a virtual environment, it's very important not to overload your virtual host. Note that there is a difference between overloading and over allocating virtual cores. It's a common practice to allocate more virtual cores across the virtual machines than there are physical/logical cores (logical when HyperThreading is enabled), but this has to be done with the knowledge and understanding of the load on the host machine. Overloading happens when so many cores are allocated and in use that the hypervisor is unable to schedule a core to a virtual machine when needed. This results in pauses and poor performance.</p> <p>In a clustered environment this is even more important, because when a cluster node dies, or is removed for upgrades etc, virtual machines will move to another node in the cluster - if that node is already overloaded then you will soon start hearing the complaints from users!</p> <p>The best explanation I have found on how hypervisors allocate cores is this article - <a href="https://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/">https://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/</a> - it's Hyper-V specific (we use Hyper-V here) but much of the information also applies to VMWare.</p> <h3>Virtual Disks</h3> <p>When creating separate virtual disk volumes for your virtual machines, try to put those virtual drives on different physical drives, so they are not competing for the same I/O. Use fixed size virtual disks.</p> <h2>Continua CI Configuration Tuning</h2> <p>Continua CI is not immune to performance problems, we're always working to make it faster and consume less resources. There are however a few things that can be tuned in Continua CI to improve performance.</p> <h3>Repository Branch Settings</h3> <p>Use specific branch patterns to narrow down the number of repository files and folders which are monitored and downloaded. With repositories which use folder-based branches, such as Subversion and TFS, consider moving old branches to a separate archive folder in your repository which will not match the branch patterns. Note that you can use more than one Continua CI repository per actual repository. Some users will have multiple projects in one repository, but only need to build a single one for each configuration. Make use of relative paths, where supported by your repository type, to limit your repository to a single project folder. This can significantly speed up repository initialisation and changeset updating. </p> <h3>Repository Polling</h3> <p>Continua CI polls repositories periodically to detect new commits. Each time this occurs, Continua CI invokes the command line client for that repo, and parses the output of that process. Some clients use a surprising amount of CPU. The git client, for example, uses around 8% CPU per instance on our production server while checking for commits. Most of the time, these processes only run for a very short amount of time (when no changes are detected), however if you have a lot of repositories, these small cpu spikes can add up.</p> <p> There are a couple of options to keep this under control.</p> <p> 1) Set the appropriate polling interval for your repositories. If changes to a repository occur infrequently, then there's no point polling frequently.</p> <p>2) Set the Server property Server.RepoMonitor.MaxCheckers property. This controls how many version control client processes are spawned concurrently, the default (5) is quite conservative so you should only need to lower this on a very low spec system. If you have plenty of spare CPU capacity, then you can increase this value, however if you do then monitor CPU usage to make sure you don't overload the server.</p> <p>3) Manual polling, using post commit hooks. This reduces CPU usage on the server, by only polling for repository changes when requested and has the added benefit of reducing the load on your version control server. This does take some setting up, and depends very much on the capabilities of your version control system. I'll take a look at post commit hooks in a future blog post.</p> <h3>Repository Path Filtering</h3> <p>Repository Path Filtering is an option on all repository types, with the exception of Mercurial (*I'll explain why shortly). What this filtering does is allow you to limit which files get added to the server's repository cache. This filtering has a few benefits, less disk space used on the server and the agents, less network I/O when transferring the changes from the server to the agent, and less I/O when checking out the source into the build workspace.</p> <p>A typical use case for these rules is when you have files in your repository that rarely change and are not needed for the build process (design docs, deployment notes etc). No point adding them to the repo cache if you don't use them.</p> <p>Changes to these rules won't affect files that are already in the repository cache, but it will avoid committing changes to those filtered out files to the repo cache. The best bang for buck with these filters will come if the repository is reset (the cache is rebuilt, so filtered out files are never committed to the cache), however that can be an expensive operation, so unlike other repository settings, changing these rules will not force a reset.</p> <p>* These filters don't apply to Mercurial repositories, as we use Mercurial for our repository cache. When you point Continua CI at a Mercurial repository, it just clones it to the server (repo cache), and then clones it to the agents (repo cache) without any modifcations.</p> <h3>Repository Rules</h3> <p>Each Stage has a settings tab called Repository Rules. These rules apply when checking out the source from the agent's repository cache(s) to the build workspace. Only check out the source you need, this will improve performance. If a stage doesn't need the source at all (for example, it's only working with artifacts from previous stages), then just blank out the Repository Rules field.</p> <p>Don't leave logging of the repository rules turned on unless you are debugging the rules. Logging the files exported to the workspace can be a real performance killer.</p> <h3>Workspace Rules</h3> <p>Similar to Repository Rules, these rules control which files are transferred between the server and agent's build workspace folders, and back again. Only transfer files back to the server's workspace that you actually need, like build artifacts, reports etc. </p> <p>Don't leave logging of the workspace rules turned on unless you are debugging the rules. Logging the files transferred can be a real performance killer.</p> <h3>Actions</h3> <p>Avoid logging too much information. For example, verbose logging on MSBuild should be avoided unless debugging build issues. Output logged from actions is queued and sent back to the server to be written to the build log, this causes high network and disk I/O.</p> <h3>Disk Space</h3> <p>Disk space is quite often at a premium (especially with SSD's), and it's important to keep on top of it. This is where the Clean up Policies come into play. Continua CI allows you to specify a global clean up policy for both the server and the agents, however it can be overridden at the Project or Configuration level. The clean up policy controls how long to keep old builds and their associated workspaces around. The clean up policy is highly configurable - use it to keep control over disk space. Bear in mind that the work of cleaning up old builds is quite I/O and database intensive, so be sure to schedule it to run during a quite period</p> <h3>Anti-virus Software</h3> <p>Anti-virus software can be a major performance killer, and in instances, an application killer. If I had a dollar for every time anti-virus software turned out to be the cause of a problem with Continua CI or FinalBuilder, well that would be some serious beer money at least!</p> <p>If you have anti-virus software installed on your server or agents, be sure to add exclusions from real-time scanning for the server's share folder, and the agent's workspace folder. Add scheduled scans on those folders instead. Also, when using the bundled PostgreSQL database, add an exclusion for C:\ProgramData\VSoft\ContinuaCI\PostgreSQLDB &nbsp;- otherwise you may experience database corruption.</p> <p>You should also consider adding an exclusions for the hg.exe in the "C:\Program Files\VSoft Technologies\ContinuaCI Agent\hg" folder. We found in testing that this will speed up the processing of the repostiory rules substantially (testing with windows defender). </p> <h3>Version Control Clients</h3> <p>Avoid installing tools like TortoiseSVN or ToirtoiseHG on your server or agent machines as these programs do background indexing (for icon overlays) and can also cause file/folder access issues.</p> <h2>Wrapping Up</h2> <p>I intend to revise this post as I learn more about performance tuning, especially in a virtual environment. If you have any techniques or tweaks that helped speed up your CI Server please feel free to share them with us (and fellow users).</p>754Hybrid Version Controlhttps://www.finalbuilder.com/resources/blogs/postid/689/hybrid-version-controlMercurialTue, 04 May 2010 15:37:00 GMT<div style="margin: 0px; background-image: initial; background-attachment: initial; background-color: #ffffff; background-origin: initial; background-clip: initial;"> <div>Distributed verison control systems are gaining in usage and popularity, but many organisations still use traditional centralised VCSs like Subversion and Visual Source Safe. Recently I've been using a hybrid setup and getting many of the benefits of a DVCS without needing to move the whole team to a new VCS platform.</div> <div> </div> <div>When I started with VSoft a few months back my first chunk of work was to create Mercurial actions for the upcoming FinalBuilder 7. It was the first time I'd used a DVCS and after the initial shock I became quite fond of it.</div> <div> </div> <div>Internally we use a CVCS - Surround SCM. If you've never used Surround you can think of it as VSS done right. It uses similar concepts and abstractions but without all the pain and frustration. We're considering moving to a DVCS but haven't yet worked out if the increased flexibility is worth the extra overhead for our relatively small dev team.</div> <div> </div> <div>After getting to know Mercurial though, I knew what I was missing out on. Primarily for me that was version control of my local changes. This is becoming more important as FinalBulder 7 gets closer to completion and breaking the build becomes a bigger deal. I also like being able to easily clone and sync my repositories locally as a basic backup strategy.</div> <div> </div> <div>The setup I've come up with couldn't be simpler. If I'm going to work on a VS.NET solution I will:</div> <ul> <li style="list-style-type: square;">check out the solution from Surround</li> <li style="list-style-type: square;"><span style="font-family: 'Courier New';">hg init</span> the solution directory to create a repository</li> <li style="list-style-type: square;"><span style="font-family: 'Courier New';">hg commit -A</span> which adds/removes all file changes to the repository</li> <li style="list-style-type: square;">work locally, committing whenever I feel like it</li> <li style="list-style-type: square;">when the solution is in a state where it won't break the build, check it in to Surround</li> </ul> <div> </div> <div>To make things easier I've set up this alias in my Mercurial.ini:</div> <div style="margin-left: 40px;"><span style="font-family: 'Courier New';">[alias]</span></div> <div style="margin-left: 40px;"><span style="font-family: 'Courier New';">cam = commit -A -m</span></div> <div style="margin-left: 40px;"> </div> <div>So to commit I just <span style="font-family: 'Courier New';">hg cam "commit message"</span>.</div> <div> </div> <div>Recently I started checking my .hg directory into Surround as well. That allows me to maintain the history of all my local commits, without cluttering up the Surround check-in logs.</div> <div> </div> <h3 style="font-family: Tahoma, Arial, Helvetica; font-size: 12px; font-weight: normal; color: #003366;"><strong>Caveats</strong></h3> <div>There's some things to be aware of when using a hyrbid system like this. While you could still use Mercurial for merging work from different developers it is much more complex than in a pure DVCS setup. I'll leave it to you to work out the details: it's not something I plan on ever doing. </div> <div> </div> <div>If you're checking your .hg folder in to your CVCS you need to be careful that it doesn't become corrupted through concurrent updates/merging. Of course if it does become corrupted you don't lose much by deleting and recreating it, because the check-in history for your major changes is in the CVCS's check-in logs.</div> <div> </div> <h3 style="font-family: Tahoma, Arial, Helvetica; font-size: 12px; font-weight: normal; color: #003366;"><strong>Other uses for Mercurial</strong></h3> <div>Because Mercurial repositories are so easy to create I've started using them for all sorts of things. For example, at home I have a perl script that runs nightly and exports the contents of my Wordpress blogs to XML. Previously I included the date in the filename and ended up with (literally) hundreds of files in my backup directory. </div> <div> </div> <div>Now I've set up a Mercurial repository, removed the date component from the filename and have my backup script commit after it downloads the latest version. The directory is now a lot cleaner as well as being smaller, because Mercurial only stores the changes between each night's backup.</div> <div> </div> <h3 style="font-family: Tahoma, Arial, Helvetica; font-size: 12px; font-weight: normal; color: #003366;"><strong>Further reading</strong></h3> <div>While running a hybrid system doesn't give you all the advantages of a pure DVCS it is a major improvement over a plain CVCS. It also allows you and your team to get comfortable with the DVCS methodology before moving away from CVCS completely.</div> <div> </div> <div>For more information on Mercurial, see:</div> <ul> <li style="list-style-type: square;">Joel Spolsky's <a href="http://hginit.com/" style="text-decoration: none; color: #064f93; font-family: Verdana, Arial, sans-serif; font-weight: normal;">hg init</a></li> <li style="list-style-type: square;"><a href="http://hgbook.red-bean.com/" style="text-decoration: none; color: #064f93; font-family: Verdana, Arial, sans-serif; font-weight: normal;">Mercurial: The Defininitve Guide</a></li> <li style="list-style-type: square;"><a href="http://mercurial.selenic.com/wiki/QuickReferenceCardsAndCheatSheets" style="text-decoration: none; color: #064f93; font-family: Verdana, Arial, sans-serif; font-weight: normal;">Mercurial Cheat Sheets</a></li> </ul> </div> <p>  </p> 689Version Control Systemshttps://www.finalbuilder.com/resources/blogs/postid/483/version-control-systemsFinalBuilder,Git,Mercurial,Subversion,TFSWed, 18 May 2005 15:01:00 GMT<p>In our recent <a href="/finalbuilder">FinalBuilder</a> customer survey, one question we asked was “What Version Control System do you currently use, or plan to use in the next 12 months”.  The question allowed mulitple answers, so the bars aren't a percentage.  I've removed the actual numbers because the relative widths of the bars is what is important.</p> <p>So, here are the results:</p> <p><img src="/blogimages/version_control_systems_finalBuilder.png" /></p> <p>So, interesting results, eh?  Obviously Microsoft Team System is high on the agenda for us, and looks like a lot of people will be taking a serious look at the new verison control system.</p> <p>We'd expect the Subversion bar to continue to grow at a very high rate - last year when we ran a similar survey only 3% of people were using Subversion. SourceGear Vault has also grown a lot, Vault was down the bottom last year, and this year it's really moved up the rankings to just behind CVS.  Most of the other VCS's have been pretty steady.</p> <p>Comments welcome.</p> 483