VSoft Technologies BlogsVSoft Technologies Blogs - posts about our products and software development.https://www.finalbuilder.com/resources/blogsContinuous Integration Server performancehttps://www.finalbuilder.com/resources/blogs/postid/754/continuous-integration-server-performance.NET,Continua CI,Delphi,FinalBuilder,Git,Mercurial,Web DevelopmentMon, 11 Sep 2017 14:31:58 GMT<p>Continuous Integration Servers are often underspecified when it comes to hardware. In the early days of Automated Builds, the build server was quite often that old pc in the corner of the office, or an old server in the data center that no one else wanted. Developers weren't doing many builds per day, so it worked, it was probably slow but that didn't seem to matter much.</p> <p>Fast forward 20 years, and the Continuous Integration Server is now a critical service. The volume and frequency of builds has increased dramatically and a slow CI server can be a real problem in an environment where we want fast feedback on that code we just committed (even though it "worked on my machine"). Continuous Deployment only adds to the workload of the CI server.&nbsp; </p> <p>In this post, I'm going to cover off some ideas to hopefully improve the performance of your CI server. I'm not going to cover compilation, unit tests etc. (which can be where a lot of the time is spent). Instead, I'll focus on the environment, machine configuration and some settings on your Continua CI configurations.</p> <h2>Hardware Requirements</h2> <p>It's impossible to provide hard and fast specs for hardware or virtual machines, as it varies greatly depending on the expected load.</p> <p>There are a bunch of things you can tweak that may improve performance. I will touch on some key points for virtual hosts, but I'm not going to go too deep into tuning virtual hosts, that's not my area of expertise. Of course, dedicated physical machines would be the ideal, but these days, even if you do get dedicated hardware for CI/CD, it's most likely going to be as a virtual host (hyper-v or vmware) rather than an OS installed on bare metal (do companies still provision a single os on bare metal servers these days?). Virtualisation brings in a whole bunch of benefits, but it also brings with it some limitations that cannot be ignored.</p> <p>Continuous Integration environments are mostly I/O bound and Continua CI is no different in that regard. So let's look at the various resources used by CI/CD.</p> <h3>CPU</h3> <p>It's unlikely that CPU will be a limiting factor in the performance of your CI server, unless you are running other CPU intensive tasks on your server. If that's the case, then move your CI server to dedicated hardware, or at least a dedicated virtual host. </p> <p>At a minimum you should have at least 2 cores on the server. On our production server, which is a virtual machine (on Hyper-V 2012R2) with 4 virtual cores and dynamic RAM, the Windows resource monitor shows that average CPU usage usually sits around 2% when idle (no running builds, measured on the guest OS using resource monitor on the Continua Server Service). With 10 concurrent builds running, the Continua CI server service was using around 6% cpu.</p> <p>Adding another 4 cores made very little difference. The Hyper-V host machine, which is also running a bunch of agent VM's, has plenty of CPU capacity, with the average CPU usage round 5-7%. Cutting down the number of cores to 2 did make a slight difference, with the VM showing slightly higher CPU usage, however no discernible difference in build times.</p> <p> This is obviously not very scientific, but it did demonstrate (well to me at least) that CPU is not the limiting factor. I set the server VM back to 4 cores and left it at that. Our Hyper-V host machines are a few years old now, and have 7200 rpm SAS hard drives (in Raid 10) rather than SSD's (they were still too expensive when we bought the machines).</p> <p>On a Continua CI Agent, we recommend at least 2 cpu cores, and limit the concurrent builds running on the agent to 1 per core. This isn't a hard and fast rule, just a convention we adhere to here (based on some performance testing). You may want to add extra cores depending on what compilers or tools you are running during your build process. The only way to know if this is needed is to monitor cpu on the agent machine while a build is running.</p> <h3>I/O</h3> <p>The most used resources are disk read/write and network read/write. Poor I/O performance will really slow down your builds.</p> <h4>Disk</h4> <p>It goes without saying, but use the fastest disks you have available to you. If you can afford it, new generation nvme/pcie SSD's are the way to go. They are still quite expensive for larger capacities though. At the very least, use a separate disk for the operating system and software installation, and another disk for your Continua CI Server's share folder (or the agents workspace folder on agent machines). This is where most of the I/O happens during builds. This recommendation applies whether running on dedicated hardware or in a virtual machine.</p> <p> If you are running the server and agent machines on the same virtual host (as we do for our production environment) then this is very important to get right. Poor I/O performance in virtualised environments is not uncommon - having agents and the server fighting for a slice of the same I/O pie is not a good idea.<br /> <br /> On the agent machines, good disk performance is critical. When a build is started on the agent, the first thing it does is create a workspace folder. It then exports the source code from the repository cache(s) (Mercurial repo which was cloned from the server) to that folder, using the repository rules (more on this later). This workspace initialisation phase can be very slow if you have poor I/O performance.</p> <h4>Network</h4> <p>Continua CI uses networking to transfer files, repository changes etc between the server and the agents. Poor network performance will impact on build initialisation times (updating the agents repo cache, build workspace) and on build completion times (transferring workspace changes back to the server). Logging between the agent and the server will also be impacted by poor network performance.</p> <p>By default, Continua CI uses SMB to transfer files, source code (repository caches) between the server and the agents. When the server's share folder is not accessible by SMB, Continua CI will try to use SSH/SFTP (Continua CI installs it own specialised SSH service). In high latency networks (for example if the agent is remote from the server), SSH/SFTP may perform better than SMB.</p> <p>You can force an agent to use SSH/SFTP by setting the agents ServerFileTransport.ForceSSH property to true.</p> <h3>Database</h3> <p>Continua CI supports PostgreSQL (the default) or Microsoft SQL Server. If you chose to use MSSQL, we recommend running it on a separate well specified machine. MSSQL is quite heavy in it's use of RAM and disk I/O - it's best run on a machine that has been tuned to run it properly. I'm not going to go into that here, that's a whole other topic on an area that I'm definitely not an expert.</p> <p>The PostgreSQL database server that is installed by default (unless you select otherwise) with Continua CI is much more more frugal when it comes to resources. On our main Continua CI server, PostgreSQL typically using around 60MB of ram. Contrast that with SQL Server running on my dev machine, not used or touched for weeks and it's using 800MB! PostgreSQL can also be tuned, we have tried to provision it with sensible defaults that strike a balance between performance and resource usage. If you need to tune PostgreSQL, then we recommend installing your own PostgreSQL instance and pointing Continua CI at it.</p> <p>Currently the Continua CI installer doesn't provide any options for the database install location (C:\ProgramData\VSoft\ContinuaCI\PostgreSQLDB ), this is something we are looking at for a future release, that will make it possible to put the database on it's own drive. For now, it's possible to move the database to another location by using a symlink, we have a few customers who have done this successfully. Contact support if you need help with this.</p> <h2>Virtualisation Tips</h2> <h3>Virtual CPU Cores</h3> <p>In a virtual environment, it's very important not to overload your virtual host. Note that there is a difference between overloading and over allocating virtual cores. It's a common practice to allocate more virtual cores across the virtual machines than there are physical/logical cores (logical when HyperThreading is enabled), but this has to be done with the knowledge and understanding of the load on the host machine. Overloading happens when so many cores are allocated and in use that the hypervisor is unable to schedule a core to a virtual machine when needed. This results in pauses and poor performance.</p> <p>In a clustered environment this is even more important, because when a cluster node dies, or is removed for upgrades etc, virtual machines will move to another node in the cluster - if that node is already overloaded then you will soon start hearing the complaints from users!</p> <p>The best explanation I have found on how hypervisors allocate cores is this article - <a href="https://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/">https://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/</a> - it's Hyper-V specific (we use Hyper-V here) but much of the information also applies to VMWare.</p> <h3>Virtual Disks</h3> <p>When creating separate virtual disk volumes for your virtual machines, try to put those virtual drives on different physical drives, so they are not competing for the same I/O. Use fixed size virtual disks.</p> <h2>Continua CI Configuration Tuning</h2> <p>Continua CI is not immune to performance problems, we're always working to make it faster and consume less resources. There are however a few things that can be tuned in Continua CI to improve performance.</p> <h3>Repository Branch Settings</h3> <p>Use specific branch patterns to narrow down the number of repository files and folders which are monitored and downloaded. With repositories which use folder-based branches, such as Subversion and TFS, consider moving old branches to a separate archive folder in your repository which will not match the branch patterns. Note that you can use more than one Continua CI repository per actual repository. Some users will have multiple projects in one repository, but only need to build a single one for each configuration. Make use of relative paths, where supported by your repository type, to limit your repository to a single project folder. This can significantly speed up repository initialisation and changeset updating. </p> <h3>Repository Polling</h3> <p>Continua CI polls repositories periodically to detect new commits. Each time this occurs, Continua CI invokes the command line client for that repo, and parses the output of that process. Some clients use a surprising amount of CPU. The git client, for example, uses around 8% CPU per instance on our production server while checking for commits. Most of the time, these processes only run for a very short amount of time (when no changes are detected), however if you have a lot of repositories, these small cpu spikes can add up.</p> <p> There are a couple of options to keep this under control.</p> <p> 1) Set the appropriate polling interval for your repositories. If changes to a repository occur infrequently, then there's no point polling frequently.</p> <p>2) Set the Server property Server.RepoMonitor.MaxCheckers property. This controls how many version control client processes are spawned concurrently, the default (5) is quite conservative so you should only need to lower this on a very low spec system. If you have plenty of spare CPU capacity, then you can increase this value, however if you do then monitor CPU usage to make sure you don't overload the server.</p> <p>3) Manual polling, using post commit hooks. This reduces CPU usage on the server, by only polling for repository changes when requested and has the added benefit of reducing the load on your version control server. This does take some setting up, and depends very much on the capabilities of your version control system. I'll take a look at post commit hooks in a future blog post.</p> <h3>Repository Path Filtering</h3> <p>Repository Path Filtering is an option on all repository types, with the exception of Mercurial (*I'll explain why shortly). What this filtering does is allow you to limit which files get added to the server's repository cache. This filtering has a few benefits, less disk space used on the server and the agents, less network I/O when transferring the changes from the server to the agent, and less I/O when checking out the source into the build workspace.</p> <p>A typical use case for these rules is when you have files in your repository that rarely change and are not needed for the build process (design docs, deployment notes etc). No point adding them to the repo cache if you don't use them.</p> <p>Changes to these rules won't affect files that are already in the repository cache, but it will avoid committing changes to those filtered out files to the repo cache. The best bang for buck with these filters will come if the repository is reset (the cache is rebuilt, so filtered out files are never committed to the cache), however that can be an expensive operation, so unlike other repository settings, changing these rules will not force a reset.</p> <p>* These filters don't apply to Mercurial repositories, as we use Mercurial for our repository cache. When you point Continua CI at a Mercurial repository, it just clones it to the server (repo cache), and then clones it to the agents (repo cache) without any modifcations.</p> <h3>Repository Rules</h3> <p>Each Stage has a settings tab called Repository Rules. These rules apply when checking out the source from the agent's repository cache(s) to the build workspace. Only check out the source you need, this will improve performance. If a stage doesn't need the source at all (for example, it's only working with artifacts from previous stages), then just blank out the Repository Rules field.</p> <p>Don't leave logging of the repository rules turned on unless you are debugging the rules. Logging the files exported to the workspace can be a real performance killer.</p> <h3>Workspace Rules</h3> <p>Similar to Repository Rules, these rules control which files are transferred between the server and agent's build workspace folders, and back again. Only transfer files back to the server's workspace that you actually need, like build artifacts, reports etc. </p> <p>Don't leave logging of the workspace rules turned on unless you are debugging the rules. Logging the files transferred can be a real performance killer.</p> <h3>Actions</h3> <p>Avoid logging too much information. For example, verbose logging on MSBuild should be avoided unless debugging build issues. Output logged from actions is queued and sent back to the server to be written to the build log, this causes high network and disk I/O.</p> <h3>Disk Space</h3> <p>Disk space is quite often at a premium (especially with SSD's), and it's important to keep on top of it. This is where the Clean up Policies come into play. Continua CI allows you to specify a global clean up policy for both the server and the agents, however it can be overridden at the Project or Configuration level. The clean up policy controls how long to keep old builds and their associated workspaces around. The clean up policy is highly configurable - use it to keep control over disk space. Bear in mind that the work of cleaning up old builds is quite I/O and database intensive, so be sure to schedule it to run during a quite period</p> <h3>Anti-virus Software</h3> <p>Anti-virus software can be a major performance killer, and in instances, an application killer. If I had a dollar for every time anti-virus software turned out to be the cause of a problem with Continua CI or FinalBuilder, well that would be some serious beer money at least!</p> <p>If you have anti-virus software installed on your server or agents, be sure to add exclusions from real-time scanning for the server's share folder, and the agent's workspace folder. Add scheduled scans on those folders instead. Also, when using the bundled PostgreSQL database, add an exclusion for C:\ProgramData\VSoft\ContinuaCI\PostgreSQLDB &nbsp;- otherwise you may experience database corruption.</p> <p>You should also consider adding an exclusions for the hg.exe in the "C:\Program Files\VSoft Technologies\ContinuaCI Agent\hg" folder. We found in testing that this will speed up the processing of the repostiory rules substantially (testing with windows defender). </p> <h3>Version Control Clients</h3> <p>Avoid installing tools like TortoiseSVN or ToirtoiseHG on your server or agent machines as these programs do background indexing (for icon overlays) and can also cause file/folder access issues.</p> <h2>Wrapping Up</h2> <p>I intend to revise this post as I learn more about performance tuning, especially in a virtual environment. If you have any techniques or tweaks that helped speed up your CI Server please feel free to share them with us (and fellow users).</p>754Delphi-Mocks Parameter Matchershttps://www.finalbuilder.com/resources/blogs/postid/737/delphi-mocks-parameter-matchersDelphi,GeneralGit,Open SourceTue, 22 Sep 2015 10:15:11 GMT<p> We recently updated Delphi Mocks to allow for better parameter matching on Expectations registered with the Mock. This allows the developer to place tighter controls on verifying that a mocked interface/object method is called. Below is a simple example of when the parameter matchers can be used. </p> <pre class="brush:delphi; toolbar:true; highlight:[14,15,16,17]">procedure TExample_InterfaceImplementTests.Implement_Multiple_Interfaces; var sutProjectSaver : IProjectSaveCheck; mockProject : TMock&lt;IProject&gt;; begin //Test that when we check and save a project, and its dirty, we save. //CREATE - The project saver under test. sutProjectSaver := TProjectSaveCheck.Create; //CREATE - Mock project to control our testing. mockProject := TMock&lt;IProject&gt;.Create; //SETUP - Mock project will show as dirty and will expect to be saved. mockProject.Setup.WillReturn(true).When.IsDirty; //NEW! - Add expectation that the save will be called as dirty is returning true. // As we don't care about the filename value passed to us we // allow any string to be passed to report this expectation as met. mockProject.Setup.Expect.Once.When.Save(It(0).IsAny&lt;string&gt;()); //TEST - Visit the mock element to see if our test works. sutProjectSaver.Execute(mockProject); //VERIFY - Make sure that save was indeed called. mockProject.VerifyAll; end; </pre> <p> Previously the developer writing this test would have to provide the exact filename to be passed to the mocked Save method. As we don't know what the projects filename is going to be (in our example case), we would either have to; 1. Forgo doing this test. 2. Implement a project object to test with. Both of these options are not ideal. </p> <p>Parameter matchers resolve this situation. It is now simple to either restrict or broaden the parameters passed to mocked methods that will satisfy the expectation defined. To achieve this Delphi-Mocks offers eleven new functions; </p> <pre class="brush:delphi; toolbar:true;">function It(const AParamIndx : Integer) : ItRec; function It0 : ItRec; function It1 : ItRec; function It2 : ItRec; function It3 : ItRec; function It4 : ItRec; function It5 : ItRec; function It6 : ItRec; function It7 : ItRec; function It8 : ItRec; function It9 : ItRec; </pre> <p>The first "function It(const AParamIndx : Integer) : ItRec;" allows the developer to specify the index of the parameter they wish to set for the next expectation setup of a mock method. It(0) will refer to the first parameter, It(1) the second and so forth. Note that the reason for specifying the parameter index is that Delphi's parameter evaluation order is not defined, so we could not rely on the parameters being evaluated in order (which is what we did when we initially wrote this feature). Interestingly, with the 64 bit Delphi compiler, parameter evaluation does appear to happen in order, but we could not be certain this will always be the case.&nbsp;</p> <p>The other ten functions It0 through to It9 are simply wrappers of the index call passing the index in their name. All these functions return an ItRec. The ItRec has the function structure;</p> <pre class="brush:delphi; toolbar:true;">ItRec = record var ParamIndex : cardinal; constructor Create(const AParamIndex : Integer); function IsAny&lt;T&gt;() : T ; function Matches&lt;T&gt;(const predicate: TPredicate&lt;T&gt;) : T; function IsNotNil&lt;T&gt; : T; function IsEqualTo&lt;T&gt;(const value : T) : T; function IsInRange&lt;T&gt;(const fromValue : T; const toValue : T) : T; function IsIn&lt;T&gt;(const values : TArray&lt;T&gt;) : T; overload; function IsIn&lt;T&gt;(const values : IEnumerable&lt;T&gt;) : T; overload; function IsNotIn&lt;T&gt;(const values : TArray&lt;T&gt;) : T; overload; function IsNotIn&lt;T&gt;(const values : IEnumerable&lt;T&gt;) : T; overload; {$IFDEF SUPPORTS_REGEX} //XE2 or later function IsRegex(const regex : string; const options : TRegExOptions = []) : string; {$ENDIF} end; </pre> <p>Each of the functions creates a different matcher. For example the IsAny&lt;T&gt; will cause the expectation to be met when the parameter passed to the mock is of any value that has the type T. In the example above this type would be a string. You will also notice that each function returns the type T. This is so that each call can be placed within the mock methods call directly. Doing so helps with making sure parameter types match the testing value.</p> <p>IsEqualTo&lt;T&gt; requires that the parameter matches exactly to the value passed into the IsEqualTo&lt;T&gt;. This could be used to restrict the expectation to a tighter test of the functionality under test.</p> <pre class="brush:delphi; toolbar:true;">//Match on the filename being "temp.txt" only. mockProject.Setup.Expect.Once.When.Save(It(0).IsEqualTo&lt;string&gt;('temp.txt')); //VERIFY - Make sure that save was indeed called. mockProject.VerifyAll; </pre> <p>In the future we are looking to provide &ldquo;And&rdquo;\&rdquo;Or&rdquo; operators. These operators might also live on the ItRec and allow combining with as many other matchers using the same type.</p> <pre class="brush:delphi; toolbar:true;">//Match on the filename being "temp.txt" or "temp.doc" only. mockProject.Setup.Expect.Once.When.Save( It(0).Or(It(0).IsEqualTo&lt;string&gt;('temp.txt'), It(0).IsEqualTo&lt;string&gt;('temp.doc')); //VERIFY - Make sure that save was indeed called. mockProject.VerifyAll; </pre> <p>There might be a better way to make the resulting code a bit cleaner. It would make the tests easier to read, instead of using regex which is also possible in this case. As a result we believe this would be a good edition to the library.</p> <p><a href="https://github.com/VSoftTechnologies/Delphi-Mocks">Feel free to clone the repository from GitHub</a>. If you have some time to spare submit a pull requests or two with your ideas/improvements. We believe this is a great little project worthy of some attention. Let us know what you think of the changes so far.</p>737TFS/VSO Task for Running FinalBuilder Projecthttps://www.finalbuilder.com/resources/blogs/postid/736/finalbuilder-vso-task.NET,DelphiFinalBuilder,Git,TFSWed, 16 Sep 2015 16:33:00 GMTToday we are announcing the new build step for Team Foundation Build 2015. This task will allow users of TFS on-prem and VSO to run FinalBuilder projects on Team Foundation Build agents. The task itself is open source and can be found on <a href="https://github.com/VSoftTechnologies/FinalBuilder-VSO">GitHub</a>.<br /> <br /> Those who use TFS on-prem will be very familiar with our XAML build activity already. This activity took a great deal of confusion out of the XAML build process. Changing a build progress from a complex workflow into a simple to maintain FinalBuilder project. The time and effort saved is huge, especially considering the "default" XAML workflow looks like this:<br /> <br /> <img alt="" src="https://www.finalbuilder.com/blogImages/jason/FinalBuilder-VSO/OldXAMLProcess.png" style="vertical-align: middle; text-align: center; width: 441px; height: 800px;" /><br /> <br /> Thankfully Microsoft have improved on their build system with the release of Team Foundation Build 2015. You can read more about this at <a href="https://msdn.microsoft.com/Library/vs/alm/Build/feature-overview">Team Foundation Build 2015</a>. In summary, the new build system greatly simplifies the build process into a list of tasks to perform. <br /> <br /> The FinalBuilder task is our custom task for TFS Build 2015. It offers TFS script builders the ability to still have a simplified overview of their build process while still gaining the power of FinalBuilder and all its supported actions. With FinalBuilder TFS build script creators are able to perform a wide number of tasks that would otherwise require breaking out powershell and diving into the TFS agent environment variables. <br /> <br /> When installed, the FinalBuilder Task gives users the following UI. All of the properties present in the UI are easily accessible from within any FinalBuilder script run by the task. FinalBuilder also gives simple access to a list of files that triggered the build.<br /> <br /> <img alt="" src="https://www.finalbuilder.com/blogImages/jason/FinalBuilder-VSO/FinalBuilderTaskOptionsAll.png" style="vertical-align: middle; text-align: center; width: 518px; height: 300px;" /> <br /> <h2>Installation and Usage</h2> The steps to adding custom build activities to your TFS and VSO instances are quick and easy. We have created a <a href="https://github.com/VSoftTechnologies/FinalBuilder-VSO">GitHub Repository</a> for explaining how to install, and use our FinalBuilder VSO task. <h2>Repository Clone</h2> To clone this repository use the following command line. You will require git to be installed and available on your path.<br /> <pre class="brush:shell; gutter:false; highlight:3">> mkdir VSoft > cd VSoft > git clone https://github.com/VSoftTechnologies/FinalBuilder-VSO.git Cloning into 'FinalBuilder-VSO'... remote: Counting objects: X, done. remote: Compressing objects: 100% (X/X), done. remote: Total X (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (X/X), done. Checking connectivity... done. </pre> With the repository cloned we require the TFS Extensions Command Line Utility (tfx-cli). It comes as a Node Package Manager (npm) package. Npm comes with both the node.js and io.js installer. Download the installer for your Windows platform and run it.<br /> <br /> To check that NPM is working correctly you can use the npm version command<br /> <pre class="brush:shell; gutter:false; highlight:1">> npm -v 2.10.1 </pre> Now your able to install the tfx-cli package using npm. Install this globally so that its accessable on the command line. The command line for this is as follows;<br /> <pre class="brush:shell; gutter:false; highlight:1">> npm install -g tfx-cli tfx-cli@0.1.11 C:\Users\<username>\AppData\Roaming\npm\node_modules\tfx-cli ├── os-homedir@1.0.1 ├── async@1.4.2 ├── colors@1.1.2 ├── minimist@1.2.0 ├── node-uuid@1.4.3 ├── q@1.4.1 ├── read@1.0.7 (mute-stream@0.0.5) ├── validator@3.43.0 ├── shelljs@0.5.3 ├── vso-node-api@0.3.4 └── archiver@0.14.4 (buffer-crc32@0.2.5, lazystream@0.1.0, async@0.9.2, readable-stream@1.0.33, tar-stream@1.1.5, glob@4.3.5, lodash@3.2.0, zip-stream@0.5.2) </pre> To test that tfx-cli is working correctly and is on the path use the tfx command.<br /> <pre class="brush:shell; gutter:false; highlight:1">> tfx Copyright Microsoft Corporation tfx <command> [<subcommand(s)> ...] [<args>] [--version] [--help] [--json]                         fTfs                       fSSSSSSSs                     fSSSSSSSSSS      TSSf         fSSSSSSSSSSSS      SSSSSF     fSSSSSSST SSSSS      SSfSSSSSsfSSSSSSSt   SSSSS      SS  tSSSSSSSSSs      SSSSS      SS   fSSSSSSST       SSSSS      SS fSSSSSFSSSSSSf    SSSSS      SSSSSST    FSSSSSSFt SSSSS      SSSSt        FSSSSSSSSSSSS                     FSSSSSSSSSS                        FSSSSSSs                         FSFs    (TM) commands:    build         manage task extensions and builds    help         command help    login         login and cache credentials. types: pat (default), basic         login <collection url> [--authtype <authtype>] [options]    parse         parse json by piping json result from another tfx command         parse <jsonfilter> [options]    version         output the version         version [options] Options:    --help    : get help on a command    --json    : output in json format.  useful for scripting </pre> For tfx-cli to upload a task to TFS it needs to be logged in. We can do this once so that all following commands will use the some credentials. The method used depends on whether your using VSO or an On Prem installation.<br /> <h2>On Premises Login</h2> For on premises TFS basic authentication will need to be enabled. The tfx-cli project has a great guide on how to achieve this Using tfx against Team Foundation Server (TFS) 2015 using Basic Authentication.<br /> <br /> Once TFS has been configured to use basic authentication use the tfx-cli login command to connect to TFS. You will be prompted for the TFS collection URL to connect to, and the username and password for accessing that collection.<br /> <pre class="brush:shell; gutter:false; highlight:1">> tfx login --authType basic Copyright Microsoft Corporation Enter collection url > http://<server>:<port>/tfs/<collection> Enter username > <user>@<domain> Enter password > <password> logged in successfully </pre> With a successful login subsequent commands will not require us to provide the credentials again.<br /> <h2>Visual Studio Online (VSO) Login</h2> For VSO login you need a personal access token setup under your account. There is a great article to configure an access token located at Using Personal Access Tokens to access Visual Studio Online.<br /> <br /> With the personal access token configured use the tfx-cli login command to connect to VSO. You will be prompted for the TFS collection URL to connect to, and access token for accessing that collection.<br /> <pre class="brush:shell; gutter:false; highlight:1">> tfx login Copyright Microsoft Corporation Enter collection url > https://<vsoname>.visualstudio.com/<collection> Enter personal access token > <access token> logged in successfully </pre> With a successful login subsequent commands will not require us to provide the credentials again.<br /> <h2>Uploading Task</h2> Once logged into TFS we are able to upload the FinalBuilder task to the server. Tasks are uploaded to the server, the server will then pass them onto agents requried to run those tasks.<br /> <br /> To upload the task use the tfx-cli tasks upload command. Each command shown below is a sub-command of the previous, so order does matter here. The overwrite option is included so that any previously installed version is overwritten. Note however the highest version number of the task will win when running builds.<br /> <br /> Note: This command is run under the directory in which this repositry was cloned to (i.e. FinalBuilderTFS).<br /> <pre class="brush:shell; gutter:false; highlight:1">> tfx build tasks upload ./FinalBuilder --overwrite Copyright Microsoft Corporation<br /> task at: ./FinalBuilder uploaded successfully! </pre> To test that the FinalBuilder task is now installed on the builds page for teh collection the task was uploaded to. Create a new empty Team Foundation Build definition. After clicking "Add build step" a FinalBuilder task should appear in the "Build" category.<br /> <h2>Further Steps</h2> For more information on the following subjects please follow the links;<br /> <br /> How to configure the build task refer to <a href="https://github.com/VSoftTechnologies/FinalBuilder-VSO/blob/master/docs/TaskUI.md">Task UI</a>.<br /> How to install FinalBuilder on an agent refer to <a href="https://github.com/VSoftTechnologies/FinalBuilder-VSO/blob/master/docs/InstallingFinalBuilder.md">Installing FinalBuilder</a>.<br /> How to create a FinalBuilder VSO agent refer to Creating a <a href="https://github.com/VSoftTechnologies/FinalBuilder-VSO/blob/master/docs/FinalBuilderVSOAgent.md">VSO FinalBuilder Agent</a>.736Team Foundation Server XAML Buildshttps://www.finalbuilder.com/resources/blogs/postid/735/team-foundation-server-xaml-buildsFinalBuilder,Git,TFSTue, 15 Sep 2015 16:16:45 GMT<h2>Team Foundation Server XAML Builds</h2> <p>Today we have released an update for Team Foundation Server XAML activities for FinalBuilder 7 and 8. These updates are to deal with conflicts caused by GAC installing these activities.</p> <p>Since 2010 Team Foundation Server has allowed custom activities to allow provide extra functionality in XAML build workflows. For those that have not had the (dis)pleasure, XAML workflows can be difficult to work with. To alleviate this pain point we implemented XAML build templates and custom activities to allow people to run any FinalBuilder script within the build workflow without the need to edit XAMAL. Customers we have talked to say this simplifies their TFS XAML build workflows a great deal.</p> <p>In previous releases, FinalBuilder’s installer automatically installed the TFS XAML activity into the GAC. The version of the activity installed was based on the TFS Agent detected on the machine in question. Installing into the GAC was done to simplify the process. This way the developer would simply use the activities in their build workflow and it would be picked up through the GAC.</p> <p>With the introduction of the new TFS build in TFS 2015 and TFS-Git in TFS 2013 this is no longer advisable. In the case of TFS-Git Workflows, assembly conflicts can cause assembly load issues.  If the activity requires a different assembly version to those already loaded by the TFS agent, you would see assembly load errors with lib2gitsharp. With TFS 2013 having five updates this is increasingly possible .</p> <p>The way to avoid any assembly loading issue for custom activities is to use “version control paths for custom XAML activities”. To be clear, we have left the GAC installation option in both FinalBuilder 7 and 8. We however do recommend switching to using custom XAML activity paths, especially if you’re using TFS-Git.</p> <p>To this end, the FinalBuilder installers now give the option as to whether to install the XAML build activities into the GAC or not. Only FinalBuilder 7 will automatically install TFS activities into the GAC for TFS 2012 and earlier agents, with FinalBuilder 8 you much chose whether to GAC install the assemblies.</p> <p>Note that if activities were previously installed in the GAC restarting the TFS Build Controller is required. This refreshes the build controller and releases any assemblies that it may have previously loaded.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/FinalBuilder-XAML-Setup.png" style="width: 387px; height: 300px; vertical-align: middle; text-align: center;" /></p> <h2>Creating a build definition</h2> <p>The creation of the build definition is exactly the same as before. If you’re interested in how to setup a build definition from scratch using the FinalBuilder XAML templates please review the “<a href="/resources/blogs/finalbuilder-and-team-foundation-server-2013">FinalBuilder and Team Foundation Server</a>” article.</p> <h2>Custom XAML activities</h2> <p>Version control paths for custom XAML activities is a feature in TFS XAML build controllers. This feature allows the build controller to source all assemblies required for an activity from a known location. If a required assembly is missing from this location the standard .Net assembly lookup methodology is used.</p> <p>Using version control paths for custom XAML activities requires the activity assemblies to be added to a repository on the TFS system. This repository can be shared with other code, or can be a repository just for the assemblies. As the custom folder does not change based on the build being performed, we suggest a separate TFS repository for the custom activities. The repository can also be either a TFS or Git-TFS repository. Both will work the same.</p> <p>To add the custom activity assemblies to a repository connect to the repository through team explorer in visual studio.</p> <p>In the source control view create a new folder that will hold the custom activity assemblies.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/CustomActivitiesInRepo.png" style="text-align: center;" /></p> <p>In the GAC sub-folder in your FinalBuilder installation (typically “%ProgramFiles(x86)%\FinalBuilder 8\GAC\”) there are folders for each version of TFS custom activities are provided for. From the folder relating the TFS version copy all assemblies contained within into the newly created repository folder.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/FinalBuilderGACFolder.png" style="text-align: center;" /></p> <p>From the source control explorer add these files to source control.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/CustomActivitiesFolderWithAssemblies.png" style="text-align: center;" /></p> <h2>Updating the XAML Build Controller</h2> <p>Next the build controller needs to be configured use the repository location. In the builds tab of team explorer click on the “Actions” link. A drop down will appear with the option to “Manage Build Controllers…”. Click the “Manage Build Controllers…” menu item.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/ManageBuildControllers.png" style="text-align: center;" /></p> <p>From the build controllers window that opens select the build controller responsible for the FinalBuilder builds. If there is more than one controller simply follow these steps for each controller. Next click the “Properties” button.</p> <p>This will present the build properties dialogue in which the “Version control path to custom assemblies” can be set. Select the folder that was created in the repository that is responsible for the custom activity assemblies. Now confirm this change by clicking OK on the build controller properties dialogue.</p> <p style="text-align: center;"><img alt="" src="https://www.finalbuilder.com/blogImages/jason/TeamFoundationServerXAMLBuilds/VersionControlPathSet.png" style="width: 400px; height: 400px; text-align: center;" /></p> <p>This has now setup the build controller to source custom assemblies from the configured repository folder. The most recent checked in assemblies will always be sourced. Therefore keeping this repository folder in sync with the custom activities used in build workflows is very important.</p> <p>Once the custom path is set the build controller can run builds using the custom activities. To test simply queue a build using the FinalBuilder custom activities. </p> 735Version Control Systemshttps://www.finalbuilder.com/resources/blogs/postid/483/version-control-systemsFinalBuilder,Git,Mercurial,Subversion,TFSWed, 18 May 2005 15:01:00 GMT<p>In our recent <a href="/finalbuilder">FinalBuilder</a> customer survey, one question we asked was “What Version Control System do you currently use, or plan to use in the next 12 months”.  The question allowed mulitple answers, so the bars aren't a percentage.  I've removed the actual numbers because the relative widths of the bars is what is important.</p> <p>So, here are the results:</p> <p><img src="/blogimages/version_control_systems_finalBuilder.png" /></p> <p>So, interesting results, eh?  Obviously Microsoft Team System is high on the agenda for us, and looks like a lot of people will be taking a serious look at the new verison control system.</p> <p>We'd expect the Subversion bar to continue to grow at a very high rate - last year when we ran a similar survey only 3% of people were using Subversion. SourceGear Vault has also grown a lot, Vault was down the bottom last year, and this year it's really moved up the rankings to just behind CVS.  Most of the other VCS's have been pretty steady.</p> <p>Comments welcome.</p> 483