VSoft Technologies Blogs

rss

VSoft Technologies Blogs - posts about our products and software development.

Delphi/Rad Studio desperately needs a proper package/library/component manager. A package manager provides a standardized way of consuming third party libraries. At the moment, use of third party libraries is very much adhoc, and in many cases this makes it difficult to move projects between machines, or to get a new hire up and running quickly.

Other developement environments, like the .net and javascript eco systems, recognised and solved this problem many years ago. Getting a .net or javascript project up an running, in a new working folder or new machine is trivial.

With Delphi/Rad Studio, it's much harder than it should be. In consulting work, I've made it a point to see how clients were handling third party code, and every client had a different way. The most common technique was... well, best described as adhoc (with perhaps a readme with the list of third party products to install). Getting that code compiling on a CI server was a nightmare.

Existing Package Managers

Embarcadero introduced their GetIt package manager with XE8, and the GetIt infrastructure has certainly has made the installation of RAD Studio itself a lot nicer. But as a package manager for third party libraries, it comes up short in a number of areas.

There is also Delphinus, which is an admirable effort, but hasn't gotten much traction, possibly due to it being strongly tied to github (you really need github account to use it, otherwise you get api rate limiting errors).

Rather than pick apart GetIt or Delphinus, I'd like to outline my ideas for a Delphi package manager. I spend a lot of time working with .net (nuget) and javascript (npm, yarn), so they have very much influenced what I will layout below.

I have resurrected an old project (from 2013) that I shelved when GetIt was announced, and I have spent a good deal of time thinking about package management (not just in Delphi), but I'm sure I haven't thought of everything, I'd love to hear feedback from people interested in contributing to this project, or just potential users.

Project Ideals

These are just some notes that I wrote up when I first started working on this back in 2013, I've tried to whip them into some semblance of order for presentation here, but they are just just rough outline of my ideas.

Open Source

The Project should be Open Source. Of course we should welcome contributions from commercial entities, but the direction of the project will be controlled by the community (ie users). The project will be hosted on GitHub, and contributions will be made through Pull Requests, with contributions being reviewed by the Steering committee (TBA).

Public Package Registry

There will be a public website/package server, where users can browse the available packages, and package authors can upload packages. This will be a second phase of the project, with the initial phase being focused on getting a working client/package architecture, with a local or network share folder for the package source.

The package registry should not be turned into a store. Once a public package registry/server is available, evaluation packages could be be allowed, perhaps by providing a fee (web hosting is not free). Commercial vendors will of course be able to distribute commercial packages directly to their customers, as the package manager will support hosting of packages in a shared network or local directory. Package meta data will include flags to indicate if the packages are commercial, eval or free/open source. Users will be able to decide which package types show up in their searches.

Package Submission

Package submission to the public registry should be a simple process, without filling in and signing and faxing of forms! We will follow the lead of nuget, npm, ruby etc on this. There should be a dispute process for package names, copyright infringement etc. There will also be the ability to assign ownership of a package, for example when project ownership changes.

Package Authors will be able to reserve a package prefix, in order to prevent other authors from infringing on their names or copyrights. For example, Embarcadero might reserve Emb. as their prefix, TMS might reserve TMS. as theirs. (of course I'm hoping to get both on board). The project will provide a dispute resolution process for package prefixes and names.

Delphi specific challenges

Delphi presents a number of challenges when compared to the .net or nodejs/javascript world.

Compatibility

With npm, packages contain source (typically minimized and obfuscated) which is pure javascript. Compatibility is very high.

With Nuget, packages contain compiled (to .NET IL) assemblies. A package might contain a few different versions, that target different the versions of the .net framework. Again, compatibility is pretty good, an assembly compiled against .net 2.0 will work on .net 4.7 (.net core breaks this, but it has a new compatibility model, netstandard).

If we look at Delphi, binary compatibility between Delphi compiler versions is pretty much non existent(yes, I know about 2006/7 etc). The dcu, dcp and bpl files are typically only compatible with the version they were compiled with. They are also only compatible with the platform they were generated for (so you can't share dcu's between 32 and 64 bit windows, or between iOS and Android). So we would need to include binaries for each version of Delphi we want our library to support. This also has major implications for library dependencies. Where as npm and nuget define dependencies as a range of versions, a binary dependency in Delphi would be fixed to that specific version. There is a way to maintain binary compatibility between releases, provided the interfaces do not change, however exactly what the rules are for this is hard to come by, so for now we'll ignore that possibility.

That limits the scope for updating to newer versions of libraries, but that can also be overcome by including the source code in package, and providing on the fly compilation of the library during install. My preference would be for pre-compiled libraries, as that speeds up the build process (of course, since that's an area I have a particular interest in). In Continuous Integration environments, you want to build fast and build often, rebuilding library code with each CI build would be painful (speaking from experience here, 50% of time building FinalBuilder is building the third party libraries).

There's also the consideration of Debug vs Release - so if we are including binaries, compiled for Release would be required, but Debug optional? The size of a package file could be problematic. If the package contains pre-compiled binaries for multiple compiler versions, it could get rather large. So perhaps allow for packages that either support a single compiler version, or multiples? The compilers supported would be exposed in the package metadata, and perhaps also in the package file name. Feedback, ideas around this would be welcome.

Package files would be (like with other package managers), a simple zip file, which include a metadata (xml) file which describes the contents of the package, and folders containing binaries, source, resources etc. Packages will not contain any scripts (ie to build during install) for security reasons (I don't want to be running random scripts). We will need to provide a way to compile during install (using a simple dsl to describe what needs to be done), this still needs a lot of thought (and very much involves dependencies).

Library/Search Paths

Say goodbye to the IDE's Library path. It was great back in 1995, when we had a few third party libraries and a few projects and we just upgraded the projects to deal with library versioning (just get on the latest). It's simply incompatible with the notion of using multiple versions of the same libraries these days.

I rarely change major versions of a library during the lifespan of a major release of my products, I might however take minor updates for bugfixes or performance improvements. The way to deal with this is simply to use the Project Search path. Project A can use version 1 of a library, Project 2 can use version 9, all quite safely (design time components do complicate this).

Where a project targets multiple platforms, installing a package should install for all platforms it supports, but it should be possible for the user to specify which platforms they need the package installed for.

Design time Component Installation

The Rad Studio IDE only allows one version of a design time package to be installed at a time. So when switching projects, which might use different versions of a component library, we would need a system that is aware of component versions, and can uninstall/install components on the fly, as projects are loaded.

I suspect this will be one of the biggest project hurdles to overcome, it will requires someone with very good open tools api knowledge (ie, not me!).

Dependencies

Libraries that depend on other libraries will need to specify those dependencies in a metadata file, such that they can resolved during installation. As I mentioned above, binary compatibility issues make dependency resolution somewhat more complicated, but not insurmountable. The resolution algorithm will need to take into account compiler version and platform. The algorithm will also need to handle when a package is compiled from source, for example, binary only packages should not be allowed to depend on source only packages (to ensure compatibility). If we end up with install time package compilation, then some serious work will be needed on the dependency tree algorithm to work our what else needs to be done during install (ie, do any dependencies need to be recompiled?).

This is certainly more complicated than other platforms, and a significant amount of work to get right (ps, if you think it isn't, you haven't considered all the angles!)

General Considerations

Package Install/Restore

The user should be able to choose from a list packages to install. When installing the package, this would be recorded either in the dproj, or a separate file alongside the drproj. The install process will update the project search paths accordingly. Package meta data would control what gets added to the search paths, my preference would be for 1 folder per package, as that would keep the search path shorter which improves compile times.

When a project is loaded, the dproj (or packages config file) would be checked, and any missing packages restored automatically. This should also handle the situation where a project is loaded in a different IDE version.

Security

We should allow for signing of packages, such that the signatures can be verified by the client(s). Clients should be able to chose whether to only allow signed packages, or allow signed and unsigned, and what to do when signature verification fails. This will allow users certainty in the authenticity and integrity of the package (ie where it comes from and whether it's been modified/tampered with).

Clients

It is envisaged that will be at least 2 clients, a command line tool, and a Rad Studio IDE plugin. Clients will download packages, add those packages to project/config search paths. A local package cache will help with performance, avoiding repetitive package downloads and also reduce disk space demands. The clients will also detect available updates to packages, and package dependency conflicts.

Command line Client

The command like tool will be similar to nuget or npm, which provide the ability to create packages, install or restore missing packages, update packages etc. The tool should allow the specification of compiler versions and platforms, as this is not possible to detect from the dproj alone. This is where the project is currently focused (along with the core package handling functionality).

RAD Studio IDE Client

An IDE plugin client will provide the ability to search for, install, restore, update or remove packages, in a similar manner to the Nuget Visual Studio IDE support (hopefully faster!). This plugin will share the core code with the the command line client (ie, it will not call out to the command line tool). I have not done any work on this yet (help wanted).

Delphi/Rad Studio Version Support

Undecided at the moment. I'm developing with XE7, but it's possible the code will compile with earlier versions, or be made to compile with minor changes.

Summary

Simply put, I want/need a package manager for Delphi, one that works as well as nuget, npm, yarn etc. I'm still fleshing out how this might all work, and I'd love some feedback, suggestions, ideas etc. I'd like to get some people with the right skills 'signed up' to help, particularly people with open tools api expertise.

Get Involved!

I have set up a home for the project on GitHub - The Delphi Package Manager Project - RFC. We'll use issues for discussion, and the wiki to document the specifications as we develop them. I have created a few issues with things that need some dicusssion. I hope to publish the work I have already done on this in the next few days (needs tidying up).

Today we released a FinalBuilder 8 update with Visual Studio 2019 and MSBuild 16 Preview support. So far, for the most part Visual Studio 2019 seems to operate (well, from our point of view) pretty much the same as 2017. There were some changes to the MSBuild location, but other than that it all seems to work fine. Since it's based on the preview, it's subject to change and or breakage at any time.

 

Back in December 2016, I posted some ideas for some Delphi language enhancements. That post turned out to be somewhat controversial, I received some rather hostile emails about how I was trying to turn Delphi into C#. That certainly wasn't my intent, but rather to modernize Delphi, in a way that helps me write less, but more maintainable code. Nearly 2 years later, Delphi 10.3 Rio actually implements some of those features.

I'm not going to claim credit for the new language features, and the syntax suggestions I made were pretty obvious ones, but I like to think I perhaps spurred them on a bit ;) My blog post had over 12K views, so there was certainly plenty of interest in new language features, and from what I have seen out there on the interwebs they have for the most part been well received.

So lets take a look at which suggestions made the cut for 10.3 - referencing my original post.

Feature Implemented Comments
Local Variable Initialisation No  
Type Inference Yes! For inline variables only, Confuses code insight!
Inline variable declaration, with type inference and block scope Yes, Yes and Yes! Confuses code insight!
Loop variable inline declaration Yes! Confuses code insight!
Shortcut property declaration No  
Interface Helpers No  
Strings (and other non ordinals) in Case Statements No  
Ternary Operator No  
Try/Except/Finally No  
Named Arguments    
Variable method arguments No  
Lambdas No  
Linq No Depends on lambdas and interface helpers.
Async/Await No  
Non reference counted interfaces No  
Attribute Constraints No  
Operator overloading on classes. No  
Improve Generic Constraint No  
Fix IEnumerable No  
Yield return - Iterator blocks No  
Partial classes No  
Allow Multiple Uses clauses No  
Allow non parameterized interfaces to have parameterized methods No  

 

So, 3 out of 23. To be honest, I was pleasantly surprised when I found out about them, given the pace of language change in the past. I'm hopeful this is just the start of things to come and we get to see Delphi evolve and catch up with other modern programming languages. I have a bunch of other language features I'd like to see, and received lots of suggestions from other users. 

We're still using Delphi XE7 for FinalBuilder 8, and I rarely change compiler versions during the life of a major product version. So I'll only get to use the new language features when I get fully stuck into FinalBuilder 9 & Automise 6. I'm in the process of getting Delphi 10.3 compatible versions of all the third party libraries (commercial and open source) - as and long time delphi user will know, that's always more difficult than it should be!  

TLDR; Our forums have moved to https://www.finalbuilder.com/forums

After years of frustration with Active Forums on Dotnetnuke, we finally got around to moving to a new forums platform. 

The old forums had zero facilities for dealing with spammers, and sure enough every day, spammers would register on the website and post spam on the forums. Even after turning on email verification (where registration required verifying your email), spammers would verify their emails and post spam.

The old forums were also terrible at handling images, code markup etc, and will often completely mangle any content you paste in.

So the hunt was on for a new platform. I've lost count of the number of different forum products I've looked at over the years, none of which totally satisfied my needs/wants. I've even contemplated writing my own, but I have little enough free time as it is, and would much rather focus on our products. 

Discourse  looked interesting, so I installed it on a Ubuntu Server 18.04 virtual machine (it runs in a Docker container). After some initial trouble with email configuration (it didn't handle subdomains properly) it was up and running. I'm not great with linux, I've tinkered with it many times over the years but never really used it for any length of time. I was a little apprehensive about installing Discourse, however their guide is pretty good and I managed just fine. 

The default settings are pretty good, but it is easy to configure. After experimenting with it for a few days (there are a LOT of options), we we liked it a lot, and decided to go with it. 

Discourse is Excellent at handling bad markup, I'm astounded at how well it deals with malformed html and just renders a good looking post (most of the time). Inserting images is a breeze, the editor will accept Markdown or html, and gives an accurate preview while you are writing a post. Posting code snippets works well usng the same markup as github, with syntax highlighting for a bunch of languages (C#, delphi, javascript, vbscript, xml etc). The preview makes it easy to tell when you have things just right. Discourse also works very well on mobile, although our website does not (the login page is usable) - more work to be done there (like a whole new site!). 

Discourse is open source (GPL), so you can either host it yourself (free) or let Discourse.org host if for you (paid, starting at $100pm). Since we had spare capacity on our web servers (which run hypver-v 2016) I chose to host it ourselves. That was 11 days ago. 

My goal was to import the content from the old forums, there are 12 years of valuable posts there which I was loath to lose. 

The first challenge was that Discourse requires unique emails, and our dotnetnuke install did not. After 12 years of upgrades, our database was in a bit of a sorry state. There were many duplicate accounts (some users had 5 accounts), I guess if you can't remember your login you just create a new one, right? I can't totally blame users for that, the password reset email system was unreliable in the past (it should be ok now, check your spam folder!). So we cleaned up the database, removed old accounts that had no licenses and no forum posts. 

The next challenge was enabling single sign on with the website. Someone had written a dotnetnuke extension for it, but I wasn't able to get it working (it was written for an older version), so I spent 2 days writing my own (and almost losing the will to live!). Once that was sorted, I got to work on importing the data. Discourse does have a bunch of import code on github - none of which are for dotnetnuke, and they are all written in Ruby (which I have zero experience with). Fortunately, Discourse does have a rest api - so using C# (with dapper & restsharp) I set about writing a tool to do the import. Since Discourse doesn't allow you to permanently delete topics, this needed to work first time, and be restartable when an error occurred. This took 4 days to write, much of which was just figuring out how to get past the rate limits Discourse imposes. I did this all locally with a back up of the website db and a local discourse instance. The import took several hours, with many restarts (usually due to bad content in the old forums, topics too short etc). 

Backing up the local instance of Discourse was trivial, as was restoring it on the remote server (in LA). We did have to spend several hours fixing a bunch of posts, and then some time with sql fixing dates (editing a post sends it to the top of a category). I did also have to ssh into the container to "rebake" the posts to fix image url issues. Fortunately theres is a wealth of info on Discourse's own forums - and search works really well!

We chose not to migrate the FinalBuilder Server forum (the product was discontinued in 2013) or the Action Studio forum (which gets very few posts).  

 

I'm sure we'll still be tweaking the forums over the next few weeks, but on the whole we are pretty happy with how they are. Let us know what you think (in the Site Feedback forum!).

Version 1.9 is now out of beta and available as a stable release. Thank you to those of you who have already tried out the beta - especially those who reported issues.

This version brings major changes to the notifications system. We redesigned it using a common architecture, that makes it much easier to add new notification publisher types. Where previously, only email, XMPP and private message notifications were available, there are now publishers for Slack, Teams, Hipchat and Stride. And we can now add more (let us know what you need).

We are no longer limited to one publisher of each type. You may, for example, have different email servers for different teams on your company. You can set up two email publishers, one for each server, and set up subscriptions so that notifications from different projects go to different email servers. Likewise for different Slack workspaces, Teams channel connectors and so on.

We have also improved the XMPP publisher to support sending notifications to rooms. Subscriptions have been improved, allowing you to specify a room and/or channel for this and other publishers.

User preferences have been updated allowing each user to specify a recipient id, username or channel per publisher.

You can see some metrics on the throughput of each publisher (number of messages on queue, messages sent per second, average send time, etc.) on the Publishers page in the Administration area. This also shows real-time counts of any errors occurring while sending messages and also any messages waiting on a retry queue due to rate limiting or service outages. This allows you to know when you need to upgrade rate limits or make other service changes.

The Templates page has been updated. Templates are now divided into a tab per publisher. The list of available variables for each event type has been moved to a expandable side panel.

This release is built on .Net Framework version 4.7.2, which has allowed us to upgrade a number of third party libraries, including the database ORM and PostgreSQL drivers. This has noticeably improved performance, as well as providing us with a richer platform to build future features on. The setup wizard will prompt for you to install .Net Framework version 4.7.2, before continuing with the installation.

Note that applications running on .Net 4.7.2 do not run on versions of Windows prior to Windows Server 2008R2 and Windows 7 SP1. We are also dropping the 32-bit server installer. This is mainly to reduce testing overheads. We will still be releasing 32-bit agents for those who are using 16-bit compilers.

We will continue to provide bug fixes to Continua CI version 1.8.1 for while to give you time to migrate from older platforms.

 

 

I'm not usually one for publishing roadmaps, mostly because I don't like to promise something and not deliver. That said, we've had a few people ask recently what is happening with Continua CI. 

Disclaimer - nothing I write here is set in stone, our plans may change.


A few weeks ago, I wrote up a "roadmap" for Continua CI on the whiteboard in our office. Continua CI 1.8.x has been out for some time, but we have been working on 2.x for quite a while. The length of time it is taking to get some features out is a cause of frustration in the office, that lead to a lengthy team discussion, the result was a "new plan". 

One of the reasons we had been holding features back for 2.x, is they required a change in the system requirements. Many of the third party libraries we use have dropped .net 4.0 support, so we were stuck on old versions. So rather than wait for 2.0 we will release 1.9 on .net 4.7.2. This will allow us to release some new features while we continue working on 2.0, and to take in some bug fixes from third party libraries.

This is "The Plan" :

Version .NET Framework x86/x64 Min OS Version UI Features
1.8.x 4.0 both Windows Server 2003R2 MVC 4  
1.9.0 4.7.2 x64 Windows Server 2008R2 MVC 5 New Notifications types
1.9.1 4.7.2 x64 Windows Server 2008R2 MVC 5 Deployment Actions
1.9.2 4.7.2 x64 Windows Server 2008R2 MVC 5 Import/Export
2.0.0 netcore 2.1 x64 Windows Server 2012 MVC 6 New Architecture
3.0.0 netcore x.x x64 Windows Server 2012 TBA New User Interface

 

Let's break down this plan.

1.9.0 Release

The 1.9 Release will built on .net 4.7.2, which allowed us to take updates to a number of third party libraries, most notably NHibernate and Npgsql (postgress driver). These two libraries factor heavily in the performance improvements we see in 1.9.0. 

The major new feature in 1.9.0 will be a completely redesigned notifications architecture. In 1.8, notifications are quite limited, offering only email, xmpp and private messages. There was very little shared infrastructure between the notification types, so adding new notification types was not simple. You could only use 1 mail server and 1 xmpp server.

In 1.9.0, notifications are implemented as plugins*, using a common architecture that made it much easier add new notification types. You can also define multiple notification publishers of the same type, so different projects can use different email servers for example.

Notification Types :  Private message, Email, XMPP, Slack, Hipchat, Stride. More will follow in subsequent updates (let us know what you need).

*We probably won't publish this api for others to use just yet, as it will be changing for 2.0 due to differences between the .net framework and .net core.

If you are running Continua CI on a 32-bit machine, then start planning your migration. Supporting both x86/x64 is no longer feasable, and dropping x86 support simplifies a lot of things for us (like updating bundled tools etc).  We will continue supporting 1.8.x for a while, but only with bug or security fixes. The minimum OS version will be the same as for the .Net Framework 4.7.2 - since Windows Server 2003R2 is out of support these days, it makes sense for us to drop support for it. 

1.9.1 Release

Deployment focused actions.  

  - AWS S3 Get/Put
  - Azure Blob Upload, Publish, Cloud Rest Service, Web Deploy
  - Docker Build Image, Push Image, Run Command, Run Image
  - File Transfer (FTP, FTPS, SFTP)
  - SSH Action
  - SQL Package Export, Package Extract, Package Import, Package Publish
  - SSH Run Script
  - Web Deploy

These actions are all mostly completed, but are waiting on some other (UI) changes to make them easier to use. We'll provide more detail about these when they closer release.

Note : These actions will only be available to licensed customers, not in the free Solo Edition.

1.9.2 Release

One of the most requested features in Continua CI, is the ability to Export and Import Continua CI Projects and Configurations. This might be for moving from a proof of concept server to a production server, or simply to be able to make small changes, and import configurations into other projects. The file format will be YAML.

Continua CI 2.0 Release - .net core.

We originally planned to target .net framework 4.7 with Continua CI 2.0, but with .net core improving significantly with netcore 2.0 and 2.1, the time is right to port to .net core. The most obvious reason to target .net core is cross platform. This is something we have wanted to do for some time, and even explored with mono, but were never able to get things working in a satisfactory manner. It's our hope that .net core will deliver on it's cross platform promise, but for now it's a significant amount of work just to target .net core. So that said, our plans for Continua CI 2.0 is to get it up and running on .net core on Windows only, without losing any functionality or features. During the port we are taking note of what our Windows dependencies are for future reference. 

The current (1.8.x) architecture looks like this :

Browser <----> IIS(Asp.net with MVC)<--(WCF)-->Service <--(WCF)-->Agent(s)

With .net core, it's possible to host asp.net in a service process, and that is what we have chosen to do. This cuts out the WCF layer between IIS and the service. .net core doesn't have WCF server support, and to be honest I'm not all that cut up about it ;) That said, we still need a replacement for WCF for communication between the agents and the server. We're currently evaluating few options for this.

Continua CI 2.0  architecture currently looks like this :

Browser <----> Service(hosting asp.net core 2.1/mvc)<--(TBD)--> Agent(s)

The current state of the port is that most of the code has been ported, the communication between the agents and the server is still being worked on, and none of the UI has been ported. We do have asp.net core and mvc running in the service. There are significant differences between asp.net/mvc and asp.net core/mvc, so we're still working through this, I expect it will take a month or so to go through and resolve the issues, then we can move on to new features. 

Continua CI 2.0 - new features.

Rest API. This is something we had been working on for a while, but on the .net framework using self hosted Nancy (in the service, running on a separate port from IIS). Once we made the decision to port to .net core, we chose to just use asp.net rather than Nancy. Fortunately we were able to use much of what was already done with nancy on asp.net core (models, services etc) and we're currently working on this right now..

Other features - TBA

Continua CI 3.0 - A new UI

Asp.net MVC has served us well over the years, but it relies on a bunch of jQuery code to make the UI usable, and I'll be honest, no one here likes working with jQuery! Even though we ported much of the javascript to typescript, it's still hard to create complex UI's using jQuery. The Stage Editor is a good example of this, even with some reasonably well structured javascript, it's still very hard to work on without breaking it. The UI is currently based on Bootstrap 3.0, with a ton of customisations. Of course Bootstrap 4.0 completely breaks things so we're stuck on 3.0 for now.

So it's time to change tack and use an SPA framework. We've done proof of concepts with Angular and React, and will likely look at Vue before making a decision - right now I'm leaning towards React. Creating a new user interface is a large chunk of work, so work will start on this soon (it's dependent on the rest api). We're likely to look at improving usability and consistency in the UI, and perhaps a styling refresh. 

Linux & MacOS Agents - with .net core running on these platforms, this is now a possibility. We looked at this several times before with Mono, but the api coverage or behavor left a lot to be desired. We do still have some windows specific stuff to rework in our agent code, and Actions will need to filtered by platform but this is all quite doable.

Summing up

We're making a big effort here to get features out more frequently, but you will notice I haven't put any timeframe on releases outlined above, they will be released when ready. We expect a 1.9.0 Beta to be out in the next week or so (currently testing the installer, upgrades etc), and we'll blog when that happens (with more details about the new notifications features). Note that it's highly likely there will be other releases in between the ones outlined above, with bug fixes and other minor new features as per usual. We have a backlog of feature requests to work from, many of which are high priorities, so we're never short of things to do (and we welcome feature requests). 

In version 1.8.1.870 of Continua CI, we have added new archiving functionality to the workspace and repository rules.

Builds can generate a lot of output files: binary library files or report files, for example. Copying a large number of these files back to the server at the end of the build can take time. Manually downloading each individual artefact from the server can be a tedious task, so compressing these files into a handy bundle makes sense.

Previously, you would have needed to use actions, such as the Seven Zip action, in your build stages to zip these files. The compression can now be performed as part of the agent-to-server workspace rules.

To compress a set of files in the agent workspace to an archive in the server workspace, specify a file with a zip extension on the left-hand side of a agent-to-server workspace rule.

e.g.



    Libraries.zip < Output/**.dll


Note that the all the usual operators are taken into account when compressing files so, in the above example, the directory structure is preserved. Likewise, using the <- operator will cause all matching files to be flattened into the root folder of the zip file.

Doubling up with the << operator will delete any existing zip file before compressing to a new file. Without the << operator, multiple sets of files can be added to the same archive file.

e.g.



    Reports.zip < Output/**.html
	Reports.zip < Output/**.css


You can also compress files into subfolders within the zip file using the new : operator

e.g.



    Reports.zip:/css < Output/**.css


Once files have been compressed at the end of one stage, you may need to access the contents of zip files in the next stage. Additionally, you may wish to unpack a zip file from your repository at the start of a stage. The : operator facilitates the extracting of zip files in server-to-agent workspace rules and repository rules.

To extract a set of files from an archive in the server workspace to a folder in the agent workspace, specify a file with a zip extension on the left-hand side of a server-to-agent workspace rule. Ensure that you follow the ‘zip’ with a : operator, otherwise the zip file will just be copied.

e.g.



    Libraries.zip: > Libraries


This also works for repository rules.

e.g.



	$Source.MyRepo$/Documents.zip: > Docs/Main


Note that the all the usual operators >, >>, -> and --> have the same meaning when extracting files as they have when copying file; signifying whether to preserve the directory structure within the zip file and whether to empty the destination folder.

You can also specify a pattern after the : operator, allowing you to filter the extracted files.

 e.g.



	Libraries.zip:/plugins/**.dll > Libraries/Plugins
	$Source.MyRepo$/Documents.zip:**.md > Docs/Markdown


See the Workspace Rules documentation for further details on the new archive rules syntax.

SSL standards are changing, and older SSL/TSL protocols are slowly being deprecated, or even turned off by some services. This post shows how to enable TLS 1.2 support in Continua CI.

Yesterday, we started getting reports that the Github Status event handler, and the Github Status action in Continua CI had stopped working.

Sure enough, in our testing here we were able to confirm the case. While testing this under the debugger, the error we were seeing was rather strange : "The request was aborted: Could not create SSL/TLS secure channel.".

After some research, we found this error was due to being unable to negotiate a common protocol between the client and the server.

Now Continua CI 1.x is built with .NET 4.0 (v2 will be on 4.7.1) - we know that .NET 4.0 doesn't support TLS 1.2, and a quick check of the github api server using SSLLabs shows that they now only support TLS 1.2.


I wondered if this was announced by github - turns out they did announce this 3 weeks ago :

Weak cryptographic standards removal notice

and yesterday they permanently disabled TLS 1.0 and 1.1

Weak cryptographic standards removed

Anyway, back to Continua CI. The good news is that there is a way to enable TLS 1.2 support in Continua CI. Note that this only works when running on Windows Server 2008 or later (Server 2003 does not support TLS 1.2 at all, and we will be dropping support for it with v2).

1) Install .Net Framework 4.5 or later - since all 4.x frameworks effectively replace 4.0 - 4.5 has support for TLS 1.2

2) Edit %ProgramFiles%\VSoft Technologies\ContinuaCI\Server\Continua.Server.Service.exe.config on the server and %ProgramFiles%\VSoft Technologies\ContinuaCI Agent\Continua.Agent.Service.exe.config on each agent - add the following line in appSettings section :

       <add key="Continua.Service.SecurityProtocolType" value="Tls|Tls11|Tls12" />
    
Note that the key supports the following values: Ssl2|Ssl3|Tls|Tls11|Tls12|Default

Default = Ssl3|Tls
Multiple protocols can be separated with |
The value "Tls|Tls11|Tls12" will allow Continua CI to work with services that do not support or have not enabled TLS 1.2, and with services that only support TLS 1.2 .

3) Open Regedit and add the following value to : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 : SchUseStrongCrypto type DWORD value 1

4) Restart the Server and Agent Services.

5) You may need to restart your server(s) for the registry change to take effect.

One last note : This change also effects the communication between the Continua CI Server and agents, if you make the change on the server, make sure you make a compatible change on the agents.

The Windows 10 Fall Creators Update has only been out a few hours, but we're already getting questions about it. 

In our limited testing, FinalBuilder 8 and Automise 5 run fine.

I've only been running the Fall Update for a few hours, but so far I have not noticed any issues. The applications I use daily all run fine. 

The "Windows 10 Creators Update" (ie the one before the Fall Update - stupid release naming imho) broke the Delphi debugger when using runtime packages. Aparently the issue was caused by a library loader optiimisation, not taking into account that dll's can have multiple import tables. I never did see a full explaination or acknowledgement of the problem from Microsoft.

This only affected the debugger (all native code debuggers, not just Delphi), which would load and unload each dll many times (based on the number of imports, for FinalBuilder's core package, it was in the hundreds). Sometimes the application would launch, only for the debugger to crash, sometimes it would just hang, sometimes the Delphi IDE would get out of memory errors. 

For me, this was a big issue, since FinalBuilder and Automise use runtime packages. This affected all versions of Delphi, even the latest 10.2 (Tokyo). Embarcadero did eventually ship an update to 10.2 that mostly resolved the problem (not an easy thing as it involved major linker changes), but that didn't help us as we're using an older version (for reasons I won't go into here!).

So since April 2017, I've been really hamstrung when it comes to debugging. Fortunately we discovered the issue before the Creators Update was installed on our other Delphi development machines (and it's a been a constant battle with windows update nagging to install it ever since) so we were still able debug, just not on my dev machine. Frustrating to say the least. 

The good news is that the Fall Update (mostly) fixes the problem.  I still see some dlls/packages getting unloaded and reloaded again, but the application launches and I can debug. 

As far as windows functionality in the Fall Update goes, well the Task Manager has a new GPU section on the performance tab which is mildly interesting, but since I don't use a Pen, or wear a VR headset while working, I'm not noticing much to get excited about. Hopefully, it's just a lot of bug fixes and performance enhancements, minus the show stoppers!! 

In this post, I'm going to look at how to structure a FinalBuilder project so that it will run on your dev machine, or on your Continua CI Server without modification. This allows the best of both worlds, develop and debug your build process on your development machine, and then later run it on your CI server.

I'm going to assume you are familiar with FinalBuilder to speed this along.

Version Control

The very first thing we need to do is add our FinalBuilder project to our version control system. This will ensure that the Continua CI agent will be able to access the projects.

We typically create a Build folder in each repository that has our FinalBuilder projects, installer scripts etc. Make sure you save your projects in uncompressed format (ie, .fbp8 for FinalBuilder 8), as that will make it possible to diff project file changes using your usual diff tool (we use Beyond Compare 4).

So in a typical repository, you might have a folder structure that looks something like this :

        \Build
        \Src
        \Docs
        \Help
        \Tools
        \Output
    

Note the Output folder is ignored by our version control system (via .gitignore or .hgignore for example). Don't forget to add ignores for the finalbuilder log file as you don't want to commit that to the repo.

This is the file structure I'll be using as the example in this post.

Build Workspace

Continua CI runs each build in a separate, clean build workspace folder. The reason for this is to allow multiple builds of the same configuration to be run concurrently (for example, building different branches at the same time). This means our folder structure above will be rooted differently for each build. So to deal with this, we will define some FinalBuilder Project Variables.

FinalBuilder Variables

 

FinalBuilder Project Variables

 

 

You can of course modify these variables to suite your needs. The REPO variable has a default value of "..", we will use the Path Manipulation action to expand that relative path when our build starts, using the FBPROJECTDIR variable as the base path. This will give us the root folder of our repository, which we can then use as the basis for other path variables.

 

Side Note: FinalBuilder does not support using unrooted relative paths. The best practice is to fully expand the relative path before using it. Relative paths need to be relative to something, and in windows that is typically the Current Directory for the process, however this is not viable in a multi-threaded application. Rooted relative paths (e.g "%SOURCE%\.." ) will work in most cases, however some windows api functions do not support relative paths at all.

The WORKSPACE variable will have the path to the Continua CI build workspace folder. Depending on how your repository and repository rules are structured, this might be different.

Repository Rules

Continua CI configurations can have multiple Repositories assigned to them, for example when building FinalBuilder, we have mulitple repos on the configuration, one for our source and three others for libraries. On our dev machines, we use symlinks to map the libs folder into our main repo's path, so we end up with :

        MainRepo\Src
        MainRepo\Libs <- symlink to LibsRepo\
    

This makes it easier to deal with search paths in Delphi, as we can just use simple relative paths. With .Net, working with libraries is so much simpler due to package managers like nuget or packet.

At the start of a Continua CI Stage, the source code is exported from the Repository Cache (a Mercurial repo) for each repository associated with the Configuration. How and what gets exported is controlled by the Repository Rules for that Stage.

The default rules looks like this :

        $Source$ >> Source
    

This will export all associated repos to a folder structure like this

        workspace\source\repo1
        workspace\source\repo2
    

In this example, I'm going to change the rules to mirror the folder structure on our dev machines :

Repository Rules

This results in the following folder structure :

        workspace\build
        workspace\src
        workspace\libs
    

Note that you can also use rules to limit which parts of the repository are exported to the workspace, so, for example, if you have files in the repository that are not needed for the build process, you can avoid exporting them and save some time (I/O is usually the biggest overhead) using exclusion rules.

Getting back to our FinalBuilder variables, we can see from the above folder structure, that our WORKSPACE variable will have the same value as the REPO variable. If your folder structure is different then you will need to adjust accordingly.

Continua CI Stage Actions

Ok, so now we have a basic FinalBuilder project, and Repository Rules set up to export our source to the build workspace, now it's time to call FinalBuilder. Continua CI has a FinalBuilder Action for doing just that.

Continua CI - FinalBuilder Action

We need to tell the action where our FinalBuilder project file lives. This needs to be anchored by the workspace foler, e.g $Workspace$\Build\FB\FinalBuilderBuild.fbp8

In the Variables tab, we will check the "Automatically apply Continua CI variable values to matching FinalBuilder variables." Option. If you want to send any protected Continua variables (ie passwords, api keys etc) to FinalBuilder, then check the "Save sensitive variable values to context XML file for use by FinalBuilder." option as well. Lastly, on the Environment Variables tab, we are passing some environment variables that are used in delphi search paths to FinalBuilder.

Continua CI - FinalBuilder Action

We also set some environment variables that are needed for our build process, Continua CI makes this painless.

Continua CI - Set Environment Variables

FinalBuilder Project

Remember that we want to be able to run the project on our dev machine and from Continua CI. So some logic is required to make sure it behaves the same way in both environments.

I added some targets to the project, Init, Build and Test.

The Init target is a dependency of the Default target, so it will always run first. This is where we initialise the variables, detect if the project is running from Continua CI or not, and set version info. We also create the Output folder here.

FinalBuilder - Init Target

Back in the Default target, we call the Build and Test targets. This is wrapped in a try/finally block and we export the FinalBuilder log file to %OUTPUT%\BuildLog.html in the finally section. We can register this file as an artifact and create a report definition in Continua CI to make it easy to view in the Continua CI UI. I've covered this before in an earlier blog post, so I won't cover it here.

FinalBuilder - Init Target

Note that you don't have to do the whole build/test/deploy all in one FinalBuilder project. When building FinalBuilder, we do the build & test in one project, and the deploy in another that is called from the Deploy stage. This avoids the temptation to deploy from a developer workstation!

When we build Continua CI (on Continua CI of course), we have separate Build, Obfuscate, Test, Package & Deploy Stages that use different FinalBuilder projects. This allows us to run different parts of the build process on different agent machines. The .net obfuscation tool we use is very expensive, so we only have a single license which is installed on one agent machine, so the obfuscation stage will only run on that agent machine (vm). Continua CI selects the best agent based on agent properties and the Stages Agent Requirements. I'll cover this in more detail in another blog post.

I have added this example project to the Examples repository on GitHub.