VSoft Technologies Blogs

Using Python in FinalBuilder

Wed, 04 Jun 2008 14:00:00 GMT

Today FB 6.1 was released as a beta version. One of the major new features in this version is Python language support. We're leveraging Microsoft's IronPython project for this, and because version 2 of IronPython is built on the Dynamic Language Runtime, we're expecting that as other DLR languages become available, we'll be able to support them in FinalBuilder as well.

The Python language is pretty neat, and the fact that IronPython interacts seamlessly with the .NET Framework makes it pretty powerful by itself. It shares this feature with PowerShell.

However, what really makes Python awesome is the amount of extra library functionality that comes with the standard language distribution.

Example

This is a totally arbitrary example. Say your build process needs to calculate a SHA1 hash. FinalBuilder only supports MD5 at the moment. Can Python save the day?

Step One: Install Python Libraries

You need to install the Python standard libraries in order to get access to their many functions. You get this as part of the standard Windows Python distribution. Grab the "Windows Binary installer" from http://python.org/download/

Step Two: Tell FinalBuilder

You need to tell FinalBuilder where the Python library is installed. If you launch FinalBuilder and go to Tools -> Options -> FinalBuilder -> Script Options, as shown, you will see this options page:

Enter the path to the Python library directory, as shown here. The options page says you need to restart, but you actually only need to do this if you've run a Python script since you started FinalBuilder.

Step Three: Run Script Action

I use the Action filter to search for the all-purpose Run Script action, which I can then add to my project:

Step Four: Type in Script

Here's my SHA1 script code. I entered this in the OnExecute event, under the Script Editor tab at the bottom of the main FinalBuilder IDE. Make sure you set the script language to "Python", as shown:

Note that the FBVariables object behaves just like a normal Python dictionary.

Click here to download a sample project file.

Step Five: Run it!

Here's the log output from the SHA1 Run Script:

Where do I go from here?

This was a pretty basic example, but the Python library provides you with literally hundreds of similar pieces of functionality. Take a look at the Python Library Reference for a complete list.

Can't I just use the .NET Framework?

Yes! The .NET Framework also provides you with a wealth of built-in functionality. Using it from IronPython is pretty simple, too. Here's Python code that performs the same functionality, written to use the .NET Framework instead of the Python library:

(It's a bit more complicated, mostly because I couldn't find a neat .NET-based way to convert a byte array to a hexadecimal string. Note that this approach does not require the Python libraries, it will work out of the box with FinalBuilder.)

Both sample scripts are included in the downloadable example.

It would also be possible to rewrite the .NET version to use PowerShell, instead. I'll leave that as an exercise.

Hate Team Foundation Server?

Thu, 15 May 2008 14:00:00 GMT

According to Google Analytics statistics for the past month, a number of people found the FinalBuilder web site in Google search results for the query hate "Team Foundation Server"

I didn't know so many people felt that way!

(For the record, we think that FinalBuilder complements Team Foundation Build in very productive ways.)

Delphi: Workaround for TThread Synchronize/WaitFor Deadlock

Wed, 23 Apr 2008 14:00:00 GMT

FinalBuilder is written in a combination of Delphi and C#. This is a blog post about a Delphi RTL threading issue that I recently ran into and had to work around.

We recently had some bug reports of FinalBuilder deadlocking at shutdown under certain circumstances. I could reproduce the problem, but I was baffled for a long time because - according to the call stacks - the program should have been fine.

The issue I'd run into has been around for years, and is summed up nicely in this Borland QC note: Synchronize and WaitFor methods of TThread can lead to a DeadLock in logically correct applications.

The Problem

The simple summary is: If you ever have a call stack which includes both Synchronize and WaitFor, and the thread you are waiting on also calls Synchronize at some point, you will occasionally get a deadlock.

Explaining why is kind of tricky, but if you read the QC case you'll get the gist of it. The problematic sequence of events looks like this:

Thread A calls Synchronize(MethodA)

Thread B calls Synchronize(MethodB)

Then, inside the context of the Main Thread:

Main thread calls CheckSynchronize() while processing messages
CheckSynchronize is implemented to batch-process all waiting calls(*). So it picks up the queue of waiting calls (containing MethodA and MethodB) and loops through them one by one.
MethodA executes in the main thread's context. Assume MethodA calls ThreadB.WaitFor
WaitFor calls CheckSynchronize to process any waiting calls to Synchronize
In theory, this should then process ThreadB's Synchronize(MethodB), allowing Thread B to complete.
However, MethodB is already a possession of the first CheckSynchronize call, so it never gets called.

DEADLOCK!

The Solution

The ideal solution is to refactor your program so it will never call CheckSynchronize or WaitFor from inside an already-synchronized method call. For FinalBuilder, this was not a good option - all it takes is one Synchronized method call to call ProcessMessages and you're letting in dozens of potential deadlocks.

The solution I came up with is to write a "SafeSynchronize" method that only allows one thread at a time to get on the Synchronize method queue. This comes with a performance hit, but prevents the deadlock.

Here's a new class which implements the SafeSynchronize method:

  TSafeSyncThread = class(TThread)
  private
    procedure SafeSynchronizeInner;
  protected
    procedure SafeSynchronize(const AThreadMethod: TThreadMethod);
  end;

implementation

{ TSafeSyncThread }

var
  FSingleSynchronize : TCriticalSection;
  FSingleSynchronizeTarget : TThreadMethod;

procedure TSafeSyncThread.SafeSynchronize(const AThreadMethod: TThreadMethod);
begin
  FSingleSynchronize.Acquire;
  FSingleSynchronizeTarget := AThreadMethod;
  Synchronize(SafeSynchronizeInner);
end;

procedure TSafeSyncThread.SafeSynchronizeInner;
var
  targetMethod : TThreadMethod;
begin
  targetMethod := FSingleSynchronizeTarget;
  FSingleSynchronize.Release; // Now we're inside a CheckSynchronize call, it's safe to let another thread join the queue
  targetMethod;
end;

Basically, this serializes access to the Synchronize queue. The performance hit is potentially quite large if you have a lot of concurrent SafeSynchronize calls and a busy main thread. FinalBuilder is designed in a way which minimizes this, so it's less of an issue for us.

I put together a quick test project which demonstrates the issue (at least on Delphi 2007.) You can get it here if you want to have a look. It also contains SafeSyncThread.pas if you want to incorporate the workaround into your own projects. This code comes with no guarantees and no warranty, obviously.

If anyone has a better (preferably faster) solution to this problem, which doesn't require rewriting the RTL, please let us know. :).

PS The test project example contains some fairly awful cut-and-paste code reproduction, which - for the record - I normally avoid like the plague. I used it here because the methods are very simple, and I couldn't see an easy way to rearchitect it.

(*) FWIW, I actually quite like the way that CheckSynchronize is implemented, using an InterlockedExchange to swap the old queue object with a brand new one. That way you can process the entire old queue at once, without granular locking. Check the source code if you want to see what I mean. It's just a shame about the deadlock. (back)

FinalBuilder/Automise Keyboard Shortcuts

Tue, 15 Apr 2008 14:00:00 GMT

Brent was watching me fly through some FinalBuilder operations the other day and he suggested I write a blog post about the best keyboard shortcuts to use.

Even as a power user/developer, you can sometimes use a tool every day and not know the best keyboard shortcuts for it. I only learned many of the keyboard shortcuts for Visual Studio after I attended a special session at TechEd last year.

I don't want to drown you in information here, so I'm going to list four shortcuts that are essential to know, and another four that are very useful.

Essential

If you own a keyboard and use FinalBuilder or Automise, the following keyboard shortcuts are essential. They will improve your productivity ten-fold!

Key	Context	Description
Ctrl-I	Main IDE Window	Focus the Actions filter field to search for a new action. If the filter field is already focused, this keystroke hides the Actions sidebar instead.^*
Ctrl-Right	Selected action(s)	Indent the selected action(s) as new children of the action above them.
Ctrl-Left	Selected action(s)	Outdent the select action(s) relative to their parent action
Ctrl-Up/Down	Selected action(s)	Move selected action(s) up and down

In my opinion, using the Ctrl-Arrow keys to rearrange actions is essential (drag and drop is just too fiddly!)

^* The "hide sidebar if currently focused" feature is new in FB 6.

Very Useful

The following key combinations are very useful when you are typing in a text field while editing an action:

Key	Description
F3	Define a new variable on-the-fly.
F2	Pop up an advanced edit dialog with lists of variables, PropertySets, etc.
F12	Pop up a list of suggested variable replacements.
Ctrl-Space	Automatic completion for variable and PropertySet names.

For FB 6, we also did a lot of work on the Tabbing behaviour in the IDE. You should be able to use Tab to navigate the main IDE elements very easily.

FinalBuilder 6 - Output Monitors

Thu, 03 Apr 2008 14:00:00 GMT

Introduction

Output Monitors give you a quick way to watch action output and change the behaviour of the action depending on what you see. This lets you extend built-in actions in powerful ways.

Sample Project

Unlike my normal "pie in the sky" examples, this one is based on a real problem. This morning we had a customer in our chatroom who had a problem deploying to QA:

when a user's connection to the database breaks, or if a user is accessing some of the files in QA and connected to a different database (different to the one we are checking against) the FinalBuilder project will try and deploy the files and get a sharing violation which causes the build to fail

Summary: the customer needed a way to tell if one or more files were open on a remote machine.

SysInternals Tools Are Awesome

SysInternals make PsTools and a raft of other excellent Windows "power administrator tools". If you haven't checked them out, you should. It's no wonder that Microsoft bought the company.

For my solution to the above problem, I made use of two great SysInternals tools:

PsExec (part of PsTools) can deploy and run an executable on a remote machine. There's a PsExec action built into FinalBuilder, although you still need to download and install PsTools.
Handle is a great little command line tool that dumps (among other things) all of the open files on a system. An excellent debugging aid.

The Plan

Use PsExec to deploy and execute handle.exe on the remote computer, and use FinalBuilder to scan the output for the file(s) we need to check for.

The Project

Here's the complete project, including my verbose comments. Because of the powerful Output Monitor (and the awesome Sysinternals tools), almost everything happens inside the PsExec action:

Output Monitor

To check for our file(s), we need to configure an Output Monitor on the PsExec action. We have two variables:

fileName - The filespec or filename that we want to check for. This variable is set by the user.
fileInUse - The details of the file that's in use, if there is one. This variable is set when the PsExec action runs.

Every action in FinalBuilder 6 has an option on the Runtime tab called "Monitor Action Output":

Click the button to edit the Output Monitors for that action:

For our monitor (shown above), we're checking for any content which contains %fileName% and saving the first match to the variable fileInUse.

There are lots of other options for output monitor behaviour. Click on the dropdown for a full list:

... For example, if we wanted a list of all matching lines, we could use the "Save All Matches to Variable" behaviour.

As configured, the output monitor supports wildcard matching with * and ?. If we wanted it to be really powerful, we could change it to use a regular expression - grep-in-a-box!

Project File

You can download the FinalBuilder 6 project file here.

Alternatives

Sometimes you need more powerful output processing, and Output Monitors are not up to the task. You can always use the OnStatusMessage script event to run some script every time the action outputs data. I actually did a blog post about this a couple of weeks ago.

FinalBuilder 6 : PowerShell (Part Three, Custom Actions)

Wed, 02 Apr 2008 14:00:00 GMT

This is the last in a series of blog posts about PowerShell support in FinalBuilder 6.

Part One showed how to use one-liners to create powerful PowerShell execute conditions.
Part Two showed how to use PowerShell script events to parse output and change the behaviour of an action.

For this last post, I'm going to walk through creating a custom action which scans the Event Log and reports the frequency of messages from different sources.

The Script

Here are the parts of the PowerShell script that I'm going to use to implement the new action.

1) A custom function

WMI uses dmtf date formatting. This is a PowerShell function to create a dmtf datetime string representing (Now - $x hours) ago.

function dmtfWithin([int] $hours)
{
[System.Management.ManagementDateTimeconverter]::ToDmtfDateTime `
([DateTime]::Now.AddHours(- $hours))
}

2) Read Action Properties

Action properties are loaded using the same syntax as other custom actions. If you've used ActionStudio before then this will look familiar. Coming in-the-pipeline are some CmdLets to make this look more like PowerShell and less like .NET.

# Set up parameters
$computerName = $Context.ExpandExpression($Context.Properties.PropertyAsString("ComputerName"), $True)
$hours = $Context.Properties.PropertyAsString("Hours")

3) Process Event Log

This is where the dirty work happens - build a WQL filter string, get the WMI objects for the event log entries, and build a hashtable mapping event source names to their frequencies. The particularly neat line is the last one with the % sign - that's shorthand for "for each".

# Read results
$filter = "TimeWritten > '" + (dmtfWithin($hours)) + "'"
$results = Get-WmiObject -computer $computerName -class Win32_NTLogEvent -filter $filter
$freqTable = @{}
$results | % { $freqTable[$_.SourceName] += 1 }

4) Log the Results

Log the contents of our hash table, sorted and formatted into a table:

# Log Results
$freqTable.GetEnumerator() | sort -descending Value | ft Value, Name -autosize -wrap

5) Set Result to True

The Result parameter determines whether or not to fail the action.

$Result = $True

Creating the custom action

I recorded a screencast showing how to create the custom action in ActionStudio. Click here to view in a new window.

Running the custom action

Click here to view this screencast in a new window.

Download

Click here to download a zip file with the custom action package. The unzipped .fbap file needs to be saved in the ActionDefs directory inside the FinalBuilder 6 program directory.

More Resources

The example action package "FBScriptExamples.fbap" comes with FinalBuilder, and contains some more sample PowerShell actions which can be loaded in ActionStudio.

The ActionStudioManual.pdf file (located in the FinalBuilder Program directory) contains a reference section with built-in custom action types and methods, and also some more information on creating PowerShell actions.

Some Notes

This is a pretty simple example, but (like the others) it is fairly simple to extend it to cover lots of other Windows-related tasks.
FYI, there is a dedicated Get-EventLog command in PowerShell. In this instance, I chose to use WMI instead because WMI can read from remote machines, and can read across all event logs. If you use the dedicated command, you need to choose a single event log to read from.

FinalBuilder 6 : PowerShell (Part Two, Script Events)

Wed, 26 Mar 2008 05:00:00 GMT

PowerShell Script Events

This is Part Two in a series about the new PowerShell support in FinalBuilder 6. In Part One, we learned how to create an action Execute Condition using PowerShell.

In this article we will show how to use PowerShell as a scripting language in the FinalBuilder IDE, in order to customize the behaviour of a built-in FinalBuilder action. We'll also showcase some of the powerful string processing features in PowerShell.

Script Events

When you add an action to a FinalBuilder project, you also add at least three script events:

BeforeAction - This script event is called before the action runs.
OnStatusMessage - This script event is called each time the action outputs something to the log.
AfterAction - This script event is called after the action has finished running.

You can access the script events by clicking on the "Script Editor" tab in FinalBuilder. For this blog post, we're going to use PowerShell to create an implementation for each of these script events.

PsLogList

I'm going use scripting to extend the built-in FinalBuilder action for the PsLogList tool. PsLogList is part of the PsTools suite from SysInternals/Microsoft. It dumps the contents of the event log on a local or remote machine.

We're going to customize the action so it parses the list of log entries to automatically extract data about the frequency of events from different sources.

Output format

Here's an example log entry, as output from PsLogList:

[16664] Service Control Manager
Type: INFORMATION
Computer: DYNAMO
Time: 3/03/2008 3:09:18 PM ID: 7035
User: NT AUTHORITY\SYSTEM
The IMAPI CD-Burning COM Service service was successfully sent a start control.

For our example, the piece of data want to extract would be "Service Control Manager" in the expression above. To extract this, we can use the following Regular Expression:

^\[\d+\] (.+)$

Regular expressions are supported directly in PowerShell. You can create a regular expression using a code snippet like this:

$headerExpr = [regex] "\[\d+\] (.+)"

(If you ever need help remembering Regular Expression syntax, I recommend www.regular-expressions.info.)

Hash tables

The other piece of PowerShell functionality that we're going to make use of is the hash table. We're going to create an empty PowerShell hash table, and then use it as a dictionary to store event source names along with the frequency that each source appears.

Here's the script used to create a new empty hashtable:

$hashTable = ${}

To increment the frequency of an item in the table, we can use the increment operation += on the hashtable entry. If there is no entry, PowerShell returns $null, but ($null + 1) == 1 so it will work properly.

$hashTable[$name] += 1

Scripting it up

Here are the complete script events: (download as project.)

BeforeAction Script Event

# Create empty hash table for use in action (we need to store it in a Project Variable)
$FBVariables["FrequencyTable"] = @{ }

OnStatusMessage Script Event

# When output comes from the action, check if it is an Event header line

$headerPattern = "^\[\d+\] (.+)$"
$RegexOptions = [System.Text.RegularExpressions.RegexOptions]

foreach ( $match in [regex]::Matches( $StatusMessage.MessageText, $headerPattern, $RegexOptions::MultiLine ) )
{
$appName = $match.Groups[1].Value.Trim()
$freqTable = $FBVariables["FrequencyTable"]
$freqTable[$appName] += 1
}

AfterAction Script Event

# Log the frequencies, sorted by value
"Frequencies for different sources"
$FBVariables["FrequencyTable"].GetEnumerator() | sort -descending Value | ft Value, Name -autosize -wrap
#
# (the .GetEnumerator() call in the expression above is a trick which allows us to sort the contents of the hashtable.)

Download the Project

Click here to download a sample project with this action.

The downloaded project has one more feature: it suppresses the normal output from PsLogList. Only our custom information is shown in the log.

Action output

Here's the log output from when the action is run normally:

Frequencies for different sources
Value Name
----- ----
29 Service Control Manager
3 Application Popup
2 W32Time
2 IPSec
2 EventLog
2 WinHttpAutoProxySvc
1 DCOM
1 AeLookupSvc
1 Tcpip6
1 Tcpip
1 redbook

Why this is Useful

OK, so this example isn't particularly relevant for automated builds. However, this technique (or one like it) can be used to parse information from any tool which is run by FinalBuilder.

Notes

Having a script event run every time data is logged decreases the performance of the action. For actions which log very large amounts of data, it's better to use Log To Variable and then process the output all-at-once in a separate action.

You'll note that the actual regular expression call used in the script event is different to the example expression showed at the beginning. We have to call the .NET RegularExpression class directly in order to use MultiLine mode. (If anyone from the PowerShell Team is reading this, please try and find a way to set the expression mode with simpler syntax!)

You can actually do everything shown in the example with JavaScript instead of PowerShell, using a Regexp object and an associative array. However, some steps (like automatically printing the results) are much more complex to do with JavaScript.

You could also get the same results from a PowerShell action by using get-wmi and the Win32_NTLogEvent class. That will be the subject of upcoming blog post Part Three: Custom actions.

FinalBuilder 6 : PowerShell (Part One, Execute Conditions)

Tue, 18 Mar 2008 05:00:00 GMT

Introduction

PowerShell support in FinalBuilder 5.5 was limited to the Execute PowerShell Action. For FinalBuilder 6, we've made PowerShell a "first class" scripting language, alongside JavaScript and VBScript.

This is the first in a series of blog posts which show the different ways PowerShell can integrate with FinalBuilder. Today's example is the Execute Condition.

Execute Conditions

Every action in FinalBuilder has a field called the "Execute Condition", on the Runtime property page.

This lets you place a single line of script which evaluates as a boolean condition. If the script evaluates "True", the action will execute. If the script evaluates "False", the action will be skipped.

PowerShell Execute Conditions

Due to it's terse but expressive syntax, PowerShell seems to lend itself to one-line expressions (a quick Google search will confirm this.) This can be great for execute conditions.

Examples

If you want an action to only run if Notepad is executing, you can use this one-line expression:

get-Process | where {$_.name -eq "NotePad"}

If you want an action to run only if a remote computer is online, you can use this expression:

gwmi Win32_PingStatus -Filter "Address='remotePC'" |where {$_.StatusCode -eq 0}

PowerShell treats any non-null value as equivalent to $True, so any result will be regarded as True, while no result will be regarded as False.

Caveats

I think one-liners are neat. It's probably because they remind me of growing up with my Apple //e and Nibble magazine's one-liner competition. However, like anything else, one liners can be abused. They can be the stuff of programming horror stories.

My suggestions for responsible Execute Condition use:

Put something in the action comment field if the Execute Condition isn't self-documenting.
Use a script event if the Execute Condition gets longer than the text field on the Runtime page.

Next - Part Two : Script Events

( Credits : thanks to beta tester Robert for the Notepad suggestion.)

Team Foundation Build and FinalBuilder 6 (a match made in heaven)

Fri, 14 Mar 2008 05:00:00 GMT

I just posted a new article on Integrating FinalBuilder with Microsoft Team Foundation Server. The article deals with the great new Team Build integration in FinalBuilder 6.

You can now do things with FinalBuilder 6 and Team Build which are impossible to achieve with any other tool combination, and you can do it all without editing any MSBuild files at all.

Need MbUnit test statistics in your Team Build Quality Report? Do it with FinalBuilder.

Need to deploy build results via FTP? Do it with FinalBuilder.

Need to build legacy Visual Studio 2003/Delphi/C++ Builder/VB 6 projects with Team Build, and see compiler metrics in the Team Build Report? Do it with FinalBuilder.

Need smart version information management (including auto-incrementing version numbers across projects) without having to use custom C# code? Do it with FinalBuilder.

Need to do perform any of the thousands of other built-in operations, with full graphical configuration and structured logging output? Do it with FinalBuilder.

If you're interested, check out the new article here.

I also wanted to plug a great (FREE) Screencast creator called Wink, by DebugMode Software. Paul put me onto this and I used it to create the Flash videos in the article. We also have a license to an expensive commercial product, but I don't think I'll ever want to use it again now that I've found Wink.

FinalBuilder 6 Tech Article - .NET custom action API

Tue, 04 Mar 2008 06:00:00 GMT

I just posted a new tech article about the FinalBuilder 6 .NET Custom Action API.

If you've ever done any custom action development with .NET, then I strongly suggest you check this out. It's a really nifty API and it makes designing FB 6 actions incredibly simple.

In addition, it is still possible to tweak the API slightly to add functionality before FB 6 goes final - so please let us know if there's anything that you'd like to see added.

"The Bus Factor", Software Development practices, and FinalBuilder

Wed, 27 Feb 2008 06:00:00 GMT

(This blog post is shamelessly ripped off from based on Roy Osherove's praise of FinalBuilder(*) during a session at TechEd Barcelona last year.)

"[The Bus Factor] means if the developer who wrote [the project] is hit by a bus, you're screwed."

- Scott Hanselman

It's common to talk about the Bus Factor of a software project or a piece of code. Specifically, the Bus Factor is the number of people on your team who need to be "hit by a bus" (or change jobs, or change projects, or move away, or retire, or get sat on by an elephant) before the project is in trouble. The hallmark of a bad Bus Factor is having specialised, hard to access, knowledge which is only understood by one or two people on your team.

FinalBuilder encourages a good Bus Factor for builds.

To help demonstrate this, here's the Nant build process for one of my favourite .NET Open Source tools, NUnit:

(Start scrolling now...)

(**)

The build process is 2,688 lines of XML, spread amongst 33 XML files. I combined them all into a PDF to create the images shown above. NUnit uses NAnt, but if you're using MSBuild then your build process will look very similar.

Comparatively speaking, this a simple build process. The entire NUnit codebase is only around 70 thousand lines of C# code. The build above doesn't do anything particularly complex. Imagine how much worse this gets if either the codebase, or the level of functionality, needs to grow.

Would you want to be a new developer in charge of working on that build process?

For comparison sake, I recreated most of the NUnit build functionality in FinalBuilder (this is based on a rolling demo that we used at TechEd Barcelona.)

The FinalBuilder project doesn't do absolutely everything that the NAnt script does, but it could be extended to fill in the gaps without changing the appearance drastically from what you see here. The FinalBuilder project also does do some things that the NAnt project does not do - namely version control integration and FTP deployment.

Ask yourself: Which type of build process would I like to see deployed in my organization?

Nitpicker's Corner

This isn't an entirely fair comparison, because lots of the details about how the FinalBuilder project works are hidden away under the GUI abstraction. That's my the point, though: In FinalBuilder the project details aren't all in your face, but they're easy to find and understand if the person who wrote the build process happens to get hit by a bus.

Footnotes:

(*) For the record, we didn't ask Roy to praise FinalBuilder, and we definitely didn't pay him to do so. He said it all off his own bat. I hope he doesn't mind me quoting him. (back)

(**) I don't mean to pick on NUnit here. Naturally, OSS development involves different factors which come into play when choosing tools. That said, we'd be happy to offer the NUnit project a free license (or two) of FinalBuilder for use with internal build and configuration management. (back)

Safely Using Relative Paths in Actions

Tue, 26 Feb 2008 06:00:00 GMT

Using relative paths in a project is a good idea - they make a project more maintainable, readable and portable.

However, lately I've seen quite a few sample FinalBuilder projects where people are using this kind of relative path in their actions:

MySolutionFile.sln

.\MyProject\Installer\Project.iss

..\..\MyOtherProject\MyProject.dproj

The idea is that if the file is located relative to the project file path, the relative path will resolve to the right directory.

Great idea, right? Unfortunately, no.

These relative paths rely on the current Windows working directory for the process. Unfortunately, FinalBuilder and Automise only have limited control over this property. Various API calls (made by actions, scripts or the IDE) might change this directory unexpectedly.

The safe way to use relative paths in a FinalBuilder or Automise project is to use the variable %FBPROJECTDIR% (called %PROJECTDIR% in Automise.) This variable evaluates to the directory which contains the current project file.

Unsafe	Safe
MySolution.sln	%FBPROJECTDIR%\MySolution.sln
.\MyProject\	%FBPROJECTDIR%\MyProject\Installer\Project.iss
..\..\MyOtherProject\MyProject.dproj	%FBPROJECTDIR%\..\..\MyOtherProject\MyProject.dproj

We recommend defining new project variables to prevent these paths from becoming too long and unwieldy, and to make your project easier to read.

For instance, you could define a project variable called InstallerPath and set it to %FBPROJECTDIR%\MyProject\Installer. Turn on the "Macro" option for the InstallerPath variable to make sure variable references are expanded each time the variable is read.

XML Actions and the "XPath returned no node" error.

Mon, 25 Feb 2008 06:00:00 GMT

(It's been a while since I blogged about anything, so I thought I'd write about this while my coffee kicks in on a Monday morning.)

An issue which comes up quite often is people seeing this error when they try to use XML actions in FinalBuilder or Automise:

XPath returned no node : /Project/Import

The reason is because of the way MSXML deals with document namespaces. If your XML file uses XML namespaces (and most do), then you need to tweak the action slightly.

I'm not going to go into XML Namespaces in detail. There are better places to learn about them. I'll just show you an example:

Example

Say you have a Visual Studio 2005 .csproj file (which is really just an XML file wearing a funny hat.) You want to iterate over all of the imported project names with an XML Node Iterator action. If you try to use the XPath /Project/Import, then you'll see the error I showed above.

Here's the first line of the .csproj file:

This means the document has a default XML Namespace, which is "http://schemas.microsoft.com/blahblahblah". This is why MSXML can't follow the XPath properly.

To fix this, open the XML action and navigate to the "MSXML Parser" page.

Turn on the "Automatically use namespace prefixes..." checkbox, and type a letter in the field for the default Namespace. I used "x" in the screenshot shown above.

Now change your XPath from /Project/Import to:

/x:Project/x:Import

It works! Namespace prefix "x" now refers to the default namespace in the document.

Pedantic Details

There is a way to specify the namespace URI as part of the XPath, without using this prefix feature, but the resulting XPaths are enormous and hard for humans to read.

If you have namespace prefixes defined as attributes in the document root node, ie s:xmlns="http://www.myfunkyurl.com/namespace", then turning on this option will let you use the prefixes in your XPath.

This is the best solution we've come up with so far. We're planning to revisit this in future and (hopefully) make it all automatic without any manual configuration or custom prefixes.

MSBuild Wish List vs FinalBuilder Feature Set

Fri, 07 Dec 2007 03:47:46 GMT

The MSBuild Team recently posted a "wish list" of potential new MSBuild features, and encouraged people to vote on which ones they would like to see in future versions. The results came out last week.

We have a lot of respect for MSBuild here at VSoft Technologies. We're committed to making our own MSBuild support top-notch so you can easily interoperate between FinalBuilder and MSBuild projects. However, I wouldn't recommend MSBuild for end-to-end build automation.

I couldn't help but compare the top ten most popular MSBuild feature requests with existing FinalBuilder features.

1) Debugger

FinalBuilder gives you a full graphical debugging environment - breakpoints, stepping, watches, pause and resume.

You also get a structured log which quickly allows you to move between your project's structure and it's output. No need to scan text log files for line numbers and target names.

2) and 3) These both relate to converting existing non-MSBuild-friendly formats to MSBuild. This problem is more or less bypassed if you use FinalBuilder, because the built-in action types give you "first class" access to building these project types.

4) Visualization

This is another feature that FinalBuilder gives you "out of the box" - FinalBuilder projects are represented as structured trees, and refactoring is often as simple as drag-and-drop.

5) Native Code / VC Support

FinalBuilder has built-in support for Visual C++, at the same level of integration as other tools.

6) Distributed Builds

FinalBuilder doesn't do this either. Yet.

7) Extensible reuseable inline tasks

Every single action in FinalBuilder has extensibility built in via VBScript, JavaScript, and soon PowerShell. (Ssh! Don't tell anyone I told you!) If that's not enough, with ActionStudio you can whip up a custom action type in minutes, using one of these scripting languages or .NET.

For further extensibility, resuseable action lists can be easily created and called from different parts of your project.

8) Multiprocessor Peformance

Async Action Groups let you automatically schedule parts of your build to run in parallel with others, allowing you to take advantage of multiprocessor architectures.

9) Extensible Functions

See No. 7. The inclusion of scripting throughout a FinalBuilder project makes it easy to extend specific fields.

10) (Editing UI)

This is FinalBuilder's core feature. The FinalBuilder IDE gives you a clear graphical layout of your build process, and combined with debugging and structured logging you have a quick and easy way to move between project structure and project output.

"I have Microsoft Team System, why do I need FinalBuilder?"

Wed, 07 Nov 2007 14:19:44 GMT

I'm writing this entry from a delegate work area at TechEd Barcelona. We've been talking to a lot of people over the past two days about FinalBuilder.

Many people have said "Oh, we have Team System. We already have automated builds." The surprising thing is how many people don't realise what they're missing out on, or that FinalBuilder can integrate directly with Team System.

With Team System you can do this in your build:

Get a fresh copy of your project's source
Build one or more Visual Studio 2005 solutions
Run Visual Studio tests & Visual Studio code coverage analysis
Deploy built applications to a network share

... that's about it.

Not that this is a bad thing - sometimes all you need to do is to build a solution, and you're done. It's still an automated build, and it's a world better than F5 as a build process.

However, with FinalBuilder you can easily extend Team Build to:

Change the build process without editing XML
Automatically update and increment version information
Build solutions with any version of Visual Studio or MSBuild
Create, edit or remove Work Items based on build outcomes
Deploy to multiple locations, or via FTP
Build third party projects
Build installers
Run tests using different test frameworks
Build multiple versions or editions of the same product
Build legacy code (possibly using different tools)
Retrieve source code from legacy source repositories
Generate structured build logs, with the ability to instantly jump to any points of failure.

FinalBuilder slots directly into Team Build as a build engine, in place of (or beside) MSBuild.

At TechEd, many people have recognised that they face one or more of these scenarios. A few have no other build requirements, but a lot of people have said "Oh, we've been doing that manually" or "Yes, we've been doing this with MSBuild but it's hard and it keeps breaking."

These are the cases where FinalBuilder can help create a true end-to-end build and release system, one which saves time and money and helps safeguard build quality and avoid last-minute surprises.

... and no more XML files!

Blogging from TechEd in Barcelona

Mon, 05 Nov 2007 06:00:00 GMT

After a great two weeks holidaying in Germany and Spain, I'm back at work as we set up our booth at TechEd Barcelona. I'm looking forward to seeing some demos of new Microsoft technologies.

We also have some exciting things to show in our booth here - a FinalBuilder build integrated with Team Foundation Server, some sneak peeks of the next FinalBuilder version, and some nifty Automise demos. Come by the Exhibitor Hall if you get a chance.

Automising the Calculator

Fri, 10 Aug 2007 01:06:55 GMT

NB: The project includes delay actions so you can see what's going on. Disable them, and it will run even faster!

Want to try it yourself? Download the Automise 2 Release Candidate, and the Calculator Demo project file (the demo is also included in the Automise 2 installation, under Examples\GUI Automation.

This demo may not be particularly useful, but it shows off some of the power of the GUI Automation actions included in Automise 2. The demo project was created and tested in a matter of minutes. I've written a longer article on the subject, if you are interested in reading some more.

CruiseControl.NET Plugin Released

Thu, 19 Jul 2007 05:00:00 GMT

EDIT: CruiseControl.NET version 1.3 now ships with an integrated FinalBuilder Task. No more need for a plugin.

A few people have been enjoying this for a while, but you can now download an installer for the FinalBuilder CruiseControl.NET plugin. The plugin makes FinalBuilder available as a CruiseControl.NET build engine for continuous integration. The plugin is free, and works with both FinalBuilder 3 and 4 (although FB 3 users will need the Professional Edition.)

The current version of the plugin is for CruiseControl.NET 1.0, and we will make sure it keeps up to date with the current stable release.

Runtime Assembly Reference Loading and the CLR

Fri, 06 Jul 2007 05:00:00 GMT

Following on from my earlier post about Assembly resolution and CLR versions, I have an observation to share about the .NET Just-In-Time compiler and the resolution of Assembly references. It's a fairly simple observation, but it came as a surprise to me.

DISCLAIMER: This is just an observation. I can't find any real concrete specification of the behaviour I'm describing. My supporting documents are this PowerPoint presentation, and this doctoral thesis. If you have better references, or consider this either blindingly obvious or blindingly wrong, please post in the comments.

I used to think that if "Assembly A" references another assembly, "Assembly B", then Assembly B is loaded (or at least validated) when Assembly A is loaded. If Assembly B is missing, Assembly A won't run. Am I right?

It appears I am not right. The rule is that Assembly B will not normally be resolved (ie checked for existence and loaded) until the JIT compiler encounters some code which references a type belonging to that assembly. You could delete Assembly B entirely and Assembly A would work perfectly, providing it didn't touch Assembly B's types in any of the codepaths which you take.

I put together a simple pair of C# projects which demonstrate this behaviour. You can download them here.

The projects build two assemblies, ResolutionTest.exe and MissingAssembly.dll (such creative names!)

MissingAssembly contains one class, MissingClass. ResolutionTest contains a static Program class and another class, MainClass. Here is the code for MainClass:

(This is the executive summary version. The actual source has some other attributes and comments about inlining. I'll get to that in a minute.)

Here's what happens when it runs:

No great surprises. A MainClass object is instantiated. The constructor calls the SubMethod() method, which then creates a new object of type MissingClass from the MissingAssembly.

Now, suppose we delete MissingAssembly:

An exception is raised because it can't find the MissingAssembly object. However, the MainClass() constructor still executes. The FileNotFoundException occurs when the call to SubMethod() is made. The JITter has examined SubMethod() and said "I need a MissingClass which is held in assembly MissingAssembly. I'd better go find that...", and then failed to find it.

Why is this worth knowing?

Because (I think) it means that you can "cheat" when binding assemblies. Previously I'd used late binding via Reflection (like this) when I didn't know if something might or might not be installed on the end user's machine. As far as I can tell, this approach is not wholly necessary - If the "third party" type references are all encapsulated nicely away from the "main" code, it should be possible to just use exception handling to react if the types aren't available at JIT-time.

It's also worth knowing because even if you call Assembly.Load() at runtime without any problems, it doesn't mean all the references have resolved. The loaded assembly could still break when you go to use it.

Some things to remember:

This isn't something that you can should use every day. You can dig yourself a big hole of shoddy design by doing this. In my case, I just needed a clean way to disable some functionality and fail cleanly when optional assemblies weren't available.
The assemblies still have to be available when the source is compiled to IL. You can't compile IL which references types or assemblies that do not exist.
This knowledge can help you optimize your code, by controlling when extra loading and JITting is performed.
There is a chance that the JITter will inline lightweight method calls. If MainClass.SubMethod() is inlined into the constructor, then it's possible that the constructor will fail. I originally tried to avoid this by attaching MethodImplOptions.NoInlining to SubMethod(). However, in the end I couldn't make it inline the code under any circumstances. It's possible that the JITter notices the unloaded type inside the method, and decides not to inline for that reason.
It's a bit of a hack, but you could dynamically link to one of several available assemblies at runtime, by going through a series of alternative wrapper classes (each referencing different assemblies) and finding the one which loads successfully. If all of the wrappers implemented a common interface, you could then use the components interchangeably with acceptable performance. A more generalised CLR-supported model of this is described in the doctoral thesis I linked to above (he demonstrates dynamically choosing an alternative to Windows.Forms for GUI display when running Rotor on OS X or Linux.)
This probably doesn't work if you've used ngen.exe, for obvious reasons.

Happy coding!

Automise, FinalBuilder and XML Formatted Data

Thu, 05 Jul 2007 22:36:17 GMT

Now and again people comment that they think FinalBuilder or Automise uses proprietary binary formats. I sympathise entirely. Binary formats are a pain to work with in version control, and a format like XML can be great when you need to make a sweeping change or tweak something. (If the people who make our help authoring tool are reading this post, please take note!)

The good news is that all of FinalBuilder and Automise's data is available via XML or raw text, you just need to know how to look for it!

The project files are XML

Both Automise and FinalBuilder save their project files as XML. If you save in the .FBP5 / .ATP2 "uncompressed" project format, then the project file is actually just an XML document. The XML isn't exquisitely formatted for human eyes, but a machine will be able to read it just fine..

If you save in the .FBZ5 / .ATZ2 "compressed" project format, the project file is compressed using industry-standard zip compression. If you rename the file to .ZIP, and open it in a zip program (like 7zip), then you can view or extract the uncompressed project file inside.

To choose whether to save "compressed" or "uncompressed" by default, go to Tools -> Options -> General IDE Options -> Design Time Options:

Project logs can be exported as XML

OK, you've got me. Automise and FinalBuilder project log files (.fbl5 and .log2) are a proprietary database format. In the old days, FinalBuilder 1 & 2 used a text formatted log file (or so I'm told.) This led to performance problems with large log files and long running projects. Hence the move to an embedded database engine.

However, you can still export any project log as XML. To export a single log, go to the History tab, select a log, and click "Export Log to XML" (or HTML, or plain text.) To export the current log automatically as part of a running project, use the Export Log action:

Logs can be exported as text, HTML, or XML. XML formatted logs can be further tweaked via XSLT. It is possible to pass data from FB/AT to the XSL stylesheet, via the use of extra XSLT parameters. If you're interested in writing an XSL transform for the Export Log action, take a look at the "Stylesheets" directory in the FinalBuilder or Automise program directory. It contains the XSL files which are used to do the exports for text, xml and html.

The Other Project Files are INI Files

Any extra project files (like the FBD data file, or the FBV persistent variables file) are actually INI files which can be viewed in any text editor. Go ahead and try it!

... armed with this knowledge, you should be diffing, merging, checking in, parsing, cutting, pasting and transforming in no time!

Four different flavors of .NET Assemblies and the CLR

Tue, 03 Jul 2007 05:00:00 GMT

(This is a technical post about some of the "gotchas" when coding for .NET on 64- and 32-bit architectures.)

The short story:

The TeamFoundation client assemblies are x86-only assemblies. They don't work when called from an x64 .NET Application.

Even though they're in the GAC, you'll get a FileNotFoundException if the CLR tries to load them.

My story:

I already knew this, but it took me most of the day to realise that was my problem.

The long story:

Microsoft's .NET CLR 2.0 supports three different processor architectures: x86, x64 and IA-64. This means that there are three different CLR environments. x64 machines can host the x86 CLR alongside the native 64-bit CLR.

Although the CLR executes architecture-agnostic Common Intermediate Language (aka MSIL) code, any assembly can be marked as only suitable for one of the particular architectures.This is what happens when you choose a "Platform" type for a Visual Studio project. Choosing 'Any CPU' means that the project Assemblies are marked as language-agnostic CIL code.

It all plays out like this:

CLR Type	Can Be Hosted From	Can Load These Assembly Types
x86	x86 or x64 Operating Systems	x86, "Any CPU"
x64	x64 Operating Systems	x64, "Any CPU"
IA-64	Itanium Operating Systems	IA-64, "Any CPU"

Or, alternatively:

Assembly Type	Can Run On
"Any CPU" / MSIL	Any CLR
"x86"	x86 CLR Only
"x64"	x64 CLR Only
"IA-64"	IA-64 CLR Only

To find out if an assembly is x86-only, you can use the CorFlags.exe utility which ships with the .NET SDK. You can also use this tool to tweak the assembly type, although doing this will break strong named assemblies and generally cause chaos.

When loading a.NET-based process, the .NET Framework decides which version of the CLR to load based on the assembly type. Processes based on "Any CPU" assemblies will load in the native CLR for the host system.

This can lead to some funny situations. For example, the ASP.NET Development Server application is 32-bit only, so any web sites you run on it will run in the x86 CLR. On the other hand, on x64 IIS can run either 32- or 64-bit ASP.NET applications. This means you can move your ASP.NET application from the Development Server to IIS on the same machine, and all of a sudden you will get a BadImageFormatException when an assembly refuses to load any more.

The Twist:

Sometimes you get a FileNotFoundException instead. This is when you try to reference an Assembly which is in the GAC.

... and you think "That's odd, I thought that assembly was in the GAC." and you look in the GAC browser, and there it is:

... the key is the "Processor Architecture" column on the righthand side.

As of .NET Framework 2.0, the CLR actually has four different GACs - one for each architecture, and one for "Any CPU" MSIL Assemblies. You can see this quite literally if you disable the GAC Shell extension, and browse to the GAC directory.

Each version of the CLR can only load assemblies which exist in its own GAC, or in the architecture-agnostic MSIL GAC. If you try to load an assembly from a different GAC, you'll see the FileNotFoundException.

Windows Vista compatibility

Wed, 23 May 2007 05:00:00 GMT

Now that FinalBuilder 5.5 is out, I've posted a short article detailing some of the things to keep in mind when using FB with Windows Vista and User Account Control (UAC.) The article can be found here.

On a more techy subject, most developers are probably already well and truly up to speed with Vista integration by now, but here are a few resources that I found invaluable while getting a grip on the concepts of UAC, manifests, elevation, virtualization, and the elevation prompt.

User Account Control Overview at Microsoft TechNet. A good semi-technical (code-free) overview of UAC.
Evolving the Software Organism: Default Namespaces and UAC Manifests in Windows Vista. Covers some gotchas when embedding a UAC manifest.
Kenny Kerr's Windows Vista for Developers (Part 4). Great code summary of the two ways to request UAC elevation in Vista. Creating a process with an elevated token, or creating an elevated out-of-process COM object.
Windows Vista Application Development Requirements for User Account Control Compatibility. 500-pound-gorilla Word document, covers everything you ever wanted to know (and then some.)

Happy Vista-ing!

Mixing up FinalBuilder with Team System (no more XML files!)

Thu, 17 Aug 2006 14:00:00 GMT

One of the comments that we frequently hear from users is "I couldn't do xyz with MSBuild, but I can now with FinalBuilder."

This is great, but hardly surprising. FinalBuilder includes out-of-the-box support for many more tools and processes than MSBuild. Not to mention that it has a GUI to set up your build process - no more editing XML files!

Lately, though, an increasing number of our users are moving to Microsoft Team System. One of the cornerstones of the Team System process is the Team Foundation Build engine, which is built on top of MSBuild.

Now, the Team Foundation Build wizard is fine if you only need to create simple build processes. Get some files, build some solutions, deploy some binaries to a file share. If you need to do anything more, though (and who doesn't?), then you're stuck back in XML land... The same applies if you want to edit your build processes after you've created them.

Plain XML editing, with no built-in support for third-party tools, no support for data processing, no support for parallelisation, FTP transfers, Active Scripting, structured exceptions, recursion or Windows system functionality...

FinalBuilder to the rescue!

The Team Foundation Build engine allows you to add custom MSBuild tasks at any stage of your build process. This means that, with just a couple of lines of XML, you can move your entire Team Build process to FinalBuilder! FinalBuilder can be used in place of MSBuild to provide functionality at every stage of the Team Foundation build process (fetch, compile, test & deploy.) What's more, it all fits perfectly within the Team System framework!

To see how, take a look at the walkthrough I just wrote: Integrating FinalBuilder with Microsoft Team Foundation Build.

... Happy building!

File Dependencies

Thu, 17 Nov 2005 01:18:00 GMT

Have you ever wished you can have your build check whether or not files have changed, so you know whether or not to rebuild?

With FinalBuilder 4, you now can! Check out the new File Dependency action (you will need the latest public beta.)

This is a skeleton action list that does two things:

It synchronizes a local copy of the source files (via SurroundSCM Get.)
It checks the file dependencies to see if anything has changed. Lets zoom in on the File Dependency action:

In Files and Folders to check, I'm telling FinalBuilder to look at the modification dates of all the main FinalBuilder source and resource files. Because I've chosen Recurse into subdirectories, FinalBuilder will scan the subdirectories of each for all files of those types.

So, how do we know if the files have changed?

As you can see, I've told FinalBuilder to check for any files which have changed since the last time the action was run. In the Options section, I've elected to have FinalBuilder ignore any files created within two minutes of the last run, to compensate for jitter in the modification dates.

You can see in the greyed-out “Use date from file“ section that I could have elected to compare the modification dates of the compiled libraries, in order to be really sure whether or not to rebuild.

In fact, you can build a simple continous integration tool with just one more action:

Example1 re-schedules itself (using the Schedule Add action and the Windows Scheduler) to run 10 minutes after it finishes, but only starts a build if there are files that have changed.

Example 2 is even simpler - it loops continously checking for changed files (with a short delay in between each loop.) I also put in a separate check so the help file is only rebuilt if it has changed.

CruiseControl.NET

Wed, 05 Oct 2005 02:46:00 GMT

I've also been working on a FinalBuilder plugin for the open source continuous integration tool, CruiseControl.NET.

We're still testing the plugin right now, but it will be released in time for FinalBuilder 4. You can add a FinalBuilder project to CruiseControl.NET with just two XML tags!

If you would like to beta test, please let us know.

New XML Actions

Wed, 05 Oct 2005 02:32:00 GMT

I'm now into my fourth month here at VSoft, so it must be about time for a first blog post!

I've been working on all kinds of new actions for FinalBuilder 4. I think lots of you who requested features will be very happy with the upcoming release (and everyone else should find something useful, too.)

Today I just wanted to demonstrate two new XML actions : the XML Iterator and the XML Read from Variable Action. Combined with the already strong XML support in FinalBuilder, you can work with data from almost any XML file. Allow me to demonstrate:

Here's the project:

The XML file I'm iterating is a 3Mb chunk of the Javanese edition of the Wikipedia, exported as XML. I'm going to catalogue all the users who made edits, and record their usernames as well as any comments they recorded. I don't want to record revisions by the automatic MediaWiki bots.

So, I iterate through the XPath /mediawiki/page/revision, saving the absolute XPath in to each revision in the 'Page' FinalBuilder variable:

Once I have the absolute XPath to each node, I can read the 'Comment' and 'Contributor' tags into FinalBuilder variables. Here's the action which reads 'Comment':

The 'Comment' tag is optional, so I use a Try... Catch block to blank out the comment if there isn't one.

Together with a 'Write Text File', these actions are all you need to create a list of contributors and their comments.

This sort of technique can be used to parse any kind of XML file as part of your build process. If you need to, you can even create an XML file which gives instructions to your build process. But who'd want to do that?