Beta Feedback needed!

Continua CI Discussion
1

Hi All

We desperately need some feedback. We're using Continua CI in house to build itself and a bunch of open source projects, and it's working really well, but then we know how to use it  

The problem is we really have no idea if any beta tester has actually managed to build anything with it. Please let us know how it went.  I realise the the doco is still a bit thin, we will get it all filled out before release, if there is anything that you need help with to progress then please do let us know. If you have any installation issues then we really need to hear about them. 

We're zeroing in on a release candidate in the next week or two.. so we do need your help. 

2

Hello,

 In general I'm impressed what I see in Continua. It still seems to be bit shaky, there are incomplete things (mostly in the docs), and some rough edges, but in general I like it very much.

Some miscellaneous observations:

* While the balloon error messages shown in the web interface are certainly nice looking, it's rather infuriating that you cannot copy text from them. It makes reporting and/or discussing any errors bit hard.

* The actions need some more inline help, examples and so forth. There is the image shown besides some fields indicating it can accept Query Expressions, but nothing really explaining what and how the field should contain. Take for example the FinalBuilder -action and the variable -tab - it could certainly benefit from some guidance as what's expected and can be put in the field. Another example would be the Visual Studio -actions Artifacts -tab, I have zero idea how it should be used. Yes there are search paths, for what and how they should be used? There is also instruction text "Relative to workspace", but it's not really clear to me to what it refers to. Also the context for the Output Directory -field is not clear, is it intended to be the directory where the artifacts should be moved, or the output directory of the project, should it be relative or absolute? These are the things new user wonders about the actions, and some assistance inline would be beneficial there.

* The number of available actions are bit dissapointing. Most glaring omissions are for example Delphi compilers, and further string and file/path manipulation actions. Currently the set offered is in many cases _almost_ enough to replace FinalBuilder -scripts, but not quite.

* Using of FinalBuilder scripts should be easier especially related to specifying the variables. Something like what's offered by the FBS would be most welcome. Also consider limiting the variables offered for use by Continua for example adding the capability to explicitly specify a variable as "interface variable" in the FB script.

In more general level some best practices, ideas and guidance on how to migrate FBS based infrastructure in general level would be beneficial (in addition to feature level instructions). We have a complex FB(S) driven build system, which builds software, signs the binaries, creates localization dictionaries for them, builds support material, and installers. The whole package (binaries, support files from version control, dictionaries, support material in number of formats, and multiple installers) is constructed in a specific directory structure and finally copied to a network share from where it's taken for tests and installations etc. It's not clear for us how should we create this kind of functionality in Continua. The Artifacts -concept is bit unclear also for us, but would not appear to be the suitable answer - our typical build folder is bit over 2 gigabytes in size, and contains approximately 9400 files in 300 folders, so registering them as artifacts would probably not be a good idea.

3

Hi,

I just noticed one other issue that probably should be addressed: Detecting the Visual Studio 2010 does not work correctly.

 

Microsoft is using the Visual Studio shell as part of other products. Among these products is the SQL Server Management Studio 2012. Installing SQL Server Express 2012 installs significant portion of the Visual Studio to the system - including the devenv.com at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE Continua uses to determine if Visual Studio is installed. However the parts installed definitely won’t allow building software, so there is distinct possibility this causes issues where Continua allocates stages to an agent that is not really capable of building them.

 

I recall that the VS shell is used also in other new products, so this may not be isolated to SQL Server Express.

 

4

Hi Olli

Great feedback, thank you!

We’ll try to get all of these issues addressed asap.

The balloon error popup is being changed as I type to not close when you click on it and will have an explicit close button.

We’re looking at ways to give the actions more inline help, better default values, for example for a FinalBuilder Action it might have a default value of “$Source.YourRepo$\PathToProject\YourProject.fbp7” or something like that. Of course things will also be clearer once we actually document the actions…

With regards to FinalBuilder projects and variables… this is one of the areas that is not as good as FinalBuilder Server. The problem is that at design time we do not have access to the FinalBuilder project file because it resides in a version control repository, and is copied into the builds workspace at runtime. This is unfortnate but it is a side effect of each build having it’s own workspace (which enables multiple builds of the same configuration to be built concurrently), it’s a compromise that we feel was work making, in that we intend Continua CI to be for a wider audience than just FinalBuilder users. Of course we want FinalBuilder to work better with Continua than it does with any other CI tool… we’ll be evolving things over time to that end.

I will point out that the actions available in Continua are intended to be comparable with other CI server products, we are not trying to replace FinalBuilder. Continua will never have as many actions as FinalBuilder, and in general the actions will be much more coarse grained than FinalBuilder. If you have complex logic or fine grained tasks that you need to perform in your build process (and it sounds like you do) then FinalBuilder is still the best tool for the job. Obviously we need to do a better job of communicating that and how to go about that. Of course over time we will add more Actions (we’re still working on new actions right now).

For Delphi, you should use the FinalBuilder (which has comprehensive delphi support) or MSBuild Actions (depending on the delphi version). Embarcadero have decided that their command line story is now MSBuild… you can no longer just point dcc32 at a project and expect that it will compile without giving it a whole lot of information, some of which is not stored in the dproj and only known to msbuild support, and there are some undocumented options. I have argued against this with them but to no avail. What we probably do need to do is add default property collectors for delphi because the FinalBuilder and MSBuild actions only rely on their respective tools being available on an agent and have no way to indicate what else is needed (we’re looking at ways to improve this).

The Visual Studio property collector looks in the registry for the installation location of each version of visual studio. It’s unfortunate that the shell identifies itself as Visual Studio when it’s used for other products… we’re looking into other ways to identify whether its’s actually visual studio or something else.

The Artifacts concept is really intended for files that are the direct output of the build, for example the resulting setup.exe, or files that will be used as the direct input to another build (which will be the canse when we implement dependencies). We spend this morning discussing ways in which we can improve the management of artifacts and the output of stages in general… there’s a lot we can do but it’s not a small task so those optimisations will come in a point release.

Your feedback has certainly sparked a lot of discussion here today, it would be great if you could describe your build process in more detail, either here or in private if you prefer. Whilst we are never going to be able to cater for everyone, we would really like to make sure this is going to work for you if at all possible.

5
Posted By Vincent Parrett on 10 May 2012 11:56 PM

Your feedback has certainly sparked a lot of discussion here today, it would be great if you could describe your build process in more detail, either here or in private if you prefer. Whilst we are never going to be able to cater for everyone, we would really like to make sure this is going to work for you if at all possible.

 Hi,

 We can provide you with details of our build setup in private, and also provide access to our build scripts if they are of value to you. Please let me know if you want me to send those to you.

 

6

Hi Olli

Having a look at the build scripts would be very useful, send them to support@finalbuilder.com and we can discuss your setup in private.

7

Hi,

I’ll try to get around sending them to you later today.

 

 

8

* While the balloon error messages shown in the web interface are certainly nice looking, it's rather infuriating that you cannot copy text from them. It makes reporting and/or discussing any errors bit hard.

We've now changed the behavior of the balloon by adding a close icon and making the text selectable without closing the balloon.

* The actions need some more inline help, examples and so forth. There is the image shown besides some fields indicating it can accept Query Expressions, but nothing really explaining what and how the field should contain. Take for example the FinalBuilder -action and the variable -tab - it could certainly benefit from some guidance as what's expected and can be put in the field. Another example would be the Visual Studio -actions Artifacts -tab, I have zero idea how it should be used. Yes there are search paths, for what and how they should be used? There is also instruction text "Relative to workspace", but it's not really clear to me to what it refers to. Also the context for the Output Directory -field is not clear, is it intended to be the directory where the artifacts should be moved, or the output directory of the project, should it be relative or absolute? These are the things new user wonders about the actions, and some assistance inline would be beneficial there.

Thanks for pointing that out. We'll be updating a lot of these fields next week with (hopefully) better/easier explanations.In the mean time, if you wanted to know a bit more about artifacts you can have a read of the Register Artifact Action's wiki page: http://wiki.finalbuilder.com/display/continua/Register+Artifact+ActionThat action works the same as the artifact tabs you see in other actions.
9

With regards to FinalBuilder projects and variables.. this is one of the areas that is not as good as FinalBuilder Server. The problem is that at design time we do not have access to the FinalBuilder project file because it resides in a version control repository, and is copied into the builds workspace at runtime. This is unfortnate but it is a side effect of each build having it's own workspace


One thought concerning this:

At the time the FB projects are configured, the configuration is likely almost complete - at least in the sense that repositories (where the FB script among others reside) have been configured, therefore Continua should have access to the script at the time stages are configured.

Would it be possible to get a temporary copy of the script to an temporary location and parse the interface from that copy?
10

Hi Olli

Yes that’s certainly something we are looking at. One of the problems is that the repository initialization is asynchronous, so it may or may not be available at that time (for example cloning a 500mb mercurial repository can take a while).

11

Yes that's certainly something we are looking at. One of the problems is that the repository initialization is asynchronous,

I see the issue about repository synchronization. I was thinking more in the lines that as you know the repositories at that time and supposedly the FB script is in a path relative to an known repository, you could check out just the script to an temp location, in parallel of the primary repository initialization so it's not an issue should that still be in progress.
12

Couple of ideas for further development came up:

  • It could be useful to have a Path Existence property collector. Currently you can look up an executable(/file?) or  check access to a path. However checking existence of a directory is not possible?
  • EC2 build agents. We might want to use Amazon EC2 instances as build agents. In theory it could be done now with a preparation stage using a FinalBuilder script to fire up the prepared instances and a some sort of clean up stage using some way to shut them down, but it would be significantly easier if the support for this  would be built in to the system.
13

Hi Ollie

I’m not sure if there’s a path exists property collector, if not then we’ll add one this week, should be trivial to add.

As for EC2, we absolutely do want to add support for cloud based agents (not just EC2). It won’t be in 1.0 though, but since we’re planning frequent releases/updates it probably won’t take too long. Since there’s a cost involved in using EC2 we’ll need to add some smarts to our agent selection process (which is pretty basic at the moment). The current agent selection process only looks at agents that are online, and will only choose an agent that meets the requirements of a stage, so we’ll need to figure out how that all fits together.

One of the things we are working on at the moment is enabling SSH support between the agent and the server for repository and workspace synchronisation(using an ssh server we’re developing)… that’s a must to allow for agents outside of the local network (and it also allows us to deal with issues where the agent is no on the same domain and can’t write to the server share). We spent the last few days re-thinking how we move files/source around server & agents (and back again) and taking a general look at how to reduce the IO overhead. We have what we believe is a much better design, and we’ll be starting work on that tomorrow. It will mean pushing the release back a few weeks but will be worth it (should result in much better performance/reduced build times/lower disk space usage).

14

Hello,

 After some more playing with the current beta, some additional ideas came up.

* For debugging more complex build setups disabling a build stage would be welcome. The same applies to actions.

* Capability to re-use build stages in different project would be welcome. For example via some kind of stage pool or repository.

* Some kind of release management functionality (effectively a release stage that can be triggered separately).

Have you taken look at the build scripts I sent to you last week, and do you have any comments or questions concerning them.

15

One item I forgot there:

There is bit of a usability issue in Stage -edit view. Most views fit in reasonably sized browser window, but this is without exeption higher than the browser window. When some action is edited you must scroll down to get the save -button, without which all the changes made for the stage are lost without warning.

After the initial setup of a configuration I find that I often have to visit the Stages -editor, where it’s bit counter intuitive to have to go to the bottom of the screen to find the Save -buttons, and it’s often forgotten. There is also no affordance to figure out that without explicit Save -option at the Stage -level the changes performed to an action and acknowledged there are lost.

You could perhaps show the save button at the Configuration Wizard -header, if there are unsaved changes to an action in a stage, or alternatively save them automatically when acknowledging the action editor. At the very least show a warning if I’m about to lose changes to an action by leaving the view without saving.

16

Hi Ollie

* For debugging more complex build setups disabling a build stage would be welcome. The same applies to actions.

We'll look into this.. I though we had already implemented this so perhaps it's just a UI issue. 

* Capability to re-use build stages in different project would be welcome. For example via some kind of stage pool or repository.

This is  something that was on the feature list, but got cut in favor of getting the product out. It's high on the list for post 1.0 features.

* Some kind of release management functionality (effectively a release stage that can be triggered separately).

You can do this right now, on the Stage properties, uncheck the "Automatically promote to the next stage" checkbox on the second last stage. This will cause the build to stop on that stage, however you can manually kick of the next stage (promote). 

We do also have plans fore a Release Management module which will focus on managing the  deployment pipeline. 

* Have you taken look at the build scripts I sent to you last week, and do you have any comments or questions concerning them.

We have had a very quick look, and along with the other feedback you provided we decided to change the way we manage the build workspace and source code checkout to reduce wasted IO.

The current builds synchronise the entire workspace between stages (because it's not known at that time which agent the next stage will run on). Paul is currently working on that, there will be workspace rules that control what get's pushed between the server & agent and back again, and also what parts of the source code are checked out. This should speed up things a lot.. hopefully we'll see the results of that work next week. 

We're also working on SSH support for agents so that agents do not have to be on the domain (which is currently the case in order to access the server share). The repository caches will use SSH if they do not have access to the share, and workspace sync will use SFTP if the share is not accessable. This will also make it possible to use agents in the cloud. 

We should have another build (without the above changes) available tomorrow.

17

With regards to the Stage Editor, agreed there is more work to be done there (in particular to do with sizing), and that will happen over time. There is a Save All Stages button on the Toolbar above the action list. You can also click on the numbers at the top of the wizard pages to skip directly the page you want.

18
Posted By Vincent Parrett on 24 May 2012 04:56 AM

 You can also click on the numbers at the top of the wizard pages to skip directly the page you want.


I noticed that. However the most repeated case for me appears to be that I start to edit a configuration, click to the  Stages -item, do some changes to some action. After this I often need to go for example to define new variables etc. and if I do not explicitly save when clicking the Variables -item, the changes made to the actions are lost without notice.

 Obviously I'd prefer that no explicit save action is required, as the changes are already once acknowledged in the Action editor.

19

The issue is that there is that it should be prompting you if you go to move off the stage edit page without saving changes… it used to do that, something has broken that functionality… we’ll check it out tomorrow.

The stages etc are all edited in memory at the browser and loaded/saved in one service call… mostly I guess because of how they are serialized. When time permits we plan enhance the stage editor, right now though it works well enough and there are other issues to resolve (and limited resources).

20

Hi,

 Some items cropping up from the 1.0.0.423:

* Typo in Repositories -page or project wizard. "What is a Repositroy?"

* While the copy action now notes that the target files exist and will not be overwritten, it would be helpful if it would also include the intended target directory in the log. Now it's not possible to know from the log where it actually tried to copy the files to:

Copy [$Workspace$\Artifacts\Output] 12:42 12:42 44 milliseconds Success
Messages
Target for C:\ContinuaAgent\Ws\1038\Artifacts\Output\TestAutomationFramework.dll exists and was not overwritten
Target for C:\ContinuaAgent\Ws\1038\Artifacts\Output\TestAutomationFramework.pdb exists and was not overwritten
Target for C:\ContinuaAgent\Ws\1038\Artifacts\Output\TestSmokeClientsInstallation.csv exists and was not overwritten
Copy action completed successfully

* SVN repository gets royally confused if the repository URL is changed.

* SVN repository definitions appear to forget the password supplied if repository is edited. Validation works only when password is supplied, not in any subsequent visits to the dialog.

* I reported separately an issue of Agent workspace init failure with the SVN branch aware repository type.

* Debugging repository initialization issue is bit of a pain as the UI does not indicate any issues.

* Debugging or reporting Agent workspace initialization issues is also bit of a pain as the actual error is only available from the Diagnostics tool that should be run on the agent.

* Agent versions are not visible from the server.

* Event log page does not handle long lines gracefully. Controls are pushed out of the right edge if log contains long lines, and there is no indication that there are any controls there.