When Automise 1.0 came out, many people commented on the lack of a generic way to interact with the Windows user interface. They requested a way to interact with dialogs by sending mouse clicks and keyboard input. We listened, and the new Automise 2.0 release candidate (available for download now) includes new actions for GUI Automation:

Action Types

Action Types

Action Summary
Wait For Window or Control Use this action to have Automise wait until a specific window or a control appears on the screen. This can be useful when you're starting a new application, or if you want to monitor a running application for a dialog box.
Perform Mouse Click Use this action to move the mouse and click on a window or control anywhere on the screen.
Perform Mouse Move Use this action to hover the mouse over a window or control, without clicking.
Send Keyboard Input Type keyboard input, or send special key combinations like Control-A or Alt-F4.
Set Scrollbar Position Move any standard Windows scroll bar or scrollable control.

Example Project

I created this sample project to demonstrate the capabilities of the new actions. You can download the project here in order to test it out in Automise 2.
The project starts the Windows Calculator accessory and makes some simple computations, including changing the calculator mode to "Scientific" and moving the result into hexadecimal and back. This isn't intended to be useful, it's just a neat demo that doesn't require any special software in order to run.

Calculator Demo Project

Action Groups

The project contains these action groups:

Launch Calculator Starts the Calculator Accessory by using the Execute Program action, and waits for the main window to open.
Switch to Scientific Mode Switches the calculator to scientific mode via the View menu.
Decimal Switches the calculator to Decimal mode by pressing a button.
Calculate 22+1001... Calculates 22 + 1001, using a mixture of buttons and keypresses.
The project pauses so you can see the result.
Alternative Calculation Method This disabled action group provides an alternative method for calculation (see below.)
Switch to Hex Switches the calculator to Hexadecimal display by pressing a button.
The project pauses so you can see the result.
Switch back... Switches the calculator back to Decimal display.
Close Calculator Closes the calculator window by typing Alt-F4.

Configuring the Actions

For this example I'll show how to configure the action which presses the "2" button on the calculator. All of the actions were configured in the same way.

When you first add a "Perform Mouse Click" action to the project, it looks something like this:

Perform Mouse Click

By default, the action is set to find a window by using the Window Class and/or the Caption. However, we're going to use a different option.
If you click on the "Find a control in a running application" button (arrowed, above) then Automise will disappear and the following dialog will pop up:

Find Control...

You'll find that as you move the mouse, a red rectangle appears around the current window or control. Here is what happens if you move the mouse over the "2" button in the Calculator and hold down the Control key:

Calculator with Highlight

... you can see that the "2" button is highlighted in Red. As well as the highlight box, Automise also pays attention to the exact position of the mouse over the window.
Tip: Because of the way some windows are drawn, the highlight rectangle does not always appear. This does not stop you from control-clicking to make a selection!

After you control-click, the action's property page will look like this:

Automise now knows how to find the "2" button inside the Calculator window. This is all you need to do to configure the action. Automise generates a "Description" which it then uses to find the control again. This description is relatively simple.
Tip: The "descriptions" are actually snippets of script code. A future Automise 2 update will include a complete scripting API so that you can use these scripts in JScript or VBScript, if you want to.

If you wanted to right-click, control-click, or double-click on the control, you can set this by moving to the "Click At Point" tab.

All of the GUI Automation actions can be configured in this way. I recommend that you take a look through the demo project to see how they all work.

Alternative Calculation Method

When Automating GUI applications, you'll often find that there is more than one way to achieve something. The demo project includes an "alternative calculation" action group which shows a quicker way to enter the main calculation, using the keyboard.

If you disable the "Calculate..." action group and enable the "Alternative Calculation" group, the project will look like this:

Alternative Project

... this project does the same thing as the original project, only it's slightly faster.

Putting it all together

Here's a Flash Video that shows the project in action, running on Windows Vista:


What Now?

Download the Automise 2 Release Candidate, and the Calculator Demo project file, and try it out yourself.

Then see what uses you can find for the GUI Automation actions!

Posted in: Automise
Actions: E-mail | Permalink |