Am I right in saying that iterators are currently all synchronous - that is, items happen one after another? Is there a way to carry things out two at a time (or more? The only way I can think of is to split the list in two, double all the actions and group them in an async action group, as follows:
Original:
List iterator (A-Z)
-Copy file
Replace with:
Async action group
-List iterator (A-L)
–Copy file
-List iterator (M-Z)
–Copy file
Would it possibly make sense to add a “execute _ actions at once” option on iterators?
I’ve only just discovered this machine has two CPU’s, you see…
Steve
It is on the todo list to R&D the possibility of “asyncing” iterators to allow them to take advantage of multiple cpu’s.
One possibility is that you run your list iterator in parallel with something else, eg.
+ Async Group
+ List Iterator
| + Do Something with each list item
+ Create Installer
+ Something else here
hth…
.t8
Hi Steve,
If you’re doing things like copying files (on a single drive) then your two CPUs won’t be of much use anyhow.
It’s if you have CPU intensive activities, or things on different I/O pathways (maybe two drives, or a local and a network copy, or two FTP uploads to remote servers) that you’d see a speedup.
- Angus
Ah yeah, sort of figured as much. I was surprised not to see much difference with zipping files (around 5%) - obviously the “zip” is a relatively small part of the total work of reading source files and writing them out again. Or maybe the compress file action is multithreaded anyway…
I may try some cute refactoring just in case. Currently I have something sort of like:
Compile
-Zip
-Zip
-Zip…
Compile
-Zip
-Zip
-Zip
So I might try something like:
Compile
Async
-Sync
–Zip
–Zip
–Zip
-Compile
…
(as was suggested earlier). So many fun options to explore! 
Steve
Just out of curiosity, how far has R&D gotten on this?
I am thinking (or hoping) that the Queue structure of FB 5.5 might be "just enough" to do this.
The basic idea is to set up a Queue with the list of tasks to be done (in my case, it would be a list of Unit Tests to run).
Create an Async action group with "n" children. Each loops around popping the cppUnit solution name off the Queue and building it. Each child "completes" when the pop operation fails.
The main concerns are:
(1) is the Queue MT-safe, so that no child gets the same item from the queue, and
(2) the only way to tune the number of threads is to add or subtract children of the Async Action group
Hi Miles,
Queue/Stack actions are thread safe. However, there's no way to run the same set of child actions multiple times (ie you'll need a separate child branch for each set of actions you want to run in parallel.) At minimum, this can be a set of Run Action List actions which all call the same action.
There is, however, a very neat "maximum threads" property on the Async Action Group action. This lets you limit the number of children which run in parallel.
Using this property, you could do something like have each child of the Async gorup as a Run Action List action, with a parameter which tells it which C++ project to compile. The Async group will start X parallel copies of the action list, and as each one finishes it will start a new one until there are none left. This should give you the behaviour you're after, without even needing to define a stack/queue if you don't want to. It should be easy to fine-tune the number of threads for peak performance, as well (AFAIK, Number of Cores + 1 is usually the recommended number for compilers.)
Regards,
Angus
Angus
Glad to see you're still on the forums...
That's an interesting tip to keep in mind. In actual practice, my Queue would be the result of a file iterator, so I can't predict the number of children that would be necessary. I suppose I could put in a large number of children, all calling the same action list. There would be a few synchronization points when the async group completes and then must be repeated while stuff remainds in the queue. With your MT-safe Queues, it would definitely work.
However, the general solution is the option to make interators asynchronous, with an option to control the maximum number of threads. Keep it in your R&D list. Our site won't be hung up without this functionality, but there is more than one place where we could find this useful.