I need some advice on how to handle uploading dozens of large files to a slow FTP server.
Each file takes around 3 minutes to upload, so I’d like to do them in parallel.
It seems that if you put several FTP Upload actions in an async group all pointing to the same FTP Connection, they are still run serially. Is that true?
I had to create an async group and create a number of FTP Connections inside of it, and then use separate FTP Upload commands to upload to their own unique connections.
I would prefer to use the FTP Mirror command instead of FTP Upload, but still be able to break the FTP Mirror into async/parallel actions. However, I couldn’t find a way to specify an Include File Mask for the FTP Mirror, it only has an exclude mask. Could this be added as a feature? (I could log this separately in the feature request forum).
It would also be really nice in the FTP upload action to be able to specify a target directory (like you can in FTP Mirror). Since I had to make seperate FTP Connections for each async group of files, I also needed to duplicate the FTP change dir command into each async group before the FTP Upload. It just makes it all look messy in my project.
The FTP protocol does not support uploading or downloading multiple files over the same connection at the same time. TBH, I’m not sure that running multiple uploads in parallel will make things quicker, I guess it depends on why it’s slow (network, server io etc)?
To do parallel uploads, create a new target, with connect, upload, disconnect ftp actions, add a parameters to control which file to upload etc, then you can call it multiple times in parallel - see this example which uses a queue and multiple threads to process the queue (so you would populate the queue with a list of files to upload).
Thanks. The parallel is much faster on my machine as a single FTP upload gets me 700Kb/s however, I can run 4 or 5 in parallel and they all get the same 700Kb/s… Maybe it’s some throttling per connection server-side…don’t know…
I’ve never used the queue actions in finalbuilder before and I like how your async action just runs a single target; that’s a nice concept. Specifically for the FTP action though, if I were to put an FTP Connect and FTP Upload actions into the target (that gets run multiple times async), is it ok that they would all use the same “FTP connection name”?