Hopefully you guys can point me to a faster way to copy new/missing files onto other servers if they do not exist in the destination folder's tree.
Brief description:
Hundreds of files are actively copied into the Source folder all day long. This project, on an hourly schedule, deploys files to a QA server and a Staging server. As the number of files has increased, so has the total run time of the project. The project now takes 1.5 hours to run, which is not acceptable because it is supposed to be running on an hourly schedule.
The Source folder is a single folder. Files are only copied into the QA/Staging server if they do not exist in the tree of the Destination folder.
Destination\WorkingDestination\Processed
Destination\Bad
Destination\Scraped
To accomplish this, I use the following steps:
FileSet define [ FilesQA ] (all destination folders are defined)
FileSet to Variable [ FilesQA ] -> [ iFilesQA ]
FileSet define [ FilesStaging ] (all destination folders are defined)
FileSet to Variable [ FilesStaging ] - > [ iFilesStaging ]
FileSet define [ FilesSource ]
Iterate FileSet [ FilesSource ] (iFilesSource as variable)
Path Manipulation [iFilesSource] (extract filename)
If [%iFilesSource%] is not in [%iFilesQA%] (xcopy file to destination)
If [%iFilesSource%] is not in [%iFilesStaging%] (xcopy file to destination)
The Source folder is cleaned up every day so that it only contains the last 7 days of files.
Source currently contains 5,043 files. An average of 30 1mb files are added to Source each hour.
Looking at the logs, the If [%iFilesSource%] is not in steps have a combined run time of 1 second if the file already exists on all servers. Multiple that by 5043 and you get 84 minutes just to check the existence of the files.
Is there a better way to do this??