How to tell what is causing Automise to get stuck?

Automise Discussion
21

Hi Jonathan

Thanks for the project file and the trace file. I don’t seen anything out of the ordinary in your project… while it’s large, I have seen much larger and a lot more complex!

It seems I was a bit hasty when I said it was a deadlock… that’s not the case. It’s blocking while running a process

thread $296c (TOmniThread):
763a387a KERNELBASE.dll                                                  WaitForMultipleObjectsEx
763a3773 KERNELBASE.dll                                                  WaitForMultipleObjects
01a7dfab vsoft.core.bpl VSoft.Core.Process.Impl                      606 TFBProcess.WaitForProcessCompletion
01a7dc91 vsoft.core.bpl VSoft.Core.Process.Impl                      459 TFBProcess.DoRedirectedExecute
01a7da3e vsoft.core.bpl VSoft.Core.Process.Impl                      399 TFBProcess.Execute
018cee8b vsoft.core.bpl VSoft.Core.Actions.ExecuteBaseAction         429 TFBExecuteAbstractAction.Execute

Unfortunately I can’t tell from this which particular Execute Program Action it is, however the trace also shows

13ec OpenAirManager.exe      0  18 43 normal       C:\im_shortcuts\User_Download_US_oabundle
1cbc OpenAirManager.exe      0  4  1  normal       C:\im_shortcuts\Project Download_oabundle
2f3c OpenAirManager.exe      0  4  1  normal       C:\im_shortcuts\Project Download_oabundle

Which is a process that is called a lot from the project. My guess is that one or more of these processes have hung or are causing locking.

I can’t really advise on what to do here since I don’t know what those processes do but it might be a case of just killing those and atcmd and then running the project again.

22

Interesting. I know what the processes do…they call a program which runs in the background to download data from an external system. They are always the first programs to run. The question is why they are no longer running correctly. I can try running them manually to see what the issue might be. Thank you for the analysis!

However…sometimes lately it has made it through more of the project before getting stuck. I will try to get logs if that happens again.

“might be a case of just killing those AND atcmd and then running the project again”

Brilliant. I never thought to see if the external program was still running in Task Manager - I only checked to see if ATCMD was still running. There were 5 instances of the external program all still running in the Task Manager!

I killed those and started the batch process again for tonight. We’ll see what happens…but I am cautiously optimistic.

23

Well, the batch process got further, but it is still running 10 hours later. It should have ended within about 3 hours. I will run another madtraceprocess on it and attach the log here.

madTraceProcess4.mbr.txt (40.6 KB)

24

It’s the same issue, it’s an Execute Program action waiting for the process to finish

019ddfab vsoft.core.bpl VSoft.Core.Process.Impl 606 TFBProcess.WaitForProcessCompletion

0140 OpenAirManager.exe      0  18 43 normal       C:\im_shortcuts\User_Download_US_oabundle

Does the application write any logs or windows eventlog entries? I’m not sure there is much we can do about this, the issue appears to be with an external application.

25

Yes…this may be an issue with the external program. At least for the last couple of hangs. However, I have had it hang during the MS SQL Server Execute statement for no apparent reason. I am working on an issue with the external program, to see what I can do about it. Will keep you posted here.

Thank you for the help, again.

26

That action executes an external process - perhaps there is a network issue on your server causing general slow downs… I did see Semantic Endpoint Protection in the trace reports… perhaps that may be affecting things?

27

Is there any way to run an MS SQL Execute statement in such a way that it will ignore blocks, or wait for blocks to clear? Any parameters we can set on the Automise side?

28

No sorry. There is a query timeout option but it’s in seconds and it’s in seconds so not really ideal for very long running queries.

The only other option is to use the Execute program action to run sqlcmd and turn off wait for completion - you will need to figure out the command line options for it.

29

Just had another issue where the manual run of an Action List is hanging. I ran the madtrace report:
madTraceProcess5.mbr.txt (41.4 KB)

It is stuck on a Send Email action. It has already run the same action 84 times in this run without issue.

30

Hi Jonathan

That’s useful information, it at least gives us something to test. We’ll do some testing with this over the next few days.

Can you show the settings you have on the action either here or via a PM or email to support. The email action has a few different modes, so we would want to try and replicate the settings as closely as possible.

Is the action called in a tight loop? I am wondering if your mail server is doing some sort of rate limiting (although usually the will respond with a 45x error code).

31

I do not think the loop is that tight - it goes through a lot of actions in between each email. The SMTP settings are to use smtp.office365.com port 587, with all Authentication boxes checked and “Use Require TLS” chosen. A user ID and password is provided in the settings as well.

Please let me know if this is not enough information.

32

The fact that you are using office365 suggests to me that rate limiting might well be an issue

I will set up a test with it and see if I can get myself rate limited and see what error it reports.

33

Maybe…but when I am not running manually, it does not seem to be an issue. Also, it is not ALWAYS getting stuck. But I agree it is something to look out for. Do we pause it for a while when we hit a certain number of emails?