I’m been having some intermittent and annoying failures with FinalBuilder jobs that are launched from a batch file that runs as a scheduled task via Windows Task Scheduler.
The OS is Windows Server 2012 R2 and the FinalBuilder version is 7.0.0.3472 - it’s been happening for a while with earlier versions and upgrading to the latest build hasn’t changed anything.
Now, first off I’m not at all sure that this is a FinalBuilder bug but I figured I’d post here to see if it rang any bells or if anyone has further troubleshooting ideas since I’m running out of them myself.
The wierd thing is that they only seem to happen when the scheduled task is set to ‘run when user is logged in or not’. If the task is set to ‘run when user is logged in’ or I run the batch file from the command line, they complete fine. My hourly ‘build testing build’ tasks run fine, it’s my nightly ‘resync everything from perforce and rebuild signed release builds’ tasks that fail.
I’ve seen reports on the net that Windows Task Scheduler can be ‘unreliable’ so I tried an alternative (Splinterware’s System Scheduler) and the problem persisted.
When the build fails, it used to the Delphi Compile action that failed but since changing the scripts to use an Execute Program action of SignTool for dual signing rather than Authenticode action, SignTool can also fail.
When the Delphi Compile fails, it’s because the .cfg that’s generated by FinalBuilder is missing all our paths from the Delphi XE2Library settings in Tools options that is passed in for the -I, -U, -R options hence the compiler fails with F1026 ‘file not found’.
I did tried attaching the two .used files for comparison but I got a ‘file type not allowed’ error.
When the SignTool step fails it’s with a -1073740940 code (0xC0000374 in hex) which is STATUS_HEAP_CORRUPTION.
Bizarre, I know!
I gather than Scheduled Tasks set to ‘run whether logged in or not’ run in a special Station 0/Session 0 reserved for services rather that any users interactive Station/Session. Does FinalBuilder (FBCMD.exe) do anything different when running in that mode that could cause it’s .cfg generation to intermittently fail? Does it inject any code into the address space of programs it executes that could cause them to get heap corruption? I’m really grasping at straws here, I know 
Cheers,
Paul.
Hi Paul
I suspect this is a bug in Server 2012, see https://support.microsoft.com/en-us/kb/2968540 and
- http://serverfault.com/questions/540427/windows-server-2012-scheduled-tasks-run-using-default-profile-when-ran-session
The issue is that the user profile is not getting loaded in time, and delphi (and many other tools) rely on the user profile for environment variables, registry entries etc.
The work around appears to be to schedule another task, for example a simple no-op batch file to run a minute or so before (or at regular intervals) to get the user profile loaded into memory. Hacky, but since microsoft don’t seem to interested in fixing this issue that’s pretty much all you can do!
Thanks so much for the prompt reply, Vince!
I’m not using the Delphi Library path from the registry, but the “Use Global Library Path” option to use the paths set in the Tools section of FinalBuilder but I’m guessing they are stored in the User Profile registry so the wrong profile could still trip FinalBuilder up?
If so that could indeed explain the Delphi failures as the directories showing in the ‘bad’ Delphi actions appear to be the default ones that show in FinalBuilder after installing XE2.
The SignTool failures are a bit more perplexing but maybe SignTool can fail spectacularly with STATUS_HEAP_CORRUPTION if the User Profile is Default or if it changes whilst it’s running?
Anyway, I can try the suggested workarounds and see if that makes any difference.
Answering my own question - I see that FinalBuilder stores it’s Tools Options settings in a FinalBuilder7.ini file stored in %APPDATA%\FinalBuilder7 directory.
Since the Windows 2012 R2 bug causes %APPDATA% and %LOCALAPPDATA% to be pointing at the Default profile not the specified profile, that would do it.
I’ll try setting those directly as suggested since the ‘schedule another task’ 1 or 2 minutes before is reportedly not particularly reliable and reliable builds is what I want!
It will be interesting to see if these changes also fix the SignTool heap corruption crashes…
Cheers,
Paul.
OK, the explicit SET resolves the Delphi Library path failures. Awesome!
However, it doesn’t fix the SignTool issue. I’m using Execute Program to call signtool since I needed to dual sign before FinalBuilder got around to supporting /as
Executing external process: C:\Program Files (x86)\Windows Kits\10\bin\x86\signtool.exe
Parameters: sign /fd sha1 /f C:\Build\Certs\accredosha1.pfx /p ezy2gess /t http://timestamp.globalsign.com/scripts/timestamp.dll C:\Build\Build9\REL\Products\Saturn\AccredoSaturn.exe
Output from C:\Program Files (x86)\Windows Kits\10\bin\x86\signtool.exe
Done Adding Additional Store Program returned code : -1073740940
Run from a task set to ‘run whether user logged in or not’ and SignTool fails as above.
Run from a task set to ‘run only when user logged in’ and SignTool succeeds .
I’m using Signtool from the Windows 10 SDK. Maybe if I try the one from the Windows 8.1 SDK I’ll have more luck?
I thought I’d try using the built-in SignTool Sign Files action to see if that made any difference. It didn’t.
Command Line: signtool.exe sign /v /fd sha1 /t http://timestamp.globalsign.com/scripts/timestamp.dll /f C:\Build\Certs\accredosha1.pfx /p [suppressed] C:\Build\Build9\REL\Products\Mercury\AccredoMercury.exe The following certificate was selected: Issued to: Accredo Business Software Ltd Issued by: Thawte Code Signing CA - G2 Expires: Tue Oct 04 12:59:59 2016 SHA1 hash: xxx Done Adding Additional Store
Action Failed
I tried the Windows 8.1 SDK version of SignTool next. Suceess! It seems to run without heap corruption crashes regardless of how the scheduled task is configured.
So I’m left wondering if the Server 2012 R2 profile bug potentially has effects on the .NET runtime which the Windows 10 SDK SignTool uses (4.0?) vs. the one which Windows 8.1 SignTool uses (3.5?).
I’ll leave it configured for unattended schedules for a while and keep my fingers crossed.