Git - Fetching changesets fails

Continua CI Bugs
1

Hello,

we have lots of these messages in our ContinuaCI Log. Sometimes we end up with a detached head on out git repos and ContinuaCI seems unable to fetch the changesets. In the server logs we find lots of these error messages:

Cannot get changes for repository [<reponame>](http://<buildserver>:8080/view/2002/db29af76-7f9a-4951-bf9c-3f395d50ec47): Running 'C:\Program Files\Git\cmd\git.exe' with arguments "-c credential.helper= -c credential.helper="!\"C:\\Program Files\\VSoft Technologies\\ContinuaCI\\Server\\Continua.Git.CredentialsHelper.exe\" --key <somekey> --workdir \"X:\CiShare\Rc\db29af76\"" fetch --prune --force --all --dry-run" failed with return code 1 and error output: "Fetching origin

bash: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://<ourgitlabserver>': No such file or directory
error: could not fetch origin

Latest version is 1.9.2.902.
Git-Version is: 2.35.1.2

Is there are way to tell ContinuaCI to jump back to the head of the git-repo without checkout of the full repo? (Which happens if one changes the branch name). If we click ‘check now’ nothing happens and we are stuck on some older commit (hence the detached head I suppose)

Thanks for your help in advance!
Christian

2

Hi Christian,

Where are you seeing the detached head? If you are referring to the working folders under the Rc folder on the server, then this is nothing to be concerned with. The checkout state of these folders is changed automatically by Continua CI and can often be in a detached head state. This shouldn’t be the reason for the repository to be stuck.

The error that you have posted implies either that the username and password is incorrect or there is an issue with our credentials helper. Once this is resolved, Continua CI should be able to fetch further changesets. Can you check that the credentials entered for the repository are correct - maybe they were changed on your Git server but not updated in Continua? Otherwise, to verify whether the issue is with the credentials helper, try temporarily changing the repository authentication method to ‘Embed username and password in URL’.

3

Hey Sparky,

yes, its in the RC folder. If I don’t have to be concerned about that so the better.
About the credentials and our gitlab server: The weired bit is that the credentials are correct and it works mostly. If the credentials are not working you cannot even save the property page of the repository property dialog. Its validated green and works fine. But nevertheless these errors in the logs appear.
In order to fix the problem I shut down the service … deleted the repo folder from Rc and let ContinuaCI checkout the whole repo again.
I googled the error message about bash … could it be that something with git needs to be reconfigured?

With regards,
Christian

P.S. I will speak with our admin to temp. enable embed username and password in the URL but he might say that this is not possible for security reason

4

Hi Christian,

We understand that your administrator may not want to use the “embed username and password in the URL” authentication method as it is not secure. Note that Authentication Method is a setting on the Edit Repository dialog in Continua CI and we have suggested this only as a temporary test as you appear to be using a local git server.

We know that the error message means that git is attempting to prompt for credentials, but is not able to as it is not running interactively. The Continua CI Git Credentials Helper gets called by git when it requires credentials, if it doesn’t get them for some reason it attempts to ask for them.

It is possible to enable logging for the git credentials helper. Create a new server property on the Properties page in the Administration section of Continua CI with Namespace “Git.CredentialsManager” and Property Name “LogFilePath”. Set the Value to a path for the log files. e.g. “C:\Temp\GitCredentialsHelperLogs\GitCredentialsHelperLog###.txt”. Note that the ‘###’ is important and will be replaced by the action type, timestamp and a unique id.

Then, once you see the git error logged in the event log, check the folder for files containing the string ‘-get-’ which were created around same time. Check the file for lines containing “error”, “exception”, “failed”, “invalid” or “not found” and send us details of what you find.

It’s unusual that the repository is not recovering from this error, as it’s just a dry run. Are there other errors in the event log? Can you send us a diagnostics report? This can be generated from a link on the top of the Event Log page? Send it privately either by direct message to me here or by email to support at finalbuilder.com.

Note that there are other alternatives to the Continua CI Git Credentials Helper. You can switch the authentication method in Continua CI to “None” and use the Git Credential Manager for Windows instead. This can be installed with Git for Windows. However, to use this, you will need to log in to the server as the service user and run git commands to store the username and password.

Another option for git repositories is to use SSH. See Git over SSH for the steps to set this up, or see Use SSH keys to communicate with GitLab for more detailed steps more relevant to GitLab.

That said, we prefer to diagnose the issue you are seeing with our credentials helper first if you are able to send us further details from the logs.

5

Note that there an easier way to do this by clicking on the Reset button next to the repository on the Repositories page in the Administration section of Continua CI

6

Thank you so much for your detailed answer! I will try and check all of that and will respond to you.
Cheers,
Christian

7

Hey Dave,
enabled the creation of this logfile and managed to relate the errors from the event log with logfiles (via the key which seems to be unique and always different)

<snipped by admin for privacy>

11:26:44.583: Git details date-time '20220308092612' invalid. Difference is 32,5832284seconds.

Turns out that every single event log error refers to a logfile which ends in a date-time difference. But it only happens sometimes. Mostly the logfiles end like this:

<snipped by admin for privacy>

23:05:46.398: Done
8

I switched the authentication to ‘none’ and will see how well that works. Seems to be fine though, since I also have TortoiseGit installed which I used for peeping.

9

HI Christian,

Thanks for sending the log file. There appears to be a quite large time difference with the time encrypted in the credentials - possibly a regional issue. We’ll look into a possible cause for this on Monday.

10

Hi Christian,

This isn’t a regional issue as I had suspected. The date/time string which was output to the log corresponds to the time that the line was logged in UTC plus 32 seconds.

The credentials with the current UTC date/time are encrypted and set as an environment variable immediately before calling the git command line. In our tests this takes a few milliseconds. The git command line then calls the credential helper which decrypts the environment variable and checks the time difference before returning the credentials to git.

We would really need to see a corresponding server debug log to fully diagnose the cause of the delay, but it appears to happening with the git command line.

The only reason we can think of for such a long delay with the git command line is that the git working folder is locked by another process. Do you often have TortoiseGit or another app open and pointing at the Continua CI repository folders on the server? If so, then we do recommend leaving the Rc folders alone as this can interfere with Continua’s automatic processes.

11

Hi Dave,

interesting! Even though I seldomly use TortoiseGit, it has a desktop integration which probably triggers git commands in order to update shell icons. That could be an explanation for such delays. Since there is also git for windows installed, I should be able to just uninstall TortoiseGit without affecting ContinuaCI, right?

12

Yes, although it shouldn’t really be necessary to uninstall it, as long as it is not open and actively locking the folder.

We not certain that this is the cause of your issue, it’s just the only possible theory we have at the moment for why there could be a delay with git command line.

A corresponding server debug log may provide us with better information to confirm or discount this conjecture. Are you able to reproduce the issue with debug logging enabled on the server (and Git.CredentialsManager.LogFilePath set) so we could check what is happening on the server immediately before the error?