Univeral won't load. Stuck at Syncronizing with Git

Restarted the entire server now getting this repeating in the logs

2023-06-14 09:01:49.346 -04:00 [INF] AuthenticationScheme: Cookies was challenged.
2023-06-14 09:01:49.346 -04:00 [INF] AuthenticationScheme: Bearer was challenged.
2023-06-14 09:01:50.304 -04:00 [INF] Executing OkObjectResult, writing value of type '<>f__AnonymousType1`3[[System.Boolean, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Boolean, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.String, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]'.
2023-06-14 09:01:50.363 -04:00 [INF] Authorization failed. These requirements were not met:
DenyAnonymousAuthorizationRequirement: Requires an authenticated user.
DenyAnonymousAuthorizationRequirement: Requires an authenticated user.

You’d have to remove the git settings in the database if you don’t have access to the UI.

In terms of the log, I’d check to see if there is another log file with more information. Sometimes it ends up writing to two logs.

The other log file was just showing groom job info

I ended up restoring the whole thing from last nights backup and it’s working now, even with a git repo configured that doesn’t exist anymore.

Not sure what exactly is going on at the “Synchronizing with Git” part (other than the obvious), but it might be a good idea to have some sort of timeout there.

I agree. I opened an issue for that here: Git Sync should time out on startup · Issue #2466 · ironmansoftware/issues · GitHub

I have seen git sync hang before if using the external git client and it pops up a credential dialog.

EDIT: It looks like we do have a timeout for the external git client now of 60 seconds.

I am using the external client. The built in client doesn’t like my gitlab instance for some reason.

Was the timeout put in place in 4.0.0? If so I wouldn’t have seen that since I rolled back to a previous version.

That’s what I get for screwing around

The timeout was included 8 months ago so it’s in all recent versions of 3.x

You would also see an error in the log if it was timing out.

if (!process.WaitForExit(60000))
{
    _logger.LogError("Failed waiting for a git command after 60 seconds. Stopping the git process.");
    process.Kill();
    throw new Exception("Failed waiting for a git command after 60 seconds.");
}

So it was something else. It sat at that page for 30+ minutes.

Ok. If you ever have the bandwidth to try again, a memory dump would be helpful as it would indicate exactly where it is hanging.

I’ll see if I can reproduce the issue in my test environment.