[2026.3.1] App Crash

avatar

Version: 2026.3.1
OS: 2025 Server Datacenter (latest Windows Updates - including the Out-Of-Band one)
.Net Core: 10.0.12 - Windows Server Hosting
IIS Hosted - App Pool running as a gMSA
Database: SQLite


Whenever I attempt to browse to PSU at all, even the /admin page, I get application crashes logged and the page errors out or the application pool crashes. This is the application crash I'm getting:

Faulting application name: w3wp.exe, version: 10.0.26100.1882, time stamp: 0xc6bd45bd
Faulting module name: e_sqlite3.DLL, version: 3.53.3.0, time stamp: 0x4b3d3b00
Exception code: 0xc00000fd
Fault offset: 0x0000000000115d7c
Faulting process id: 0xB68
Faulting application start time: 0x1DD4B6FADEF5BD3
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: D:\Powershell Universal\e_sqlite3.DLL
Report Id: 840d3897-5d38-49f7-9c2b-968392a41442
Faulting package full name: 
Faulting package-relative application ID: 

(sometimes it references ntdll.dll, but that's typically a red herring)
This was happening on 2026.3.0 after upgrading from 2026.1.2, though that was routinely referencing D:\Powershell Universal\coreclr.dll.

What I've tried:

  • Verified the ZIP download hash.
  • Manually verfied files weren't blocked (I run a Get-ChildItem -Recurse | Unblock-File in the upgrade process)
  • Re-extracted and copied over our appsettings.json and web.confg.
  • Reviewed and updated the appsettings.json to contain any new settings that exist in the one from the zip
  • Upped the logging to debug in appsettings.json, I see a lot of lines about loading license information, but nothing that looks useful


The only error (tagged warning but has error in the message) in the systemLog file for PSU is the following:

2026-09-23 11:26:35.921 -04:00 [INF][UniversalAutomation.Services.UniversalConfigurationService] Reading configuration for Variable
2026-09-23 11:26:35.925 -04:00 [INF][ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider] RPC services being provided by UniversalAutomation.IPublicGitSyncService: 5
2026-09-23 11:26:36.048 -04:00 [INF][Microsoft.AspNetCore.Hosting.Diagnostics] Request starting HTTP/1.1 GET https://<internalURL>/login?returnUrl=%2Fadmin - null null
2026-09-23 11:26:36.109 -04:00 [WRN][Microsoft.AspNetCore.Session.SessionMiddleware] Error unprotecting the session cookie.
System.Security.Cryptography.CryptographicException: The payload was invalid. For more information go to https://aka.ms/aspnet/dataprotectionwarning
   at Microsoft.AspNetCore.DataProtection.Cng.CbcAuthenticatedEncryptor.DecryptImpl(Byte* pbCiphertext, UInt32 cbCiphertext, Byte* pbAdditionalAuthenticatedData, UInt32 cbAdditionalAuthenticatedData)
   at Microsoft.AspNetCore.DataProtection.Cng.Internal.CngAuthenticatedEncryptorBase.Decrypt(ArraySegment`1 ciphertext, ArraySegment`1 additionalAuthenticatedData)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
   at Microsoft.AspNetCore.Session.CookieProtection.Unprotect(IDataProtector protector, String protectedText, ILogger logger)
2026-09-23 11:26:36.334 -04:00 [INF][Microsoft.AspNetCore.Routing.EndpointMiddleware] Executing endpoint '/Login'
2026-09-23 11:26:36.354 -04:00 [INF][Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker] Route matched with {page = "/Login", action = "", controller = ""}. Executing page /Login
2026-09-23 11:26:36.372 -04:00 [INF][Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker] Executing handler method PowerShellUniversal.Login.OnGet - ModelState is "Valid"
2026-09-23 11:26:38.618 -04:00 [INF][UniversalAutomation.Services.UniversalConfigurationService] Reading configuration for Translation
2026-09-23 11:26:38.623 -04:00 [INF][UniversalAutomation.Services.UniversalConfigurationService] Reading configuration for Settings


Both reverting to the 2026.1.2 folder and reverting the VM restores functionality.

For reference, my upgrade process is as follows:

  1. Create new folder, set ACL security on the folder for the gMSA.
  2. Extract the ZIP.
  3. Unblock recursively
  4. Copy web.conf and appsettings.json from the existing PSU folder to the new folder
  5. iisreset /stop
  6. Rename folders (old one gets datetime stamped, new one gets renamed to the original path)
  7. iisreset (to start IIS)


The database/repository folders are separate and outside the path of the IIS site, and are not touched. Database access looks functional based on the systemLog, as in debug logging, I see it sucesfully executing queries:

2026-09-23 11:52:15.845 -04:00 [DBG] Creating DbConnection.
2026-09-23 11:52:15.846 -04:00 [DBG] Created DbConnection. (0ms).
2026-09-23 11:52:15.846 -04:00 [DBG] Opening connection to database 'main' on server 'D:\PSUDB\database.workflows.db'.
2026-09-23 11:52:15.846 -04:00 [DBG] Opened connection to database 'main' on server 'D:\PSUDB\database.workflows.db'.
2026-09-23 11:52:15.846 -04:00 [DBG] Creating DbCommand for 'ExecuteReader'.
2026-09-23 11:52:15.846 -04:00 [DBG] Created DbCommand for 'ExecuteReader' (0ms).
2026-09-23 11:52:15.846 -04:00 [DBG] Initialized DbCommand for 'ExecuteReader' (0ms).
2026-09-23 11:52:15.846 -04:00 [DBG] Executing DbCommand [Parameters=[@filter_UsableAsActivity='?' (DbType = Boolean)], CommandType='"Text"', CommandTimeout='30']
SELECT "w"."Id", "w"."BinaryData", "w"."CreatedAt", "w"."Data", "w"."DefinitionId", "w"."Description", "w"."IsLatest", "w"."IsPublished", "w"."IsReadonly", "w"."IsSystem", "w"."MaterializerContext", "w"."MaterializerName", "w"."Name", "w"."OriginalSource", "w"."ProviderName", "w"."StringData", "w"."TenantId", "w"."ToolVersion", "w"."UsableAsActivity", "w"."Version"
FROM "WorkflowDefinitions" AS "w"
WHERE "w"."UsableAsActivity" = @filter_UsableAsActivity
ORDER BY "w"."CreatedAt"
2026-09-23 11:52:15.846 -04:00 [INF] Executed DbCommand (0ms) [Parameters=[@filter_UsableAsActivity='?' (DbType = Boolean)], CommandType='"Text"', CommandTimeout='30']
SELECT "w"."Id", "w"."BinaryData", "w"."CreatedAt", "w"."Data", "w"."DefinitionId", "w"."Description", "w"."IsLatest", "w"."IsPublished", "w"."IsReadonly", "w"."IsSystem", "w"."MaterializerContext", "w"."MaterializerName", "w"."Name", "w"."OriginalSource", "w"."ProviderName", "w"."StringData", "w"."TenantId", "w"."ToolVersion", "w"."UsableAsActivity", "w"."Version"
FROM "WorkflowDefinitions" AS "w"
WHERE "w"."UsableAsActivity" = @filter_UsableAsActivity
ORDER BY "w"."CreatedAt"
2026-09-23 11:52:15.846 -04:00 [DBG] Closing data reader to 'main' on server 'D:\PSUDB\database.workflows.db'.
2026-09-23 11:52:15.846 -04:00 [DBG] A data reader for 'main' on server 'D:\PSUDB\database.workflows.db' is being disposed after spending 0ms reading results.
2026-09-23 11:52:15.846 -04:00 [DBG] Closing connection to database 'main' on server 'D:\PSUDB\database.workflows.db'.
2026-09-23 11:52:15.846 -04:00 [DBG] Closed connection to database 'main' on server 'D:\PSUDB\database.workflows.db' (0ms).
2026-09-23 11:52:15.846 -04:00 [DBG] 'ManagementElsaDbContext' disposed.
2026-09-23 11:52:15.846 -04:00 [DBG] Disposing connection to database 'main' on server 'D:\PSUDB\database.workflows.db'.
2026-09-23 11:52:15.846 -04:00 [DBG] Disposed connection to database 'main' on server 'D:\PSUDB\database.workflows.db' (0ms).

I will note that I don't see any logs about it talking to the main DB, just this workflow db.

The final lines in the systemLog are always:

2026-09-23 11:54:28.550 -04:00 [DBG][UniversalAutomation.Common.Interfaces.ILicenseService] Loading license information.
2026-09-23 11:54:29.129 -04:00 [INF][UniversalAutomation.Services.UniversalConfigurationService] Reading configuration for Translation
2026-09-23 11:54:29.135 -04:00 [INF][UniversalAutomation.Services.UniversalConfigurationService] Reading configuration for Settings
2026-09-23 11:54:29.139 -04:00 [DBG][UniversalAutomation.Services.ConfigurationScript] Reading configuration file settings.ps1
2026-09-23 11:54:29.140 -04:00 [DBG][UniversalAutomation.Services.ConfigurationScript] File exists. Deserializing items.
2026-09-23 11:54:32.682 -04:00 [DBG][Universal.Server.Routing.UniversalEndpointDataSource] Endpoints changed. Notifying change token.



Any help on where to start further troubleshooting would be appreciated.

All Comments (1)

avatar

Hello @SunnyJ

Thank you for providing such a detailed report and for documenting the troubleshooting you have already completed.

We have been reviewing several areas and behaviors in the latest PSU releases, and I am going to add this IIS and SQLite crash scenario to our current lab testing so we can compare the behavior under similar conditions.

The fact that reverting to 2026.1.2 restores functionality, while both 2026.3.0 and 2026.3.1 exhibit the crash, gives us a useful baseline for testing. We will also take into account the IIS hosting configuration, the gMSA identity, the SQLite database location, and the `0xc00000fd` crash being reported against `e_sqlite3.dll`.

For now, please confirm whether this is affecting a production environment and whether the system is currently stable again after reverting to 2026.1.2.

I will continue reviewing this against our lab environment and update the thread with the findings.

Best regards,
Ruben Tapia