Groom Jobs Do Not Complete and Scheduler stops processing until restarted

PowerShell Universal Version: 5.5.2

I have an issue on my Development server, where the Groom Jobs do not complete and this causes the scheduler to stop processing any new jobs. Restarting the PowerShell Universal service fixes it, but this is not a great solution. The same issue does not occur on my production server (thankfully).

After a restart, everything works normally, but it takes around 2 hours before the scheduler stops processing new jobs. Any new jobs started manually just stay in the "queued" status forever. Any scheduled jobs simply don't get queued at all.

I looked in Hangfire, and the groom schedule runs every minute or so, but the jobs never complete. Other scheduled or manual jobs do appear in the hangfire queue and do complete normally, but once the hangfire queue gets to 100 processing jobs, then everything just stops. All 100 jobs are for the Groom Service. It always stops once the processing jobs gets to 100 entries.

I restarted the service yesterday and then left it overnight so you can see the issue more clearly via the screenshots. There are exactly 100 jobs in the processing queue. And they are all the same IGroomService.Groom job. There are almost 2000 queued up jobs. 17 hours ago was when I restarted the service. The enqueued jobs are both Groom ones and the ones from the PSU scheduler.



In PSU, the jobs page shows all the jobs being successful up until the hangfire processing queue filled up, and then just the login trigger script when I logged in, and one job that gets launched via an api trigger. The scheduled jobs just never get queued in this view. They are visible in hangfire though (enqueued jobs).




I enabled debug logging and after the restart of the service, I see these errors in the system log. They repeat multiple times per second.

2026-10-21 16:06:21.226 +01:00 [WRN][UniversalAutomation.GroomService] Failed to clear job queue.
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithRowsAffectedOnlyAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at PowerShellUniversal.Extensibility.EFTable`2.SaveWithRetryAsync(DbContext context) in D:\a\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 720
   at PowerShellUniversal.Extensibility.EFTable`2.SaveWithRetryAsync(DbContext context) in D:\a\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 726
   at PowerShellUniversal.Extensibility.EFTable`2.DeleteAsync(T item) in D:\a\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 158
   at PowerShellUniversal.Extensibility.EFTable`2.DeleteAsync(T item) in D:\a\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 158
   at UniversalAutomation.GroomService.GroomJobQueue(Int32 groomDays) in D:\a\universal\universal\src\Universal.Server\Services\GroomService.cs:line 29