API Endpoint Service problems after upgrade to 2026.2.3

Published 9 days ago

Our Powershell Universal has been upgraded from 2026.2.3 from 2026.1.2. After upgrade, our API endpoints started encountering gateway timeouts. It seems that executing commands as simple as "Get-PSUSchedule -Integrated" in a custom API Endpoint hang or take excessively long to execute. The same command executed in the same environment in script or locally on the server takes about 20 seconds to execute. I increased the API endpoint timeout to 300s here and it still timed out.

2026-08-11 09:14:17.262 -04:00 [INF][Microsoft.AspNetCore.Hosting.Diagnostics] Request starting HTTP/1.1 GET <redacted> - null null
2026-08-11 09:14:17.265 -04:00 [INF][Microsoft.AspNetCore.Routing.EndpointMiddleware] Executing endpoint 'GET:<redacted>'
2026-08-11 09:14:17.287 -04:00 [INF][] GET /logicmonitorcheck <redacted>026-08-11 09:19:21.245 -04:00 [INF][Microsoft.AspNetCore.Routing.EndpointMiddleware] Executed endpoint 'GET:<redacted>'
2026-08-11 09:19:21.245 -04:00 [ERR][Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware] An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="Call canceled by the client.", DebugException="System.OperationCanceledException: The operation was canceled.")
---> System.OperationCanceledException: The operation was canceled.
--- End of inner exception stack trace ---
at ProtoBuf.Grpc.Internal.Reshape.UnaryTaskAsyncImpl[TRequest,TResponse](AsyncUnaryCall`1 call, MetadataContext metadata, CancellationToken cancellationToken)
at PowerShellUniversal.Api.ApiGrpc.ExecuteAsync(ApiRequest request, CancellationToken cancellationToken) in D:\a\powershell-universal\powershell-universal\src\PowerShellUniversal.Api\ApiGrpc.cs:line 193
at PowerShellUniversal.Api.ApiService.ExecuteEndpointAsync(HttpContext httpContext, Endpoint endpoint, Dictionary`2 variables) in D:\a\powershell-universal\powershell-universal\src\PowerShellUniversal.Api\ApiService.cs:line 792
at PowerShellUniversal.Api.ApiService.ExecuteAsync(HttpContext httpContext, Endpoint endpoint) in D:\a\powershell-universal\powershell-universal\src\PowerShellUniversal.Api\ApiService.cs:line 504
at Universal.Server.Routing.ApiEndpointSource.HandleAsync(HttpContext context, Endpoint endpoint) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Routing\ApiEndpointSource.cs:line 77
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at PowerShellUniversal.FeatureMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\FeatureMiddleware.cs:line 42
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at PowerShellUniversal.DisallowedModeMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\ModeMiddleware.cs:line 25
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at PowerShellUniversal.CspMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\CspMiddleware.cs:line 28
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 165
at PowerShellUniversal.PSUMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\PowerShellMiddleware.cs:line 15
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Universal.Server.Middleware.RegexApiMiddleware.Invoke(HttpContext httpContext, IApiService apiService) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\RegexApiMiddleware.cs:line 19
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Universal.Server.Middleware.WindowsAuthMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\WindowsAuthMiddleware.cs:line 58
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 51
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Universal.Server.Middleware.McpAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\powershell-universal\powershell-universal\src\Universal.Server\Middleware\McpAuthenticationMiddleware.cs:line 30
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\powershell-universal\powershell-universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
2026-08-11 09:19:21.246 -04:00 [INF][Microsoft.AspNetCore.Hosting.Diagnostics] Request finished HTTP/1.1 GET <redacted> - 500 null text/plain 299970.7853ms

All Comments (1)

Patrick Ouimet

Published 3 days ago

Hi Shawn,

Thank you for the detailed log excerpt on the endpoint timeout after the 2026.1.2 to 2026.2.3 upgrade.
That stack trace was the piece we needed.

I want to share what we determined so far and the short test we would like to run, with your authorization, to confirm the root cause before applying a permanent fix.

What the log tells us
The 500 fires at 299970 ms, which is your 300 s endpoint timeout to the millisecond.
The call stalls inside the reverse gRPC channel between the API worker and the PSU host process (ApiGrpc.ExecuteAsync) and never returns.

ASP.NET is the side that cancels the call at the ceiling.
There is no exception from your PowerShell code, no server-side error, only silence for the full timeout.
That signature points to a stuck authorization step on the reverse gRPC path, not an HTTP, middleware, or script problem.

What changed between 2026.1.2 and 2026.2.3
The 2026.2.3 release notes include the change: "Improved job data management APIs to use granular permissions."
Get-PSUSchedule is a job data management API, so this change is on that exact code path.
In PowerShell Universal v5, when the API is in Strict security mode, -Integrated calls are authorized through the reverse gRPC channel using the endpoint's execution context.

Our hypothesis is that the new granular privilege introduced in 2026.2.3 is not present on the identity your endpoint runs under, and the authorization step blocks rather than returning a clean Unauthorized.

This fits every detail of your log: 20 s locally as an admin account, indefinite hang from inside the endpoint, no PowerShell exception, cancellation at exactly the client timeout.
Proposed test to confirm

We would like to run one reversible test on the PSU host in a short change window, roughly 15 minutes, with one service restart and a full rollback path included.
The steps:

Snapshot appsettings.json to a backup folder on the host.
Call the endpoint once to record the failing baseline (matches what you already have in your log).
Set the PSU security mode to Permissive in appsettings.json, then restart the PowerShell Universal service.
Call the endpoint again. If it returns in seconds, the hypothesis is confirmed.
Restore the original appsettings.json and restart the service to end the window.

Permissive mode is only in effect between steps 3 and 4, and it is reverted at step 5 regardless of the outcome.

We do not recommend leaving the instance in Permissive mode, it is a diagnostic setting only, because it disables privilege enforcement for -Integrated calls across the whole instance.

Permanent fix path if the test confirms the diagnosis
Two options, both keep you on Strict security mode:
Option A, grant the specific privilege introduced in 2026.2.3 to the App Token or Role that your endpoint's identity uses, then keep Get-PSUSchedule -Integrated in the endpoint as it is today.

Option B, replace the -Integrated call with an authenticated REST call to /api/v1/schedule using $AppToken, which sidesteps the reverse gRPC authorization path entirely.

Either option preserves the security posture you get from 2026.2.3.

If the test does not confirm the hypothesis, we will pivot to a debug log capture with the appropriate categories enabled plus a process dump of the API worker while the endpoint is hung.

Best regards,

Patrick Ouimet