[Devolutions Gateway] Validate the TLS certificate at service startup, not at first handshake

[Devolutions Gateway] Validate the TLS certificate at service startup, not at first handshake

1 vote

avatar

Summary

The Gateway's two certificate sources behave very differently when something is wrong, and only one of them fails in a way an administrator can act on.

With a file-based certificate, the Gateway reads and binds the certificate at startup. One it cannot read, parse or bind stops the service, and the reason is written to a boot.stacktrace file next to gateway.json. That is exactly the behaviour you want: the problem is visible immediately, before any user tries to connect.

With TlsCertificateSource: System, nothing reads the store at startup at all. The lookup and the private key acquisition both happen inside a callback that runs per incoming handshake. The service therefore starts cleanly, opens the listener, and reports itself as healthy — and the failure only appears once per connection, in the log.

We would like the certificate-store path brought in line with the file-based path: resolve the certificate once at startup and fail loudly there.

We are aware this is a real change rather than relocating an error that already exists. The file-based path is the precedent we are asking you to match.

What we saw

Our Gateway was configured to load its certificate from the Windows certificate store. The service account had no read permission on the private key. From the outside, everything looked healthy:

PS> Get-Service DevolutionsGateway
Status   Name                DisplayName
------   ----                -----------
Running  DevolutionsGateway  Devolutions Gateway Service

PS> Get-NetTCPConnection -LocalPort 7171 -State Listen
LocalAddress   LocalPort  State
------------   ---------  -----
::             7171       Listen
0.0.0.0        7171       Listen


The startup sequence in the log was equally clean:

INFO devolutions_gateway::listener: Listening on https://[::]:7171
INFO devolutions_gateway::listener: Listening on https://0.0.0.0:7171
INFO devolutions_gateway: devolutions-gateway service started


No boot.stacktrace was written, because the service had not failed to start — it had simply never looked at the store.

The actual problem appeared only when a client connected, and then repeated for every connection thereafter:

ERROR devolutions_gateway::tls::windows: Failed to resolve TLS certificate
error="no usable certificate found in the system store;
key acquisition failures: cert[0]: failed to acquire key: Error code 80090016"

ERROR devolutions_gateway::listener: handle_https_peer failed
error="TLS handshake failed: unexpected error: no server certificate chain resolved"


0x80090016 is NTE_BAD_KEYSET. The message itself is excellent — it named the exact cause, and we fixed it within a minute of finding it. The problem is purely where and when it becomes visible.

The request

Perform the store lookup and key acquisition once at service startup, in addition to the per-handshake path. If the certificate cannot be found or its private key cannot be acquired, report it at startup with:

  • the certificate subject that was looked for
  • the store name and location that were searched
  • the account the service is running as
  • the underlying error


For example:

ERROR Cannot use configured TLS certificate.
      Subject: CN=gateway.example.com
      Store:   LocalMachine\My
      Service account: NT AUTHORITY\NetworkService
      failed to acquire key: 0x80090016 (NTE_BAD_KEYSET)
      -> Verify that the service account has read permission
         on the certificate's private key.


Whether the service should then refuse to start, as the file-based path does, or start and report unhealthy, is a design decision we have no strong opinion on. What matters is that the condition is visible before a user tries to connect, rather than being inferred from a listener that accepts connections and drops every one of them.

Why this matters

The installer has performed certificate checks since 2026.2.x, which is a good addition — but it only covers configuration made through the installer. Anything that changes afterwards is unchecked:

  • gateway.json edited directly, which is the documented way to configure the certificate store option
  • a certificate replaced or renewed in the store
  • private key permissions changed by policy or by a GPO
  • the service account changed


In all of those cases the installer's validation ran long ago, and the next indication that something is wrong is a user reporting that they cannot connect.

There is already a doctor diagnostic that performs closely related checks — SAN presence, EKU serverAuth, missing intermediates. Running an equivalent resolution step at service start would close the gap between "the installer said it was fine" and "it is fine right now".

References

Existing file-based behaviour, for comparison:
https://docs.devolutions.net/gateway/knowledge-base/troubleshooting-articles/devolutions-gateway-troubleshooting/#service-fails-to-start

The difference between the two certificate sources described above was confirmed by Devolutions support in this thread:
https://forum.devolutions.net/topics/55273/gateway-versuib-2926220-not-working

---

Related request for Devolutions Server: surface the actual reason behind "Certificate could not be verified", which is the dialog users see when this condition occurs.

All Comments (0)