Problem(s) with installation Wayk Bastion

avatar

Hi,

I have a little problem after installation of wayk bastion:
I want to use Windows Server 2019 Core edition LTSC, so localhost is not an option for initial config. Is it possible to start WaykBastion so it allows initial config for another machine, not localhost. Currently I am getting an error when trying to open the website:
forum image

All containers start without an error:

PS C:\ProgramData\Devolutions\Gateway> Start-WaykBastion -Verbose
docker rm den-mongo
Removing C:\ProgramData\docker\volumes\den-mongodata\_data\WiredTiger.lock
Starting den-mongo
VERBOSE: docker run --name den-mongo -d --restart=on-failure --network=den-network -v "den-mongodata:c:\data\db"
library/mongo:4.2-windowsservercore-1809
den-mongo successfully started
docker rm den-picky
Starting den-picky
VERBOSE: docker run --name den-picky -d --restart=on-failure --network=den-network -e
"PICKY_DATABASE_URL=mongodb://den-mongo:27017" -e "PICKY_REALM=sectra.com" -e "RUST_BACKTRACE=1" -e
"PICKY_PROVISIONER_PUBLIC_KEY_PATH=c:\picky\picky-public.pem" -v "C:\ProgramData\Devolutions\Wayk
Bastion/picky:c:\picky:ro" devolutions/picky:4.8.0-servercore-ltsc2019
den-picky successfully started
docker rm den-lucid
Starting den-lucid
VERBOSE: docker run --name den-lucid -d --restart=on-failure --network=den-network -e "LUCID_ADMIN__SKIP=true" -e
"LUCID_TOKEN__DEFAULT_ISSUER=https://wayk-somedomain.com" -e
"LUCID_API__ALLOWED_ORIGINS=https://wayk-somedomain.com" -e
"LUCID_ACCOUNT__LOGIN_URL=http://den-server:10255/account/login" -e
"LUCID_ACCOUNT__APIKEY=qUe9kCHK6BL8DTIvvENKJkmG...HLcrw" -e
"LUCID_ACCOUNT__USER_EXISTS_URL=http://den-server:10255/account/user-exists" -e
"LUCID_LOGIN__ALLOW_UNVERIFIED_EMAIL_LOGIN=true" -e "RUST_BACKTRACE=1" -e
"LUCID_ACCOUNT__REFRESH_USER_URL=http://den-server:10255/account/refresh" -e
"LUCID_ACCOUNT__SEND_ACTIVATION_EMAIL_URL=http://den-server:10255/account/activation" -e
"LUCID_DATABASE__URL=mongodb://den-mongo:27017" -e "LUCID_LOGIN__PATH_PREFIX=lucid" -e
"LUCID_LOGIN__SKIP_COMPLETE_PROFILE=true" -e "LUCID_TOKEN__ISSUERS=https://localhost:4000" -e
"LUCID_LOGIN__ALLOW_FORGOT_PASSWORD=false" -e "LUCID_LOGIN__PASSWORD_DELEGATION=true" -e
"LUCID_LOCALHOST_LISTENER=https" -e "LUCID_LOG__LEVEL=warn" -e "LUCID_API__KEY=DAO1IMY02aAl9zbN6rh....N9z5xevUI" -e
"LUCID_LOGIN__DEFAULT_LOCALE=en_US" -e "LUCID_LOG__FORMAT=json" -e
"LUCID_ACCOUNT__FORGOT_PASSWORD_URL=http://den-server:10255/account/forgot" --health-interval=5s --health-timeout=2s
--health-retries=5 --health-start-period=1s --health-cmd='curl -sS http://den-lucid:4242/healthz'
devolutions/den-lucid:3.9.5-servercore-ltsc2019
den-lucid successfully started
docker rm den-server
Starting den-server
VERBOSE: docker run --name den-server -d --restart=on-failure --network=den-network -e
"MONGO_URL=mongodb://den-mongo:27017" -e "PICKY_REALM=sectra.com" -e "LUCID_INTERNAL_URL=http://den-lucid:4242" -e
"DEN_PRIVATE_KEY_FILE=c:\den-server\den-private.key" -e "DEN_API_KEY=qUe9kCHK6BL8D.....kmGHLcrwLTv" -e
"DEN_HOST_INFO_FILE=c:\den-server\host_info.json" -e "RUST_BACKTRACE=1" -e
"LUCID_AUTHENTICATION_KEY=DAO1IMY02......rhaWp7N9z5xevUI" -e "DEN_EXTERNAL_URL=https://wayk-somedomain.com" -e
"DEN_ROUTER_EXTERNAL_URL=https://wayk-somedomain.com/cow" -e "DEN_LOGIN_REQUIRED=false" -e
"DEN_PUBLIC_KEY_FILE=c:\den-server\den-public.pem" -e "DEN_LISTENER_URL=https://wayk-somedomain.com:4000" -e
"JET_RELAY_URL=https://wayk-somedomain.com" -e "LUCID_EXTERNAL_URL=https://wayk-somedomain.com/lucid" -e
"AUDIT_TRAILS=true" -e "PICKY_URL=http://den-picky:12345" -e "PICKY_EXTERNAL_URL=https://wayk-somedomain.com/picky"
-v "C:\ProgramData\Devolutions\Wayk Bastion/den-server:c:\den-server:ro" --health-interval=5s --health-timeout=2s
--health-retries=5 --health-start-period=1s --health-cmd='curl -sS http://den-server:10255/health'
devolutions/den-server:3.6.0-servercore-ltsc2019 -l info -m onprem
den-server successfully started
docker rm den-traefik
Starting den-traefik
VERBOSE: docker run --name den-traefik -d --restart=on-failure --network=den-network -v
"C:\ProgramData\Devolutions\Wayk Bastion/traefik:c:\etc\traefik" -p 4000:4000
library/traefik:1.7-windowsservercore-1809 --file --configFile=c:\etc\traefik\traefik.toml
den-traefik successfully started
docker rm den-gateway
Starting den-gateway
VERBOSE: docker run --name den-gateway -d --restart=on-failure --network=den-network -e
"DGATEWAY_CONFIG_PATH=c:\gateway" -e "RUST_LOG=info" -e "RUST_BACKTRACE=1" -v "C:\ProgramData\Devolutions\Wayk
Bastion/den-gateway:c:\gateway:rw" -p 8080:8080 devolutions/devolutions-gateway:2021.1.4-servercore-ltsc2019
den-gateway successfully started

In the config, I gave up with external db already, is quite deafult except for External Url (https://wayk.somedomain.com) and Listener Url: (https://wayk.somedomain.com:4000).

Configuration of gw is according to documentation listening on 7171 and 8181.

Best regards,
Rok

All Comments (10)

avatar

Hi,

You can reconfigure the listener and external URLs for temporary access on a non-localhost interface like this:

Set-WaykBastionConfig -ListenerUrl "http://192.168.1.100:4000" -ExternalUrl "http://192.168.1.100:4000"

Then call Restart-WaykBastion, and perform the initial configuration using the non-localhost address. When reconfiguring for proper external access, do not forget to use https:// in both parameters, the correct ports, and the correct hostname in the external URL.

Best regards,

Marc-André Moreau

avatar

That worked, it loaded the web page and allowed me to configure...

Thank you!

Best regards,
Rok Berlec

avatar

Hi,

I have a few more questions:

  • is it possible to publish service to external and not expose the management web site? They are the same service (container) as far as I understand?


Also agent does not connect, maybe you can help me:

2021-05-26 16:51:33 NowService::service::update_monitor [DEBUG] - Waiting on update loop timeout (900 seconds)
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change called - state=Connecting
2021-05-26 16:51:33 NowService::service [DEBUG] - Den url: https://wayk.somedomain.com
2021-05-26 16:51:33 common::logging [TRACE] - >> - ipc://wayk_service_broadcast [L]: Kind: 0 Id: 1 Len: 398
2021-05-26 16:51:33 NowService::service [DEBUG] - den_client_on_state_change finished successfully
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change successfully finished
2021-05-26 16:51:33 common::logging [DEBUG] - curl url: https://wayk.somedomain.com:443/.well-known/configuration (proxy: )
2021-05-26 16:51:33 common::logging [WARN] - curl_easy_perform failure: SSL peer certificate or SSH remote key was not OK
2021-05-26 16:51:33 common::logging [WARN] - NowDenUtil_GetConfiguration failed for https://wayk.somedomain.com:443 (NETWORK)
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change called - state=Failure
2021-05-26 16:51:33 NowService::service [DEBUG] - Den url: https://wayk.somedomain.com
2021-05-26 16:51:33 common::logging [TRACE] - >> - ipc://wayk_service_broadcast [L]: Kind: 0 Id: 2 Len: 398
2021-05-26 16:51:33 NowService::service [DEBUG] - den_client_on_state_change finished successfully
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change successfully finished
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change called - state=Disconnected
2021-05-26 16:51:33 NowService::service [DEBUG] - Den url: https://wayk.somedomain.com
2021-05-26 16:51:33 common::logging [TRACE] - >> - ipc://wayk_service_broadcast [L]: Kind: 0 Id: 3 Len: 398
2021-05-26 16:51:33 NowService::service [DEBUG] - den_client_on_state_change finished successfully
2021-05-26 16:51:33 NowService::service::callbacks [DEBUG] - den_client_on_state_change successfully finished


I guess this is the error: "curl_easy_perform failure: SSL peer certificate or SSH remote key was not OK"
Anyhow, Powershell connects to the iis without a problem:

curl https://wayk.somedomain.com/.well-known/configuration

StatusCode        : 200
StatusDescription : OK
Content           : {"den_router_uri":"https://wayk.somedomain.com/cow","lucid_uri":"https://wayk.somedomain.com/lucid","picky_uri":"https://wayk.somedomain.com/picky","realm":"somedomain.com","wayk_client_id":"eqHW...
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 252
                    Content-Type: application/json
                    Date: Wed, 26 May 2021 14:37:17 GMT
                    Server: Saphir
                    
                    {"den_router_uri":"https://wayk.somedomain.com/cow","lucid_uri":"https:/...
Forms             : {}
Headers           : {[Content-Length, 252], [Content-Type, application/json], [Date, Wed, 26 May 2021 14:37:17 GMT], [Server, Saphir]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 252


avatar

Hi,

Is this a certificate signed by a private certificate authority, or a public certificate authority? One thing to double check is that the server certificate used for HTTPS contains the certificate *chain* including the leaf certificate and the intermediate CA certificate, but excluding the Root CA certificate. Cases where certificate validation works in some places but not in Wayk are most often caused by a server certificate that does not include the intermediate CA certificate: browsers or even the code used by PowerShell leverage a certificate cache, such that if you loaded an intermediate CA certificate for another site, it will be able to complete the missing certificate in the chain.

Best regards,

Marc-André Moreau

avatar

Certificate is from LetsEncrypt, I guess a trusted CA ;-)
I noticed that it works form a private nework but not through internet, so I guess it has something to do with DGateway or configuration. The .pem on DGateway is the whole chain.

Best regards,
Rok

avatar

I don't know, probably I cannot read the installation documentation anymore...
I eventually removed Dgateway as additinal container now, how do I configure the one that is build-in - den-gateway? This is the same thing isn't it? Directly through json file? In the pem file here is only a public key (I guess of internal CA)...

avatar

Hi,

The built-in Devolutions Gateway (Set-WaykBastionConfig -JetExternal $false) is basically automatically configured behind the same external URL as Wayk Bastion.

You can refer to this code sample on how to configure the separate Devolutions Gateway. It imports the public key from Wayk Bastion + imports a certificate + configures an HTTPS/WSS listener in addition to a TCP listener:
https://github.com/Devolutions/devolutions-labs/blob/master/powershell/gw_vm.ps1#L238

On the Wayk Bastion side, the equivalent command is Set-WaykBastionConfig -JetExternal $true -JetRelayUrl "https://gateway.ad.it-help.ninja"

As for the certificate validation issue, we have observed it once on our side and we are looking into it. We are still unsure as to why only this call from the Wayk Agent would have trouble with the certificate validation, and we've seen it with a letsencrypt certificate as well. Is the certificate validation failure contained to just the Wayk Agent when it attempts to make a call with Devolutions Gateway, but everything else works?

Marc-André Moreau

avatar

I have -JetExternal set to false right now:

Get-WaykBastionConfig


DisableCors               : False
DisableDbSchemaValidation : False
DisableTelemetry          : False
ExperimentalFeatures      : False
ServerExternal            : False
MongoExternal             : False
TraefikExternal           : False
JetExternal               : False
PickyExternal             : False
LucidExternal             : False
NatsExternal              : False
RedisExternal             : False
Realm                     : somedomain.com
ExternalUrl               : https://someserver.somedomain.com
ListenerUrl               : https://someserver.somedomain.com
ServerMode                :
ServerLogLevel            :
ServerCount               : 0
DenServerUrl              :
DenRouterUrl              :
DenKeepAliveInterval      : 0
DenApiKey                 : qUe9kCHK6BL8DTIvvENKaAAaJkmGHLcrwLTv
ServerImage               :
MongoUrl                  :
MongoVolume               :
MongoImage                :
TraefikImage              :
JetRelayUrl               :
JetTcpPort                : 0
JetRelayImage             :
PickyUrl                  :
PickyImage                :
LucidUrl                  :
LucidApiKey               : DAO1IMY02aAl9zbN6rhaaAAaWp7N9z5xevUI
LucidImage                :
LucidLogLevel             :
NatsUrl                   :
NatsUsername              :
NatsPassword              :
NatsImage                 :
RedisUrl                  :
RedisPassword             :
RedisImage                :
DockerNetwork             :
DockerPlatform            :
DockerIsolation           :
DockerRestartPolicy       :
DockerHost                :
DockerBaseImage           :
SyslogServer              :

When I imported the cert with Import-waykbastioncert it seems like it is imported only in traefik, is that correct? Others have different private keys and public.pem consist only of -----BEGIN/END PUBLIC KEY----- no CERTIFICATE...

avatar

Yes, you are correct, Import-WaykBastionCertificate imports the certificate inside traefik, which is then used as a reverse proxy in front of all of the other microservices (den-server, den-gateway). The certificate validation error makes no sense, because the Wayk Agent validates the certificate for other calls made to Wayk Bastion, and therefore uses the same certificate. We can't explain it yet, but in the instance where we've seen it happen, rebooting the Wayk Bastion host "solved" the issue, after which we've been unable to replicate it. Clearly something is off, but we're still trying to figure out what exactly.

Marc-André Moreau

avatar

Can you send us the certificate chain configured in Wayk Bastion (the one copied into the traefik folder) without the private key, of course, to wayk@devolutions.net so we can take a closer look at it?

Marc-André Moreau