wss://den:443/cow -> 502 Bad Gateway

Can you make sure that opnsense is configured to allow WebSocket traffic to go through? The /cow route on Wayk Den is meant for WebSockets specifically, the one that Wayk Now connects to for peer-to-peer messaging. This would explain why exactly everything but that part works.

Can you make sure that opnsense is configured to allow WebSocket traffic to go through? The /cow route on Wayk Den is meant for WebSockets specifically, the one that Wayk Now connects to for peer-to-peer messaging. This would explain why exactly everything but that part works.

Websockets are active according to opnsense, also I can throw a WS at the IP direcitly on port 4000 bypassing opnsense and get the same error. on the other hand if I throw a WSS at the domain but not cow I get different errors like an Auth fail or whatever (which is not weird considering that point is not supposed to take a socket, but it is a different error than 502)

Edit here a screen of that page of the config

Can you grab the logs of the following containers: den-traefik and den-server

docker logs den-traefik > den-traefik.log
docker logs den-server > den-server.log

Then post the logs here

2020/05/30 16:27:58 reverseproxy.go:445: http: proxy error: backend tried to switch protocol "websocket" when "" was requested


basically every 30 seconds from traefik directly into the console when executing the command.

dunno how many of the websocket errors in den-server were me trying to do dumb attempts at connecting to the cow.

I don't know much about OPNsense, but from the logs you provided, it appears to be using nginx under the hood. Nginx needs to be configured to explicitly allow WebSocket traffic on specific routes:

https://nginx.org/en/docs/http/websocket.html
https://www.tutorialspoint.com/how-to-configure-nginx-as-reverse-proxy-for-websocket

Here the /cow route is always a WebSocket connection, so you would need to add the proper configuration in nginx from OPNsense. Wayk Now is configured to attempt a WebSocket reconnection every 30 seconds, but since the WebSocket connection never succeeds, it keeps trying.

yup that settings part I have shown you is the nginx reverse proxy setting of opnsense. but the weird part is that still, when I just go directly towards the local IP, bypassing all I still get that error.

I mean if nginx/opnsense was at fault I shouldnt get 502s when going ws://192.168.x.y:4000/cow but I do.

Hi,

How exactly are you testing the WebSocket connection? Simply opening the WebSocket URL in a browser will result in a 502 bad gateway in normal circumstances. Testing that WebSockets work is a bit tricky, but I found a curl command to test it on localhost:4000:

curl --include \
     --no-buffer --http1.1 \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Host: localhost" \
     --header "Sec-WebSocket-Version: 13" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     http://localhost:4000/cow

If it works, you should see the following output:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=

You can then test WebSockets externally through the reverse proxy (just replace "den.contoso.com" by your real host in two places):

curl --include \
     --no-buffer --http1.1 \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Host: den.contoso.com" \
     --header "Sec-WebSocket-Version: 13" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     https://den.contoso.com/cow

If you can correctly see the "HTTP/1.1 101 Switching Protocols" response locally but not externally, there you have your confirmation that something needs to be done at the reverse proxy level. If the localhost test doesn't work, please post the output of the curl command that you have.

thanks for that command. that may have helped in diagnosing. the weird part was all the time that the 502 clearly came from the den and not from the opnsense/nginx as that uses styled error pages and one can see them really well (also I never exactly knew how websockets work so seeing the headers might have helped piece info together.

seems like the request itself passed mostly but apparently not the upgrade header and the websocket option seems to be buggy and needs a reboot of the whole damn thing (not even just the nginx, lol)

although it might probably be a bit more helpful if you wouldn't just get a 502 bad gateway without anything else, but rather an info which makes it clear that

1) the den shows the error to show the den actually got the request and not some proxy-side precondition
2) actually tell that someone stripped the upgrade header, like by saying that this endpoint is websocket only, which would immediately tell something slightly more exact

as in yes the request is coming through but broken

also I was testing websocket with a browser plugin, which for some super weird reason NOW correctly opens the websocket.

thanks for the help it's been a hell of a ride and lemme just say that this was SUPER weird.

I mean it's basically the exact same config just the apparently around end of april or so when I set up the den it literally just worked.

I AM CONFUSED, but whatever now I'll go ham and try the relay as well.

Edit: half an hour Later. Jet is running. and THIS with custom Den and Jet is something that truly seperates wayk from all the others.