File upload through Gateway very slow

Hello

In the past we've had at least two reports of the same problem and, interestingly, all of them relate directly to VMWare (vSphare and / or esxi).

When we've looked into it previously we were not able to recreate the issue; but clearly something is going on here when talking to VMWare.

I will ask that we reopen our internal investigation into that; but it would certainly be helpful if you could describe your environment in a little more detail:

• Relevant versions of the Devolutions stack you are using (Gateway, Devolutions Server, RDM, etc)
• Relevant versions of the VMWare components you are using
• Any particular network details (a description of the environment); particularly if there are intervening network hardware or software tools involved (like other proxies, firewalls, VPNs)
• Do you notice the same issue with other sites or services, or it seems isolated to VMWare? Or perhaps, this is the only place the problems comes up if you are not doing very large file uploads with other services.

Please, let me know if something isn't clear or you have further questions

Kind regards,

Hi Richard,

At the moment we only notice it when sending "lare" files to vcenter, so I think we can relate it to this specific case.

We use the following versions:

Remote Desktop Manager 2025.2.25.0
Devolutions Server 2025.2.8.0
Devolutions Gateway 2025.2.3

vCenter 8.0.3

If there is anything we can test or help to get this problem solved, please let me know.

Hello

Can you let me know where exactly in vCenter you're doing the uploads? Is it to a datastore, or somewhere else?

Thanks and kind regards,

Hi Richard,

The upload happens most of the time when you select Deploy OVF template.

Hello

Broadly when I made a sanity test on my side, I didn't notice much of a different uploading a large file via the web to MS Azure. Going via Gateway is a little slower; which is normal (instead of browser -> ESXi; we're going Browser > SOCKS Proxy > jetsocat.exe > JMUX -> Devolutions Gateway > ESXi); but in the two existing tickets we had the customer reported a 50x slowdown in this scenario (and your results sound similar).

I wonder if this is something specific to OVF deployment, and I'll re-open the investigation ticket on our side to try and evaluate that. Unfortunately the dev and QA teams do not have access to the OVF tools in our vCenter so this is something that requires a bit of coordination internally. Although, I would be surprised if this is specific to OVF because as far as I can tell, when it pushes the actual blobs it's just using HTTP POST; no custom VMWare / Broadcom magic.

From your side, is there a lot of variability when using (for example) Cloudflare Speediest over the Gateway versus without it? It's also using HTTP POST to test the upload as far as I can tell. I'd like to narrow this down as something VMWare / OVF specific.

Thanks and kind regards,

I did the Cloudflare speedtest with and without gateway. Both gave the same result so it seems it's related to vcenter in combination with the gateway.

Hello

I think it's specifically related to OVF deployment. If you just upload an arbitrary file to a data store on vCenter, is it slower? Our tests show that it is, but not significantly so (and expected due to the extra layers of indirection).

VMWare / Broadcom's documentation for the standalone OVF tool has a "highlight": "Uses new optimized upload and download API (optimized for vSphere 4.0 and newer)" which suggest to me they are doing something special in this case, which might have characteristics that don't work well (currently) with Devolutions Gateway.

I'm asking our QA team to re-open the investigation here and focus specifically on image deployment to see if we can reproduce and understand the traffic. I don't expect this to happen quickly because it requires significant involvement from our IT team.

In the meantime, I have a suggestion which might help or at least provide an additional data point. You could try using the standalone OVF Tool for image deployment. The documentation says that it allow you to specific a proxy. For the proxy you could create a Devolutions Gateway Tunnel session type in RDM and launch it, then point OVF Tool at the tunnel local address. It would be interesting to know if this works better or the result is broadly the same.

Thanks and kind regards,

Hi Richard,

Any update on this from the QA Team? Also Data limit for (website) entries through the Devolutions Gateway - Devolutions Forum is mentioning about slow transfer.

Hello

I'm afraid I have no update on our side. Granting access to deploy images to our internal VMWare infrastructure is not something that can be taken lightly by the IT or security teams. And unfortunately given the current state of Broadcom licensing it's non-trivial to deploy separate VMWare instances just to test this.

Let's see if we can't take it forwards in another way. I've identified some areas of the code base that might be causing an issue, but it's impossible to say without a reproduction or better diagnostics.

First, I've already asked if the problem exists when using the standalone OVF Tool and a Devolutions Gateway Tunnel. That question still stands.

Next, please try this: close RDM and then relaunch RDM with the environment variable `JETSOCAT_LOG` set to `info,jmux_proxy=debug`. From PowerShell, that would be something like

$Env:JETSOCAT_LOG="info,jmux_proxy=debug"
& 'C:\Program Files\Devolutions\Remote Desktop Manager\RemoteDesktopManager.exe`

Now open your vCenter session and confirm we're logging to %appdata%\jetsocat. Open the newest file and you should see DEBUG lines in the form <id> request tcp://host:port. If you only see the version= line and the pipe open/close, the variable didn't reach the process.

Now run an .ovf upload, make a note of the wall clock time that it started. At the same time, I'd like you to watch the CPU of the local machine (running RDM) and the CPU of the Gateway machine.

Afterwards, we should have a few different pieces of data:

1. A comprehensive log file from jetsocat
2. The approximate start/end time of the upload
3. An answer to: did the upload spike the CPU of either the client or Gateway machine? I.e. Was 1 core pegged to 100% for the duration of the upload?

If something isn't clear, please let me know

Thanks and kind regards,

We only use a content library in vcenter so I can't use the ovf tool. I tested it with firefox where I configured the proxy to use Devolutions Gateway Tunnel. When I do an upload test (not an ova, just a .bin testfile it's as slow as in Devolutions self. When I disable the proxy in Firefox and upload the same file directly to vCenter it's fast. The speed difference is huge (a file of 100 mb takes minutes throught the gateway and seconds without).

The jetsocat logfile gives no relevant debug information. During the upload I only see logs about telemetry sites that can't be reached (that's normal, these sites are not reachable from the gateway server:

2026-08-20T08:59:59.537056Z WARN jmux:scheduler: jmux_proxy: Channel opening failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060) local_id=l#30 destination_url=tcp://scapi.telemetry.broadcom.com:443 msg.reason_code=TTL_EXPIRED (0x00000006)
2026-08-20T08:59:59.537081Z WARN jmux:scheduler: jmux_proxy: Channel opening failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060) local_id=l#29 destination_url=tcp://scapi.telemetry.broadcom.com:443 msg.reason_code=TTL_EXPIRED (0x00000006)
2026-08-20T09:00:09.743071Z WARN jmux:scheduler: jmux_proxy: Channel opening failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060) local_id=l#31 destination_url=tcp://edge.microsoft.com:443 msg.reason_code=TTL_EXPIRED

I didn't see any CPU spikes.

Hello

Ok, I understand and the detail that this is not confined to OVF is an important one. I still have some theories on this.

If you don't mind, can you please repeat your test while looking at the "Network" page in Firefox dev tools. We're interested in the protocol, you may need to right-click the column headers to add the column. I want to know if the HTTP PUTs are showing HTTP/2 or something else.

You can then try going to about:config and setting `network.http.http2.enabled = false`; restart Firefox and try again.

Please let me know the results

Kind regards,

And setting this network.http.http2.enabled = false isn't doing anything it seems.

Hello

Thanks. "Isn't doing anything" is a bit ambiguous here; if you recheck the protocol column does it still show HTTP/2? i.e. the setting genuinely had no effect and the test didn't tell us something. Or it shows HTTP/1 and my theory is wrong and we need a different line of attack? Or maybe with that set, Firefox now reaches for HTTP/3, which has the same flow control and in theory has the same problem? If that's the case, `network.http.http3.enable` can control it.

Curl would let us definitively choose the protocol, but I'm not sure how to craft a curl command to push a file to VMWare.

Ultimately what I want to understand right now is:

1. If HTTP/1 proxied is fast but HTTP/2 is slow, this is an issue with how we're interacting with HTTP/2+ flow control
2. If HTTP/1 and HTTP/2 proxied are both slow, the tunnel is adding cost to something else that is round-trip bound and I probably need a capture to identify it
3. If all the above is similar, then the tunnel might not be the variable at all and we need to look somewhere else

If you can confirm any of the above it would be helpful.

Kind regards,

It still show HTTP/2

Hello again

So I've been working on this today and I want to qualify my response by saying I still don't have a repro on my side. File level access to our vCenter is not something I have, or can easily get, for the reasons given above. But I've been researching and testing against local HTTP servers and trying to reproduce the circumstances of the issue. My findings are that there may be two separate things at play here; a change that you can make on your side and a fix for a genuine issue on the Gateway side.

The background is that browsers talk to vCenter using HTTP/2, which handles file uploads very differently than HTTP/1. With HTTP/1, the browser opens a connection and sends the whole file in one continuous stream. Nothing interrupts it. With HTTP/2, the server hands out permissions in small allowances. It tells the browser "send me 64KB", the browser sends 64KB then waits for the server to say "another 64KB!" and so on. For a 100MB file, that's about 1600 back-and-forth exchanges instead of just 1. It works this way for good reason: it lets many request share a single connection, without any of them hogging it. But it also means an upload speed depends entirely on how quickly those exchanges complete.

(Side note that I used 64KB in my example because that is the default in the HTTP/2 specification, but this value is not fixed, it's chosen by the application and can indeed tune itself at runtime - if the application supports it. This is a fundamental difference to HTTP/1 where flow control is handled at the TCP level).

Through Gateway, each one of those exchanges simply has further to travel. Your machine to Gateway, Gateway to vCenter, and back again (and inside that there are many layers, the traffic is tunnelled over a local proxy and then web sockets).

Investigating this, I found that Gateway was likely adding up to 25ms of unnecessary delay in each exchange. It came from a batching optimization we made a couple of years ago that helped large continuous transfers but penalized this kind of back-and-forth traffic. I've made a fix for that and assuming it's accepted, will be in the next release. How much this helps you in reality I don't know, because the other factor affecting this is the overall RTT (round trip time). If your Gateway sits close to vCenter (a few ms RTT) it's possible our 25ms was the bulk of the penalty and the fix may bering Gateway uploads closer to direct speed. If there's real distance between Gateway and vCenter (say, 50ms), then the RTT is dominant and our fix will certainly help but it may not feel fast still.

In researching this I turned up this article. Broadcom documents raising vCenter's HTTP/2 upload allowance from 64KB to 8MB. That would take my 100MB example above from 1600 round trips to about a dozen, at which point extra distance stops mattering at all. It needs a config change on the appliance and a service restart, and I haven't tried or tested this myself. I'm going only from Broadcom's own documentation. If travel time is the dominant factor then this change is what will probably make uploads much faster.

That 64KB default has always been there, it just doesn't matter if you'r close enough to vCenter. It only becomes a bottleneck when each exchange has further to travel, which is what happens when the traffic is relayed by Gateway. So this isn't a misconfiguration in the current form, but it's a default that stops being suitable once the path gets longer.

Please, let me know if something isn't clear or you have other questions

Kind regards,