Remote desktop connection via rd gateway on non standard port

Hi,
this is the situation:

• we can connect to servers at customers, via remote desktop, but on a non standard port and only from the office ip addresses.
• we have a rd gateway setup at the office, using the office ip address.

When we create a remote desktop session to a customer, with a specific port, when at the office we can connect using the remote desktop manger.
When we are at home, what happens frequently now, we cannot connect because of the restriction of ip addresses on the customer site.
We use directaccess connections to have the laptops connected to the office network (so they can use rdm).
It's not an option to relay all internet traffic, from the pc's over the vpn to the office and from there to internet, load would be too high.

When we have a customer using this on the standard port, we can connect from home, via the rd gateway/

Anyone any suggestions on how we could get this working for people working from home?

Rgds