Please see screenshots attached.
I gotta say, everytime I try to use RDM Mac, I wanna tear my hair out.
RDM Windows and RDM Windows doesn't use the same RDP engine to establish the RDP connection in embedded mode.
In RDM Windows, by default, we use a Microsoft ActiveX.
In RDM Mac, we use an engine called FreeRDP.
That being said, in RDM Mac, could you try to check/uncheck the NLA and TLS option under the Connection tab and verify if you are able to establish the connection.
A quick search of 0xC00000BB gives me STATUS_NOT_SUPPORTED (The request is not supported.) While this is not very helpful in and of itself, this could mean a limitation of the Legacy FreeRDP engine (which use an older build of FreeRDP). Switching to Modern "might" solve the issue.
If not, and you are willing to provide a bit more info, you can enable the Session Logging and send us the resulting log afterward (ideally with the Modern engine).