Create MS SQL data source with custom administrator users (not SQL)

It is impossible. The fact that they are sysadmin means they have full access to every database including the RDM database. Even if you block them in the application there is nothing from stopping them from changing the data directly in the database.

SQL Server simply doesn't support it: https://social.msdn.microsoft.com/Forums/sqlserver/en-US/e86835e7-8ca2-471c-906d-7157488d5f8d/can-i-deny-permissions-on-a-sysadmin-account?forum=sqlsecurity

Above statement is false. You could use database encryption to avoid it completely.

But, I was not asking about restricting database access to data, because then sysadmin/DBA should know your schema, relations and many more to avoid program security and even then can make mistakes which renders database useless/broken.

I was asking if there are program capabilities in this area, because now every DBA can run private RDM, connect to our DB and login with his DBA credentials and have full access to defined connections, can make logons to our clients and steal personal data - it is against GDPR in Europe, which insist full control who can have access to personal data.
If it is impossible, it is serious flaw especially in hosting environment (where our servers are in cloud and administered by cloud staff).

If you want your sysadmins to have no access to the RDM sessions then (ie. won't even be able to connect) then use the Security Providers. Security Providers will encrypt all the session information but not things like logs for example.

If the sysadmin does know the "Shared Passphrase" or does not have access to the certificate they will never be able to connect via RDM.

They will however be able to delete logs and manipulate the RDM database directly but not see any session information.

Thanks a lot, i'll check it.