Hide SSO groups in roles

We would like to hide the synced SSO groups for people who are not administrators. The reason for this is that we work with "vault" owners who are responsible for a vault and it is up to them to give the users the needed permissions like contributor or reader. But we really want them add permissions per person and not a whole bunch of
people who might not need access to that specific vault.