RDM Agent - install on servers.

I'm still faily new to RDM and working out how to get things working.
I have RDM installed on my laptop and wondering how I can get the agent installed on the servers. I have rdm entries setup for them but if I just install the gaent to the server it says it isn't connected.
Do I need to install the agent to do remote commands?
I have read you might need the full rdm install on each server? I don't see why I'd need this as I would just use the version on my laptop?

I want to be able to pull data like the inventory from remote servers and computer management etc but they are failing. Pining etc works.
Is this what the agent is for or am i barking up the wrong tree?