1) Open Administrative Tools and then Terminal Services Configuration
2) Select connections from the left pane and then RDP-TCP
3) Click the action menu and properties
4) Select the permissions tab
5) Ensure that only admins (Full Control) and System available. Remove any other users. If you have an admin group you use for RDP access, add and allow full control.
If you wish you can also disable other features such as drive mapping, windows printer mapping etc. These can be found by clicking the client settings tab.