Changing the port number used by ssh from the default 22 can help increase the security of the ssh server.
To do this open up the config file
Scroll down to the part which reads
Uncomment the line and change the value to your preferred one, making sure that the port number is not already in use.Then restart the server
# service sshd restart
Limiting SSH access by user or group
All valid users on the system are allowed to log in and enjoy the benefit of SSH but a more secure policy is to allow only a predetermined list of users or groups to log in.
To do this, log in as root and open the SSH configuration file
# vi /etc/ssh/sshd_config
Scroll down and locate the line which starts
And append to it the users you wish to allow, as example
AllowUsers anton james george
You can also use the same method to allow members of a valid administration group to log in.
Or you can add admin to the where this is a valid user
When done restart the server with
$ sudo service sshd restart
See also SSH Harden shell and Install OpenSSH