1 - Check the last users connection on the host
From console digit:
# last
This command will show you all user that logged successfully in the host.
This is a good point to start to understand if somebody get an unauthorised access on your server.
This is one example of the command :
root@servername~# last
root pts/0 10.10.50.3 Mon Jan 30 13:03:45 2017 still logged in
root pts/0 10.10.50.3 Mon Jan 30 13:00:27 2017 - Mon Jan 30 13:03:44 2017 (00:03)
root pts/0 10.10.50.3 Mon Jan 30 12:50:42 2017 - Mon Jan 30 13:00:26 2017 (00:09)
root pts/0 10.10.50.3 Fri Jan 27 09:17:39 2017 - Fri Jan 27 16:44:53 2017 (07:27)
2 – Check the running processes with TOP
Personally I prefer top to “ps aux” just because top will provide you some extra information that can be useful.
With top you can verify if some application are using more resource than usual.
With this command you can also find other useful information, like free Ram, average load, number of task, CPU load ect.
# top
top - 11:33:21 up 21 days, 41 min, 1 user, load average: 0.02, 0.02, 0.05
Tasks: 90 total, 2 running, 88 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 6808648 total, 4629072 free, 306808 used, 1872768 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5962784 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 125104 3648 2408 S 0.0 0.1 1:40.58 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.89 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:01.84 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
As default TOP will update himself every second with the latest information, but will show you only a limited number of application
If you’d like see all application are running the best is to add more:
# top | more
Optional: Kill a process
If you need to kill a process you can do it in 2 ways
The easly one is to digit
# pidof processname
Example to kill all process of apache
# pidof apache
However you can also kill a specific pid digit the command “kill” followed to the pid number that you can find in “Top” or “ps aux”
Example:
# kill 3445
3 - Check SSH attempt connections:
Another step is to check the ssh logs to understand is somebody is trying to get access to the server,
You can check the access log to the server ( SSH ) in this way
From console digit:
# tail -n 500 /var/log/secure | grep ‘sshd’ ( if you are using a Centos/RedHat distribution based )
# tail -n 500 /var/log/auth.log | grep sshd ( if you are using a Debian distribution based )
This is one example:
root@servername:~# tail -n 1000 /var/log/auth.log | grep sshd
Jan 30 13:24:12 servername sshd[17655]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan 30 13:26:43 servername sshd[17796]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.50.3 user=root
Jan 30 13:26:45 servername sshd[17796]: Failed password for root from 10.0.50.3 port 14238 ssh2
Jan 30 13:26:51 servername sshd[17804]: Accepted password for root from 10.0.50.3 port 14332 ssh2
This command will show you the log ( from the last 500 lines ) of all the attempt to get into the ssh server (failed or authorized) with important information such as the username was trying to get
access source IP
Note:
If you need to read backward the log you need to increase the number of line maybe to 1000 or more depend of the server use because this logfile store all access to the server ( FTP SSH Webmin ldap, and other… ) also after X time the server create a new file to keep the file size small, that means maybe you have to open some old logs.
3a Optional: Last command by user
In case someone get access to the server probably he sent some command trough the ssh.
The follow command will show the last 100 line of command digit on the console for the current user
tail -n 100 ~/.bash_history | more
you can use nano instead of tail to read all the log.
nano ~/.bash_history | more
if you’d like you can check also command digit from other user. To be able to do that you have to visit they own home folder, generally is /home
Example:
tail -n 100 /home/username/.bash_history | more
4 - Netstat :Verify the current connections from and to the server
Another step is to verify the connection to your server with netstat
From console digit:
# netstat | more
this command is will show you 2 part the first is “Active Internet connections” (w/o servers) and the second is “Active UNIX domain sockets (w/o servers)”
check both carefully because if you got a malicious script is trying to sending spam mail or try to attach other server you can easily find here
The point of the situation:
After this 4 step if you server has been hacked you should find something, but to be sure is better do some extra checks
5 - Check IP Ban / Blacklist for the Public IP Address
The best web is google a bit and find a couple website that allow you to verify if your ip is banned.
One of the website im using is: http://whatismyipaddress.com/blacklist-check
If your public ip is banned, probably that means that your server was doing something wrong like SSH Bruten force, email spam, DDos.
However that don’t means that you server has been hacked but maybe one of the website or web application has been hacked or do something wrong.
That means that you should check the website
6 - Crontab: scheduled jobs
Another best practice is to check the cronjobs ( for all users )
Maybe a malicious script or application could be seen here
# cd /var/spool/cron/crontabs/ && grep . * ( for Debian distribution based )
# cd /var/spool/cron && grep . * ( for Centos/RedHat distribution based )
this command will show you all the cron jobs for all users, that can show you some more information to understand if there is some malicious script is running at the reboot or every X time
7 - Antivirus scan
A good practice is to install an antivirus and run a scan when you think that the system can have virus.
To install the AV on
debian:
Apt-get install clamav-scan
Redhat:
yum install clamav-server clamav-data clamav-update clamav-filesystem clamav clamav-scanner-systemd clamav-devel clamav-lib clamav-server-systemd
now you need to update the AV:
# freshclam
This command will start the update of the virus definition, soon the update has been completed you can run the scan typing
# clamscan –r / | grep FOUND >> /var/log/av/AvReport.txt
Adding the second part eventually virus report will be logged into the file AvReport.txt
NOTE: I suggest to avoid to setup a schedule or run the AV in realtime, not just for an eventual performance degradation but because the AV if find a false-positive will be delete automaticly a file and I don’t think this is a good idea in some kind of servers
