The first thing almost everyone does, faced with a suspect server, is reinstall everything. It is the understandable reaction and, in most cases, it is also the mistake that costs you control of the problem.
The reason is simple: if you do not know how they got in, reinstalling is pointless. They get back in the same day, through the same door, and this time you do not even have the logs to work it out.
The rule worth more than any tool
Preserve the evidence, then act. The moment you find something suspicious, before deleting anything do three things: copy the file aside, save the logs, and note the time. An infected file you have deleted is a file you will never be able to analyse, and it is often the only piece that tells you who got in.
Only afterwards, and with the evidence safe, do you start cleaning up.
The signals you must not ignore
Before looking for infected files, learn to recognise that something has happened. The quietest symptoms, in order of probability:
Anomalous CPU usage. An installed miner runs at full tilt and has no recognisable name. On a server that should be idle, a process that grinds the CPU is the first signal. If you do not know which process is normal for your service, you have no baseline, and that is already a flaw.
Unknown outbound connections. Outbound traffic is the channel the attacker uses to carry your data out or to download the second stage. A server that has no reason to contact the outside world should not have permanent outbound connections.
Files modified in the last few days. The files an attacker touches have a recent modification date, and files that should not exist have the wrong owner or the wrong date.
Cron. It is the most common persistence mechanism of all, and also the most ignored. A crontab -l with a line you do not recognise is an alarm.
The commands, in order of use
1. Processes: what is actually running
# I processi ordinati per consumo CPU, i primi 15
ps aux --sort=-%cpu | head -15
# Solo i processi con un percorso eseguibile, con il comando completo
ps -eo pid,ppid,user,%cpu,%mem,etimes,args --sort=-%cpu | head -20
# I processi lanciati dopo il boot, con il percorso assoluto del binario
ps -eo pid,lstart,args | grep -v "^\s*PID" | head -20
The columns that matter are etimes (how many seconds the process has been running) and args (the full command line). A process with a low etimes that consumes a lot of CPU and has a name like [kdevtmpfsi] is not a system process: it is a miner with a disguised name.
A detail worth a check of its own: compare the ppid. If a process that should be launched by systemd has an unknown process as its parent, the chain is suspect.
2. The filesystem: the suspicious files
# File modificati negli ultimi 10 giorni, esclusi cache e pseudo-filesystem
find /var/www /home /tmp -mtime -10 \
-type f \( -name "*.php" -o -name "*.js" -o -name "*.sh" \) \
-not -path "*/cache/*" 2>/dev/null | head -50
# Gli stessi file, con la data di modifica in chiaro
find /var/www -mtime -10 -type f -name "*.php" \
-printf '%TY-%Tm-%Td %TH:%TM %u %p\n' 2>/dev/null | sort -r | head -40
The -printf is the part you need: %TY-%Tm-%Td for the date, %u for the owner. A file dated yesterday with owner www-data in a directory that should be yours is the clue.
# File senza proprietario (uid non mappato): spesso lasciano questi
find / -nouser -o -nogroup 2>/dev/null | grep -vE "^/(proc|sys|dev|run)" | head -30
# File eseguibili in directory che non dovrebbero contenerne
find /tmp /var/tmp /dev/shm -type f -perm -u+x 2>/dev/null | head -30
# Sorgenti PHP con funzioni sospette: eval, base64_decode su input, gzinflate
grep -rlnE "eval\s*\(|base64_decode\s*\(\s*\\\$_(GET|POST|REQUEST)|assert\s*\(\s*\\\$|gzinflate\s*\(" \
/var/www --include="*.php" 2>/dev/null | head -20
The last one is the Swiss army knife. Almost all malicious PHP code goes through one of these functions, because they are needed to decode and execute something that statically looks harmless. A base64_decode($_POST[...]) followed by eval is a webshell, and there is no legitimate case for it in a theme or a plugin.
3. Persistence: where it hides
# Cron di tutti gli utenti
for u in $(cut -d: -f1 /etc/passwd); do echo "== $u"; crontab -u "$u" -l 2>/dev/null; done
# Cron di sistema
cat /etc/crontab; ls -la /etc/cron.*/ 2>/dev/null
# Script di avvio e profili, eseguiti a ogni login
ls -la /etc/profile.d/ /etc/init.d/ /etc/rc.local 2>/dev/null
grep -rInE "curl|wget|nc |bash -i|sh -c" /etc/profile.d/ /etc/rc.local 2>/dev/null
# Chiavi SSH autorizzate: controlla sempre *chi* può entrare
cat ~/.ssh/authorized_keys 2>/dev/null
find / -name authorized_keys -not -path "*/proc/*" 2>/dev/null | head
That for u in ... is there because a non-privileged user with their own crontab is the place an attacker keeps persistence when they cannot write to /etc.
And the authorized_keys should be checked every time, even if you suspect nothing: it is the most common way to maintain access, and one extra key is impossible to spot by eye.
4. Network: where it comes from and where it goes
# Connessioni attive con il processo che le tiene
ss -tunap 2>/dev/null | head -30
# Chi ascolta, e su quali indirizzi
ss -tulnp 2>/dev/null
# Connessioni verso l'esterno, per un server che dovrebbe starsene tranquillo
ss -tnp state established 2>/dev/null | grep -vE "127\.0\.0\.1|::1" | head -20
# DNS: modifiche recenti a resolv.conf sono un classico di DNS hijacking
stat /etc/resolv.conf; cat /etc/resolv.conf
# Hostname sospetto: alcuni malware si impostano un nome riconoscibile
hostnamectl 2>/dev/null | head -5
If your server resolves names to a DNS you do not recognise, all outbound traffic passes through someone else. It is a quieter attack than most, because it touches almost no file.
5. The right question: how they got in
This is the part that reinstalling never solves for you. The vectors, in order of real frequency:
# Web: file PHP scritti di recente nella document root
find /var/www -type f -name "*.php" -mtime -30 -printf '%TY-%Tm-%Td %p\n' 2>/dev/null | sort
# Log di accesso: gli status insoliti (200 su .php strani, 4xx da tentativi)
grep -E " (404|403|500) " /var/log/nginx/access.log 2>/dev/null | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# Login SSH falliti: da quanti indirizzi diversi
grep "Failed password" /var/log/auth.log 2>/dev/null | awk '{print $NF}' | sort | uniq -c | sort -rn | head
That uniq -c | sort -rn on the URLs requested with an error is the most informative technique of all. An automated attacker looking for a vulnerability leaves a very regular footprint: hundreds of requests to URLs that do not exist. Seeing those URLs tells you what they were looking for, and therefore what you have.
The order of operations
Once you have gathered the information, the order matters. Do it in the wrong order and you destroy the evidence and learn nothing.
First. Copy the suspicious files and the logs off the box. Put the server behind a firewall with only the connections you really need, without touching the filesystem. If the compromise is active, every minute counts.
Then. Rotate the credentials, all of them: user passwords, SSH keys, API tokens, database credentials, and the keys in the cloud. All of them, not the ones you remember. The assumption is that something you had has been read, and you have no way of knowing what.
Then again. Rebuild from a clean image, rather than repairing. The "I clean the files" approach works as long as you find them; when it stops working it is because there is a persistence you did not look for. Rebuilding takes longer than ten minutes and removes the doubt.
Finally. Close the door they came in through. Patches, rotated credentials, or the CMS bug. Without this, the rebuild is only a hire: the next attack gets in through the same gap.
The role of preventive defence
All of this is useful, but the real work is having made it hard to get there. The four things that make the difference, in order:
Updates. The first entry vector is still software with a patch available. A server 200 days behind on updates is a server that has already decided when it will be breached.
Minimal access. Nobody should be able to write to the document root except the deploy user. No service should run as root. Every excess permission is one more opportunity for the attacker.
Logs, and someone who looks at them. A production server without logs that get read is a server that has violated the privacy of all its users without anyone knowing. Automatic detection matters more than retention.
A backup you have tested. A backup that has never been restored is a hypothesis, not a backup. If you do not know how long it takes to get back on your feet, you do not have a plan: you have a wish.
The point worth keeping in mind is this: when you find an infected file, you are looking at the result of someone who has been inside your server for weeks or for months. The file you find is not the entry point, it is a consequence. The real work is not cleaning up, but understanding why that file was there, and who put it there.
If your server hosted a WordPress, the entry point was probably a plugin vulnerability: in that case the search path is described in Finding vulnerabilities in WordPress. And if what you are looking at is an administration panel, the most common entry points besides that one are SQL injections, which also explain why a file ends up where it should not be.