ModSecurity and increasingly high CPU usage by httpd Apache process
We've run over a massive server slowdown over the first weeks of a migration from CEntOS 7 to AlmaLinux 8. Here is the overview of the situation.
Issue
On a Plesk 18 server with multiple websites running, httpd processes are using a lot of CPUs and requests are sometimes slow. The situation is getting worse over time. Plesk is running with normal settings (no major custom settings) and has ModSecurity 2.9 on Apache with Comodo ruleset.
What is really going on
Request are using a lot of CPUs because of ModSecurity ruleset. Disabling ModSecurity temporarily proves it.
The file at /var/lib/mod_security/apache-ip.pag is growing fast, and is slowing down the requests the more it grows.
Solutions
Solution 1 : use another ruleset
Switching to ModSecurity 3.0 on nginx with OWASP ruleset fixed the issue. But it is a completly different ruleset.
In that case, it might be a good idea to make sure that requests are going through nginx as much as possible, by making sure all website use FPM application served by nginx. But all requests seems to go through nginx anyway, so I am not sure this is required.
The following SQL query to psa database can give a quick list of all websites not using this PHP handler:
SELECT d.name, wsp.value FROM domains AS d
LEFT JOIN dom_param as dp ON dp.dom_id = d.id AND param = "webServerSettingsId"
LEFT JOIN WebServerSettingsParameters as wsp ON wsp.webServerSettingsId = dp.val AND wsp.name = "nginxServePhp"
WHERE wsp.value = "false"
Solution 2 : empty the file
An empty or lightweight apache-ip.pag file should also solve the issue. For exemple, running this command daily would keep the database manageable:
cat /dev/null > /var/lib/mod_security/apache-ip.dir
cat /dev/null > /var/lib/mod_security/apache-ip.pag
service httpd restart
But emptying this file means loosing data, and might also weaken security since this data could be use by ModSecurity rules.
Solution 3 : manage apache-ip.pag correctly
Managing the apache-ip.pag correctly, or at least understanding why it doesn't behave correctly would be the best solution. However, we didn't get there.
Possible paths toward this solution would be :
- More exhaustive logs analysis
- Smart regular cleanup scripts using
modsec-sdbm-util(or something similar) to keepapache-ip.pagat a clean state.