Green computing with firewall country DROPs

How to reduce log-filling and cpu-cycle-wasting traffic by blocking bad behaving countries with the help of IPdeny and shorewall. This is the typical scenario for me :

I'm greeted on login with this :

Last login: su maalis  6 14:26:17 EET 2016 on pts/2
Last failed login: su maalis  6 14:27:55 EET 2016 from xxx.xxx.xxx.xxx on ssh:notty
There were 4354335 failed login attempts since the last successful login.

And my /var/log/secure is filled with this :

Mar  6 14:21:24 mail sshd[30744]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xxx.xxx.xxx  user=root
Mar  6 14:21:41 mail sshd[30744]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xxx.xxx.xxx  user=root
Mar  6 14:21:46 mail sshd[30746]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xxx.xxx.xxx  user=root
Mar  6 14:22:02 mail sshd[30746]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xxx.xxx.xxx  user=root
Mar  6 14:22:06 mail sshd[30753]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xxx.xxx.xxx user=root

And the /var/log/nginx/dbut-access.log is swarmed with script kiddies trying to find my state of the art administration tools.

xxx.xxx.xxx.xxx - - [11/Feb/2016:00:22:49 +0200] "HEAD /edit/fckeditor HTTP/1.1" 404 - "-" "-"
xxx.xxx.xxx.xxx - - [16/Feb/2016:22:50:39 +0200] "GET /phpmyadmin/scripts/setup.php HTTP/1.0" 404 226 "-" "-"
xxx.xxx.xxx.xxx - - [17/Feb/2016:06:47:41 +0200] "GET /phpmyadmin HTTP/1.1" 404 208 "-" "Python-urllib/2.7"
xxx.xxx.xxx.xxx - - [21/Feb/2016:12:03:51 +0200] "GET /joomla/administrator/index.php HTTP/1.1" 404 228 "-" "-"
xxx.xxx.xxx.xxx - - [28/Feb/2016:02:46:33 +0200] "GET /blog/wp-admin/ HTTP/1.1" 404 212 "-" "-"

IP to country CIDR blocks

A wonderful project called IPdeny maintains a database of ipv4/6 network blocks per country. Blocks are represented in CIDR notation for ease of use in multitude of firewall and networking tools. Following script will toggle the presence of country blocklist in shorewall parameters file.

#!/bin/bash
set -e
[[ $1 =~ ^[a-z]{2,2}$ ]] || ( echo -e "Toggle country block list in /etc/shorewall/params \n $0 <ccTLD>" && exit 1 )

zone=$1
url="http://www.ipdeny.com/ipblocks/data/countries/$zone.zone"
echo "HEAD list for $zone from $url"
curl -s -S --fail -I "$url" > /dev/null

parameter_name=$(echo "COUNTRY_$zone"|tr '[[:lower:]]' '[[:upper:]]')

if [ "$(grep -c "^$parameter_name" /etc/shorewall/params)" = "1" ] ; then
  sed -i "/^$parameter_name/ d" /etc/shorewall/params
  echo "removed $parameter_name"
else
  blocklist=$(curl -s -S --fail "$url"|tr '\n' ','|sed 's/,$//'|sed "s/^/$parameter_name=/")
  echo "$blocklist" >> /etc/shorewall/params
  echo "added $parameter_name"
fi 

Add a country with few blocks. CN works just as an example, I've never stated 99% of waste landing on my servers originates from .cn.

[root@centos7 swblöck]# ./shorewall_country.sh cn
HEAD list for cn from http://localhost/cn.zone
added COUNTRY_CN
[root@centos7 swblöck]#

Add rule(s) into /etc/shorewall/rules

DROP        net:$COUNTRY_CN $FW

Performance bogged?

Testing perf by HTTP benchmarking a kvm virtualized CentOS 7 running nginx over a virtio network interface. CPU load is well under 1.0 on client (8 cores i7-6700k) and server (2 cores). Client will request a 50 KB HTML file without gzipping and by default siege and ab will make a new connection for every request. ab is also tested using keep-alive (-k).

CN DROP rules are toggled on/off and it has ~6500 blocks. Finally netem will add some latency and jitter in the last two tests tc qdisc add dev eth0 root netem delay 20ms 2ms

software	cn block in use	trans/sec	throughput MB/sec
siege -c4 -t60s -b	on	10529	350
siege -c4 -t60s -b	off	14055	467
ab -s100000	on	8190	280
ab -s100000	off	10772	369
ab -k -s100000	on	23400	803
ab -k -s100000	off	23500	805
network delay 20±2ms, siege -c4 -t60s -b	on	63	2.11
network delay 20±2ms, siege -c4 -t60s -b	off	63	2.11

Like in any benchmark these results have very little to do with real world performance :)

• ab reports request average request time rise from 0.043 to 0.093 (+50 microseconds) when blocklist is enabled and
• throughput for ab and siege drops about 25%.
• enabling keep-alive for ab shows significant performance boosts as netfilter can skip almost all rules for ESTABLISHED connections.
• 50 microseconds delay introduced by the blocklist rules becomes negligible as you "add some distance" i.e. 20±2 ms with netem.

Unscientific result in a nutshell : On the tested virtual machine each country block drop rule adds about 5 ps delay for each new connection. For long-living connections (file transfers, ssh, etc.) performance is unaffacted as connection is in ESTABLISHED state. If blocklist is HUUUUGE and therefore delay becomes noticeable, a web service could mitigate part of it with a short keep-alive(2s) when client requests related resources or fondness for minute long keep-alives could even help with the subsequent page requests.