Fail2ban and Cloudflare Site Specific Banning

Tags
cloudflarefail2ban
Created
Sep 2, 2022 2:16 PM
Status
Done

Intro

I’ve been working on trying to get fail2ban to work with Cloudflare. I ended up just using Cloudflare instead to protect logins. But some people still want to use fail2ban.

GridPane Post - My Original Post and Root Issue

Here’s the original post on GridPane

Hello,

If you’re using the Fail2Ban + Cloudflare integration, it will only work for domain zones owned under the account you generated the Cloudflare token. It will not work for domain zones to which you have access as a member.

This is troublesome if you prescribe to the “client should retain all” methodology, or you can’t put a domain under your account.

The current endpoint is specific to accounting rules, which aren’t used for member access accounts or domains.

https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules

Instead, fail2ban should be looking up the domain name zone id and then using the following endpoint, which is domain-specific and not account-specific.

https://api.cloudflare.com/client/v4/zones/<zoneid>/firewall/access_rules/rules

The only problem is getting the domain name to do a zoneid lookup. However if there was a helper application that would take the logpath, cloudflare credentials, IP to be banned, and message then this could function.

I’ve put in a Roadmap request Fail2ban and Cloudflare Zones under Member Accounts - Stack Feature Requests - GridPane 3

Resolution

The issue only exists with GridPane’s OLS stack, so we really just need to address OLS right now.

You can add to the OLS configuration using the /var/www/<site>/ols/blocks.conf file. So I just made another logFormat config.

accesslog $VH_ROOT/logs/domain.com.access.fail2ban.log {
  useServer               0
  logFormat               "%t %h %T - %v "%r" %>s %b %T "%{Referer}i" "%{User-agent}i"
  logHeaders              7
  rollingSize             1M
  keepDays                0
  compressArchive         1
}

This config will generate an additional access log with the proper format for Fail2ban. After you add the above code, you need to regenerate the OLS config by running the following command

gpols site domain.com

If you visit domain.com a new log entry will be present in /var/www/domain.com/logs/domain.com.access.fail2ban.log

Point your fail2ban configuration here, and you can now correctly ban IPs using Cloudflare under the Cloudflare WAF for the individual domain.

Resolution Part 2 - Logrotate

Since we are now creating an additional set of logs, we need to ensure they’re being rotated as, we don’t want them to stick around forever and ultimately take up too much space.

Fortunately, Openlitespeed does this automatically; the above code has a line keepDays where you can specify how many days to keep the logs. It’s currently set to 0, which results in the default 30 days. I couldn’t find specific documentation but looking at how the GridPane logs rotate this seems to be true.

Back and Forth Discussion on the GridPane Community Forum

Another GridPane Customer having the same issue.

Hey @jtrask

Reinvigorating this thread as it's just come up for me again today that we need this! I did some research and poking and I've come up with a bit of a plan.

tl;dr - the fail2ban action needs to know the website domain name, but I don't know how to get that through as a variable in the action. This could either come from the name of the log file that was being read, or as a variable that fail2ban has (but I couldn't find one)

My thought process has been this:

  • The filter watches a log file for the site (configured in jail.local)
  • jail.local calls the action
  • The action makes the cloudflare API call

The subtlety here is that the action has no idea what domain the filter was applied on. If it did then adding / modifying the ban action to be two calls would be fairly straightforward.


actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
            -d '{"mode":"block","configuration":{"target":"<cftarget>","value":"<ip>"},"notes":"Fail2Ban <name>"}' \
            <_cf_api_url>

This would become two calls in bash

  1. to fetch the domain zone with account and zone id
  • jq the accountid or zone id (depending on your preference) out of this
  1. Send the IP block to the account OR to zone (depending on preference)

My issue right now is I don't know enough about fail2ban to be able to get the FQDN of the site that triggered the rule (even the log file name would do as this contains the FQDN).

My Response #1

You hit the nail on the head, I've got some code for getting the zoneid using jq and setting the zone per page higher. There is also code for adding firewall rules which you'll need as Cloudflare Access Rules are user specific (https://api.cloudflare.com/#ip-access-rules-for-a-user-get-ip-access-rules) and won't work for domains you're a member of.

https://github.com/managingwp/cloudflare-wordpress-rules/blob/main/cloudflare-wordpress-rules.sh

>My issue right now is I don't know enough about fail2ban to be able to get the FQDN of the site >that triggered the rule (even the log file name would do as this contains the FQDN).

Also there is this thread on how to grab data in the log file via the failregexp command

https://github.com/fail2ban/fail2ban/issues/1935

# jail.local or filter.d/myfilter.conf
failregex = ^failure from <HOST>, port: <F-PORT>\\d+</F-PORT>, protocol: <F-PROTO>\\S+</F-PROTO> - <F-REASON>.*</F-REASON>$

# myaction.conf:
actionban = echo ip=<ip> port=<F-PORT> proto=<F-PROTO> reason="<F-REASON>" >> /tmp/test.log

It's all here, you just need to wrap it all up. You'd need to have a firewall rule called fail2ban that would block and you would add and remove IP's from it. You could use the 1 free IP List which might be easier. (https://developers.cloudflare.com/firewall/cf-firewall-rules/rules-lists/)

actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \\
            -d '{"mode":"block","configuration":{"target":"<cftarget>","value":"<ip>"},"notes":"Fail2Ban <name>"}' \\
            <_cf_api_url>

I'd use a shell script here and have the option of banning and ip on a domain and unbanning an ip on a domain.

The GridPane Customer Goes Further Response #2

Thanks @jtrask good info to start.

Here's where I am at (verbose as I can be in case its useful to others in the future).

I'm looking at GP OLS Logs and they look like this for POST requests:

"156.96.151.132 - - [03/Sep/2022:09:20:46 +1000] "POST /xmlrpc.php HTTP/2" 403 705 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"

This gives us the IP Address of the originating host, but not the domain name of the website being called in GP, so I have had to change the OLS log format to display it so it can be pulled by failregex.

To do that I've had to mess with the GP managed /usr/local/lsws/conf/vhosts/*/vhconf.conf and change the logFormat to include %v

Original:

logFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"

New:

logFormat "%h %v %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"

Side note that this is impractical to do on every vhost - maybe Team GridPane can modify to have this as standard @stevebell @aon perhaps can you comment on how this would be possible?

Now we have a log line as follows: 

"156.96.151.132 mydomain.com - - [03/Sep/2022:09:20:46 +1000] "POST /xmlrpc.php HTTP/2" 403 705 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"

In the /etc/fail2ban/filter.d/wordpress-xmlrpc.conf I have modified to include this domain using the F- Notation from your link

Original: failregex = <HOST> .* "POST /xmlrpc.php .* 200 New: failregex = <HOST> <F-FQDN>\S+</F-FQDN> .* "POST /xmlrpc.php .* 200

I created my own version of the /etc/fail2ban/action.d/gpcloudflare.conf to then echo the output the domain

actionban = echo host=<HOST> OR <ip> site=<F-FQDN> taget=<cftarget> That's as far as I have been today, and because the vhconf.conf gets regenerated and is not persistent I think I'm stuck.

My Response #2 about the Nginx + OLS Log differences

This is something I was going to submit as a support ticket to GridPane support. This is something that should be fixed for OLS.

GridPane Client’s Response #3

I actually asked already. <GridPane Employee Name> said it won't be done " The reason for not adding the site name in the log is because its redundant for normal use cases, we already know which site's log we are reading. "

My Response #3

Interesting. <GridPane Employee Name>isn't wrong, however, the Nginx log format includes the $http_host variable, which is technically redundant. It would be great if the logformats were the same across both web servers, less work for everyone involved. I created this so maybe @aon can see if this is something that can be addressed. Hopefully they don't take out the $http_host as it's technically redundant, otherwise, we have to find another way to address this issue. Upvoty Standard Log Output for Nginx and OLS - Stack Feature Requests - GridPane 2 Right now Nginx and OLS do not output the same log format. This causes issues for software such as Fail2ban and GoAccess, requiring different configuration for each server type or just not working at all. GP OLS logFormat "%h %l %u %t "%r" %&

GridPane Employee Response

Hi Jordan,

That is indeed interesting, providing a consistent logging format would be beneficial for interoperability on both stacks. I am going to open an internal ticket to discuss this.

It's an enhancement/feature request, although small, it would come after any current ongoing development/bug fixes. You should check if it's possible to use the file name in the meantime.

GridPane Client Response #4

I've posted a question in stackoverflow What log file triggered the fail2ban action? - Stack Overflow 1  and as a discussion in github Use logfile path / name in the action? · Discussion #3350 · fail2ban/fail2ban · GitHub 2

GridPane Client Response #5

Summary of the response at Use logfile path / name in the action? · Discussion #3350 · fail2ban/fail2ban · GitHub "Unfortunately not - the information about logfile where it is occurred isn't stored in the ticket at the moment.One can surely patch fail2ban (basically enough to patch fail2ban/server/filter.py )" There's no chance I'm going to patch the core files of fail2ban, so is there a chance of another interim workaround for the log file format?