Containers

Secure your server from known baddies using CrowdSec & Swag

PTS

Jan 14, 2024 • 11 min read

Although it doesn't have a fancy GUI, SWAG is my go-to reverse proxy. One of the things which drew me to it is it just works, and comes with fail2ban nicely integrated.

I like fail2ban. It's pretty straightforward. I'd heard of CrowdSec, but found it a little more finicky to set up the bouncers and integrations. Well, SWAG now has its own mod and settings to integrate seamlessly with CrowdSec, and this guide will show you how.

My Setup

Synology DS920+
Docker version 20.10.23
Docker-compose version v2.9.0-6413

Prerequisites

Before we get started, you'll need the following:

Docker and docker-compose installed on your machine
The ability to SSH / use CLI/terminal on your machine
Some sort of access and relevant permissions to manage and create new directories
A working SWAG instance (click here for info on how to set that up if you need it)

Setting up the CrowdSec Container

This is pretty straightforward to do. First, we're going to create our required directories:

SSH into your machine and navigate to the directory you create your docker app config folders
Type the following:

mkdir crowdsec && cd crowdsec && mkdir config data && touch docker-compose.yml

this creates our crowdsec directory, the config and data folders inside it, and the file we'll need to modify

Open the docker-compose.yml to edit it, and copy paste the following:

networks:
  default:
    name: #insert the network your SWAG container is attached to
    external: true

services:
  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - GID= #insert GID here
      - COLLECTIONS=crowdsecurity/nginx crowdsecurity/http-cve crowdsecurity/whitelist-good-actors
      - CUSTOM_HOSTNAME=myservername #change
    volumes:
      - ./config:/etc/crowdsec:rw
      - ./data:/var/lib/crowdsec/data:rw  
      - /var/log:/var/log/host:to
      - $SWAGNGINXLOG:/var/log/swag:ro #change `$SWAGNGINXLOG` to point at your swag's nginx log mapped directory (found in `/config/log/nginx`)
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true

Things to be aware of or modify in the above:

Add the docker network name your existing SWAG instance is attached to (line 3)
Add a GID for a group which has access to read SWAG and host logs (line 11)
Change your CUSTOM_HOSTNAME to any name you like (line 13)
Make sure the /var/log/swag:ro points at the correct mapped folder on your NAS

Let's get the container running. From the same directory your docker-compose.yml is located:

docker-compose -p "crowdsec" up -d

Once it's created, follow along with the logs:

docker logs -f crowdsec

Look out for the line msg="Starting processing data" at which point you want to STOP the container.

We now want to modify some CrowdSec files which will have been populated and should look like this:

Locate aqcuis.yaml and replace the contents with the following:

filenames: 
  - /var/log/swag/*
labels:
  type: nginx
---
filenames: 
  - /var/log/host/auth.log*
labels:
  type: syslog

Save it and back out so you're back in your CLI.

Let's also take a moment to create an alias for the commands we'll want to run inside the container. Type the following and press enter:

alias cscli="docker exec -t crowdsec cscli"

💡

Note that unless you save your aliases to a .bashrc file or similar, they need to be created for every SSH session

Start the container again, and follow the logs
You want to wait until you see something like the following

level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
level=info msg="Adding file /var/log/swag/access.log to datasources" type=file
level=info msg="Adding file /var/log/swag/error.log to datasources" type=file
level=info msg="Adding file /var/log/swag/unauthorized.log to datasources" type=file

Type the following into your CLI:

cscli metrics

Something like this should be returned:

If you don't have anything in acquisition or bucket, it may simply be that your logs haven't had anything new written

You can use commands such as cscli decisions list to check what local bans/decisions have been made, and cscli decisions alerts inspect <alertid> to get more information on a particular alert.

At this time we have one more command to run:

cscli bouncers add bouncer-swag

This will give you an API key

Save the API key this spits out - we'll need it soon

Integrating with SWAG

We need to add a couple of lines to our SWAG docker-compose.yml:

services:
  swag: #2-in-1 SSL certs from letsencrypt, and fail2ban
    image: ghcr.io/linuxserver/swag
    container_name: swag
    cap_add:
      - NET_ADMIN
    environment:
      - PUID=
      - PGID=
      - URL=
      - SUBDOMAINS=
      - VALIDATION=
      - DNSPLUGIN=
      - EMAIL=
      - ONLY_SUBDOMAINS=
      - DOCKER_MODS=linuxserver/mods:swag-crowdsec
      - CROWDSEC_API_KEY=$SWAGAPICROWDSEC
      - CROWDSEC_LAPI_URL=http://crowdsec:8080
      - CROWDSEC_MODE=stream
      - CROWDSEC_VERSION=v1.0.7
      - CROWDSEC_CAPTCHA_PROVIDER=recaptcha      
      - CROWDSEC_SITE_KEY=$CROWDSECSITEKEY
      - CROWDSEC_SECRET_KEY=$CROWDSECSECRETKEY
    volumes:
      - ./swag:/config
    ports:
      - 443:443

❗

Note that the above only includes environment variables specific to Crowdsec, and that CROWDSEC_VERSION must be locked to v1.0.7 or you will experience auth issues when signing in to your reverse-proxied services

DOCKER_MODS must have the crowdsec mod (you can separate mods by using | if you already have others)
As of March 2024 the crowdsec version is set and locked due a recent bouncer update which has made it incompatible with later versions
Insert the API key we got in the last step
Provided your SWAG and CrowdSec containers are on the same docker network, you can keep the LAPI_URL the same
I prefer stream as the mode, but you can change this to live if you want

Captchas

I originally tried to set this up without a captcha provider but it led to huge timeouts when serving my sites, both locally and externally. I couldn't find a way to turn this off, so I set one up.

The link here will give you the options of which captcha to use, I set up the Google recaptcha (v3) which works well and is very quick.

Back in the compose, make sure you:

add your provider (if not recaptcha)
add your SITE_KEY
add your SECRET_KEY

IPv4 vs IPv6

SWAG automatically tries to access captcha on both IP modes. If you only use IPv4 then you need to change the following to avoid a loop:

Inside your SWAG /config folder, navigate to /nginx/resolver.conf
Add ipv6=off to the end of the line. Most will look like this:

resolver 127.0.0.11  valid=30s ipv6=off;

Spinning it all up

Recreate your SWAG container
Follow the logs, and at the end you should see nginx: [alert] [lua] crowdsec_nginx.conf:4):8: [Crowdsec] Initialisation done or something very similar
You should also notice that you now have a crowdsec folder in your root swag config folder:

This is where the settings for your bouncer are stored. Any changes you want to make here will always be overwritten by the environment variables in your SWAG docker-compose. If you want to change any, a list of the variables can be found here.

Let's go back to our cli and see how things are in crowdsec with another cscli metrics.

In the time it's taken me to write this guide, we can see we have an alert, so let's see what cscli decision list returns:

Then for more info, cscli alerts inspect 2 returns:

Good to know it works! However, how would I have known otherwise?

Adding Notifications

The folder tree includes a /notifications folder which has various options for configuring notifications:

After setting one up, you then need to visit the profiles.yaml to uncomment notifications and the method of your choice. Below, I chose email:

Now I will receive an email when an IP gets banned, however there are other methods such as Discord notifications etc. which some users may prefer. To do this:

Create a discord.yaml in the /notifications folder, and copy-paste the following:

type: http

name: discord

log_level: info

format: |
  {
    "content": null,
    "embeds": [
      {{range . -}}
      {{$alert := . -}}
      {{range .Decisions -}}
      {{if $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":flag_{{ $alert.Source.Cn | lower }}: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}. <https://www.shodan.io/host/{{$alert.Source.IP}}>",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680",
        "image": {
          "url": "https://www.mapquestapi.com/staticmap/v5/map?center={{$alert.Source.Latitude}},{{$alert.Source.Longitude}}&size=500,300&key=<MAPQUEST_API_KEY>"
        }
      }
      {{end}}
      {{if not $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":pirate_flag: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}. <https://www.shodan.io/host/{{$alert.Source.IP}}>",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680"
      }
      {{end}}
      {{end -}}
      {{end -}}
    ]
  }

url: https://discord.com/api/webhooks/<DISCORD_WEBHOOK_KEY>

method: POST

headers:
  Content-Type: application/json

You'll need to add a MAPQUEST_API_KEY (line 21)
Don't forget to create your Discord webhook integration on your Discord server to replace DISCORD_WEBHOOK_KEY on line 38
In your profile.yaml you will need to add - discord under notifications:

Adding Dashboard Visualisation

We're now going to add both Prometheus and Grafana to our stack. What will this do? Well Prometheus is a scraper which will pull various metrics (like the ones we find by typing cscli metrics) and Grafana will put them into pretty graphs and charts for us.

Preparing to modify our CrowdSec compose

Prometheus and Grafana have folder requirements.

In SSH/CLI, navigate to your crowdsec directory
Copy/paste the following

mkdir prometheus grafana && cd prometheus && mkdir data etc && cd etc && touch prometheus.yml && cd .. && cd ..

This creates our Prometheus and Grafana folders, creates data and etc folders inside Prometheus, and creates a prometheus.yml inside the etc folder

Changing the CrowdSec compose

Prometheus and Grafana don't need to be on the same network as our SWAG instance (unless you want to reverse proxy them) so we're going to create a new crowdsec network and add CrowdSec, Prometheus and Grafana to them, all in the next compose:

networks:
  default:
    name: #insert the network your SWAG container is attached to
    external: true
  crowdsec:
    name: crowdsec
    ipam:
      config:
        - subnet: 172.50.1.0/29 #make sure the subnet isn't already created or covered by another docker network

services:
  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - GID= #insert GID here
      - COLLECTIONS=crowdsecurity/nginx crowdsecurity/http-cve crowdsecurity/whitelist-good-actors
      - CUSTOM_HOSTNAME=myservername #change
    volumes:
      - ./config:/etc/crowdsec:rw
      - ./data:/var/lib/crowdsec/data:rw  
      - /var/log:/var/log/host:to
      - $SWAGNGINXLOG:/var/log/swag:ro #change `$SWAGNGINXLOG` to point at your swag's nginx log mapped directory (found in `/config/log/nginx`)
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    networks:
      - default
      - crowdsec

  prometheus:
    image: bitnami/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/data:/opt/bitnami/prometheus/data
      - ./prometheus/etc/prometheus.yml:/etc/prometheus/prometheus.yml
    security_opt:
      - no-new-privileges=true
    ports:
      - 6699:9090
    restart: unless-stopped
    depends_on:
      crowdsec:
        condition: service_started
    networks:
      - crowdsec

  grafana:
    image: grafana/grafana
    container_name: grafana
    volumes:
      - ./grafana:/var/lib/grafana
    ports:
      - 3010:3000
    depends_on:
      prometheus:
        condition: service_started
    restart: unless-stopped
    networks:
      - crowdsec

Copy/paste the above into your crowdsec docker-compose, and save it

Configuring Prometheus

Run docker-compose -p "crowdsec" up -d and watch the prometheus container's logs. Assuming no errors, run docker stop prometheus
Navigate to and open your prometheus.yml and copy paste the following into it:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "crowdsec"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["crowdsec:6060"]

save it, then restart the prometheus container docker start prometheus
switch to your browser and enter your host/NAS IP:6699

Click on Status > Targets

What we're looking for here is our crowdsec metrics state to be UP.

Adding the data source to Grafana

Still in your browser, open Grafana by typing host/NASIP:3010 (obviously don't type this verbatim, use the IP for your host or NAS)

In the left panel click Connections
Start typing 'prometheus' in to the search box, then when it shows up, click the Prometheus data source

Click Add new data source top right

Enter a name for the data source, and the URL, which should be http://prometheus:9090
Scroll down to the bottom and click Save & Test and you should see a successful response

Adding dashboards

Grafana dashboards can sometimes be confusing to set up. If you're up to it then go ahead and start pulling metrics yourself.

If however you want a little help, visit this link which will take you directly to a couple of .json dashboards. The most useful is probably the overview.

Go ahead and click Crowdsec Overview.json and then click the copy button
Head back to Grafana

Click Dashboards in the left panel, followed by New top right and then Import in the drop down box

Paste the .json you copied from GitHub into the box, then click Load

You'll notice that the `Name` has been auto-filled for you

Change the name if you wish
Select the correct data-source from the drop down box (prometheus-crowdsec) then hit Import

Something like this should be visible after clicking `Import`

Go back to the repo and follow the same steps to import any other dashboards you want

Final Thoughts

It's true that CrowdSec has many moving parts, but which part of server/network security doesn't? In my case at least, it's already proven it adds value.

Users can take a look at the various bouncers available on their site (using your Cloudflare proxy for instance) and add those in, or check out other parsers for say Home Assistant or Authelia.

Happy bouncing!