We are happy to announce the official release of CrowdSec v.1.0.x which introduces several improvements to the previous version, including a major architectural change: the introduction of a local REST API.

This local API allows all components to communicate together in a more efficient way, supporting more complex architectures, while keeping it simple for mono-machines users. It also makes the creation of bouncers (the remediation component) much simpler and renders them more resilient to upcoming changes, limiting the necessary maintenance time.

In the new 1.0 release, while the Debian package should be out soon, the CrowdSec architecture has been deeply remodeled:

All CrowdSec components (the agent reading logs, cscli for humans, and bouncers to deter the bad guys) can now communicate together via a REST API, instead of reading or writing directly in the database. With this new version, only the local API service will interact with the database (supports SQLite, PostgreSQL and MySQL).

In this tutorial, I am going to cover how to install and run CrowdSec on a Linux server:

• Setup
• Testing of detection capabilities

• Setup of a bouncer
• Observability

Environment

The machine I used for this test is a Debian 10 buster t2.medium EC2.

To make it more relevant, let’s start by installing nginx on it:

$ sudo apt-get update$ sudo apt-get install nginx

I’ve configured the security groups so that both ssh (tcp/22) and http (tcp/80) can be reached from the outside world. This will be useful to simulate attacks later on.

Installation

First of all, let’s grab the latest version of CrowdSec:

$ curl -s https://api.github.com/repos/crowdsecurity/crowdsec/releases/latest | grep browser_download_url| cut -d ‘”‘ -f 4  | wget -i –

(note: you can also download it on our GitHub page)

Here is the installation process:

$ tar xvzf crowdsec-release.tgz $ cd crowdsec-v1.0.0/$ sudo ./wizard.sh -i

The wizard is here to help the user during installation and configuration. And I have to say it’s very helpful!

First of all, it’s going to identify services present on the machine:

Here, the wizard allows us to choose which services to monitor. I’m going to go with the default option and monitor all three services: Nginx, sshd and the Linux system itself.

For each service, the wizard identifies the associated log files and asks us to confirm it (we’re going to follow the defaults again):

Caption: nginx log files

Once the services and the associated log files have been properly identified (which is crucial, as this is where CrowdSec will get its information from), the wizard is prompting with suggested collections.

A collection is a set of configurations that aims at creating a coherent ensemble to protect a technological stack. For example, the crowdsecurity/sshd collection contains a parser for sshd logs and a scenario to detect ssh bruteforce and ssh user enumeration.

The suggested collections are based on the services that we decided to protect. 

One last step completed by the wizard is to deploy generic whitelists that will prevent from banning private IP addresses. It is also reminding the user that CrowdSec itself is in charge of detecting any malevolent IP but will not ban a single one of them. You need to download a bouncer to be able to block attacks. That’s a very important thing to remember: CrowdSec detects attacks, bouncers block them. 

Now that the initial setup is done, CrowdSec should be up and running.

First steps with Crowdsec

We installed CrowdSec and we should already have coverage for common internet background noise, let’s check this out!

Attacking our web server with wapiti

First of all, I’m going to simulate a web application vulnerability scan on my nginx service using wapiti (you need to do it from an external ip, keep in mind that private IPs are whitelisted by default):

ATTACKER$ wapiti   -u http://34.248.33.108/[*] Saving scan state, please wait… Note========This scan has been saved in the file /home/admin/.wapiti/scans/34.248.33.108_folder_b753f4f6.db…

On our freshly equipped machine, we can see the attacks in the logs:

We can see here that my IP triggered different scenarios :

• crowdsecurity/http-path-traversal-probing : detect path traversal probing attempts patterns in the URI or the GET parameters
• crowdsecurity/http-sqli-probbing-detection : detect obvious SQL injection probing attempts patterns in the URI or the GET parameters

Bear in mind that the attacked website is an empty nginx server,the scanner would perform a lot of other actions that would lead to further detections if this was a real website.

Checking results with cscli

One of the main tools to interact with the CrowdSec service is cscli, and one of its features is the visualization of active decisions and past alerts :

When using the command cscli decisions list we can see active decisions at any given time, while cscli alerts list is going to show us a list of past alerts (even if decisions are expired or if the alert didn’t lead to a decision).

We can also inspect a given alert to get more details with cscli alerts inspect -d <ID> (the ID is displayed in the left column of alerts list).

cscli offers various other features, but one that might be relevant right now is to see exactly which parsers and scenarios are installed in the default setup:

Observability

Observability (especially a software that might take defensive countermeasure) is always a key point for a security solution, and – besides the obvious “tail the logfile” – CrowdSec offers two ways to achieve this : metabase dashboards, and prometheus metrics.

Observability : Dashboard

cscli (again!) allow us to deploy a dashboard relying on metabase and docker.

First of all, let’s install docker following their official documentation.

 (note: if you’re using an AWS EC2 instance as in the howto, be sure to expose tcp/3000 to be able to access your dashboard)

cscli dashboard setup enables you to deploy a new metabase dashboard running on docker with a random password:

Observability: Prometheus Metrics

While some users love visual dashboards, others might prefer different kinds of metrics, and this is where CrowdSec’s Prometheus integration comes into play.

One way to visualize these metrics is via cscli metrics:

The cscli metrics only expose a subset of Prometheus metrics. These are important ones for a system administrator. A more detailed description of the metrics can be found in the dedicated documentation section. Without going into details, you can see that the metrics are split into various sections :

• Buckets: How many buckets of each type were created, poured and/or have overflowed since the daemon startup
• Acquisition: How many lines/events were read from each of the specified sources, and whether or not they were parsed and/or poured to buckets later
• Parser: How many lines/events were delivered to each parser, and whether or not the parser succeeded into processing the mentioned events
• Local api: How many times each route was hit and so on

However, viewing Crowdsec’s Prometheus metrics via cscli metrics, even if convenient, doesn’t do justice to Prometheus. It would be out of scope to deep dive into Prometheus, but we can however have a quick glance at what CrowdSec’s Prometheus metrics look like in Grafana:

Defending ourselves: Bouncers

So far, we looked into CrowdSec’s detection capabilities and how to gain observability on what is going on. However, to protect ourselves, we need to be able to block attackers, and this where bouncers have a major part to play: CrowdSec detects, bouncers deter.

Bouncers are working by querying CrowdSec’s API to know when to block an IP. You can download bouncers directly on the Hub: 

Here, we are going to use cs-firewall-bouncer: it will ban directly any malevolent IP at firewall level (using iptables or nftables)

(note: if you used your IP to simulate attacks, unbanning your IP before going further is recommended: sudo cscli decisions delete -i X.X.X.X)

Installing the bouncer

First, let’s download the selected bouncer from the Hub:

$ wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.5/cs-firewall-bouncer.tgz$ tar xvzf cs-firewall-bouncer.tgz$ cd cs-firewall-bouncer-v0.0.5/

The bouncer can be installed with a simple install script:

(note: the install script will check whether you have iptables or nftables installed and/or ask you to install it if it’s missing)

As stated earlier, bouncers communicate with CrowdSec via a REST API, and we can check that the bouncer is registered on the API:

The last command (sudo cscli bouncers list) shows our newly installed bouncer!

Testing the bouncer

Warning: before going further with the next step, ensure that you have another IP available to access your machine and not kick yourself out (using your smartphone internet connection will work)

Now that we have a bouncer to protect us, let’s try again, shall we?

Trying to access the server at the end of the scan :

ATTACKER$ curl –connect-timeout 1 http://34.248.33.108/curl: (28) Connection timed out after 1001 milliseconds

Now, let’s see how it turns out rom the defender point of vie :

For the most curious, the cs-firewall-bouncer uses either nftables or iptables. When using nftables (as it’s the case on debian 10 by default), it is creating and maintaining two tables named crowdsec and crowdsec6 (for ipv4 and ipv6 respectively).

$ sudo nft list ruleset…table ip crowdsec { set crowdsec_blocklist { type ipv4_addr elements = { 3.22.63.25, 3.214.184.223,     3.235.62.151, 3.236.112.98,     13.66.209.11, 17.58.98.156, …                        } } chain crowdsec_chain { type filter hook input priority 0; policy accept; ip saddr @crowdsec_blocklist drop }}table ip6 crowdsec6 { set crowdsec6_blocklist { type ipv6_addr } chain crowdsec6_chain { type filter hook input priority 0; policy accept; ip6 saddr @crowdsec6_blocklist drop }}

You can change the firewall backend used by the bouncer in /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml by changing the mode from nftables to iptables for example (ipset is required for iptables mode).

The CrowdSec team would love to hear your feedback about this latest release. If you are interested in testing the software or would like to get in touch with them, below are a few useful links. Happy 2021!

• Download CrowdSec v.1.0
• Website
• GitHub
• Gitter