Welcome to the second part of our tutorial on how to set up and secure a multi-server CrowdSec Security Engine installation. In the first part, I walked you through the setup of CrowdSec Security Engines across multiple servers, with one server serving as the parent and two additional machines forwarding alerts to it.

In this part, I will address security issues posed by clear HTTP communication in the previous multi-server Security Engine installation. To solve this, I propose establishing the communication between Security Engines over encrypted channels. This solution allows server-2 or server-3 to trust the server-1 identity and avoid man-in-the-middle attacks.

Using self-signed certificates

First, you need to create a certificate. This can be achieved with the following one-liner.

bash
openssl req -x509 -newkey rsa:4096 -keyout encrypted-key.pem -out cert.pem -days 365 -addext "subjectAltName = IP:172.31.100.242"

For now, the Security Engine is not able to ask for the passphrase of the private key when starting. So, you have the choice to decipher the private key by hand each time you start or reload the Security Engine or store the key unencrypted. In any way, to strip the passphrase, you can use the following:

bash
openssl rsa -in encrypted-key.pem -out key.pem

Then, the unencrypted key file can be safely deleted after the Security Engine is started.

On server-1, you need to configure the Security Engine to use the generated certificate. As seen below, the tls.cert_file and tls.key_file options in the api.server section of the following /etc/crowdec/config.yaml excerpt is set to the generated certificate file.

yaml
api:
  server:
    log_level: info
    listen_uri: 10.0.0.1:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    online_client: # Crowdsec API credentials (to push signals and receive bad 

    tls:
      cert_file: /etc/crowdsec/ssl/cert.pem
      key_file: /etc/crowdsec/ssl/key.pem

On the client side, configuration changes happen in two files. First, modify /etc/crowdec/config.yaml to accept self-signed certificates by setting the insecure_skip_verify to true.

You also need to change HTTP for HTTPS in the /etc/crowdsec/local_api_credentials.yaml file in order to reflect the changes. This small change has to be done on all three servers (server-1, server-2, and server-3).

Note: Please keep in mind that this LAPI configuration has to be done on server-1 as well if it's used as a log processor too.

yaml
url: https://10.0.0.1:8080/
login: <login>
password: <password>

Side note: Obviously using self-signed certificates doesn't provide any confidence over ownership on the LAPI server. Servers using the service (server-2 or server-3 in this setup) are still vulnerable to man-in-the-middle attacks, but at least this setup provides encrypted communications. That's the reason why the InsecureSkipVerify option is needed.

Using a Certificate Authority-issued certificate

Let's Encrypt, or services like Amazon ACM, can be leveraged to workaround the InsecureSkipVerify, by issuing a certificate for a fully qualified domain name that can be added to /etc/hosts or to a local DNS server. /etc/crowdsec/local_api_credentials.yaml can then be filled with this specified fully qualified domain name.

This indeed works and prevents the InsecureSkipVerify option from being set. This ensures that communication between client and server can't be tampered with as long as the DNS configuration can be trusted, but should still be considered as a workaround.

Using a PKI

The process of configuring and managing an SSL Public Key Infrastructure (PKI) falls outside the scope of this tutorial, but I highly recommend you take a look at the official OpenSSL documentation. The simple PKI scenario is enough for this Security Engine setup.

Following the OpenSSL documentation, there are a few things worth mentioning.

To be usable in our CrowdSec TLS scenario, the certificate requests have to be issued with a subject alternative name corresponding to the IP of the Crowdsec LAPI server. This can be done by positioning the SAN environment variable when invoking OpenSSL for the certificate request (see step 3.3 in the OpenSSL simple PKI scenario).

bash
SAN=IP:10.0.0.1 openssl req -new -config etc/server.conf -out certs/crowdsec.csr -keyout certs/crowdsec.key

The public part of the root and the signing certificates (bundle file created at step 4.5 in the OpenSSL simple PKI scenario) have to be added to the local certificate store before starting the CrowdSec Security Engine. In this setup, this is required to connect to the LAPI server. There are many ways to do so, golang sources specify where certificates are expected, or you can use the SSL_CERT_FILE environment variable in the systemd service file to specify where to find the certificate when launching the Security Engine.

Updated note on CrowdSec and TLS authentication

After the first publication of this article, we added a new feature to the Security Engine you are now able not only to secure communication over TLS but also ensure authentication with certificate. In the official documentation you can find a great example that shows how TLS authentication can be done using certificates between Security Engines or between Security Engine and Remediation Component.

Conclusion

This article gives some highlights on how to secure communications between different CrowdSec Security Engine installations. The considered use case is Security Engine installations in a private network, but this can also be deployed on a public network with communication over the internet. In such a case, a third-party certificate would easily do the trick.

Depending on the needs, I proposed three different ways to achieve secure TLS communications between your Security Engines — using self-signed certificates, using certificates issued by a Certificate Authority, and using an SSL Public Key Infrastructure.

The first scenario, with self-signed certificates, only applies if you want to ensure encrypted communication with no need for authentication. The second scenario proposed may only be considered as a workaround when you have the possibility to modify local DNS resolutions. The third proposed scenario is the most complicated but would fit in most use cases and may be the way to go when security concerns are high.

I hope this tutorial comes in handy. Thanks for reading and stay tuned!

If you have any questions or feedback, don’t hesitate to reach out to us on our community platforms on Discord and Discourse.