Detection Engineering Part 1: Setting up Elastic, Kibana and Fleet Server for SIEM and EDR

This post was authored by Fareed.

Introduction

In this quick guide, we will walk you through the process of installing and setting up Elastic 8 with Kibana and Fleet. Elastic Stack is a powerful open-source solution that enables efficient data management and analysis, along with robust security capabilities. By following this guide, you'll be able to get started with Elastic 8 swiftly, connect it with Kibana, and leverage Fleet to connect to the advanced security features provided by Elastic Security. Let's dive in and unlock the potential of these essential tools for your data management and security needs.

Requirement

To successfully install and set up Elastic 8 with Kibana and Fleet, you will need to prepare 2 Ubuntu servers: 

The first server will host Elastic and Kibana, which are essential components of Elastic Stack. Elastic provides powerful search and analytics capabilities, while Kibana offers a user-friendly interface for visualizing and exploring data. 

The second server will be dedicated to hosting the Fleet Server, a centralized management tool that allows you to deploy and manage Elastic agents across your infrastructure.

Install Elastic

For the Elastic and Kibana installation, we will use the first Ubuntu server. The IP address of my server is 192.168.206.130.

Download and Install Elasticsearch

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch |sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install elasticsearch

Save security autoconfiguration information

After the installation is completed, the last output will be like this:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is: +n+to6c9RoElFkHnR-vt

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

Copy the whole content and save it in a new text file for future reference as it contains information that we needed.

Configure Elasticsearch

You can configure your Elastic search by editing /etc/elasticsearch/elasticsearch.yml file. In my case, I leave it default.

Start and test Elasticsearch

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Then to test the availability of Elasticsearch, run this command:

curl -X GET -k https://elastic:<PASSWORD>@localhost:9200

Replace the <PASSWORD> with your own based on the saved autoconfiguration information.

If it is alright, you will see no errors.

Install Kibana

Run this command to install Kibana:

sudo apt install kibana
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Running the binary elasticsearch-create-enrollment-token will generate a token. Copy the whole token and we will use the token in the next command. For example, my token is like below:

eyJ2ZXIiOiI4LjguMSIsIm<MyZDU5NzcwMTNjIiwia2V5IjoiR1JJTXRvZ0JrSXlrMHRiOEtoVEc6<REDACTED>X2ZDOFVYdG9UamFEMC1QbG9jWVhXUSJ9

Run this command to set up Kibana and enter the enrollment token above.

sudo /usr/share/kibana/bin/kibana-setup

Then start Kibana using these commands

sudo systemctl enable kibana
sudo systemctl start kibana

After that, install nginx to be used to host the Kibana application

sudo apt install nginx
sudo nano /etc/nginx/sites-enabled/

In the configuration, modify the location setting with this line:

location / {
                # First attempt to serve request as file, then
                proxy_pass http://127.0.0.1:5601;
                # as directory, then fall back to displaying a 404.
                # try_files $uri $uri/ =404;
        }

Then restart nginx:

sudo systemctl restart nginx
sudo systemctl enable nginx

Now your Kibana is ready. In my case, my address is http://192.168.206.130/.

Elastic login page

The credential is elastic: +n+to6c9RoElFkHnR-vt. Replace +n+to6c9RoElFkHnR-vt with your password given in the autoconfiguration.

Install Fleet

For the Fleet server installation, we will use the second Ubuntu server. The IP address of my server is 192.168.206.140.

Copy SSL cert from Elastic server to Fleet Server

In this lab tutorial, we will use the self-sign certificate for our SSL. So we need to do a few things.

In the elastic server, run this command to setup the web server on port 8080 to transfer the certificate into the Fleet Server:

sudo su
cd /etc/elasticsearch/certs
python3 -m http.server

In the Fleet Server, download the cert and put it into /usr/local/etc/ssl/certs/elastic/ location:

mkdir -p /usr/local/etc/ssl/certs/elastic
curl http://192.168.206.130:8000/http_ca.crt -o /usr/local/etc/ssl/certs/elastic/http_ca.crt

We're done with the self-sign certificate.

Install Fleet integration in Elastic

Go to the burger icon, in the Management menu, select Integrations.

Go to Integrations page

Search for Fleet Server in the integrations search bar:

Search for Fleet Server

On the page of the Fleet Server, click the "Add Fleet Server" button.

Fleet Server integration page

In the configuration of the integration, leave it by default and click the "Save and continue" button.

Configure integration

After finishing up the integration creation, Elastic will ask you to add an Elastic agent to the Fleet Server which is the Ubuntu server that we prepare for the Fleet Server. Click "Add Elastic Agent to your hosts" to proceed.

Click Add Elastic Agent

During the enrollment process for the Fleet, keep all configurations at their default values except for number 3. Please include the URL, which should be the IP address of the dedicated Ubuntu server intended for the installation of the Fleet Server.

Use the Fleet Server IP address in the URL

Then click Add host and click Generate service token button.

Elastic then will give options of commands to be installed in the OS. Choose Linux Tar as we use the Ubuntu server for this setup.

Copy the whole line and add another parameter:

\ --fleet-server-es-ca=/usr/local/etc/ssl/certs/elastic/http_ca.crt \ --insecure

Thus, the final command will be like this:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-
8.8.1-linux-x86_64.tar.gz 
tar xzvf elastic-agent-8.8.1-linux-x86_64.tar.gz 
cd elastic-agent-8.8.1-linux-x86_64 
sudo ./elastic-agent install --fleet-server-es=https://192.168.206.130:9200  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2ODY2ODMxNTcyMzk6QkpYeTVWenVTUHkwemdGT0Y4bjVZdw --fleet-server-policy=f27845e0-0a1a-11ee-a6fc-d10607a4c7d4 --fleet-server-es-ca-trusted-fingerprint=35afb655b26673c49f6813dca0140752b4eab75e8089a0e3a244732d5977013c --fleet-server-port=8220 --fleet-server-es-ca=/usr/local/etc/ssl/certs/elastic/http_ca.crt --insecure

Change the IP address with your own.

After that, the Elastic agent will be successfully installed in the Fleet Server.

The installation of the agent in the Fleet Server is successful

In the conclusion of our installation and setup guide, we will take full advantage of the Fleet Server we prepared earlier. The Fleet Server serves as a centralized hub for agent installation and communication, enabling seamless integration with Windows Logging for SIEM (Security Information and Event Management) and Elastic Security.

With the Fleet Server in place, you can easily deploy Elastic agents on your Windows endpoints, ensuring comprehensive data collection for security monitoring. The agents will gather valuable information from various sources, including Windows logs, system metrics, and network data, providing you with deep insights into potential security threats and anomalies.

Install the Fleet agent on Windows

To install Fleet Agent on Windows, go to the Fleet page. Then, click "Add agent".

Add new agent
Then, select a policy. If you don't have one, create a new policy. In my case, I put my policy name as "Windows" for the Windows endpoint dedicated.

Select policy

For number 3, select Windows and copy the whole Powershell lines:
Select Windows installation script

Copy the Powershell lines and paste them into the Windows endpoint that you want to monitor. Make sure to run the Powershell in Administrator mode.

Don't forget to add the "--insecure" parameter at the final line of the script.

The script will look like below:
$ProgressPreference = 'SilentlyContinue' Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.8.1-windows-x86_64.zip -OutFile elastic-agent-8.8.1-windows-x86_64.zip 
Expand-Archive .\elastic-agent-8.8.1-windows-x86_64.zip -DestinationPath . 
cd elastic-agent-8.8.1-windows-x86_64 
.\elastic-agent.exe install --url=https://192.168.206.140:8220 --enrollment-token=ZmhKRnRvZ0JrSXlrMHRiOHVDZ046NjF6Z21RMUdTdXVqQUVTbUdOMG0zZw== --insecure

Agent successfully installed

So after this, we just add integration in the policy for more features and capabilities. We will touch this in the next series.

Extra

In the next part of this series, we will delve into the topic of "Harnessing Elastic Security, Sysmon, Windows Logs, and Aurora Lite for Threat Monitoring." We will explore how to utilize Elastic Security alongside Sysmon, a powerful Windows system monitoring tool, and Aurora Lite. Together, these tools provide an integrated solution for threat monitoring, enabling you to proactively detect and respond to potential security incidents. Stay tuned for the next installment, where we will uncover the synergies between these tools and showcase practical use cases for effective threat monitoring and response.