Elastic Stack

The Elastic Stack (often called ELK) is Elasticsearch for storage and search, Logstash for processing, Kibana for dashboards and Beats or Elastic Agent for shipping data. Use it for central logs and search in a lab that can spare the RAM. It is heavy compared with Loki and Alloy, and Graylog is the all-in-one alternative. These notes follow the 9.5 documentation.

Documentation: Elasticsearch Docker install · Filebeat

Host Setting #

Elasticsearch needs a larger memory map limit:

sudo sysctl -w vm.max_map_count=1048576
sysctl vm.max_map_count

Make it permanent in /etc/sysctl.d/. Elastic’s current page says to raise it to 1048576 when the default is lower. Older guides used 262144.

Single Node (Docker) #

From the Elastic docs:

docker network create elastic
docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:<VERSION>

The initial elastic password and a Kibana enrolment token print once on first start. Save them, or regenerate:

Test with the CA certificate and the elastic credentials from a variable or .netrc, not typed on the command line:

curl --cacert http_ca.crt --netrc https://localhost:9200

Security is on by default, with HTTPS, and the -m 1GB limit is only for a quick trial. Pin the image tag. A production-like compose file with Kibana and ELASTIC_PASSWORD was not verified here.

Ports #

Common API Calls #

These are standard Elasticsearch calls, not checked against the 9.5 reference in this pass:

Run them with curl against https://localhost:9200/<PATH> using the CA certificate and credentials as above, or in Kibana’s Dev Tools.

Filebeat #

Install the .deb from artifacts.elastic.co and match the version to your Elasticsearch:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-<VERSION>-amd64.deb
sudo dpkg -i filebeat-<VERSION>-amd64.deb

Minimal /etc/filebeat/filebeat.yml:

filebeat.inputs:
  - type: filestream
    id: system-logs
    paths:
      - /var/log/*.log

output.elasticsearch:
  hosts: ["https://<ES_HOST>:9200"]
  username: "filebeat_writer"
  password: "${FILEBEAT_PASSWORD}"
  ssl.enabled: true
  ssl.ca_trusted_fingerprint: "<CA_FINGERPRINT>"

Every filestream input needs a unique id. Use a dedicated writer user with limited rights, not elastic.

Logstash #

A minimal pipeline that receives Beats and writes to Elasticsearch:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://<ES_HOST>:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

Filters (grok, date) and TLS options for the output were not verified. Many homelabs skip Logstash and ship straight from Beats to Elasticsearch.

Licence and Forks #

Elastic added AGPL as an option again in 2024, alongside ELv2 and SSPL. Check the licence for the version you run before relying on this. OpenSearch is the Apache 2.0 fork of Elasticsearch 7.10 (from general knowledge, not fetched here), and Graylog’s Data Node is based on it.

Not Verified #

Kibana compose and the kibana_system password, ELASTIC_PASSWORD and ES_JAVA_OPTS environment variables, Index Lifecycle Management, filebeat test output, and Logstash filters.

See Also #

• Graylog
• Loki and Alloy
• Grafana
• Prometheus