Elasticsearch + Fluentd + Kibana Setup (EFK) with Docker
In this article, we will see how to collect Docker logs to EFK (Elasticsearch + Fluentd + Kibana) stack. The example uses Docker Compose for setting up multiple containers.
But before that let us understand that what is Elasticsearch, Fluentd, and kibana.
1. Elasticsearch :- Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
2. Kibana:- Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data
3. Fluentd:- Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.
How to setup EFK stack Step by Step :-
STEP 1:- First of all create a
docker-compose.yaml file for EFK stack. In this demo here we are using Opendistro docker images for security , but you can use official image.
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
soft: 262144 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
- 9600:9600 # required for Performance Analyzer
STEP 2:- Then create a folder name called fluentd and in that folder create
Dockerfile . it looks like
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "3.5.2"]
STEP 3:- After that create a folder conf also create a fluent.conf file inside the fluentd directory. it looks like
@type memory # or file
In this config you can remove user and password if you are not using opendistro images and change your hosts . Now run the docker compose file by this command.
docker-compose up -d
STEP 4:- Finally EFK stack is ready now lauch your application and send the logs into Elasticsearch. Here i am using nginx and attached the
In this config use your fluentd-address and give the tag name for kibana index pattern.
STEP 5:- Now Confirm Logs from Kibana Dashboard so go to
http://localhost:5601/ with your browser. Then, you need to set up the index name pattern for Kibana. Please specify
fluent* to Index name or pattern and press
Here you can see that your index pattern created and now you can see your application logs by going to
Reference links:- https://docs.fluentd.org/container-deployment/docker-compose
Originally published at http://blog.logicwind.com on February 8, 2020.