Elasticsearch + Fluentd + Kibana Setup (EFK) with Docker

In this article, we will see how to collect Docker logs to EFK (Elasticsearch + Fluentd + Kibana) stack. The example uses Docker Compose for setting up multiple containers.
But before that let us understand that what is Elasticsearch, Fluentd, and kibana.

1. Elasticsearch :- Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

2. Kibana:- Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data

3. Fluentd:- Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.

How to setup EFK stack Step by Step :-

STEP 1:- First of all create a docker-compose.yaml file for EFK stack. In this demo here we are using Opendistro docker images for security , but you can use official image.

STEP 2:- Then create a folder name called fluentd and in that folder create
Dockerfile . it looks like /fluend/Dockerfile

STEP 3:- After that create a folder conf also create a fluent.conf file inside the fluentd directory. it looks like /fluend/conf/fluent.conf

In this config you can remove user and password if you are not using opendistro images and change your hosts . Now run the docker compose file by this command.

STEP 4:- Finally EFK stack is ready now lauch your application and send the logs into Elasticsearch. Here i am using nginx and attached the logging tag

In this config use your fluentd-address and give the tag name for kibana index pattern.

STEP 5:- Now Confirm Logs from Kibana Dashboard so go to http://localhost:5601/ with your browser. Then, you need to set up the index name pattern for Kibana. Please specify fluent* to Index name or pattern and press Create button

Here you can see that your index pattern created and now you can see your application logs by going to discover section

Reference links:- https://docs.fluentd.org/container-deployment/docker-compose

Originally published at http://blog.logicwind.com on February 8, 2020.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store

We craft digital products for your business growth. Development | Marketing | Branding | Strategic Partnership