{"id":1818,"date":"2017-02-01T09:48:59","date_gmt":"2017-02-01T09:48:59","guid":{"rendered":"http:\/\/cloudkul.com\/blog\/?p=1818"},"modified":"2017-06-14T11:33:26","modified_gmt":"2017-06-14T11:33:26","slug":"setting-elasticsearch-logstash-kibana-centralized-logging","status":"publish","type":"post","link":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/","title":{"rendered":"Setting up Elasticsearch, Logstash and Kibana for centralized logging"},"content":{"rendered":"

For proper analysis of a server or a system, logs<\/strong> play a very important role. By reading logs we can know about\u00a0 system incidents\u00a0 which happened in a particular time. Logs help us to detect the problems in the system and troubleshoot the problems. Logs<\/strong> are generally stored in files<\/strong>. For different services different log files are generated. If we have many systems and\u00a0 we need to analyze the logs,then it will be difficult to manage all the logs<\/strong>. So it is best practice to store the logs in a centralized<\/strong> sever.<\/p>\n

As logs are saved in files, sometimes it is hard to analyze the data in text format. If we can extract the data from logs and convert the data in table, graph, pie-chart<\/strong> form it will be easy for us to analyze the data. It is easier to analyze the data in detailed picture form rather than reading the text based log files.<\/p>\n

In case of apache\u00a0 log, contents of the log file increases every time some one visit the website. So for proper analysis we should able to read the logs in real time. So the centralized server should be able to fetch the data from the servers in real time.<\/p>\n

The above requirements are fulfilled by the ELK<\/strong> (Elasticsearch, Logstash and Kibana<\/strong>) stack<\/strong>. Elasticsearch<\/strong> is used as a centralized log server where as Logstash<\/strong> is used to send log data to Elasticsearch in real time. Kibana<\/strong> is used for view the complex log data in picture form.<\/p>\n

In this tutorial I will show you how to setup\u00a0 ELK stack. I am using ubuntu 14.04<\/strong> in all the servers.<\/p>\n

Setting up ELK stack<\/strong><\/p>\n

Suppose I have 3 different servers running apache service and I need to get the log files from each server. So, I need to setup Logstash in each of the server. Logstash<\/strong> will read the apache log<\/strong> files and send it to the Elasticsearch<\/strong> server. Elasticsearch will save the logs in unique index<\/strong> defined by the Logstash. I need another server which will use Kibana<\/strong> to fetch the data from the Elasticsearch<\/strong> and visualize the data.<\/p>\n

\"\"<\/a><\/p>\n

Install Java
\n<\/strong><\/p>\n

The ELK<\/strong> stack needs java to run the applications. So you need to install java in all the servers. Before installing any of the application make sure Java<\/strong> is installed.<\/p>\n

sudo apt-add-repository ppa:webupd8team\/java\r\nsudo apt-get update\r\nsudo apt-get install oracle-java8-installer\r\n#verfify your java version\r\njava -version\r\n\r\n<\/pre>\n
Install Kibana<\/strong><\/p>\n
Login to you server where you want to setup kibana<\/p>\n
# Download the deb package for ubuntu\r\nsudo wget https:\/\/artifacts.elastic.co\/downloads\/kibana\/kibana-5.1.2-amd64.deb\r\nsudo dpkg -i kibana-5.1.2-amd64.deb\r\n\r\n\r\n<\/pre>\n
Open \/etc\/kibana\/kibana.yml <\/strong>and change the following.<\/p>\n
server.host: \"192.168.1.80\"\r\n# IP of the Elasticsearch server\r\nelasticsearch.url: \"http:\/\/192.168.1.56:9200\"<\/pre>\n
Now, restart the kibana service.<\/p>\n
sudo service kibana restart<\/pre>\n
Install Elasticsearch<\/strong><\/p>\n
Login to your server where you want to setup Elasticsearch<\/p>\n
# Download the deb package for ubuntu\r\nsudo wget https:\/\/artifacts.elastic.co\/downloads\/elasticsearch\/elasticsearch-5.1.2.deb\r\nsudo dpkg -i elasticsearch-5.1.2.deb\r\n<\/pre>\n
Open \/etc\/elasticsearch\/elasticsearch.yml<\/strong>\u00a0 and change the following.<\/p>\n
network.host: 192.168.1.56\r\n<\/pre>\n
Now,restart the Elasticsearch service.<\/p>\n
sudo service elasticsearch restart<\/pre>\n
Install Logstash<\/strong><\/p>\n
You need to install Logstash for each\u00a0 server. Installation and configuration is same for all the server, you just need to define different index name for Elasticsearch<\/strong>.<\/p>\n
# Download deb package for ubuntu\r\nsudo wget https:\/\/artifacts.elastic.co\/downloads\/logstash\/logstash-5.1.2.deb\r\nsudo dpkg -i logstash-5.1.2.deb<\/pre>\n
Create a file\u00a0 \/etc\/logstash\/conf.d\/apache.conf<\/strong> and copy the following content.<\/p>\n
input {\r\n    file {\r\n        # Give path to your log file\r\n        path => '\/var\/log\/apache2\/mysite_access.log'\r\n    }\r\n}\r\n\r\nfilter {\r\n    grok {\r\n        #match => { \"message\" => \"%{COMBINEDAPACHELOG}\" }\r\n        match => { \"message\" => \"%{NOTSPACE:clientip} \\- \\- \\[%{NOTSPACE:date} \\+%{INT}\\] \\\"%{WORD:method} \/%{NOTSPACE:request} %{NOTSPACE:httpversion}\\\" %{INT:response} %{INT} \\\"%{NOTSPACE:fullurl}\\\" \\\"%{GREEDYDATA:misc}\\\"\" }\r\n    }\r\n}\r\n\r\n#output {\r\n#    stdout { codec => rubydebug }\r\n#}\r\noutput {\r\n    elasticsearch {\r\n         # Elastic server IP and port\r\n         hosts => [\"192.168.1.56:9200\"]\r\n         # choose an index name\r\n         index => \"client1-apache\"\r\n    }\r\n}\r\n<\/pre>\n
For different server change the index<\/strong>. Make sure the path to the log file is correct in the input<\/strong> section. In filter section you can define your own pattern and get the logs in key value format.<\/p>\n
Now, restart the Logstash<\/strong> service and use the configuration<\/strong> file we wrote for sending logs to Elasticsearch.<\/p>\n
sudo service logstash restart\r\nsudo \/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/conf.d\/apache.conf<\/pre>\n
If there is no error in configuration, you will see output like this<\/p>\n
\"\"<\/a><\/p>\n
Now, In the browser open Kibana<\/strong> server, in my case it is http:\/\/192.168.1.80:5601<\/strong>.<\/p>\n
Go to Management -> Index Patterns -> Add New<\/strong> and add your index<\/strong> name., the index<\/strong> name you added in the logstash<\/strong> configuration file.<\/p>\n
\"\"<\/a><\/p>\n
After you create your index<\/strong> you can able to visualize your logs in Kibana<\/strong> Dashboard. Do this for other servers also and you will be able to view logs by index<\/strong> name. If you have any query regarding\u00a0 ELK<\/strong> stack<\/strong> setup<\/strong>, you can ask me in the comment.<\/p>\n
 <\/p>\n
 <\/p>\n
 IN CASE OF ANY QUERY,CONTACT US<\/a><\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
For proper analysis of a server or a system, logs play a very important role. […]<\/a><\/p>\n","protected":false},"author":8,"featured_media":1835,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[],"yoast_head":"\nSetting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Setting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul\" \/>\n<meta property=\"og:description\" content=\"For proper analysis of a server or a system, logs play a very important role. [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/\" \/>\n<meta property=\"og:site_name\" content=\"Cloudkul\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-01T09:48:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-06-14T11:33:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cloudkul.com\/blog\/wp-content\/uploads\/2017\/02\/6i2gm2uH.png\" \/>\n\t<meta property=\"og:image:width\" content=\"848\" \/>\n\t<meta property=\"og:image:height\" content=\"422\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"suranjan horrow\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/\",\"url\":\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/\",\"name\":\"Setting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul\",\"isPartOf\":{\"@id\":\"https:\/\/cloudkul.com\/blog\/#website\"},\"datePublished\":\"2017-02-01T09:48:59+00:00\",\"dateModified\":\"2017-06-14T11:33:26+00:00\",\"author\":{\"@id\":\"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/a653191c790a89b07d4b4aaefc3e2809\"},\"breadcrumb\":{\"@id\":\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/cloudkul.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Setting up Elasticsearch, Logstash and Kibana for centralized logging\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/cloudkul.com\/blog\/#website\",\"url\":\"https:\/\/cloudkul.com\/blog\/\",\"name\":\"Cloudkul\",\"description\":\"Host your eCommerce Store on AWS with Optimized Performance\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/cloudkul.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/a653191c790a89b07d4b4aaefc3e2809\",\"name\":\"suranjan horrow\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/dfd7f87bc7d62c1426e1f4c07653ff00?s=96&d=https%3A%2F%2Fs.gravatar.com%2Favatar%2F6148c37469011bc2f8e491ca8f5de495%3Fs%3D80&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/dfd7f87bc7d62c1426e1f4c07653ff00?s=96&d=https%3A%2F%2Fs.gravatar.com%2Favatar%2F6148c37469011bc2f8e491ca8f5de495%3Fs%3D80&r=g\",\"caption\":\"suranjan horrow\"},\"sameAs\":[\"http:\/\/webkul.com\"],\"url\":\"https:\/\/cloudkul.com\/blog\/author\/suranjan-horrow869\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Setting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/","og_locale":"en_US","og_type":"article","og_title":"Setting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul","og_description":"For proper analysis of a server or a system, logs play a very important role. [...]","og_url":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/","og_site_name":"Cloudkul","article_published_time":"2017-02-01T09:48:59+00:00","article_modified_time":"2017-06-14T11:33:26+00:00","og_image":[{"width":848,"height":422,"url":"https:\/\/cloudkul.com\/blog\/wp-content\/uploads\/2017\/02\/6i2gm2uH.png","type":"image\/png"}],"author":"suranjan horrow","twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/","url":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/","name":"Setting up Elasticsearch, Logstash and Kibana for centralized logging - Cloudkul","isPartOf":{"@id":"https:\/\/cloudkul.com\/blog\/#website"},"datePublished":"2017-02-01T09:48:59+00:00","dateModified":"2017-06-14T11:33:26+00:00","author":{"@id":"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/a653191c790a89b07d4b4aaefc3e2809"},"breadcrumb":{"@id":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/cloudkul.com\/blog\/setting-elasticsearch-logstash-kibana-centralized-logging\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/cloudkul.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Setting up Elasticsearch, Logstash and Kibana for centralized logging"}]},{"@type":"WebSite","@id":"https:\/\/cloudkul.com\/blog\/#website","url":"https:\/\/cloudkul.com\/blog\/","name":"Cloudkul","description":"Host your eCommerce Store on AWS with Optimized Performance","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/cloudkul.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/a653191c790a89b07d4b4aaefc3e2809","name":"suranjan horrow","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/cloudkul.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/dfd7f87bc7d62c1426e1f4c07653ff00?s=96&d=https%3A%2F%2Fs.gravatar.com%2Favatar%2F6148c37469011bc2f8e491ca8f5de495%3Fs%3D80&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dfd7f87bc7d62c1426e1f4c07653ff00?s=96&d=https%3A%2F%2Fs.gravatar.com%2Favatar%2F6148c37469011bc2f8e491ca8f5de495%3Fs%3D80&r=g","caption":"suranjan horrow"},"sameAs":["http:\/\/webkul.com"],"url":"https:\/\/cloudkul.com\/blog\/author\/suranjan-horrow869\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/posts\/1818"}],"collection":[{"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/comments?post=1818"}],"version-history":[{"count":14,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/posts\/1818\/revisions"}],"predecessor-version":[{"id":2823,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/posts\/1818\/revisions\/2823"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/media\/1835"}],"wp:attachment":[{"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/media?parent=1818"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/categories?post=1818"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudkul.com\/blog\/wp-json\/wp\/v2\/tags?post=1818"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}