How to setup kibana,elasticsearch,logstash server,redis

A. Elasticsearch


sudo wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.deb
sudo dpkg -i elasticsearch-1.5.2.deb

sudo update-rc.d elasticsearch defaults 95 10

sudo /etc/init.d/elasticsearch restart

Elasticsearch is now installed. Let's edit the configuration:

sudo vi /etc/elasticsearch/elasticsearch.yml

network.host: localhost

sudo service elasticsearch restart

sudo update-rc.d elasticsearch defaults 95 10



B. Kibana

wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz

tar xvf kibana-*.tar.gz

vi ~/kibana-4*/config/kibana.yml

host: "localhost"

sudo mkdir -p /opt/kibana

Now copy the Kibana files into your newly-created directory:

sudo cp -R ~/kibana-4/ /opt/kibana/

Kibana can be started by running /opt/kibana/bin/kibana, but we want it to run 
as a service. Download a Kibana init script with this command:

cd /etc/init.d && sudo wget https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/bce61d85643c2dcdfbc2728c55a41dab444dca20/kibana4

Now enable the Kibana service, and start it:

sudo chmod +x /etc/init.d/kibana4

sudo update-rc.d kibana4 defaults 96 9

sudo service kibana4 start



C. Setup Logstash

 sudo wget http://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.0-1_all.deb

sudo dpkg -i logstash_1.5.0-1_all.deb

sudo update-rc.d logstash defaults 95 10

sudo /etc/init.d/logstash restart


D. Create ssl key for clients


sudo mkdir -p /etc/pki/tls/certs sudo mkdir /etc/pki/tls/private

If you don't have a DNS setup—that would allow your servers, that you will gather logs from, to resolve the IP address of your Logstash Server—you will have to add your Logstash Server's private IP address to the subjectAltName (SAN) field of the SSL certificate that we are about to generate. To do so, open the OpenSSL configuration file:

sudo vi /etc/ssl/openssl.cnf

subjectAltName = IP: logstash_server_private_ip

cd /etc/pki/tls;

sudo openssl req -x509 -batch -nodes -days 365 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

E. Create logstash config files for getting data from client machine

1. LogStash for LumberJack:

vi /etc/logstash/conf.d/01-lumberjack-input.conf

input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } }

1. grok syslogs into the correct format:

vi /etc/logstash/conf.d/10-syslog.conf

filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } }

1. configure LogStash to store it’s logs in ElasticSearch

vi /etc/logstash/conf.d/30-lumberjack-output.conf

output { elasticsearch { host => localhost } stdout { codec => rubydebug } }

E. install redis server

sudo apt-get install redis-server

Search for the following line in /etc/redis/reis.conf and replace

bind 127.0.0.0

with

bind 0.0.0.0