Install Elastic Stack 8 (ELK) on Amazon Linux 2

5 min readOct 23, 2022

This guide will discuss how to deploy ELK stack on Amazon Linux 2. ELK stack is a combination of three projects, Elasticsearch, Logstash and Kibana.

Elasticsearch is a log analytics engine while Logstash is a logs processing pipeline that helps transport and aligns the logs from different sources to Elasticsearch. Kibana, on the other hand, is the web front end that is used for the visualization of the logs after the analysis is done by Elasticsearch.

Pre-requisites

We will need the following in order to successfully archive our objectives:

Amazon Linux 2
OpenJDK/Oracle Java
2 CPU, 4GB RAM
Ports 9200, 5601, 5044

Install Elasticsearch on Amazon Linux 2

Before we can install Elasticsearch on Amazon Linux 2, we need to have Java installed on our system. Install OpenJDK as shown below:

sudo yum -y install java-openjdk-devel java-openjdk

Add the ELK repository on Amazon Linux 2.

cat <[elasticsearch-8.x]

name=Elasticsearch repository for 8.x packages

baseurl=https://artifacts.elastic.co/packages/8.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

Import the GPG key.

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Update the system cache.

sudo yum clean all

sudo yum makecache

Install Elasticsearch on Amazon Linux 2.

sudo yum -y install elasticsearch

Verify that Elasticearch has been installed successfully:

[root@amazon-linux ~]# rpm -qi elasticsearch

Name        : elasticsearch

Epoch       : 0

Version     : 8.0.1

Release     : 1

Architecture: x86_64

Install Date: Thu Mar  3 22:12:37 2022

Group       : Application/Internet

Size        : 1100107680

License     : Elastic License

Signature   : RSA/SHA512, Thu Feb 24 18:00:40 2022, Key ID d27d666cd88e42b4

Source RPM  : elasticsearch-8.0.1-1-src.rpm

Build Date  : Thu Feb 24 14:03:12 2022

Build Host  : packer-virtualbox-iso-1636998457

Relocations : /usr

Packager    : Elasticsearch

Vendor      : Elasticsearch

URL         : https://www.elastic.co/

Configure Elasticsearch on Amazon Linux 2

After the installation, you may need to configure Elasticsearch to add your cluster name and the bind-address. Edit the file /etc/elasticsearch/elasticsearch.yml

$ sudo vi /etc/elasticsearch/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

#

# Use a descriptive name for your cluster:

#

cluster.name: elk-cluster

#

# ---------------------------------- Network -----------------------------------

#

# By default Elasticsearch is only accessible on localhost. Set a different

# address here to expose this node on the network:

#

network.host: 0.0.0.0

#

# By default Elasticsearch listens for HTTP traffic on the first free port it

# finds starting at 9200. Set a specific HTTP port here:

#

http.port: 9200

#

# For more information, consult the network module documentation.

You can also modify Java memory options for Elasticsearch by editing the file /etc/elasticsearch/jvm.options

$ sudo vi /etc/elasticsearch/jvm.options

-Xms1g

-Xmx1g

The lines above set the maximum memory size to 1GB.

Start and enable Elasticsearch service.

sudo systemctl enable --now elasticsearch.service

Verify service status

[root@amazon-linux ~]# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

   Active: active (running) since Sun 2021-08-15 13:05:05 UTC; 16s ago

     Docs: https://www.elastic.co

 Main PID: 2637 (java)

   CGroup: /system.slice/elasticsearch.service

           ├─2637 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encod...

           └─2818 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller



Aug 15 13:04:39 amazon-linux systemd[1]: Starting Elasticsearch...

Aug 15 13:05:05 amazon-linux systemd[1]: Started Elasticsearch.

[root@amazon-linux ~]#

Verify that Elasticsearch is working:

$ curl http://127.0.0.1:9200 



  "name" : "amazon-linux",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "DzAF-DclTYqC9zG9S5uWow",

  "version" : 

    "number" : "7.14.0",

    "build_flavor" : "default",

    "build_type" : "rpm",

    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",

    "build_date" : "2021-07-29T20:49:32.864135063Z",

    "build_snapshot" : false,

    "lucene_version" : "8.9.0",

    "minimum_wire_compatibility_version" : "6.8.0",

    "minimum_index_compatibility_version" : "6.0.0-beta1"

  ,

  "tagline" : "You Know, for Search"

Test Elasticsearch indexing:

$ curl -X GET "localhost:9200"



  "name" : "amazon-linux",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "DzAF-DclTYqC9zG9S5uWow",

  "version" : 

    "number" : "7.14.0",

    "build_flavor" : "default",

    "build_type" : "rpm",

    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",

    "build_date" : "2021-07-29T20:49:32.864135063Z",

    "build_snapshot" : false,

    "lucene_version" : "8.9.0",

    "minimum_wire_compatibility_version" : "6.8.0",

    "minimum_index_compatibility_version" : "6.0.0-beta1"

  ,

  "tagline" : "You Know, for Search"

Install Logstash on Amazon Linux 2

After a successful installation and configuration of Elasticsearch on Amazon Linux 2, we now proceed to the next element, which is Logstash.

Install Logstash with the command below:

sudo yum install logstash -y

Edit the Logstash configuration file to add the input and output parameters.

$ sudo vi /etc/logstash/conf.d/logstash.conf



input 



  beats 



    port => 5044



  





output 



  elasticsearch 



    hosts => ["localhost:9200"]



    manage_template => false



    index => "%[@metadata][beat]-%[@metadata][version]-%+YYYY.MM.dd"

Start and enable Logstash

systemctl enable --now logstash

Install and Configure Kibana on Amazon Linux 2

The next item to be installed will be Kibana.

Kibana exists in the ELK repo that we had configured earlier, we shall therefore proceed to install the package directly.

sudo yum -y install kibana

Configure Kibana by adding the port binding parameters:

$ sudo vim /etc/kibana/kibana.yml

server.host: "0.0.0.0"

elasticsearch.url: "http://localhost:9200"

Start and enable Kibana service:

systemctl enable --now kibana

Verify that the service is up.

$ systemctl status kibana

● kibana.service - Kibana

   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)

   Active: active (running) since Sun 2021-08-15 13:15:04 UTC; 7s ago

     Docs: https://www.elastic.co

 Main PID: 2941 (node)

   CGroup: /system.slice/kibana.service

           ├─2941 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"

           └─2955 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"



Aug 15 13:15:04 amazon-linux systemd[1]: Started Kibana.

Allow Kibana through the firewall:

sudo firewall-cmd --permanent --add-port=5601/tcp

sudo firewall-cmd --reload

Access the Kibana dashboard by http://server-ip:5601.

You can now start adding your data and shipping logs using beats such as Filebeat, Metricbeat etc.

Install Filebeat on Amazon Linux 2

Filebeat is a beat that is used to send log files to the ELK cluster for different applications and services.

Run the command below to install Filebeat on Amazon Linux 2.

sudo yum install filebeat

Enable modules for Filebeat. This enables the applications that will ship their logs to Elasticsearch. To check the available modules, run the command below:

sudo filebeat modules list

Enable a module, such as the Nginx module:

sudo filebeat modules enable system

Run the Filebeat setup to initialize the Filebeat process

$ sudo filebeat setup

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.



Index setup finished.

Loading dashboards (Kibana must be running and reachable)

Loaded dashboards

Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.

See more: https://www.elastic.co/guide/en/machine-learning/current/index.html

Loaded machine learning job configurations

Loaded Ingest pipelines

Start Filebeat service

sudo service filebeat start

Go to the Kibana dashboard and verify that you can see the metrics for the enabled module.

That’s all for this setup.

Wrap Up

We have successfully set up ELK stack on Amazon Linux 2. You can use this stack to monitor logs for your distributed systems. This helps in improving business productivity by allowing system admins to plan for their systems in terms of resources, security, etc.

https://www.computingpost.com/install-elastic-stack-8-elk-on-amazon-linux-2/?feed_id=16138&_unique_id=6354e1b80aa16