Elasticsearch Monitoring
Elasticsearch offers the monitoring feature to monitor the health of the cluster. It is responsible for collecting the metrics from each and every node and storing them in Elasticsearch indices. Elasticsearch contains elasticsearch.yml file inside the config folder, where we can set all settings associated with monitoring for each node. In addition, these settings can also be set in dynamic cluster settings wherever possible.
Note that the monitoring metrics stores in local indices as default
Whenever we monitor a cluster, we need to collect the data from elasticsearch nodes, Kibana instances, Logstash nodes in a cluster and store in elasticsearch indices. This monitoring feature helps to ensure the health and performance of your elasticsearch cluster.
We need to check the cluster settings to start monitoring of Elasticsearch. Look at the following example how could it be done –
In the stack, each component monitors itself. After that it forwards those documents to the elasticsearch production cluster for indexing (storage) and routing. The process of indexing and routing is handled by exporters and collectors. So, first of all, we need to know about the collector and exporter. Let’s understand what the collectors and exporters are?
Collectors
In elasticsearch, collectors are used to obtaining the data from the public API, which it chooses to monitor. The collectors run once per each collection interval. Once the data is collected, it is delivered in bulk to the exporters to be sent to the monitoring cluster.
Remember one important thing that – only one collector per data type is gathered. However, each collector can create zero or multiple monitoring documents.
Exporters
Exporters are used to take data and route it to the monitoring cluster. The data taken by the exporter is collected from any Elastic Stack source. Elasticsearch allows us to configure more than one exporter, which can be configured at both node and cluster level. Although the default and general setup are to use a single exporter. In elasticsearch, there are two types of exporters, which are local and http.
1. local –
The local exporter is responsible for routing the data back into the same cluster.
2. http –
On the other hand, http exporter is the most preferred exporter. This allows us to route the data into any supported elasticsearch cluster accessible through the HTTP protocol.
Remember that the exporters have to set up certain elasticsearch resources (like templates and ingest pipeline) before route the monitoring data. Because only then, they can route monitoring data.
As we already told, the monitoring data is stored by elasticsearch indices collected from a cluster. These indices use the index templates to be configured that can be retrieved using _template API.