Elasticsearch Integration

Prerequisites

• Middleware Host Agent installed on the machine that will collect metrics. See the Installation Guide.
• Elastic superuser privileges to allow stats access. Elastic provides a built-in superuser role and an elastic superuser account; follow Elastic guidance to create users/roles with the minimum necessary privileges.

Setup

1 Create database credentials

Create a YAML file with your Elasticsearch endpoint and (if enabled) credentials. If your cluster is not password-protected, omit username and password. Example: /home/ubuntu/elasticsearch-creds.yaml

1elasticsearch:
2  endpoint: http://localhost:9200
3  username: YOUR_USERNAME
4  password: YOUR_PASSWORD

Docker Agent: place the YAML inside /var/log on the Agent host. Docker Agents might not support dynamic volume binding. Quick verify (from the Agent host):

1# If auth is enabled, add: -u USER:PASS
2curl -s http://<es-host>:9200/_cluster/health?pretty
3curl -s http://<es-host>:9200/_nodes/stats?filter_path=nodes.*.jvm,cluster_name | head

These endpoints confirm reachability/permissions and return cluster health and node statistics.

2 Access Integrations

In Middleware, go to Installations → All Integrations → Elasticsearch (the original page mistakenly references MySQL—use Elasticsearch here).

3 Enable Integration

Select the host (where the Host Agent is running), provide the credential file path from Step 1, and Save.

Visualize Your Data

• Default Elasticsearch dashboard: Open Dashboard Builder → Elasticsearch – Metrics (Runtime) for ready-to-use graphs.
• Create widgets: In a dashboard, choose Add New Widget → elasticsearch to build custom charts (e.g., disk I/O per node, CPU, JVM heap, shard sizes, indexing pressure).

Alerts

Create alerts on any Elasticsearch metric. Choose Database as the detection method and Elasticsearch as the database type; pick a metric and set conditions (e.g., high indexing_pressure, low disk available, rising thread_pool.tasks.queued).

Metrics Collected

Troubleshooting

Integrations menu is missing

• Your user likely lacks Installation permissions. Ask an admin to grant the Installation permission in Settings.

Endpoint unreachable from the Host Agent From the Agent host, run:

1curl -s http://<es-host>:9200/_cluster/health?pretty

• If this fails, fix DNS/firewall/routing or use the correct scheme/port (e.g., https:// behind proxies or Elastic Cloud). The health API should return a JSON status (green/yellow/red).

Authentication/authorization errors (401/403)

• Confirm the YAML creds are correct and have sufficient privileges. Elastic’s superuser role has full access, but Elastic recommends creating narrowly-scoped roles where possible.

To verify what your user can do, call:

1curl -s -u USER:PASS http://<es-host>:9200/_security/user/_privileges

• (Returns the logged-in user’s privileges.)

Docker Agent can’t read the credentials file

• Ensure the file is inside /var/log on the Agent host when using the Middleware Docker Agent (dynamic volume binding may not be supported). Update the integration to point to that path.

Metrics look partial or stale Check the _nodes/stats endpoint directly to ensure the cluster is returning JVM/OS/thread pool stats:

1curl -s http://<es-host>:9200/_nodes/stats?filter_path=nodes.*.os,nodes.*.jvm,nodes.*.thread_pool | head

• If output is limited, review your user’s privileges or cluster settings.