Kafka

The Kafka Integration offers real-time insights into your Kafka pipeline performance. Actively monitor your Kafka brokers, partitions, consumer groups, and JMX metrics in our default dashboard or with custom widgets.

For more information on the Kafka Integration, navigate to the Kafka Demo Project repository.

Prerequisites

Middleware Host Agent (MW Agent) v 1.5.0+ must be installed on your local machine. To install the MW Agent, see our Installation Guide.

Setup

Step 1: Download Jar File

Download the opentlemetry-jmx-metrics.jar file here.

Step 2: Access Integrations

Login to your Middleware account, navigate to the Kafka integration, and connect Kafka:

Step 3: Add Kafka Host

Add your JMX Server and Broker Host configuration to instrument the Kafka Host:

For more information on each input field, navigate to the Host Configuration Details section.

Visualize Your Data

Default Kafka Dashboard

Quickly access your Kafka data with Middleware’s default Kafka dashboard. Navigate to the Dashboard Builder and select the Kafka Dashboard:

Metrics Collected

Create your own widget from scratch. Navigate to the Dashboard Builder and select the dashboard you would like to create a widget in. Select Add New Widget and choose the Kafka data source.

MetricDescriptionReceiver
kafka.brokersNumber of brokers in a given clusterkafkametrics
kafka.consumer_group.lagCurrent approximate lag of consumer group at partition of topickafkametrics
kafka.consumer_group.lag_sumCurrent approximate sum of consumer group lag across all partitions of topickafkametrics
kafka.consumer_group.membersCount of members in the consumer groupkafkametrics
kafka.consumer_group.offsetCurrent offset of the consumer group at partition of topickafkametrics
kafka.consumer_group.offset_sumSum of consumer group offset across partitions of topickafkametrics
kafka.partition.current_offsetCurrent offset of partition of topickafkametrics
kafka.partition.oldest_offsetOldest offset of partition of topickafkametrics
kafka.partition.replicasNumber of replicas for partition of topickafkametrics
kafka.partition.replicas_in_syncNumber of synchronized replicas of partitionkafkametrics
kafka.topic.partitionsNumber of partitions in topic.kafkametrics
kafka.message.countThe number of messages received by the brokerjmx
kafka.request.countThe number of requests received by the brokerjmx
kafka.request.failedThe number of requests to the broker resulting in a failurejmx
kafka.request.time.totalThe total time the broker has taken to service requestsjmx
kafka.request.time.50pThe 50th percentile time the broker has taken to service requestsjmx
kafka.request.time.99pThe 99th percentile time the broker has taken to service requestsjmx
kafka.request.queueSize of the request queuejmx
kafka.network.ioThe bytes received or sent by the brokerjmx
kafka.purgatory.sizeThe number of requests waiting in purgatoryjmx
kafka.partition.countThe number of partitions on the brokerjmx
kafka.partition.offlineThe number of partitions offlinejmx
kafka.partition.under_replicatedThe number of under replicated partitionsjmx
kafka.isr.operation.countThe number of in-sync replica shrink and expand operationsjmx
kafka.max.lagThe max lag in messages between follower and leader replicasjmx
kafka.controller.active.countThe number of controllers active on the brokerjmx
kafka.leader.election.rateLeader election ratejmx
kafka.unclean.election.rateUnclean leader election ratejmx

Host Configuration Details

The following describes all input fields required to start streaming Kafka data to Middleware:

Input FieldDefinition
JMX Server HostHost name of JMX service URL
JMX PortPort where JMX service URL is built
Service NameName for differentiating between Kafka brokers
Jar PathPath where opentlemetry-jmx-metrics.jar is located on your local
Broker HostHost name of broker URL
Broker PortPort where broker URL is built
UsernameOptional unless using Kafka Broker with authentication type SASL plaintext
PasswordOptional unless using Kafka Broker with authentication type SASL plaintext

Next Steps

Need assistance or want to learn more about Middleware? Contact our support team in Slack.