Introduction

Apache Kafka's distributed architecture makes it powerful but complex to manage. The right tooling is essential for monitoring consumer lag, inspecting messages, managing topics, and debugging production issues. This article covers four Kafka management tools: AKHQ, Kafka UI, Kowl, and Offset Explorer.

Kafka Tools: AKHQ, Kafka UI, Kowl, Offset Explorer

AKHQ (formerly KafkaHQ)

A comprehensive Kafka web UI with topic management, consumer group monitoring, and schema registry support:

docker-compose.yml

version: "3.8"

services:

akhq:

image: tchiotludo/akhq:latest

environment:

AKHQ_CONFIGURATION: |

akhq:

connections:

local:

properties:

bootstrap.servers: "kafka:9092"

schema-registry:

url: "http://schema-registry:8081"

connect:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- name: connect

url: "http://kafka-connect:8083"

ports:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- "8080:8080"

depends_on:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- kafka

Key features :

  • Topic browser with search and filter

  • Consumer group monitoring with lag visualization

  • Schema registry viewer (Avro, Protobuf, JSON Schema)

  • Kafka Connect management

  • Message producer (publish test messages)

  • Partition reassignment viewer

  • Node and broker monitoring

AKHQ advanced configuration

akhq:

connections:

production:

properties:

bootstrap.servers: "broker1:9092,broker2:9092,broker3:9092"

security.protocol: SASL_SSL

sasl.mechanism: SCRAM-SHA-512

sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username='admin' password='${SECRET_AKHQ_PASSWORD}';"

schema-registry:

url: "https://schema-registry:8081"

basic-auth-username: admin

basic-auth-password: "${SECRET_SCHEMA_REGISTRY_PASSWORD}"

Kafka UI (by Provectus)

A lightweight, open-source Kafka web UI:

docker-compose.yml

services:

kafka-ui:

image: provectuslabs/kafka-ui:latest

ports:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- "8081:8080"

environment:

KAFKA_CLUSTERS_0_NAME: local

KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081

KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: connect

KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://kafka-connect:8083

SERVER_PORT: 8080

Key features :

  • Dashboard with cluster overview and broker health

  • Topic browser with partition visualization

  • Consumer groups with lag chart

  • Schema registry viewer

  • Kafka Connect management

  • Message filtering with custom predicates

  • Dark and light themes

Multi-cluster configuration

KAFKA_CLUSTERS_0_NAME: staging

KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: staging-kafka:9092

KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL

KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: SCRAM-SHA-512

KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password='pass';"

KAFKA_CLUSTERS_1_NAME: production

KAFKA_CLUSTERS_1_BOOTSTRAPSERVERS: prod-kafka:9092

Kowl (now Redpanda Console)

Developer-friendly Kafka browser with a focus on data exploration:

docker-compose.yml

services:

redpanda-console:

image: docker.redpanda.com/redpandadata/console:latest

ports:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- "8082:8080"

environment:

KAFKA_BROKERS: kafka:9092

SCHEMA_REGISTRY_ENABLED: "true"

SCHEMA_REGISTRY_URLS: "http://schema-registry:8081"

Key features :

  • Fast message search and filtering with custom predicates

  • Consumer group monitoring with lag alerts

  • Schema registry with compatibility checking

  • Message publish with schema validation

  • Kafka Connect management

  • Data masking for sensitive fields

  • Role-based access control

Data masking configuration

server:

maskedFields:

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- fieldName: "password"

maskingStrategy: "redact"

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- fieldName: "email"

maskingStrategy: "hash"

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- fieldName: "ssn"

maskingStrategy: "hash_last_four"

Message search predicate examples

Find messages with specific JSON field:

$.user.email == "test@example.com"

$.order.total > 100.0

$.status in ["pending", "processing"]

Offset Explorer (formerly Kafka Tool)

A desktop GUI application for Kafka (Windows/macOS/Linux):

Key features :

  • Tree-based cluster browser

  • Topic partition viewer with message content

  • Consumer group offset management

  • Message producer

  • Import/export topics

  • SSL/SASL authentication support

  • Time-based message search

CLI Tools

The Kafka distribution includes essential CLI tools:

Topic management

kafka-topics.sh --bootstrap-server localhost:9092 --list

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic events --partitions 6 --replication-factor 3 --config retention.ms=604800000

Consumer group monitoring

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-processor

Message inspection

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning --max-messages 10

Performance testing

kafka-producer-perf-test.sh --topic test --num-records 100000 --record-size 1024 --throughput 10000 --producer-props bootstrap.servers=localhost:9092

kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic test --messages 50000

Comparison

| Feature | AKHQ | Kafka UI | Kowl (Console) | Offset Explorer |

|---------|------|----------|----------------|-----------------|

| Type | Web UI | Web UI | Web UI | Desktop |

| Message search | Basic | Basic | Advanced | Basic |

| Schema registry | Yes | Yes | Yes | No |

| Kafka Connect | Yes | Yes | Yes | No |

| Data masking | No | No | Yes | No |

| RBAC | No | No | Yes | No |

| Multi-cluster | Yes | Yes | Yes | Yes |

Recommendations

  • Full-featured management : AKHQ for comprehensive cluster administration.

  • Quick monitoring : Kafka UI for lightweight cluster health checks.

  • Message debugging : Kowl (Redpanda Console) for powerful message search and filtering.

  • Desktop preference : Offset Explorer for a native desktop experience.

  • Scripting : Kafka CLI tools for automation and CI/CD tasks.

For development and staging environments, Kafka UI or Kowl provide excellent web-based management with minimal setup. For production, AKHQ offers the most comprehensive feature set, while Redpanda Console provides the best message debugging experience.