Learning Kafka and Connection Kafka To Golang Application

Contents

• Learning Kafka and Connection Kafka To Golang Application
  • Contents
  • Installation
  • Quick Start
  • Kafka CLI

Installation

To install Kafka package, you need to install Go and set your Go workspace first.

1. You first need Go installed (version 1.18+ is required), then you can use the below Go command to install Kafka.

$ go get -u github.com/confluentinc/confluent-kafka-go/kafka

2. Import it in your code:

import "github.com/confluentinc/confluent-kafka-go/kafka"

3. You first need Makefile

# install windows
C:\> choco install make

# install linux
$ sudo apt-get install make

# install MacOS with Homebrew
$ brew install make

Verify Version Makefile

# verify that Makefile is installed correctly by checking the version
$ make --version

4. You first need Docker Desktop installed.

5. You first need Docker Compose.

Windows and macOS

• Docker Compose is included in Docker Desktop for Windows and macOS.

Linux

• For Ubuntu and Debian, run:

$ sudo apt-get update
$ sudo apt-get install docker-compose-plugin

• For RPM-based distros, run:

$ sudo yum update
$ sudo yum install docker-compose-plugin

Verify Version Docker Compose

# verify that Docker Compose is installed correctly by checking the version
$ docker compose version

6. You first need Kafka CLI or Web UI

# install kafka cli, make sure all container running
$ make compose

# install kafka web ui, make sure all container running, wait 1 minutes, then open to browser http://localhost:9021
$ make compose-ui

Quick Start

• Producer

# application producer, run:
$ make producer

• Consumer

# application consumer1, run:
$ make consumer1

# application consumer2, run:
$ make consumer2

Kafka CLI

Note that you need to set the corresponding binding tag on Operating System. For example: winpty, sudo or etc

# entry into docker container kafka, run:
$ make exec

Create Topic

$ kafka-topics --bootstrap-server localhost:9092 --create --topic YOURTOPIC

Create Topic with specific configuration

$ kafka-topics --bootstrap-server localhost:9092 --create --topic YOURTOPIC --replication-factor 1 --partitions 3 --config "cleanup.policy=compact" --config "delete.retention.ms=100"  --config "segment.bytes=204800" --config "min.cleanable.dirty.ratio=0.01"

List Topic

$ kafka-topics --bootstrap-server localhost:9092 --list

Delete Topic

$ kafka-topics --bootstrap-server localhost:9092 --delete --topic YOURTOPIC

Alter Topic

$ kafka-topics --bootstrap-server localhost:9092 --alter --topic YOURTOPIC --partitions 3

List Detail Topic

$ kafka-topics --bootstrap-server localhost:9092 --describe --topic YOURTOPIC

Send Message Producer

$ kafka-console-producer --bootstrap-server localhost:9092 --topic YOURTOPIC

Send Message Producer with Key

# example: >1:test
$ kafka-console-producer --bootstrap-server localhost:9092 --topic YOURTOPIC --property "parse.key=true" --property "key.separator=:"

Received Message Consumer

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --from-beginning

Received Message Consumer Group

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --group YOURGROUP --from-beginning

Received Message Consumer Group with Key

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --group YOURGROUP --from-beginning --property "print.key=true"

List Consumer Group Offset

$ kafka-consumer-groups --bootstrap-server localhost:9092 --all-groups --all-topics --describe

List Consumer Group Offset 1

$ kafka-consumer-groups --bootstrap-server localhost:9092 --group YOURGROUP --describe