Kafka configuration

Apache Kafka is a distributed streaming platform. It is useful for building real-time streaming data pipelines to get data between the systems or applications. Another useful feature is real-time streaming applications that can transform streams of data or react on a stream of data.

On GNU/Linux

Arch Linux

$ yay -S kafka
$ sudo pip install kafka-python

Ubuntu

A very useful guide (and recommended) can be readed in tecadmin.net, just follow it until step 3.

TL;DR

$ sudo apt update
$ sudo apt install default-jdk
$ sudo pip install kafka-python

$ wget http://www-us.apache.org/dist/kafka/2.4.0/kafka_2.13-2.4.0.tgz

$ tar xzf kafka_2.13-2.4.0.tgz
$ mv kafka_2.13-2.4.0 /usr/local/kafka

$ echo """[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target""" >> /etc/systemd/system/zookeeper.service

$ echo """[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target
""" >> /etc/systemd/system/kafka.service

$ systemctl daemon-reload

Daemon

Start daemon, on Ubuntu must be zookeeper.service instead of zookeeper@kafka.service.

systemctl enable kafka.service zookeeper@kafka.service
systemctl start kafka.service zookeeper@kafka.service

Create topic, on Ubuntu the complete path moust be specified: /usr/local/kafka/bin/kafka-topics.sh

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic binary
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic binary0
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic binary1
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic binary2
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic binary3
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic eeg
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic aux
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic eeg0
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic eeg1
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic eeg2
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic eeg3
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic marker
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic annotation
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic command
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic feedback

Change retention

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name binary --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name binary0 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name binary1 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name binary2 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name binary3 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name aux --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name eeg --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name eeg0 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name eeg1 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name eeg2 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name eeg3 --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name marker --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name annotation --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name command --alter --add-config retention.ms=1000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name feedback --alter --add-config retention.ms=1000

Windows

Who cares?