Study04 README Contents

Research the Kafka Broker and Apache Kafka Streams

Kafka is an event streaming platform.

Kafka Storage & Processing:

• Storage Layer
  • Concepts: topic, partition, broker
  • Kafka Broker
• Processing Layer
  • Concepts: stream, table, global table
  • Kafka Streams

The Docker container 'ptc-kafka' runs a Kafka Server with a single Kafka Broker.

The sections of this project:

1. Docker Build
2. Producer, Transformer, Consumers
3. Comparer and Counter
4. Kafka Scripts

---

Java source code. Packages:

application sources :  kp
test sources :  kp.processors

Java API Documentation ●  Java Test API Documentation

---

❶ Docker Build

Action:

1. With batch file "01 Docker build and run.bat" build the image and
start the container with the application.

1.1. Docker image is built using these files: Dockerfile and compose.yaml.

A screenshot of the created Docker containers.

Back to the top of the page

---

❷ Producer, Transformer, Consumers

Docker containers for stream processing applications: Producer, Transformer, and two Consumers

2.1. Producer in the container ptc-producer produces to the topics 'prod-1' and 'prod-2'.
It sends 'ProducerRecord' objects using the Kafka client 'KafkaProducer'.
The producer method: kp.producers.KpProducer::produceRecordsSet.

Log from the container ptc-producer.

2.2. Transformer in the container ptc-transformer directs records from the producer topics ('prod-1' and 'prod-2') to the consumer topics ('cons-1' and 'cons-2').
The topology method: kp.processors.KpTransformer::initializeTopology.

Log from the container ptc-transformer.

2.3. Consumer consumes the 'ConsumerRecord' objects with the Kafka client 'KafkaConsumer'.
The consumer method: kp.consumers.KpConsumer::consumeRecords.

Consumer in the container ptc-consumer-1 consumes from topic 'cons-1'.

Log from the container ptc-consumer-1.

Consumer in the container ptc-consumer-2 consumes from topic 'cons-2'.

Log from the container ptc-consumer-2.

Back to the top of the page

---

❸ Comparer and Counter

Docker containers for stream processing applications: Comparer and Counter

3.1. Comparer in the container ptc-comparer compares the records in the producer topics with the records in the consumer topics:

• 'prod-1' versus 'cons-1'
• 'prod-2' versus 'cons-2'

The topology method: kp.processors.KpComparer::initializeTopology.

A screenshot of the log from the container ptc-comparer.

3.2. Counter in the container ptc-counter counts the records in topics 'prod-1', 'prod-2', 'cons-1', and 'cons-2'.
The topology method: kp.processors.KpCounter::initializeTopology.

A screenshot of the log from the container ptc-counter.

Back to the top of the page

---

❹ Kafka Scripts

Action:

1. With batch file "02 Kafka list topics.bat" run in Docker the Kafka script for topic listing.
2. With batch file "03 Kafka consume.bat" run in Docker the Kafka consumer console on topic 'cons-1'.

4.1. The batch file "02 Kafka list topics.bat".
A screenshot of the console log from the run of the batch file "02 Kafka list topics.bat".

4.2. The batch file "03 Kafka consume.bat".

Console log from the run of the batch file "03 Kafka consume.bat".

Back to the top of the page

---