Prerequisites and Setup

To start building a Spring Boot Kafka consumer producer example, you need to have a basic understanding of Java and Spring Boot. Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming apps. You can learn more about Spring Boot and its features in our Spring Boot tutorial.

The required dependencies for this project include spring-boot-starter-kafka and kafka-clients. You can add these dependencies to your pom.xml file if you are using Maven. The spring-boot-starter-kafka dependency includes the necessary libraries for building Kafka producers and consumers.

Here is an example of a basic Kafka producer configuration class:

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;

@Configuration
public class KafkaProducerConfig {
 @Value("${kafka.bootstrap-servers}")
 private String bootstrapServers;

 // Create a KafkaTemplate to send messages to Kafka
 @Bean
 public KafkaTemplate kafkaTemplate() {
 // Create a KafkaTemplate with the bootstrap servers // this is necessary to connect to the Kafka cluster
 KafkaTemplate template = new KafkaTemplate<>();
 template.setBootstrapServers(bootstrapServers);
 return template;
 }
}

The KafkaTemplate is used to send messages to Kafka. You can learn more about KafkaTemplate and its configuration options in our Kafka Template Configuration article.

To test the Kafka producer, you can create a simple test class that sends a message to a Kafka topic. The expected output should be the message sent to the Kafka topic:

Message sent to Kafka topic: Hello, World!

You can learn more about building a Spring Boot Kafka consumer in our Spring Boot Kafka Consumer article.

Kafka Fundamentals and Concepts

Kafka’s architecture is based on a distributed system, where data is stored in **topics**. A **topic** is a stream of related messages, and it can be thought of as a category or a feed. Each **topic** is split into **partitions**, which are ordered, immutable logs that store messages. The **partitions** are distributed across multiple **brokers**, which are the servers that make up a Kafka cluster.

A **broker** can be either a leader or a follower, and each **partition** has one leader and zero or more followers. The leader is responsible for handling all read and write requests for a **partition**, while the followers replicate the leader’s data and take over as the new leader if the current leader fails. This ensures that data is highly available and fault-tolerant. To learn more about setting up a Kafka cluster, visit our guide on setting up a Kafka cluster.

The **producers** send messages to a **topic**, and the messages are then stored in the corresponding **partitions**. The **consumers** subscribe to a **topic** and read the messages from the **partitions**. The KafkaProducer class is used to send messages to a **topic**, while the KafkaConsumer class is used to subscribe to a **topic** and read messages. The **offset** is used to keep track of the last message that was read by a **consumer**.

Kafka provides a high-throughput and scalable messaging system, making it suitable for big data and real-time data processing applications. The **strong consistency** model ensures that all **brokers** have the same view of the data, and the **weak consistency** model allows for higher throughput but may result in inconsistencies. Understanding these **strong** and **weak consistency** models is crucial for designing and implementing a Kafka-based system, and further reading on Kafka consistency models can provide more insight into this topic.

Step-by-Step Guide to Creating a Kafka Consumer

To create a Kafka consumer in a Spring Boot application, you need to configure the consumer properties and implement a listener to process the incoming messages. The Spring Kafka project provides a simple way to integrate Kafka with Spring Boot. You can learn more about the Spring Kafka project and its features in our Spring Kafka tutorial.

The first step is to add the necessary dependencies to your pom.xml file if you are using Maven or your build.gradle file if you are using Gradle. You need to include the Spring Kafka and Spring Boot dependencies.

To implement a Kafka consumer, you need to create a KafkaListener that will listen to a specific topic and process the incoming messages. Here is an example of a simple Kafka consumer:

package com.example.kafkaconsumer;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class KafkaConsumer {
 @KafkaListener(topics = "my-topic", groupId = "my-group")
 public void receiveMessage(String message) {
 // Process the incoming message
 System.out.println("Received message: " + message);
 // You can also use a logger to log the message
 }
}

The KafkaListener annotation is used to specify the topic and group ID for the consumer. The receiveMessage method will be called for each incoming message.

When you run the application, the Kafka consumer will start listening to the specified topic and processing the incoming messages. The expected output will be:

Received message: Hello, World!
Received message: Foo, Bar!

For more information on how to configure and implement a Kafka producer in a Spring Boot application, you can refer to our Spring Boot Kafka producer example.

Full Example of a Spring Boot Kafka Consumer Producer Application

To create a **Kafka** consumer and producer in a **Spring Boot** application, you need to configure the **Kafka** properties and create a **Kafka** template. The Kafka configuration is crucial for setting up the connection to the **Kafka** cluster.

The **Kafka** consumer will listen to a specific topic and consume messages, while the producer will send messages to a topic. The KafkaTemplate class is used to send messages to a **Kafka** topic.

The following example demonstrates a complete **Spring Boot** application that produces and consumes **Kafka** messages:

package com.example.kafka;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;

@SpringBootApplication
public class KafkaApplication implements CommandLineRunner {

 @Autowired
 private KafkaTemplate<String, String> kafkaTemplate;

 public static void main(String[] args) {
 SpringApplication.run(KafkaApplication.class, args);
 }

 @Override
 public void run(String... args) throws Exception {
 // Send a message to the Kafka topic
 String topic = "example-topic";
 String message = "Hello, Kafka!";
 // We use the KafkaTemplate to send the message to the topic
 kafkaTemplate.send(topic, message).get();
 
 // Consume the message from the Kafka topic
 // For more information on consuming Kafka messages, see our Kafka consumer example
 }
}

The expected output will be:

Hello, Kafka!

This example demonstrates a basic **Kafka** consumer and producer in a **Spring Boot** application. For more information on **Kafka** configuration and **Spring Boot**, see our Kafka configuration example and introduction to Spring Boot.

Common Mistakes and Troubleshooting Tips

When working with Kafka and Spring Boot, several common issues can arise. Identifying and resolving these issues is crucial for a smooth development process. One of the primary concerns is handling deserialization errors.
For more information on Kafka configuration, visit our Spring Boot Kafka configuration guide.

Mistake 1: Incorrect Deserialization

Incorrect deserialization can lead to SerializationException. The following example demonstrates the incorrect way to deserialize a Kafka message:

public class KafkaConsumer {
 @KafkaListener(topics = "my-topic")
 public void receiveMessage(String message) { // WRONG
 // process the message
 }
}

This will result in a SerializationException because the deserializer is not specified. The correct way to deserialize a Kafka message is:

public class KafkaConsumer {
 @KafkaListener(topics = "my-topic")
 public void receiveMessage(@Payload String message, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) String key) {
 // process the message
 // We use @Payload to specify the deserializer for the message body
 }
}

The expected output will be the deserialized message:

Received message: Hello, World!

Production tip: Always specify the deserializer for the Kafka message to avoid SerializationException.

For further reading on Kafka Consumer configuration, visit our Kafka Consumer configuration guide.

Mistake 2: Missing groupId Configuration

Missing groupId configuration can lead to ConsumerGroupException. The following example demonstrates the incorrect way to configure a Kafka Consumer:

@Bean
public ConsumerFactory<String, String> consumerFactory() { // WRONG
 return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}

This will result in a ConsumerGroupException because the groupId is not specified. The correct way to configure a Kafka Consumer is:

@Bean
public ConsumerFactory<String, String> consumerFactory() {
 Map<String, Object> props = new HashMap<>();
 // We specify the groupId to identify the consumer group
 props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
 return new DefaultKafkaConsumerFactory<>(props);
}

For more information on Spring Boot and Kafka integration, visit our Spring Boot Kafka integration guide.

Best Practices for Deploying a Kafka Application to Production

When deploying a Kafka application to production, optimizing and securing the application is crucial. This involves configuring the KafkaConsumer and KafkaProducer to handle high volumes of data and ensuring that the application can recover from failures. To achieve this, developers should focus on implementing retry mechanisms and error handling strategies.

Production tip: Implement a retry mechanism with exponential backoff to handle temporary failures when producing or consuming messages from Kafka topics.

Configuring the acks setting for the KafkaProducer is also essential to ensure that messages are properly replicated across Kafka brokers. For more information on configuring the KafkaProducer, see our article on configuring a Kafka producer in a Spring Boot application.

Production tip: Set the acks setting to all to ensure that the KafkaProducer waits for all in-sync replicas to acknowledge the message before considering it sent.

To further secure the Kafka application, developers should implement authentication and authorization using SASL or SSL/TLS. This involves configuring the KafkaConsumer and KafkaProducer to use the chosen authentication mechanism.

Production tip: Use SSL/TLS to encrypt communication between the KafkaConsumer and KafkaProducer and the Kafka cluster, and implement role-based access control to restrict access to Kafka topics.

Monitoring the Kafka application is also crucial to ensure that it is performing as expected. Developers can use tools like Prometheus and Grafana to monitor Kafka metrics and detect any issues before they become critical. For more information on monitoring a Kafka application, see our article on monitoring Kafka with Prometheus and Grafana.

Testing Strategies for Kafka Consumer and Producer Applications

When developing **Kafka** consumer and producer applications, testing is crucial to ensure the reliability and performance of the system. To achieve this, you can use various **testing frameworks** and tools, such as **JUnit** and **Testcontainers**. For more information on setting up a **Kafka** cluster, refer to our article on setting up a Kafka cluster with Spring Boot.

To test **Kafka** consumer applications, you can use the KafkaConsumer class to subscribe to topics and consume messages. You can also use the KafkaProducer class to send messages to topics. When testing **Kafka** producer applications, you can use the KafkaProducer class to send messages to topics and verify that they are consumed correctly.

Here is an example of a test class that uses **Testcontainers** to start a **Kafka** container and test a **Kafka** consumer application:

public class KafkaConsumerTest {
 @ClassRule
 public static KafkaContainer kafka = new KafkaContainer("5.4.3");

 @Test
 public void testConsumeMessage() {
 // Create a Kafka consumer
 Properties props = new Properties();
 props.put("bootstrap.servers", kafka.getBootstrapServers());
 props.put("group.id", "test-group");
 props.put("key.deserializer", "org.springframework.kafka.support.serializer.StringDeserializer");
 props.put("value.deserializer", "org.springframework.kafka.support.serializer.StringDeserializer");
 KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
 
 // Subscribe to a topic
 consumer.subscribe(Collections.singleton("test-topic"));
 
 // Send a message to the topic
 KafkaProducer<String, String> producer = new KafkaProducer<>(props);
 producer.send(new ProducerRecord<>("test-topic", "test-message"));
 
 // Consume the message
 ConsumerRecords<String, String> records = consumer.poll(100);
 // We expect one record with the test message
 assertEquals(1, records.count());
 // Verify the message
 assertEquals("test-message", records.iterator().next().value());
 }
}

The expected output of this test will be:

1 record consumed with value: test-message

For further reading on **Kafka** consumer and producer configuration, refer to our article on configuring Kafka consumers and producers with Spring Boot. Additionally, you can use **Mockito** to mock the **Kafka** consumer and producer classes and test your application’s business logic.

Key Takeaways and Conclusion

When building a Spring Boot Kafka consumer producer application, it is essential to understand the key concepts and best practices. The KafkaTemplate class is used for sending messages to Kafka topics, while the KafkaListener annotation is used for consuming messages from Kafka topics. Proper configuration of consumer groups and topic partitions is also crucial for a scalable and fault-tolerant application.

To ensure reliable message delivery, implement idempotent producers and idempotent consumers to handle duplicate messages. Additionally, consider implementing retry mechanisms and dead letter queues to handle message processing failures. For more information on error handling strategies, refer to our article on Spring Boot Kafka Error Handling.

When designing the consumer producer application, consider the acks configuration to balance durability and throughput. The acks=all configuration provides the highest level of durability, but may impact performance. It is also essential to monitor the application using metrics and logging to detect potential issues and optimize performance.

By following these best practices and understanding the key concepts, developers can build a scalable and reliable Spring Boot Kafka consumer producer application. The spring-kafka project provides a comprehensive set of tools and configurations to simplify the development process. By leveraging these tools and configurations, developers can focus on building a robust and efficient application that meets the requirements of their use case.

Advanced Topics and Future Directions

As we continue to explore the capabilities of Kafka in our Spring Boot application, we can leverage advanced features such as log compaction and idempotent producers. The KafkaProducer class provides a range of configuration options, including the ability to specify a acks setting to control the number of acknowledgments required for a successful send operation. This can be particularly useful in scenarios where data integrity is paramount. For more information on configuring Kafka producers, see our article on Configuring Kafka Producers in Spring Boot.

Another key area of exploration is the use of Kafka Streams for real-time data processing. By utilizing the KStream and KTable APIs, developers can create complex data pipelines that integrate with their existing Kafka infrastructure. This can be particularly useful for applications that require low-latency data processing and aggregation. The KafkaStreams class provides a simple and intuitive API for building these types of applications.

Looking to the future, there are several emerging trends and technologies that are likely to have a significant impact on the development of Kafka applications. One area of interest is the integration of machine learning and artificial intelligence with Kafka, enabling the creation of more sophisticated and autonomous data processing systems. The use of cloud-native technologies, such as Kubernetes and containerization, is also likely to play a key role in the deployment and management of Kafka clusters.

As the Kafka ecosystem continues to evolve, it is likely that we will see new features and capabilities emerge that will further enhance the functionality and performance of our Spring Boot applications. By staying up-to-date with the latest developments and advancements in Kafka technology, developers can ensure that their applications remain scalable, efficient, and effective in meeting the needs of their users. For further reading on Kafka and its applications, see our article on Kafka Use Cases and Applications.

Read Next

• Java Algorithms
• Mastering SQL
• Spring Boot Tutorials
• Spring Batch Guide

Pillar Guide: Spring Boot Tutorials Hub — explore the full learning path.

You Might Also Like

Mastering Spring Boot JPA Pagination and Sorting
Mastering Spring Boot Actuator Monitoring
Spring Boot with Redis Caching Tutorial: A Comprehensive Guide (2026)