(Spring) Kafka - one more arsenal in a distributed toolbox
Nakul Mishra at Spring I/O 2018
Kafka is an open-source distributed commit log addressing low latency, high throughput, scalability, fault-tolerance, and disk-based retention. It can be used to build tracking systems, messaging systems, high performance streaming platforms, real-time analysis, audit log….you name it. In our case, it’s been used to build a scalable event-store and messaging platform that stores billions of messages. To ingest historical data in our event store, we opted for Spring Kafka due to easy integration with Spring framework and enhanced testing support introduced by it. In this talk, we’re taking a closer look at this magic combo and explain essential as well as more advanced Spring-Kafka concepts.
We will look at stuff such as producer, consumer, transactions, etc. and how Spring Kafka maps those concepts using KafkaMessageListenerContainer, Kafka template, @KafkaListener, @KafkaHandler, Kafka Transaction Manager, @SendTo, etc. in Spring's world. We will also demonstrate how to use spring-Kafka-test for developing and running your unit tests against embedded Kafka server.
Furthermore, we will dig into some of the enhancements, such as synchronizing a Kafka transaction with some other transaction, configuring generic MessageConverter bean for publishing/consuming messages and detecting asynchronous consumers when they are idle; introduced by SpringKafka. Last but not the least we will highlight some potential pitfalls that one should watch out for when going to production with Kafka.
We will look at stuff such as producer, consumer, transactions, etc. and how Spring Kafka maps those concepts using KafkaMessageListenerContainer, Kafka template, @KafkaListener, @KafkaHandler, Kafka Transaction Manager, @SendTo, etc. in Spring's world. We will also demonstrate how to use spring-Kafka-test for developing and running your unit tests against embedded Kafka server.
Furthermore, we will dig into some of the enhancements, such as synchronizing a Kafka transaction with some other transaction, configuring generic MessageConverter bean for publishing/consuming messages and detecting asynchronous consumers when they are idle; introduced by SpringKafka. Last but not the least we will highlight some potential pitfalls that one should watch out for when going to production with Kafka.