Tag: kafka streams

Poison Pills in Kafka (I)

What is a poison pill? A “poison pill” is a record that always fails when consumed, no matter how many times it is attempted. They come in different forms: Corrupted records. Records that make your consumer deserializer fail (e.g., an Avro record whose writer schema is not compatible with the consumer reader schema). The problem with a poison pill is unless the consumer eventually handles it, it blocks the consumption of the topic/partition that contains it, halting the consumer progress. What can we do with poison pills? There are many different strategies to deal with […]

By Javier

Timeouts in Kafka clients and Kafka Streams

IMPORTANT: This is information is based on Kafka and Kafka Streams 1.0.0. Past or future versions may defer. As with any distributed system, Kafka relies on timeouts to detect failures. Those timeouts can be sent by clients and brokers that want to detect each other unavailability. The following is a description of the configuration values that control timeouts that both brokers and client will use to detect clients not being available. The original design for the Poll() method in the Java consumer tried to kill two birds with one stone: Guarantee consumer liveness Guarantee progress […]

By Daniele Zedda • 18 February


By Daniele Zedda • 18 February

Share on