50 Posts

  • Poison Pills in Kafka (I)

    What is a poison pill? A “poison pill” is a record that always fails when consumed, no matter how many times it is attempted. They come in different forms: Corrupted records. Records that make your consumer deserializer fail (e.g., an Avro record whose writer schema is not compatible with the consumer reader schema). The problem with a poison pill is unless the consumer eventually handles it, it blocks the consumption of the topic/partition that contains it, halting the consumer progress. What can we do with poison pills? There are many different strategies to deal with […]

  • Kafka defaults that you should re-consider (I)

    Image taken from http://www.htbdpodcast.com There is a vast number of configuration options for Apache Kafka, mostly because the product can be fine-tuned to perform in various scenarios (e.g., low latency, high throughput, durability). These defaults span across brokers, producers and consumers (plus other sidecar products like Connect or Streams). The guys at Kafka do their best to provide a comprehensive set of defaults that will just work, but some of them can be relatively dangerous if used blindly, as they might have unexpected side effects, or be optimized for a use case different to yours. […]

  • Timeouts in Kafka clients and Kafka Streams

    IMPORTANT: This is information is based on Kafka and Kafka Streams 1.0.0. Past or future versions may defer. As with any distributed system, Kafka relies on timeouts to detect failures. Those timeouts can be sent by clients and brokers that want to detect each other unavailability. The following is a description of the configuration values that control timeouts that both brokers and client will use to detect clients not being available. The original design for the Poll() method in the Java consumer tried to kill two birds with one stone: Guarantee consumer liveness Guarantee progress […]

  • Simon Sinek on Leadership

    Anyone could be a leader if there was no cost. True leaders willingly pay a price, to sacrifice self interest, to have the honor to lead.

    Simon Sinek

By Daniele Zedda • 18 February


By Daniele Zedda • 18 February

Share on