From Sleep to Speed: Making Rdkafka Sync Operations 16 Times Faster

Table of Contents

• 1 The Problem with Sleep in Synchronous Operations1.1 How Sleep Was Used in rdkafka1.2 Why Sleep is Inefficient1.3 Validating Assumptions
• 1.1 How Sleep Was Used in rdkafka
• 1.2 Why Sleep is Inefficient
• 1.3 Validating Assumptions
• 2 The Role of Condition Variables and Mutexes2.1 Mutexes2.2 Condition Variables2.3 Spurious Wakeups
• 2.1 Mutexes
• 2.2 Condition Variables
• 2.3 Spurious Wakeups
• 3 Implementing Condition Variables in rdkafka3.1 Code Implementation3.2 Performance and Efficiency Gains
• 3.1 Code Implementation
• 3.2 Performance and Efficiency Gains
• 4 Implications for Ruby's Scheduler
• 5 Rdkafka and WaterDrop Synchronicity Remarks
• 6 Conclusions

As an open-source developer, I constantly seek performance gains in the code I maintain. Since I took over rdkafka from AppSignal in November 2023, I promised not only to maintain the gem but to provide a stream of feature improvements and performance enhancements. One key area where performance can often be improved is how synchronization is handled in synchronous operations. This article discusses our significant improvement in rdkafka by replacing sleep with condition variables and mutexes.

rdkafka-ruby