How (and why) we built our own streaming framework
How Adyen re-architected their platform to improve throughput and latency by developing a highly consistent Kafka-like streaming framework in-house. Adyen processes close to 5 billion transactions yearly and is rapidly growing. As a result databases were straining under load and batch processes were constantly at risk of missing their scheduled deadline. This led to Adyen deploying a streaming data flow for file generation, reporting and analytics, effectively doing away with costly batch job processing and making sharding a cinch.