How (and why) we built our own streaming framework
10:30/11:10
How Adyen re-architected their platform to improve throughput and latency by developing a highly consistent Kafka-like streaming framework in-house. Adyen processes close to 5 billion transactions yearly and is rapidly growing. As a result databases were straining under load and batch processes were constantly at risk of missing their scheduled deadline. This led to Adyen deploying a streaming data flow for file generation, reporting and analytics, effectively doing away with costly batch job processing and making sharding a cinch.
Language: English
Level: Advanced
Felix Koeslag
Developer Team Lead - Adyen
Felix is one of Adyen’s Tech leads, his main focus has been the scalability of Adyen’s payment fraud detection system and data processing infrastructure. For the latter project he has built a streaming framework focused on simplicity and correctness, which now powers all Adyen’s financial reporting for clients such as Uber, Spotify and Netflix. He is currently developing Adyen’s new banking stack which will handle the money flows from it’s global e-commerce business.
