Events Analysis with Riak, Perl and Erlang at Booking.com

At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.

Talk objectives:

- present a real use case of Riak, used here in an unusual situation.
- from the problem description to making the solution future-proof, explain the architecture choices
- give a list of steps to find the right distributed database
- make the case that turn-key solutions are not great, we always need hackability
- show some code in fun languages: Erlang and Perl (and maybe Elixir)

Target audience:

- intermediate software engineers that understand the challenge of big data, realtime analytics and distributed systems.

Developers will be happy to see some (only a bit of easy) Erlang and Perl code. Architects will understand what challenge we were facing, why we went with our solution, what it can bring to them and what are the alternatives. Sales/Marketing guys will be bored and won't understand (that's on purpose).