Evolution of Riak: resolving problems in distributed systems

This talk will cover some problems of building a distributed system framework:

• Mass adoption of scale-out with difficult developer trade-offs is hard to achieve;
• The problems of testing claims of correctness;
• Replication of activity and the impact on throughput, and replication of data can itself be a cause of data loss;
• Non-conformity of performance between nodes is very difficult to manage (one slow node).

How the Riak framework is evolving to solve these problems:

• Through the power of open source, and the capability to manage with small numbers of contributors spread globally, whilst better leveraging the BEAM eco-system to get progress for "free";
• Through extended use of property-based testing;
• Through increased specialisation of sub-components to always do "just enough" work, with better alignment between local storage layout and overall database features;
• Through reactive work distribution, and minimisation of scheduled background tasks.

OBJECTIVES

An opportunity to talk frankly about the problems that Riak has faced, and raise some optimism about how lessons learnt may give Riak a worthwhile future as an open-source Erlang project.

TARGET AUDIENCE

People interested in the theory distributed databases, the practicalities of real-world scale-out problems and their solutions.