Improving the Write Scalability of the CernVM File System with Erlang/OTP

The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service. It was developed at CERN to assist High Energy Physics (HEP) collaborations in deploying software on the worldwide distributed computing infrastructure for data processing applications, but it can also be used in other domains.

CernVM-FS is implemented as a POSIX read-only file system in user space (FUSE) and uses content-addressed storage and Merkle trees for maintaining file data and meta-data. Files are stored remotely on standard web servers and are retrieved and cached on-demand through outgoing HTTP connections only, avoiding most of the firewall issues of other network file systems.

For writing, CernVM-FS follows a publish-subscribe pattern with a single source of new content that is propagated to a large number of readers.

This talk focuses on a new publication architecture, capable of accommodating dynamic strategies for content publishing and improving the scalability of the publication phase. We chose Erlang/OTP for implementing certain components of the new architecture, due to its suitability for making scalable and fault-tolerant systems.

Talk objectives:

• Give an overview of the CernVM-FS project.
• Show a case study of using Erlang/OTP as part of a network file system architecture.

Target audience:

• People who are interested in Erlang, distributed file systems and systems programming.
• People who need to distribute many, relatively small, files to many places.