BlobSeer: Next-generation data management for large scale infrastructures. As data volumes increase at a high speed in more and more application fields of science, engineering, information services, etc., the challenges posed by data-intensive computing gain increasing importance. The emergence of highly scalable infrastructures, e.g. for cloud computing and for petascale computing and beyond, introduces additional issues for which scalable data management becomes an immediate need. This paper makes several contributions. First, it proposes a set of principles for designing highly scalable distributed storage systems that are optimized for heavy data access concurrency. In particular, we highlight the potentially large benefits of using versioning in this context. Second, based on these principles, we propose a set of versioning algorithms, both for data and metadata, that enable a high throughput under concurrency. Finally, we implement and evaluate these algorithms in the BlobSeer prototype, that we integrate as a storage backend in the Hadoop MapReduce framework. We perform extensive microbenchmarks as well as experiments with real MapReduce applications: they demonstrate that applying the principles defended in our approach brings substantial benefits to data intensive applications.
Keywords for this software
References in zbMATH (referenced in 5 articles , 1 standard article )
Showing results 1 to 5 of 5.
- Nicolae, Bogdan; Cappello, Franck: BlobCR: virtual disk based checkpoint-restart for HPC applications on iaas clouds (2013)
- Carpen-Amarie, Alexandra; Costan, Alexandru; Cai, Jing; Antoniu, Gabriel; Bougé, Luc: Bringing introspection into BlobSeer: towards a self-adaptive distributed data management system (2011)
- Nicolae, Bogdan; Antoniu, Gabriel; Bougé, Luc; Moise, Diana; Carpen-Amarie, Alexandra: BlobSeer: Next-generation data management for large scale infrastructures (2011)
- Nicolae, Bogdan; Antoniu, Gabriel; Bougé, Luc: Enabling high data throughput in desktop grids through decentralized data and metadata management: The BlobSeer approach (2009)
- Nicolae, Bogdan; Antoniu, Gabriel; Bougé, Luc: Blobseer: how to enable efficient versioning for large object storage under heavy access concurrency (2009)
Further publications can be found at: http://blobseer.gforge.inria.fr/doku.php?id=main:publications