Object Storage for High Performance I/O
Friday, June 13, 2025 2:00 PM to 6:00 PM · 4 hr. (Europe/Berlin)
Hall Y8 - 2nd floor
Tutorial
Development of HPC SkillsHigh-Performance Data AnalyticsParallel File SystemsStorage Technologies and Architectures
Information
Recently we have seen a change in the diversity of applications utilising high performance computing (HPC), from primarily computational simulation approaches, to a more varied application mix including machine learning and data analytics. With this diversification in workloads, there has also been a diversification in I/O patterns; the movements in, and requirements on, data storage. Data storage technologies in HPC, such as parallel filesystems (PFS) have long been optimised for large scale bulk operations focussed on high bandwidth with relatively low volumes of metadata operations. However, many applications now exhibit non-optimal I/O patterns for large scale PFS, with significant amounts of small I/O operations, non-contiguous data access, and increases in read as well as write I/O operations.
Parallel filesystems, such as Lustre and Storage Scale, have been optimised and extended to provided higher metadata performance and to better handle small I/O operations. However, the underlying approach of POSIX-like I/O, with block-based read and write operations, and file-level data storage, sets limitations on the overall performance and functionality that such approaches can achieve. Other I/O approaches are available, object storage being one of the foremost amongst these. Object storage approaches allows a wide range of configuration and customisation around how data is grouped together and what metadata you record alongside data elements. This can enable much richer indexing and searching of stored data along with stored data structures that match application data more naturally than files. However, moving from a file-based approach to an object-based approach requires knowledge of both how objects work and how to use common object storage systems.
This tutorial aims to address these challenges, educating attendees in the design and usage of object storage technology, as alternatives to filesystems, and using the Ceph and DAOS object stores as examples. The tutorial provides both hands-on exercise and lecture sessions to give those who attend first hand experience in using and programming for object storage technology.
Contributors:
Parallel filesystems, such as Lustre and Storage Scale, have been optimised and extended to provided higher metadata performance and to better handle small I/O operations. However, the underlying approach of POSIX-like I/O, with block-based read and write operations, and file-level data storage, sets limitations on the overall performance and functionality that such approaches can achieve. Other I/O approaches are available, object storage being one of the foremost amongst these. Object storage approaches allows a wide range of configuration and customisation around how data is grouped together and what metadata you record alongside data elements. This can enable much richer indexing and searching of stored data along with stored data structures that match application data more naturally than files. However, moving from a file-based approach to an object-based approach requires knowledge of both how objects work and how to use common object storage systems.
This tutorial aims to address these challenges, educating attendees in the design and usage of object storage technology, as alternatives to filesystems, and using the Ceph and DAOS object stores as examples. The tutorial provides both hands-on exercise and lecture sessions to give those who attend first hand experience in using and programming for object storage technology.
Contributors:
Format
On Site
Targeted Audience
The target audience for this tutorial is application developers who are writing software for large scale HPC systems, those using machine learning on large parallel systems, and those who are design and building HPC systems.
Beginner Level
40%
Intermediate Level
60%
