HDF5 Encryption
Tuesday, June 10, 2025 3:00 PM to Thursday, June 12, 2025 4:00 PM · 2 days 1 hr. (Europe/Berlin)
Foyer D-G - 2nd floor
Research Poster
Compiler and Tools for Parallel ProgrammingStorage Technologies and Architectures
Information
Poster is on display and will be presented at the poster pitch session.
As data has become one of the most valuable assets, protecting sensitive information from unauthorized access has become a critical priority for organizations across various sectors, including government, finance, healthcare, and research. While many data management solutions offer robust encryption for data at rest and in transit, the popular HDF5 format—widely used in scientific and industrial applications—lacks inherent support for encrypting sensitive data. The open nature of the HDF5 format and its lack of built-in security mechanisms present significant challenges for storing confidential data, particularly in industries such as biomedical and pharmaceuticals.
We propose a solution for securing HDF5 data by introducing encryption capabilities within the HDF5 architecture. We present a prototype implementation that enables encryption of both data and metadata stored in HDF5 files, providing protection during storage, transit and I/O without the need to decrypt entire files. The proposed encryption solution maintains the HDF5 library’s functionality and supports random access to encrypted data without altering the underlying file format. Furthermore, the solution integrates seamlessly with existing HDF5 applications, requiring no special coding effort.
We describe the limitations of previous encryption approaches in HDF5, which were constrained by the data filtering mechanism, and introduce an innovative I/O driver that handles encryption at the page level. This approach ensures comprehensive encryption while enabling transparency and efficient access for applications. The prototype leverages the GNU encryption library (gcrypt) and supports AES256 and Twofish ciphers. Future work aims to extend the solution to parallel HDF5 applications, improve performance, and offer a configurable, pluggable encryption driver to meet the evolving needs of HDF5 users.
Contributors:
As data has become one of the most valuable assets, protecting sensitive information from unauthorized access has become a critical priority for organizations across various sectors, including government, finance, healthcare, and research. While many data management solutions offer robust encryption for data at rest and in transit, the popular HDF5 format—widely used in scientific and industrial applications—lacks inherent support for encrypting sensitive data. The open nature of the HDF5 format and its lack of built-in security mechanisms present significant challenges for storing confidential data, particularly in industries such as biomedical and pharmaceuticals.
We propose a solution for securing HDF5 data by introducing encryption capabilities within the HDF5 architecture. We present a prototype implementation that enables encryption of both data and metadata stored in HDF5 files, providing protection during storage, transit and I/O without the need to decrypt entire files. The proposed encryption solution maintains the HDF5 library’s functionality and supports random access to encrypted data without altering the underlying file format. Furthermore, the solution integrates seamlessly with existing HDF5 applications, requiring no special coding effort.
We describe the limitations of previous encryption approaches in HDF5, which were constrained by the data filtering mechanism, and introduce an innovative I/O driver that handles encryption at the page level. This approach ensures comprehensive encryption while enabling transparency and efficient access for applications. The prototype leverages the GNU encryption library (gcrypt) and supports AES256 and Twofish ciphers. Future work aims to extend the solution to parallel HDF5 applications, improve performance, and offer a configurable, pluggable encryption driver to meet the evolving needs of HDF5 users.
Contributors:
Format
On DemandOn Site
