Starfish Storage

Starfish is the unstructured data management platform for high performance computing (HPC), AI/ML, and other demanding file-based workloads. Starfish provides a unified index and a common API for addressing, managing, and moving files across a diversity of file storage systems, including HPC file systems, scale-out NAS, conventional file servers, and S3-style object stores. Starfish is vendor-agnostic, supporting devices and services from all hardware, software, and cloud vendors. Common use cases include data classification, protection, archiving, migration, ROT cleanup, cost accounting, and workflow automation.

Starfish supports such a wide variety of use cases due to its super-flexible architecture that combines a metadata-rich catalog with a scale-out data mover and batch processor. The feedback loop between the catalog and the batch processor enables automated data classification, metadata-driven storage management policies, user self-service, and file-processing workflows.