The efficient, convenient, and robust execution of data-driven workflows and enhanced data management are key for productive in computer-aided RD&E. Still, the storage stack is based on the low-level POSIX I/O (or objects in cloud storage). This talk introduces chances for establishing an open community-driven next-generation storage interface in a similar fashion to the existing forums. The forum would bring together vendors, storage experts, and users to discuss key features of the API and establish governance strategies. The envisioned coarse-grained API aims to overcome current obstacles for highly parallel workflows but would be beneficial also in the domain of big data and even desktop PC. It bears the opportunity to create a new ecosystem.
The envisioned API abstracts storage and data-flow computation with the clear goal of a separation of concern between the definition of tasks and the execution on a specific hardware. Additionally, it offers means to express workflow execution and lifecycle management. This allows smart schedulers to dispatch compute and I/O across the available hardware in an efficient manner. Since the user shall not care about data and compute locations, we broaden the term liquid computing. We aim to develop a first prototype on top of existing software technology like libfabric, parallel file systems, and NoSQL solutions. The final component is how to enable vendors to do differentiating optimizations without violating the common interface.
On this page, we will soon release a detailed description of this effort.
If you are interested in this topic, subscribe to our mailing list.