The Expand Parallel File System

Sponsors

Objectives

The objective of the project is to develop and release an open-source high-performance parallel and distributed file system for HPC and Big Data environments, both for clusters and hybrid clouds. The goal is to transform the Expand parallel file system, researching and developing new mechanisms, in order to obtain a new technology that is distributed as open-source, and that can be used in operation in companies and research centres with HPC and Big Data applications. This project also aims to make Expand a high-performance file system suitable for use in both HPC and Big Data environments, in a way that differentiates it from existing products that are either adapted to the Big Data ecosystem, such as HDFS, or are adapted to HPC environments, as for example Lustre parallel file system. The objective is to obtain a product for addressing the fundamental I/O gap on current HPC and Big Data infrastructures, both for clusters and hybrid clouds, reducing installation, administration, and maintenance costs.

To achieve the former general goal, the project will pursue the following specific goals:
• O1. Improving the current prototype by including fault tolerance techniques and mechanisms for use in both HPC and Big Data environments.
• O2. Transforming the prototype into a system that can be used in production and real applications in companies and research centres with storage and high-performance needs in the input/output.
• O3. Evaluating the feasibility of the final version of Expand through relevant use cases.
• O4. Planning and coordinating project activities and to promote the use of Expand and to increase impact.

Work packages

WP1. Expand prototype for HPC and Big environments

O1. Design and develop new functionalities and API libraries for improving the current prototype of Expand with techniques for fault tolerance support, and mechanisms for use Expand in both HPC and Big Data environments, and new techniques for dynamic change in partition size.

WP2. Production release of Expand

O2. To transform the prototype into a system that can be used into production in companies and research centres with storage and high-performance needs in the input/output.

WP3. Final validation

O3. Evaluate the feasibility of the final version of Expand through relevant use cases. As use cases we propose to use the results obtained in the project entitled "Multi-source and multi-method prediction to support COVID-19 policy decision making". This project is a multidisciplinary project funded by the Community of Madrid in which several research teams from the Universidad Carlos III de Madrid, including the ARCOS group, participate. Within this project different HPC and Big Data applications are being developed that make intensive use of large volumes of data to model different COVID-19 propagation scenarios.

WP4. Coordination and scheduling of activities

O4. To plan and coordinate project activities and to promote results dissemination to increase impact.

Deliverables

WP1. Expand prototype for HPC and Big environments

• D1.1. Report on new prototype of Expand with fault tolerance support for Big Data and HPC applications



WP2. Production release of Expand

• D2.1. Final architecture of Expand
• D2.2. Installation and manual user guide



WP3. Final validation

• D3.1. Report on use cases and benchmarks
• D3.2. Report on final Evaluation



WP4. Coordination and scheduling of activities

• D4.1. Intermediate Project Report
• D4.2. Final report of the project and conclusions

The source code of the Expand Parallel File System

The source code can be reached from the GitHub repo:
https://github.com/xpn-arcos/xpn

Publications

• 2025
• Malleability and fault tolerance in ad-hoc parallel file systems,
  Authors: Dario Muñoz-Muñoz, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos, Jesus Carretero
  Journal paper: Cluster Computing
• Hierarchical and distributed data storage for Computing Continuum,
  Authors: Elias Del-Pozo-Puñal, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos,
  Journal paper: Future Generation Computer Systems
• Evaluación del sistema de ficheros Expand Ad-Hoc con aplicaciones de uso intensivo de datos DOI in progress...,
  Authors: Diego Camarmas-Alonso, Felix Garcia-Carballeira, Alejandro Calderon-Mateos, Darío Muñoz-Muñoz, Jesus Carretero,
  Conference paper: XXXV Jornadas de Paralelismo (JP25)
• Sistema de almacenamiento para computing continuum: aplicación a sistemas de información ferroviaria DOI in progress...,
  Authors: Elías Del-Pozo-Puñal, Félix García-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos,
  Conference paper: XXXV Jornadas de Paralelismo (JP25)
• Optimización de entornos de Big Data Analytics mediante sistemas de ficheros paralelos Ad-hoc DOI in progress...,
  Authors: Gabriel Sotodosos-Morales, Félix García-Carballeira, Diego Camarmas-Alonso, Alejandro Calderón-Mateos, Darío Muñoz-Muñoz, Jesús Carretero,
  Conference paper: XXXV Jornadas de Paralelismo (JP25)
• 2024
• Fault tolerant in the Expand Ad-Hoc parallel file system,
  Authors: Dario Muñoz-Muñoz, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos, Jesus Carretero,
  Conference paper: 30th International European Conference on Parallel and Distributed Computing (Euro-Par)
• Malleability in the Expand Ad-Hoc parallel file system,
  Authors: Dario Muñoz-Muñoz, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos, Jesus Carretero,
  Conference paper: 3rd EuroHPC Workshop on Dynamic Resources in HPC. Euro-Par 2024
• Tolerancia a fallos en el sistema de ficheros paralelo Expand Ad-Hoc,
  Authors: Dario Muñoz-Muñoz, Diego Camarmas-Alonso, Felix Garcia-Carballeira, Alejandro Calderon-Mateos, Jesus Carretero,
  Conference paper: XXXIV Jornadas de Paralelismo (JP24)
• Evaluación del rendimiento de un sistema de ficheros para sistemas IoT,
  Authors: Elias Del-Pozo-Puñal, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos,
  Conference paper: XXXIV Jornadas de Paralelismo (JP24)
• 2023
• A new Ad-Hoc parallel file system for HPC environments based on the Expand parallel file system,
  Authors: Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos, Jesus Carretero,
  Conference paper: 22nd International Symposium on Parallel and Distributed Computing (ISPDC)
• Evaluación de rendimiento del sistema de ficheros paralelo Expand Ad-Hoc en MareNostrum 4,
  Authors: Diego Camarmas-Alonso, Felix Garcia-Carballeira, Alejandro Calderon-Mateos, Jesus Carretero,
  Conference paper: XXXIII Jornadas de Paralelismo (JP23)
• Sistema de Ficheros Distribuido para IoT basado en Expand,
  Authors: Elias Del-Pozo-Puñal, Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos,
  Conference paper: XXXIII Jornadas de Paralelismo (JP23)

Contact

Welcome to our contact page for the Expand Parallel File System Project!
We appreciate your interest in our work and look forward to hearing from you.

Whether you have a question, a suggestion, or just want to point something out, we invite you to get in touch with us.
You can reach us via email at expand.arcos.inf.uc3m.es@gmail.com, and we will do our best to respond to your message as quickly as possible.

Thank you again for your interest in our Expand Parallel File System Project !