Roadmap¶
The current release of HyperShell is nearly feature complete. Inevitably there will be additional bug fixes, improvements, and refactorings. Below is a list of project items we’re considering for the near future.
Tutorials and Walkthroughs¶
End of 2024
We’ve been working hard for the past year to put together a series of real-world scientific use-cases with tangible data (or inputs) that users can download and run on their own to learn about all the different ways HyperShell can be used with its myriad features.
Despite a few setbacks, we plan to have topics on Bioinformatics/genomics with RNA-sequence data (likely agricultural) as a bog-standard scenario. Similarly in Astronomy with data reduction pipelines.
For the larger, extreme end of the high-throughput regime we hope to include something from Mathematics with an optimized C++ application to validate prime numbers, run at scale.
We’ll include everything in the tutorial sections here on the website. Additionally though we’re putting together an extended workshop as part of the ACCESS community of NSF-funded high-performance computing resource providers here at Purdue University. This will be simulcast among multiple institutions and we’ll hopefully publish the recording here on the website as well.
Testing and Benchmarks¶
End of 2024
We’ve done extensive realworld testing and performance analysis on numerous systems including multiple Top500 supercomputers, such as Anvil at Purdue and Summit at Oak Ridge National Lab.
We plan to add an automated, end-to-end test suite to the project to establish reproducible statistics for comparison against platforms, architectures, versions of Python, etc. We already have a private repository with scale-out tests but we hope to clean this up and abstract it to go more places.
Website¶
End of 2024
We have the hypershell.org domain and are working on a beautiful front-end website to act as a landing page for the project with additional content and information.
Refactoring¶
Early 2025
The current project is a single Python package. We have other affiliate packages in mind and would like
to keep everything under one roof as a small monorepo on GitHub. This would allow for installing
hypershell-core as a dependency in Python projects without the need to pollute them with the
command-line interface. But also extra projects, like hypershell-nextflow (see next).
Affiliate Packages¶
Early 2025
HyperShell provides high-throughput scheduling on HPC clusters where policies and practical considerations prevent direct scheduling of small tasks (e.g., with Slurm). For all the reasons one might need this kind of program, so too would a workflow system like NextFlow. We are working on a plugin to allow use of HyperShell as an execution backend for NextFlow pipelines.
Features¶
End of 2025
HyperShell is essentially feature complete. But there are a few things that may just yet be useful additions to the system. Here are a few ideas that we already have the basics for.
- Resource Monitoring
Automatic capture of resource utilization for both node-level (clients) and task-level. This would include both CPU and main memory usage of the processes as well as things like GPU usage (Nvidia, ROCm). This would function in the same way as capture of <stdout> and <stderr> for tasks. Configuration and/or command-line options would trigger client or task-level telemetry.
- Database Partitioning
For extreme scale (10M+ task) clusters using a database (likely PostgreSQL) there are performance issues with task scheduling and updates. Postgres (and its extension ecosystem) already have robust tools for helping this situation. We could build some of this into the program itself to make it easier for novice users.
- Plugin System
It might be useful to enable event-driven custom behavior when using HyperShell as a library within your project. With events like on_submit, on_schedule, or on_returned, one could register functions to affect new behavior.