.. _roadmap: Roadmap ======= The current release of `HyperShell` is nearly feature complete. Inevitably there will be additional bug fixes, improvements, and refactorings. Below is a list of project items we're considering for the near future. ------------------- Tutorials and Walkthroughs -------------------------- **End of 2024** | We've been working hard for the past year to put together a series of real-world scientific use-cases with tangible data (or inputs) that users can download and run on their own to learn about all the different ways `HyperShell` can be used with its myriad features. Despite a few setbacks, we plan to have topics on Bioinformatics/genomics with RNA-sequence data (likely agricultural) as a bog-standard scenario. Similarly in Astronomy with data reduction pipelines. For the larger, extreme end of the high-throughput regime we hope to include something from Mathematics with an optimized C++ application to validate prime numbers, run at scale. We'll include everything in the tutorial sections here on the website. Additionally though we're putting together an extended workshop as part of the `ACCESS `_ community of NSF-funded high-performance computing *resource providers* here at Purdue University. This will be simulcast among multiple institutions and we'll hopefully publish the recording here on the website as well. ------------------- Testing and Benchmarks ---------------------- **End of 2024** | We've done extensive realworld testing and performance analysis on numerous systems including multiple Top500 supercomputers, such as `Anvil `_ at Purdue and `Summit `_ at Oak Ridge National Lab. We plan to add an automated, end-to-end test suite to the project to establish reproducible statistics for comparison against platforms, architectures, versions of Python, etc. We already have a private repository with scale-out tests but we hope to clean this up and abstract it to go more places. ------------------- Website ------- **End of 2024** | We have the `hypershell.org `_ domain and are working on a beautiful front-end website to act as a landing page for the project with additional content and information. ------------------- Refactoring ----------- **Early 2025** | The current project is a single Python package. We have other affiliate packages in mind and would like to keep everything under one roof as a small monorepo on GitHub. This would allow for installing ``hypershell-core`` as a dependency in Python projects without the need to pollute them with the command-line interface. But also extra projects, like ``hypershell-nextflow`` (see next). ------------------- Affiliate Packages ------------------ **Early 2025** | `HyperShell` provides high-throughput scheduling on HPC clusters where policies and practical considerations prevent direct scheduling of small tasks (e.g., with Slurm). For all the reasons one might need this kind of program, so too would a workflow system like `NextFlow `_. We are working on a plugin to allow use of `HyperShell` as an execution backend for `NextFlow` pipelines. ------------------- Features -------- **End of 2025** | `HyperShell` is essentially feature complete. But there are a few things that may just yet be useful additions to the system. Here are a few ideas that we already have the basics for. **Resource Monitoring** Automatic capture of resource utilization for both node-level (clients) and task-level. This would include both CPU and main memory usage of the processes as well as things like GPU usage (Nvidia, ROCm). This would function in the same way as capture of and for tasks. Configuration and/or command-line options would trigger client or task-level telemetry. **Database Partitioning** For extreme scale (10M+ task) clusters using a database (likely Postgres) there are performance issues with task scheduling and updates. Postgres (and its extension ecosystem) already have robust tools for helping this situation. We could build some of this into the program itself to make it easier for novice users. **Plugin System** It might be useful to enable event-driven custom behavior when using `HyperShell` as a library within your project. With events like `on_submit`, `on_schedule`, or `on_returned`, one could register functions to affect new behavior.