hypershell.cluster


Directly embed the full cluster using one of the available thread variants (LocalCluster, RemoteCluster, AutoScalingCluster, or SSHCluster) or by using one of the provided high-level functions (run_local(), run_cluster(), or run_ssh()).

All of the parameters used below are largely the same in each instance. Configuration files have no impact when using the library; i.e., default values are not overridden by configuration.


Note

The database connection details are only specified via configuration (files or environment). To control the database configuration programmatically inject environment variables before importing hypershell.

Functions


run_local(*args, **kwargs) → None[source]

Run local cluster until completion.

All function arguments are forwarded directly into a LocalCluster thread.

Example

>>> from hypershell.cluster import run_local
>>> run_local(['echo AAA', 'echo BBB', 'echo CCC'],
...           num_threads=16, in_memory=True, no_confirm=True)

See also


run_cluster(*args, autoscaling: bool = False, **kwargs) → None[source]

Run cluster with remote clients until completion.

All function arguments are forwarded directly into either the RemoteCluster or AutoScalingCluster thread.

If autoscaling is enabled then we use the AutoScalingCluster instead of the RemoteCluster.

Example

>>> from hypershell.cluster import run_cluster
>>> run_cluster(
...     restart_mode=True, launcher='srun', max_retries=2, eager=True,
...     client_timeout=600, task_timeout=300, capture=True
... )

See also


run_ssh(*args, **kwargs) → None[source]

Run cluster with remote clients via SSH until completion.

All function arguments are forwarded directly into the SSHCluster thread.

Example

>>> from hypershell.cluster import run_ssh
>>> run_ssh(
...     nodelist=['a00.cluster', 'a01.cluster', 'a02.cluster'],
...     restart_mode=True, max_retries=2, eager=True,
...     client_timeout=600, task_timeout=300, capture=True
... )

See also

Classes


class LocalCluster(source: Iterable[str] = None, num_threads: int = 1, port: int = 50001, template: str = '{}', cores: int = None, memory: int = None, client_cores: int = None, client_memory: int = None, bundlesize: int = 1, bundlewait: int = 5, in_memory: bool = False, no_confirm: bool = False, poll: int = 30, forever_mode: bool = False, restart_mode: bool = False, max_retries: int = 1, eager: bool = False, redirect_failures: IO = None, redirect_output: IO = None, redirect_errors: IO = None, delay_start: float = 0, capture: bool = False, monitor: bool = False, client_timeout: int = None, task_timeout: int = None, task_signalwait: int = 10, ratelimit: int = None, tls: TLSConfig | None = None)[source]

Bases: Thread

Run server with single local client thread.

Parameters:

  • source (Iterable[str], optional) – Any iterable of command-line tasks. A new source results in a SubmitThread populating either the database or the queue directly depending on in_memory.

  • num_threads (int, optional) – Number of executor threads (use 0 for auto-detection). See DEFAULT_NUM_THREADS.

  • port (int, optional) – Port number for server (default: 0). See DEFAULT_PORT.

  • template (str, optional) – Template command pattern. See DEFAULT_TEMPLATE.

  • cores (int, optional) – Default number of cores to use for each task (default: none). May be overridden by inline-comment.

  • memory (int, optional) – Default memory in bytes to use for each task (default: none). May be overridden by inline-comment.

  • client_cores (int, optional) – Set core limit for client (default: all available cores).

  • client_memory (int, optional) – Set memory limit for client (default: all available memory).

  • bundlesize (int optional) – Size of task bundles returned to server. See DEFAULT_BUNDLESIZE.

  • bundlewait (int optional) – Waiting period in seconds before forcing return of task bundle to server. See DEFAULT_BUNDLEWAIT.

  • in_memory (bool, optional) – If True, revert to basic in-memory queue.

  • no_confirm (bool, optional) – Disable client confirmation of tasks received.

  • poll (int, optional) – Polling interval in seconds between database queries if no tasks are available. See DEFAULT_SERVER_POLL.

  • forever_mode (bool, optional) – Regardless of source, if enabled schedule forever. Conflicts with restart_mode and in_memory. Default is False.

  • restart_mode (bool, optional) – If source is empty, this option allows for the server to continue with scheduling from the database until complete. Conflicts with in_memory. Default is False.

  • max_retries (int, optional) – Number of allowed task retries. See DEFAULT_ATTEMPTS.

  • eager (bool, optional) – When enabled tasks are retried immediately ahead scheduling new tasks. See DEFAULT_EAGER_MODE.

  • redirect_failures (IO, optional) – Open file-like object to write failed tasks.

  • redirect_output (IO, optional) – Optional file-like object for <stdout> redirect.

  • redirect_errors (IO, optional) – Optional file-like object for <stderr> redirect.

  • delay_start (float, optional) – Delay in seconds before connecting to server. See DEFAULT_DELAY.

  • capture (bool, optional) – Isolate task <stdout> and <stderr> in discrete files (default: False).

  • monitor (bool, optional) – Track CPU cores and memory usage of each task (default: False).

  • client_timeout (int, optional) – Timeout in seconds before disconnecting from server. By default, the client waits for server tor request disconnect.

  • task_timeout (int, optional) – Task-level walltime limit in seconds (default: none).

  • task_signalwait (int, optional) – Signal escalation waiting period in seconds on task timeout. See DEFAULT_SIGNALWAIT.

  • ratelimit (int, optional) – Maximum allowed tasks per second (default: none). There is no limit on task throughput unless specified.

  • tls – (TLSConfig, optional): TLS configuration for queue interface. Clients must connect with compatible configuration. See security documentation for details.

Example

>>> from hypershell.cluster import LocalCluster
>>> cluster = LocalCluster.new(
...     ['echo AAA', 'echo BBB', 'echo CCC'],
...     num_threads=16, in_memory=True, no_confirm=True
... )
>>> cluster.join()

See also

classmethod new(*args, **kwargs) → Thread

Initialize and start the thread.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

join(timeout: float | None = None) → None

Calls Thread.join but re-raises exceptions.

stop(wait: bool = False, timeout: int = None) → None[source]

Stop child threads before main thread.


class RemoteCluster(source: Iterable[str] = None, num_threads: int = 1, template: str = '{}', cores: int = None, memory: int = None, client_cores: int = None, client_memory: int = None, bundlesize: int = 1, bundlewait: int = 5, bind: Tuple[str, int] = ('0.0.0.0', 50001), launcher: str = 'mpirun', launcher_args: List[str] = None, remote_exe: str = 'hyper-shell', in_memory: bool = False, no_confirm: bool = False, poll: int = 30, forever_mode: bool = False, restart_mode: bool = False, max_retries: int = 1, eager: bool = False, redirect_failures: IO = None, delay_start: float = 0, capture: bool = False, monitor: bool = False, client_timeout: int = None, task_timeout: int = None, task_signalwait: int = 10, ratelimit: int = None, tls: TLSConfig | None = None)[source]

Bases: Thread

Run server with remote clients via external launcher (e.g., mpirun).

Parameters:

  • source (Iterable[str], optional) – Any iterable of command-line tasks. A new source results in a SubmitThread populating either the database or the queue directly depending on in_memory.

  • num_threads (int, optional) – Number of executor threads (use 0 for auto-detection). See DEFAULT_NUM_THREADS.

  • template (str, optional) – Template command pattern. See DEFAULT_TEMPLATE.

  • cores (int, optional) – Default number of cores to use for each task (default: none). May be overridden by inline-comment.

  • memory (int, optional) – Default memory in bytes to use for each task (default: none). May be overridden by inline-comment.

  • client_cores (int, optional) – Set core limit for clients (default: all available cores).

  • client_memory (int, optional) – Set memory limit for clients (default: all available memory).

  • bundlesize (int optional) – Size of task bundles returned to server. See DEFAULT_BUNDLESIZE.

  • bundlewait (int optional) – Waiting period in seconds before forcing return of task bundle to server. See DEFAULT_BUNDLEWAIT.

  • bind (tuple, optional) – Bind address for server with port number (default: 0.0.0.0). See DEFAULT_PORT.

  • launcher (str, optional) – Launcher program used to bring up clients on other hosts. See DEFAULT_LAUNCHER.

  • launcher_args (List[str], optional) – Additional command-line arguments for launcher program.

  • remote_exe (str, optional) – Program name or path for remote executable. See DEFAULT_REMOTE_EXE.

  • in_memory (bool, optional) – If True, revert to basic in-memory queue.

  • no_confirm (bool, optional) – Disable client confirmation of tasks received.

  • poll (int, optional) – Polling interval in seconds between database queries if no tasks are available. See DEFAULT_SERVER_POLL.

  • forever_mode (bool, optional) – Regardless of source, if enabled schedule forever. Conflicts with restart_mode and in_memory. Default is False.

  • restart_mode (bool, optional) – If source is empty, this option allows for the server to continue with scheduling from the database until complete. Conflicts with in_memory. Default is False.

  • max_retries (int, optional) – Number of allowed task retries. See DEFAULT_ATTEMPTS.

  • eager (bool, optional) – When enabled tasks are retried immediately ahead scheduling new tasks. See DEFAULT_EAGER_MODE.

  • redirect_failures (IO, optional) – Open file-like object to write failed tasks.

  • delay_start (float, optional) – Delay in seconds before connecting to server. See DEFAULT_DELAY.

  • capture (bool, optional) – Isolate task <stdout> and <stderr> in discrete files (default: False).

  • monitor (bool, optional) – Track CPU cores and memory usage of each task (default: False).

  • client_timeout (int, optional) – Timeout in seconds before disconnecting from server. By default, the client waits for server tor request disconnect.

  • task_timeout (int, optional) – Task-level walltime limit in seconds (default: none). May be overridden by inline-comment.

  • task_signalwait (int, optional) – Signal escalation waiting period in seconds on task timeout. See DEFAULT_SIGNALWAIT.

  • ratelimit (int, optional) – Maximum allowed tasks per second per client (default: none). There is no limit on task throughput unless specified.

  • tls – (TLSConfig, optional): TLS configuration for queue interface. Clients must connect with compatible configuration. See security documentation for details.

Example

>>> from hypershell.cluster import RemoteCluster
>>> cluster = RemoteCluster.new(
...     restart_mode=True, launcher='srun', max_retries=2, eager=True,
...     client_timeout=600, task_timeout=300, capture=True
... )
>>> cluster.join()

See also

classmethod new(*args, **kwargs) → Thread

Initialize and start the thread.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

join(timeout: float | None = None) → None

Calls Thread.join but re-raises exceptions.

stop(wait: bool = False, timeout: int = None) → None[source]

Stop child threads before main thread.


class AutoScalingCluster(source: Iterable[str] = None, num_threads: int = 1, template: str = '{}', cores: int = None, memory: int = None, client_cores: int = None, client_memory: int = None, bundlesize: int = 1, bundlewait: int = 5, bind: Tuple[str, int] = ('0.0.0.0', 50001), launcher: str = '', launcher_args: List[str] = None, remote_exe: str = 'hyper-shell', in_memory: bool = False, no_confirm: bool = False, forever_mode: bool = False, restart_mode: bool = False, max_retries: int = 1, eager: bool = False, redirect_failures: IO = None, delay_start: float = 0, capture: bool = False, monitor: bool = False, client_timeout: int = None, task_timeout: int = None, task_signalwait: int = 10, ratelimit: int = None, poll: int = 30, policy: str = 'fixed', period: int = 60, factor: float = 1, init_size: int = 1, min_size: int = 0, max_size: int = 1, tls: TLSConfig | None = None)[source]

Bases: Thread

Run cluster with autoscaling remote clients via external launcher.

Parameters:

  • source (Iterable[str], optional) – Any iterable of command-line tasks. A new source results in a SubmitThread populating either the database or the queue directly depending on in_memory.

  • num_threads (int, optional) – Number of executor threads (use 0 for auto-detection). See DEFAULT_NUM_THREADS.

  • template (str, optional) – Template command pattern. See DEFAULT_TEMPLATE.

  • cores (int, optional) – Default number of cores to use for each task (default: none). May be overridden by inline-comment.

  • memory (int, optional) – Default memory in bytes to use for each task (default: none). May be overridden by inline-comment.

  • client_cores (int, optional) – Set core limit for clients (default: all available cores).

  • client_memory (int, optional) – Set memory limit for clients (default: all available memory).

  • bundlesize (int optional) – Size of task bundles returned to server. See DEFAULT_BUNDLESIZE.

  • bundlewait (int optional) – Waiting period in seconds before forcing return of task bundle to server. See DEFAULT_BUNDLEWAIT.

  • bind (tuple, optional) – Bind address for server with port number (default: 0.0.0.0). See DEFAULT_PORT.

  • launcher (str, optional) – Launcher program used to bring up clients on other hosts. See DEFAULT_AUTOSCALE_LAUNCHER.

  • launcher_args (List[str], optional) – Additional command-line arguments for launcher program.

  • remote_exe (str, optional) – Program name or path for remote executable. See DEFAULT_REMOTE_EXE.

  • in_memory (bool, optional) – If True, revert to basic in-memory queue.

  • no_confirm (bool, optional) – Disable client confirmation of tasks received.

  • forever_mode (bool, optional) – Regardless of source, if enabled schedule forever. Conflicts with restart_mode and in_memory. Default is False.

  • restart_mode (bool, optional) – If source is empty, this option allows for the server to continue with scheduling from the database until complete. Conflicts with in_memory. Default is False.

  • max_retries (int, optional) – Number of allowed task retries. See DEFAULT_ATTEMPTS.

  • eager (bool, optional) – When enabled tasks are retried immediately ahead scheduling new tasks. See DEFAULT_EAGER_MODE.

  • redirect_failures (IO, optional) – Open file-like object to write failed tasks.

  • delay_start (float, optional) – Delay in seconds before connecting to server. See DEFAULT_DELAY.

  • capture (bool, optional) – Isolate task <stdout> and <stderr> in discrete files (default: False).

  • monitor (bool, optional) – Track CPU cores and memory usage of each task (default: False).

  • client_timeout (int, optional) – Timeout in seconds before disconnecting from server. By default, the client waits for server tor request disconnect.

  • task_timeout (int, optional) – Task-level walltime limit in seconds (default: none). May be overridden by inline-comment.

  • task_signalwait (int, optional) – Signal escalation waiting period in seconds on task timeout. See DEFAULT_SIGNALWAIT.

  • ratelimit (int, optional) – Maximum allowed tasks per second per client (default: none). There is no limit on task throughput unless specified.

  • poll (int, optional) – Polling interval in seconds between database queries if no tasks are available. See DEFAULT_SERVER_POLL.

  • policy (str, optional) – Autoscaling policy (either ‘fixed’ or ‘dynamic’). See DEFAULT_AUTOSCALE_POLICY

  • period (int, optional) – Period in seconds between autoscaling events. See DEFAULT_AUTOSCALE_PERIOD

  • factor (float, optional) – Autoscaling factor. See DEFAULT_AUTOSCALE_FACTOR

  • init_size (int, optional) – Initial size of cluster (number of clients). See DEFAULT_AUTOSCALE_INIT_SIZE

  • min_size (int, optional) – Minimum size of cluster (number of clients). See DEFAULT_AUTOSCALE_MIN_SIZE

  • max_size (int, optional) – Maximum size of cluster (number of clients). See DEFAULT_AUTOSCALE_MAX_SIZE

  • tls – (TLSConfig, optional): TLS configuration for queue interface. Clients must connect with compatible configuration. See security documentation for details.

Example

>>> from hypershell.cluster import AutoScalingCluster
>>> cluster = AutoScalingCluster.new(
...     restart_mode=True, max_retries=2, eager=True,
...     client_timeout=600, task_timeout=300, capture=True,
...     launcher='srun -Q -A standby -t 01:00:00 --exclusive --signal=USR1@600'
... )
>>> cluster.join()

See also

classmethod new(*args, **kwargs) → Thread

Initialize and start the thread.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

join(timeout: float | None = None) → None

Calls Thread.join but re-raises exceptions.

stop(wait: bool = False, timeout: int = None) → None[source]

Stop child threads before main thread.


class SSHCluster(source: Iterable[str] = None, nodelist: List[str] = None, num_threads: int = 1, template: str = '{}', cores: int = None, memory: int = None, client_cores: int = None, client_memory: int = None, bundlesize: int = 1, bundlewait: int = 5, bind: Tuple[str, int] = ('0.0.0.0', 50001), launcher: str = 'ssh', launcher_args: List[str] = None, remote_exe: str = 'hyper-shell', export_env: bool = False, in_memory: bool = False, no_confirm: bool = False, poll: int = 30, forever_mode: bool = False, restart_mode: bool = False, max_retries: int = 1, eager: bool = False, redirect_failures: IO = None, delay_start: float = 0, capture: bool = False, monitor: bool = False, client_timeout: int = None, task_timeout: int = None, task_signalwait: int = 10, ratelimit: int = None, tls: TLSConfig | None = None)[source]

Bases: Thread

Run server with individual external ssh clients.

Parameters:

  • source (Iterable[str], optional) – Any iterable of command-line tasks. A new source results in a SubmitThread populating either the database or the queue directly depending on in_memory.

  • nodelist (List[str], required) – List of hostnames for launching clients. See also: NodeList.

  • num_threads (int, optional) – Number of executor threads (use 0 for auto-detection). See DEFAULT_NUM_THREADS.

  • template (str, optional) – Template command pattern. See DEFAULT_TEMPLATE.

  • cores (int, optional) – Default number of cores to use for each task (default: none). May be overridden by inline-comment.

  • memory (int, optional) – Default memory in bytes to use for each task (default: none). May be overridden by inline-comment.

  • client_cores (int, optional) – Set core limit for clients (default: all available cores).

  • client_memory (int, optional) – Set memory limit for clients (default: all available memory).

  • bundlesize (int optional) – Size of task bundles returned to server. See DEFAULT_BUNDLESIZE.

  • bundlewait (int optional) – Waiting period in seconds before forcing return of task bundle to server. See DEFAULT_BUNDLEWAIT.

  • bind (tuple, optional) – Bind address for server with port number (default: 0.0.0.0). See DEFAULT_PORT.

  • launcher (str, optional) – Launcher program used to bring up clients on other hosts. Defaults to ‘ssh’. Use launcher_args to provide command options.

  • launcher_args (List[str], optional) – Additional command-line arguments for launcher program.

  • remote_exe (str, optional) – Program name or path for remote executable. See DEFAULT_REMOTE_EXE.

  • export_env (bool, optional) – If enabled, embed local configuration as environment variables and forward with client launch command to remote hosts.

  • in_memory (bool, optional) – If True, revert to basic in-memory queue.

  • no_confirm (bool, optional) – Disable client confirmation of tasks received.

  • poll (int, optional) – Polling interval in seconds between database queries if no tasks are available. See DEFAULT_SERVER_POLL.

  • forever_mode (bool, optional) – Regardless of source, if enabled schedule forever. Conflicts with restart_mode and in_memory. Default is False.

  • restart_mode (bool, optional) – If source is empty, this option allows for the server to continue with scheduling from the database until complete. Conflicts with in_memory. Default is False.

  • max_retries (int, optional) – Number of allowed task retries. See DEFAULT_ATTEMPTS.

  • eager (bool, optional) – When enabled tasks are retried immediately ahead scheduling new tasks. See DEFAULT_EAGER_MODE.

  • redirect_failures (IO, optional) – Open file-like object to write failed tasks.

  • delay_start (float, optional) – Delay in seconds before connecting to server. See DEFAULT_DELAY.

  • capture (bool, optional) – Isolate task <stdout> and <stderr> in discrete files (default: False).

  • monitor (bool, optional) – Track CPU cores and memory usage of each task (default: False).

  • client_timeout (int, optional) – Timeout in seconds before disconnecting from server. By default, the client waits for server tor request disconnect.

  • task_timeout (int, optional) – Task-level walltime limit in seconds (default: none). May be overridden by inline-comment.

  • task_signalwait (int, optional) – Signal escalation waiting period in seconds on task timeout. See DEFAULT_SIGNALWAIT.

  • ratelimit (int, optional) – Maximum allowed tasks per second per client (default: none). There is no limit on task throughput unless specified.

  • tls – (TLSConfig, optional): TLS configuration for queue interface. Clients must connect with compatible configuration. See security documentation for details.

Example

>>> from hypershell.cluster import SSHCluster
>>> cluster = SSHCluster.new(
...     nodelist=['a00.cluster', 'a01.cluster', 'a02.cluster'],
...     restart_mode=True, max_retries=2, eager=True,
...     client_timeout=600, task_timeout=300, capture=True
... )
>>> cluster.join()

See also

classmethod new(*args, **kwargs) → Thread

Initialize and start the thread.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

join(timeout: float | None = None) → None

Calls Thread.join but re-raises exceptions.

stop(wait: bool = False, timeout: int = None) → None[source]

Stop child threads before main thread.


Warning

While nothing prevents you from creating more than one of these cluster-mode thread instances it should be considered an error to do so. The point of the thread is to provide for asynchronous operations not to allow for multiple instances. There are several unavoidable process-wide global considerations such as signal handling, resource tracking, and encryption key management that make it an error to spawn multiple instances of the thread with differing parameters.

Constants


DEFAULT_LAUNCHER: Final[str] = 'mpirun'

Default launcher program.

DEFAULT_REMOTE_EXE: Final[str] = 'hyper-shell'

Default remote executable name.

DEFAULT_AUTOSCALE_LAUNCHER: Final[str] = ''

Default launcher program for scaling clients.

DEFAULT_AUTOSCALE_POLICY: Final[str] = 'fixed'

Default autoscaling policy.

DEFAULT_AUTOSCALE_PERIOD: Final[int] = 60

Default period in seconds between autoscaling events.

DEFAULT_AUTOSCALE_FACTOR: Final[float] = 1

Default scaling factor.

DEFAULT_AUTOSCALE_MIN_SIZE: Final[int] = 0

Default minimum size of cluster (number of clients).

DEFAULT_AUTOSCALE_MAX_SIZE: Final[int] = 1

Default maximum size of cluster (number of clients).

DEFAULT_AUTOSCALE_INIT_SIZE: Final[int] = 1

Default initial size of cluster (number of clients).