PyLauncher TaskList

Try on DesignSafe

PyLauncher TaskList#

Generating PyLauncher Tasklists for Parameter Sweeps

by Silvia Mazzoni, 2026

DesignSafe

This script expands a single command template into all combinations of parameter values, producing one shell command per combination.

This notebook shows a simple pattern for generating a PyLauncher tasklist (e.g., runsList.txt) by expanding a single command template into a full parameter sweep. The key idea is that you write one base_command with human-readable placeholders (like ALPHA, BETA, GAMMA), provide value lists for each parameter, and then programmatically produce one fully-resolved command per combination. The resulting lines are ready to paste directly into a PyLauncher task file, so PyLauncher can dispatch them as independent tasks across the allocated resources.

A second goal of the template is clean, collision-free output organization for large sweeps. The --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID" portion is a directory pattern that gives every task a unique target folder. Importantly, this path is rooted at $WORK, meaning outputs are written to the Work filesystem root location (your Work area) rather than inside the job’s execution directory (often a temporary or app-managed working directory). This choice is intentional: it reduces the burden of “packaging up” results at the end of the run.

How do the $... variables work?#

Those $-signed items (e.g., $WORK, $SLURM_JOB_ID, $LAUNCHER_JID, $LAUNCHER_TSK_ID) are environment variables. Your Python script does not “look them up” itself. Instead:

  1. PyLauncher launches each line as a shell command (typically via /bin/bash -c ... or similar).

  2. The shell expands environment variables before the command is executed.

  3. The expanded command is what actually runs on the compute node.

So the “program that knows” these values is the runtime environment + shell, not your simulate.py script.

  • $WORK is commonly provided on TACC/DesignSafe systems as an environment variable pointing to your Work filesystem location.

  • $SLURM_JOB_ID is provided by SLURM for every running batch job.

  • $LAUNCHER_JID and $LAUNCHER_TSK_ID are set by PyLauncher for each launched task (job/line id and task/slot id), which is why they’re perfect for avoiding output collisions across many tasks.

Quoting matters here: using "--output "$WORK/..."" keeps the argument as one path string, even if any component ever contained spaces (rare on HPC, but still good practice). The $... expansions still occur inside double quotes.

Why write to $WORK/... instead of the execution directory?#

When you run under Tapis/SLURM, there’s often a notion of an execution directory (where the app stages inputs and runs). Many app workflows then archive that execution directory back to storage when the job finishes.

By writing sweep outputs to $WORK/sweep_<jobid>/... (outside the execution directory):

  • Your task outputs land in a stable location designed for larger data.

  • The execution directory stays lighter, containing mostly scripts/logs rather than every sweep result.

  • When the job finishes, the end-of-job archiving step is faster and smaller because the heavy sweep results are not sitting inside what gets packaged.

In other words: you’re explicitly choosing to store results at the root of your Work area (in a job-specific folder), rather than inside the run directory that the app/Tapis may attempt to archive. This is especially helpful for large sweeps where archiving hundreds/thousands of files can dominate wall time and create unnecessary I/O load.

This python function has been added to the OpsUtils python library that is shared in DesignSafe Community.

from __future__ import annotations

from itertools import product
from pathlib import Path
from typing import Any, Dict, Iterable, List, Mapping, Sequence

import pandas as pd

def generate_task_commands(
    base_command: str,
    sweep: Mapping[str, Sequence[Any]],
    *,
    placeholder_style: str = "token",
) -> List[str]:
    """
    Expand a command template into a list of commands for all combinations.

    Parameters
    ----------
    base_command
        Command template containing placeholders for parameters. Example:

        'python3 -u simulate.py --alpha ALPHA --beta BETA --gamma GAMMA --output ".../slot_$LAUNCHER_TSK_ID"'

        Placeholders must match keys in `sweep`.

    sweep
        Mapping of placeholder -> list/tuple of values to sweep over.
        Example: {"ALPHA": [0.3, 0.5], "BETA": [1, 2]}

    placeholder_style
        How placeholders appear in `base_command`:
        - "token": placeholders are bare tokens like ALPHA, BETA (default)
        - "braces": placeholders are in braces like {ALPHA}, {BETA}

    Returns
    -------
    list of str
        One command per combination of values, in deterministic order based
        on the insertion order of `sweep`.

    Notes
    -----
    - This function does *string substitution only*; it does not validate that
      the command is runnable on your system.
    - Environment variables such as $WORK or $SLURM_JOB_ID are left untouched.
    """
    if not sweep:
        return [base_command]

    keys = list(sweep.keys())
    value_lists = [sweep[k] for k in keys]

    # Basic validation
    for k, vals in sweep.items():
        if not isinstance(vals, Sequence) or isinstance(vals, (str, bytes)):
            raise TypeError(f"sweep[{k!r}] must be a non-string sequence of values.")
        if len(vals) == 0:
            raise ValueError(f"sweep[{k!r}] is empty; provide at least one value.")

    commands: List[str] = []
    for combo in product(*value_lists):
        cmd = base_command
        for k, v in zip(keys, combo):
            if placeholder_style == "token":
                cmd = cmd.replace(k, str(v))
            elif placeholder_style == "braces":
                cmd = cmd.replace("{" + k + "}", str(v))
            else:
                raise ValueError("placeholder_style must be 'token' or 'braces'.")
        commands.append(cmd)

    return commands


def write_tasklist(commands: Iterable[str], outfile: str | Path) -> None:
    """
    Write commands to a PyLauncher tasklist file (one command per line).
    """
    outpath = Path(outfile)
    outpath.parent.mkdir(parents=True, exist_ok=True)
    outpath.write_text("\n".join(commands) + "\n", encoding="utf-8")


# ---------------------------------------------------------------------
# Example usage (edit these for your sweep)
# ---------------------------------------------------------------------

inputFilename = "simulate.py"

base_command = (
    f'python3 -u {inputFilename} '
    f'--alpha ALPHA --beta BETA --gamma GAMMA '
    f'--output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"'
)

# Define parameters for dynamic generation (add/remove keys freely)
sweep_params: Dict[str, Sequence[Any]] = {
    "ALPHA": [0.3, 0.5, 3.7],
    "BETA": [1.1, 2, 3],
    "GAMMA": ["a", "b", "c"],
}

generated_tasks = generate_task_commands(base_command, sweep_params, placeholder_style="token")

print(f"Generated {len(generated_tasks)} task commands:")
print("-" * 60)
for cmd in generated_tasks:
    print(cmd)

# Optional: write a runsList file for PyLauncher
# write_tasklist(generated_tasks, "runsList.txt")

Generated 27 task commands:
------------------------------------------------------------
python3 -u simulate.py --alpha 0.3 --beta 1.1 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 1.1 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 1.1 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 2 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 2 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 2 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 3 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 3 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.3 --beta 3 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 1.1 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 1.1 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 1.1 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 2 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 2 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 2 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 3 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 3 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 0.5 --beta 3 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 1.1 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 1.1 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 1.1 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 2 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 2 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 2 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 3 --gamma a --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 3 --gamma b --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"
python3 -u simulate.py --alpha 3.7 --beta 3 --gamma c --output "$WORK/sweep_$SLURM_JOB_ID/line_$LAUNCHER_JID/slot_$LAUNCHER_TSK_ID"

Visualize Sweep Table#

def preview_sweep_table(sweep: Mapping[str, Sequence[Any]]) -> pd.DataFrame:
    """
    Create a preview table of all parameter combinations in a sweep.

    Parameters
    ----------
    sweep
        Mapping of parameter name -> sequence of values to sweep over.
        Example:
            {"ALPHA": [0.3, 0.5], "BETA": [1, 2], "GAMMA": ["a", "b"]}

    Returns
    -------
    pandas.DataFrame
        A table with one row per parameter combination. Column order follows
        the insertion order of `sweep`.

    Notes
    -----
    - This is intended for interactive notebooks. For large sweeps, consider
      displaying only `.head()` or sampling rows.
    - Numeric formatting (e.g., 3 significant figures) is handled by display
      settings; see example below.
    """
    if not sweep:
        return pd.DataFrame()

    keys = list(sweep.keys())
    value_lists = [sweep[k] for k in keys]

    # Basic validation
    for k, vals in sweep.items():
        if not isinstance(vals, Sequence) or isinstance(vals, (str, bytes)):
            raise TypeError(f"sweep[{k!r}] must be a non-string sequence of values.")
        if len(vals) == 0:
            raise ValueError(f"sweep[{k!r}] is empty; provide at least one value.")

    rows = [dict(zip(keys, combo)) for combo in product(*value_lists)]
    return pd.DataFrame(rows)

df = preview_sweep_table(sweep_params)

print(f"Total runs: {len(df)}")
df.head(10)

Total runs: 27
ALPHA BETA GAMMA
0 0.3 1.1 a
1 0.3 1.1 b
2 0.3 1.1 c
3 0.3 2.0 a
4 0.3 2.0 b
5 0.3 2.0 c
6 0.3 3.0 a
7 0.3 3.0 b
8 0.3 3.0 c
9 0.5 1.1 a

Optional: Display Numeric Values with 3 Significant Figures#

If you want the notebook preview to show numbers with 3 significant figures, you can set a pandas display format:

import pandas as pd
pd.options.display.float_format = "{:.3g}".format
df = preview_sweep_table(sweep_params)
df
ALPHA BETA GAMMA
0 0.3 1.1 a
1 0.3 1.1 b
2 0.3 1.1 c
3 0.3 2 a
4 0.3 2 b
5 0.3 2 c
6 0.3 3 a
7 0.3 3 b
8 0.3 3 c
9 0.5 1.1 a
10 0.5 1.1 b
11 0.5 1.1 c
12 0.5 2 a
13 0.5 2 b
14 0.5 2 c
15 0.5 3 a
16 0.5 3 b
17 0.5 3 c
18 3.7 1.1 a
19 3.7 1.1 b
20 3.7 1.1 c
21 3.7 2 a
22 3.7 2 b
23 3.7 2 c
24 3.7 3 a
25 3.7 3 b
26 3.7 3 c

Tip for Large Sweeps#

If your sweep is very large, preview only a small portion:

df.sample(5, random_state=0)   # random sample of N rows
# or
df.head(5)                     # first 20 rows
ALPHA BETA GAMMA
0 0.3 1.1 a
1 0.3 1.1 b
2 0.3 1.1 c
3 0.3 2 a
4 0.3 2 b 
print('done!')
Contents