Advanced Python Function

Advanced Python Function#

This is an upgraded version of the helper that makes it easy to adapt the run command for any application, not just Python.

NOTE This function was generated by AI and serves only as a guide – needs to be tested and modified to your needs.

You can specify run_cmd, so it will use:

  • python myscript.py $SLURM_ARRAY_TASK_ID

  • or OpenSees model_${SLURM_ARRAY_TASK_ID}.tcl

  • or even mpirun mySolver …

It also writes a small README_jobs.txt. This file lists all the jobs in your array and the exact parameters they’ll use, by reading your CSV.

This is great for:

  • Documenting exactly what was run (and when),

  • Making your HPC sweeps reproducible,

  • Sharing with collaborators or supervisors.

This way it becomes a universal tool for your HPC workflows.

Flexible helper to create SLURM array scripts for any app#

def create_slurm_array_script(csv_file, job_name="ParamSweep", output_prefix="output", 
                              nodes=1, tasks=4, time="00:20:00", partition="normal",
                              run_cmd="python run_analysis.py ${SLURM_ARRAY_TASK_ID}"):
    """
    Creates a SLURM job array script + README summary from a CSV or text file.
    Lets you specify any run command (OpenSees, Python, OpenFOAM, etc.).
    """

    # Count lines and read parameters
    with open(csv_file) as f:
        lines = f.readlines()
    num_jobs = len(lines) - 1  # minus header
    header = lines[0].strip()

    # Build the SLURM script content
    script_content = f"""#!/bin/bash
#SBATCH -J {job_name}
#SBATCH -o {output_prefix}.%A_%a.out
#SBATCH -e {output_prefix}.%A_%a.err
#SBATCH -p {partition}
#SBATCH -N {nodes}
#SBATCH -n {tasks}
#SBATCH -t {time}
#SBATCH --array=2-{num_jobs + 1}

# Load your environment here
module load python3  # or OpenSees, OpenFOAM, etc.

cd $SCRATCH/my_project
cp ~/my_local_dir/{csv_file} .

# Run your application
{run_cmd}
"""

    # Write the SLURM script
    slurm_filename = f"{job_name}.slurm"
    with open(slurm_filename, "w") as f:
        f.write(script_content)

    # Write a README summary of the jobs
    readme_name = f"README_jobs_{job_name}.txt"
    with open(readme_name, "w") as f:
        f.write(f"Parameter sweep generated for {job_name}\n\n")
        f.write(f"Array tasks: 2 to {num_jobs+1}\n")
        f.write(f"CSV header: {header}\n\n")
        f.write("Job details:\n")
        for idx, line in enumerate(lines[1:], start=2):
            f.write(f"  Task {idx}: {line.strip()}\n")

    print(f"Generated SLURM script '{slurm_filename}' and job summary '{readme_name}' for {num_jobs} jobs.")

Examples of using it for different software#

  • Python (OpenSeesPy or any param study)

    create_slurm_array_script("params.csv",
                          run_cmd="python run_analysis.py ${SLURM_ARRAY_TASK_ID}")

  • OpenSees-Tcl with indexed input files

    create_slurm_array_script("params.csv",
                          run_cmd="OpenSees model_${SLURM_ARRAY_TASK_ID}.tcl")

  • OpenFOAM multi-case driver

    create_slurm_array_script("params.csv",
                          run_cmd="bash run_case_${SLURM_ARRAY_TASK_ID}.sh")

  • ADCIRC or large MPI

    create_slurm_array_script("params.csv",
                          nodes=4, tasks=128,
                          run_cmd="ibrun adcirc")

    This makes your SLURM automation truly general:

    • Any solver or script,

    • Automatically sized to your parameter file,

    • With output logs cleanly organized by array index.

What your README_jobs.txt will look like#

Parameter sweep generated for ParamSweep

Array tasks: 2 to 6
CSV header: load,height,damping

Job details:
  Task 2: 10,5,0.02
  Task 3: 20,10,0.03
  Task 4: 30,15,0.04
  Task 5: 40,20,0.05
  Task 6: 50,25,0.06

Now every time you generate a job array, you get:

  • A robust .slurm script ready for sbatch

  • A small text audit trail of what parameters are tied to each array index

Contents