Run Any OpenSees using DS Agnostic App

Run Any OpenSees using DS Agnostic App#

Running Any OpenSees Workflow with the DesignSafe Agnostic App

Silvia Mazzoni, DesignSafe, 2026

This notebook demonstrates how the DesignSafe Agnostic App can be used to run any OpenSees workflow on HPC resources — regardless of interpreter, execution model, or parallelization strategy.

Rather than focusing on the app configuration itself (covered previously), this notebook focuses on practical execution examples. The goal is to show that the agnostic app is flexible enough to support a wide range of OpenSees use cases — from simple single-thread runs to MPI and launcher-based parallel workflows.

The examples shown here are intentionally simple. They illustrate core execution patterns and only a subset of the features available in the agnostic app. Once you understand these patterns, you can adapt them to more complex analyses, parametric studies, or large-scale production workflows.

Workflows Demonstrated#

Tcl Interpreter#

Regular OpenSees (single-thread)
OpenSeesMP (MPI)
OpenSeesSP
OpenSees with PyLauncher

OpenSeesPy#

OpenSeesPy (single-thread)
OpenSeesPy with concurrent.futures (useMPI = false)
OpenSeesPy with mpi4py
OpenSeesPy with PyLauncher

By the end of this notebook, you will see that the same agnostic app can execute all of these configurations — without changing applications — simply by adjusting how the job is launched.

The key takeaway: the agnostic app does not constrain how you run OpenSees — it enables it.

Unique to OpenSeesPy on TACC#

The Tapis app uses the Stampede3-compiled OpenSeesPy build that matches the app’s Python environment. Using a pip-installed OpenSeesPy can lead to missing shared-library dependencies (e.g., unresolved .so files) at runtime.

To avoid this, include the following block at the top of your Python script. The conditional logic makes it portable: on Stampede3 it finds the compiled opensees.so, and elsewhere it falls back to the standard import.

# Import the local version of OpenSees, if it exists
if os.path.exists('opensees.so'):
    import opensees as ops;
else:
    import openseespy.opensees as ops

You can access the app via the web portal at

https://designsafe-ci.org/workspace/designsafe-agnostic-app

Even though submitting the job via the portal is not efficient when you have to submit it more than once, looking at the app input via the portal is very helpful in gaining insight into the app itself since the inputs are presented with options and documentation.

Redirecting OUTPUT#

At the top of this notebook, we enable two job options that significantly improve end-to-end turnaround time: ZIP_OUTPUT_SWITCH and PATH_MOVE_OUTPUT. Together, these settings bundle outputs into a compact archive and move results to a specified location in $WORK on Stampede3, which can reduce the “archive” phase of a job dramatically (especially when many small files are produced). In this demo, outputs are moved to "$WORK/tmp_TestTapisApps", so be sure to check that directory for your results rather than relying only on the default archive location.

# do this for ALL jobs to minimize archive-job time
tapisInputAll['ZIP_OUTPUT_SWITCH'] = 'True'
tapisInputAll['PATH_MOVE_OUTPUT'] = '$WORK/tmp_TestTapisApps'

Initialize#

Initialize Python#

# import shutil
import ipywidgets as widgets
# from IPython.display import display, clear_output, Markdown
# import textwrap, time
# import stat
from pathlib import Path
# import json

Load Specialized Utilities Library#

I have developed a collection of python defs that are intended to simplify coding.

import os,sys
PathOpsUtils = os.path.expanduser('~/CommunityData/Training/Computational-Workflows-on-DesignSafe/OpsUtils')
if not PathOpsUtils in sys.path: sys.path.append(PathOpsUtils)
from OpsUtils import OpsUtils

# We will use this utility often to view the utility functions:
OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'show_text_file_in_accordion.py')

Examples-Files Location#

# We need to break the path up for Tapis because it uses different root paths
OpsScriptsPath_Base = 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic'
OpsScriptsPath_StorageSystem = 'CommunityData'

to run scripts locally on jupyter hub:#

# Define file paths in local/jupyter terms so we can view the input files here and we can check that the directory exists!
OpsScriptsPath_Local = os.path.join(OpsScriptsPath_StorageSystem, OpsScriptsPath_Base)
OpsScriptsPath_Local = '~/' + OpsScriptsPath_Local
OpsScriptsPath_Local = os.path.expanduser(OpsScriptsPath_Local)
print('OpsScriptsPath_Local:',OpsScriptsPath_Local)
Path_OpsScriptsPath_Local = Path(OpsScriptsPath_Local)

if Path_OpsScriptsPath_Local.is_dir():
    print('\nDirectory Exists! \n Content:')
    display(os.listdir(Path_OpsScriptsPath_Local))
else:
    print('\nERROR!!! Directory DOES NOT Exist!')

OpsScriptsPath_Local: /home/jupyter/CommunityData/Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic

Directory Exists! 
 Content:

['Ex1a_verymany.Canti2D.Push.mp.tcl',
 'Ex1a.Canti2D.Push.argv.tacc.runsList.txt',
 'Ex1a.Canti2D.Push.argv.tcl.callPylauncher.py',
 'Ex1a.Canti2D.Push.mpi4py.tacc.py',
 'Ex1.Canti2D.Push.mpi.mod.tacc.py',
 'Ex1a.Canti2D.Push.tacc.py',
 'Ex1a.Canti2D.Push.mp.tcl',
 'simpleSP.tcl',
 'Ex1a.Canti2D.Push.mpi.tacc.py',
 'Ex1a.Canti2D.Push.argv.tacc.py.callPylauncher.py',
 'Ex1a.Canti2D.Push.py',
 'Ex1a.Canti2D.Push.argv.tacc.callPylauncher.py',
 'Ex1a.Canti2D.Push.mpi4py.py',
 'Ex1a_many.Canti2D.Push.mp.tcl',
 'Ex1a.Canti2D.Push.futures.tacc.py',
 'Ex1a.Canti2D.Push.tcl',
 'Ex1a.Canti2D.Push.argv.tcl.runsList.txt',
 'Ex1a.Canti2D.Push.argv.tacc.py.runsList.txt',
 'Ex1a.Canti2D.Push.argv.tcl',
 'Ex1a.Canti2D.Push.futures.py',
 'Ex1a.Canti2D.Push.mpi.py',
 'Ex1a.Canti2D.Push.argv.tacc.py']

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, 'Ex1a.Canti2D.Push.tcl')

Connect to Tapis#

force_connect = False
# force_connect = True; # REMOVE do this only if you want to restart the clock on the token.
t=OpsUtils.connect_tapis(force_connect=force_connect)

 -- Checking Tapis token --
 Token loaded from file. Token is still valid!
 Token expires at: 2026-02-14T21:44:08+00:00
 Token expires in: 3:30:11.157951
-- AUTHENTICATED VIA SAVED TOKEN --

Initialize ALL-Job Input#

tapisInputAll = {}

SLURM-Specific Input#

tapisInputAll["maxMinutes"] = 6

tapisInputAll["execSystemId"] = "stampede3"; # really the only option right now.
tapisInputAll["execSystemLogicalQueue"] = "skx-dev"
tapisInputAll["nodeCount"] = 1
tapisInputAll["coresPerNode"] = 48
tapisInputAll["allocation"] = "DS-HPC1"

tapisInputAll['archive_system']='MyData' # Options: MyData or Work

Common Input for All Jobs#

# The folder path needs to be converted into tapis-format, we have a utility for that, which is why we have to separate the system and the folder.
tapisInputAll['storage_system'] = 'CommunityData'
tapisInputAll['input_folder'] = 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic'

App-Specific Input#

app_id = 'designsafe-agnostic-app'

Define App-Info Input#

tapisInputAll["appId"] = app_id
tapisInputAll["appVersion"] = "latest" # always use latest in this Notebook Template

Check if App Exists#

If it exists you will see a version number.

current_app_version = OpsUtils.get_latest_app_version(t,app_id)
print('current_app_version',current_app_version)

current_app_version 1.3.11

Access App Schema on Tapis#

OpsUtils.show_text_file_in_accordion(PathOpsUtils, ['getAppLatestVersion.py','display_tapis_app_schema.py'])

appMetaData = t.apps.getAppLatestVersion(appId=app_id)

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List the new app')
display(here_accordion)

with here_out:
    OpsUtils.display_tapis_app_schema(appMetaData)
thisAppVersion = appMetaData.version
isPublic = appMetaData.isPublic
here_accordion.set_title(0, f'app schema: {app_id}  -- version = {thisAppVersion}    --  isPublic = {isPublic}')

Agnostic-App Input#

# do this for ALL jobs to minimize archive-job time
tapisInputAll['ZIP_OUTPUT_SWITCH'] = 'True'
tapisInputAll['PATH_MOVE_OUTPUT'] = '$WORK/tmp_TestTapisApps'

Display All Output thus far#

display(tapisInputAll)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps'}

Job-Execution Controls#

askConfirmJob = False
askConfirmMonitorRT = False

1. OpenSees – TCL Interpreter#

OpenSees_Script = 'Ex1a.Canti2D.Push.tcl'
OpenSeesMP_Script = 'Ex1a.Canti2D.Push.mp.tcl'
OpenSeesSP_Script = 'simpleSP.tcl'

# PyLauncher Files:
PyLauncher_Input_Script = 'Ex1a.Canti2D.Push.argv.tcl.callPylauncher.py'
PyLauncher_FileList_File = 'Ex1a.Canti2D.Push.argv.tcl.runsList.txt'
OpenSees_Script_wArgv = 'Ex1a.Canti2D.Push.argv.tcl'

1A. OpenSees: Single Thread#

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, OpenSees_Script)

Test File Locally#

# test here first
os.system(f'OpenSees {OpsScriptsPath_Local}/{OpenSees_Script}')

         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.7.1 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Analysis-0 execution done
Analysis-1 execution done
Analysis-2 execution done
Analysis-3 execution done
Analysis-4 execution done
Analysis-5 execution done
Analysis-6 execution done
Analysis-7 execution done
ALL DONE!!!

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSees'
tapisInput['Main Script'] = OpenSees_Script

tapisInput['UseMPI'] = 'False'
tapisInput['MODULE_LOADS_LIST'] = 'opensees,hdf5/1.14.4'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSees',
 'Main Script': 'Ex1a.Canti2D.Push.tcl',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'opensees,hdf5/1.14.4',
 'name': 'designsafe-agnostic-app_OpenSees_Ex1a.Canti2D.Push.tcl'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

1B. OpenSeesMP via Agnostic App#

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, OpenSeesMP_Script)

# test here first
os.system(f'mpiexec -np 3 OpenSeesMP {OpsScriptsPath_Local}/{OpenSeesMP_Script}')

OpenSeesMP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory
OpenSeesMP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
OpenSeesMP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSeesMP'
tapisInput['Main Script'] = OpenSeesMP_Script

tapisInput['UseMPI'] = 'True'
tapisInput['MODULE_LOADS_LIST'] = 'opensees,hdf5/1.14.4'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSeesMP',
 'Main Script': 'Ex1a.Canti2D.Push.mp.tcl',
 'UseMPI': 'True',
 'MODULE_LOADS_LIST': 'opensees,hdf5/1.14.4',
 'name': 'designsafe-agnostic-app_OpenSeesMP_Ex1a.Canti2D.Push.mp.tcl'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

1C. OpenSeesSP via Agnostic App#

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, OpenSeesSP_Script)

# test here first
os.system(f'mpiexec -np 3 OpenSeesSP {OpsScriptsPath_Local}/{OpenSeesMP_Script}')

OpenSeesSP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory
OpenSeesSP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
OpenSeesSP: error while loading shared libraries: libscalapack-openmpi.so.2.1: cannot open shared object file: No such file or directory

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSeesSP'
tapisInput['Main Script'] = OpenSeesSP_Script

tapisInput['UseMPI'] = 'True'
tapisInput['MODULE_LOADS_LIST'] = 'opensees,hdf5/1.14.4'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSeesSP',
 'Main Script': 'simpleSP.tcl',
 'UseMPI': 'True',
 'MODULE_LOADS_LIST': 'opensees,hdf5/1.14.4',
 'name': 'designsafe-agnostic-app_OpenSeesSP_simpleSP.tcl'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

1D. PyLauncher With OpenSees-Tcl via Agnostic App#

PyLauncher Input#

Yes Pylauncher is in Python, but it executes shell commands, which may be OpenSees commands.

You need 3 files:

Your OpenSees-Analysis script needs to accept and process command-line arguments – these are your analysis variables. Example (Ex1a.Canti2D.Push.argv.tcl):

set NodalMass 5.18
set dataDir DataTCL;                # set up name of data directory
set LcolList "100 120 200 240 300 360 400 480"

if {[llength $argv]>0} {
    puts "Command-Line Arguments (argv): $argv"
    # OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 5.18 --LcolList "100 120 200 240 300 360 400 480" --outDir outData33
    # OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 5.18 --LCol 100 --outDir outData133
    foreach {label value} $argv {
        if {$label == "--NodalMass"} {set NodalMass $value}
        if {$label == "--LCol"} {set LcolList "$value"}
        if {$label == "--LcolList"} {set LcolList $value}
        if {$label == "--outDir"} {set outDir $value}
    }
}

your Main Script calls the PyLauncher:

Example:

import pylauncher
pylauncher.ClassicLauncher("Ex1a.Canti2D.Push.argv.tcl.runsList.txt",debug="host+job")

The Task File containing the commands. Example (Ex1a.Canti2D.Push.argv.tcl.runsList.txt):

OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 10.99 --outDir outCase35
OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 11.19 --outDir outCase36
OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 11.39 --outDir outCase37
OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 11.59 --outDir outCase38
OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 11.79 --outDir outCase39
OpenSees Ex1a.Canti2D.Push.argv.tcl --NodalMass 11.99 --outDir outCase40
OpenSees Ex1a.Canti2D.Push.argv.tcl --LcolList 100,200,300 --outDir outCase41
OpenSees Ex1a.Canti2D.Push.argv.tcl --LcolList 105,205,305 --outDir outCase42
OpenSees Ex1a.Canti2D.Push.argv.tcl --LcolList 110,210,310 --outDir outCase43
OpenSees Ex1a.Canti2D.Push.argv.tcl --LcolList 115,215,315 --outDir outCase44
OpenSees Ex1a.Canti2D.Push.argv.tcl --LcolList 120,220,320 --outDir outCase45
....

# PyLauncher Files:
OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [PyLauncher_Input_Script,PyLauncher_FileList_File,OpenSees_Script_wArgv])

# test here first
os.system(f'OpenSees {OpsScriptsPath_Local}/Ex1a.Canti2D.Push.argv.tcl --NodalMass 10.99 --outDir outCase35')

         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.7.1 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Command-Line Arguments (argv): --NodalMass 10.99 --outDir outCase35
NodalMass 10.99
LcolList 100 120 200 240 300 360 400 480
Saving results to: outCase35
Analysis-0 execution done
Analysis-1 execution done
Analysis-2 execution done
Analysis-3 execution done
Analysis-4 execution done
Analysis-5 execution done
Analysis-6 execution done
Analysis-7 execution done
ALL DONE!!!

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'python'
tapisInput['Main Script'] = PyLauncher_Input_Script

tapisInput['UseMPI'] = 'False'

tapisInput['MODULE_LOADS_LIST'] = 'opensees,hdf5/1.14.4,python,pyLauncher' # notice the additional modules

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'python',
 'Main Script': 'Ex1a.Canti2D.Push.argv.tcl.callPylauncher.py',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'opensees,hdf5/1.14.4,python,pyLauncher',
 'name': 'designsafe-agnostic-app_python_Ex1a.Canti2D.Push.argv.tcl.callPylauncher.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

2. OpenSeesPy Interpreter#

OpenSeesPy_script_PY = 'Ex1a.Canti2D.Push.py'
OpenSeesPy_script_PY_TACC = 'Ex1a.Canti2D.Push.tacc.py'
OpenSeesPy_script_PY_TACC_PyLauncher = 'Ex1a.Canti2D.Push.argv.tacc.pylauncher.py'
OpenSeesPy_script_PY_futures = 'Ex1a.Canti2D.Push.futures.py'
OpenSeesPy_script_PY_futures_TACC = 'Ex1a.Canti2D.Push.futures.tacc.py'
OpenSeesPy_script_PY_mpi = 'Ex1a.Canti2D.Push.mpi.py'
OpenSeesPy_script_PY_mpi_TACC = 'Ex1a.Canti2D.Push.mpi.tacc.py'
OpenSeesPy_script_PY_mpi4py = 'Ex1a.Canti2D.Push.mpi4py.py'
OpenSeesPy_script_PY_mpi4py_TACC = 'Ex1a.Canti2D.Push.mpi4py.tacc.py'



# PyLauncher Files:
PyLauncher_Input_Script = 'Ex1a.Canti2D.Push.argv.tacc.py.callPylauncher.py'
PyLauncher_FileList_File = 'Ex1a.Canti2D.Push.argv.tacc.py.runsList.txt'
OpenSeesPy_Script_wArgv = 'Ex1a.Canti2D.Push.argv.tacc.py'

2A. OpenSeesPy: Single Thread#

The scripts used in this notebook, with the .tacc.py in the filename have a logical process to check whether the TACC-compiles OpenSeesPy is available – recommended for TACC.

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [OpenSeesPy_script_PY_TACC])

Test Locally#

# test here first --
os.system(f'python {OpsScriptsPath_Local}/{OpenSeesPy_script_PY_TACC}')

using OpenSeesPy wheel
['/home/jupyter/CommunityData/Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic/Ex1a.Canti2D.Push.tacc.py']
Analysis-0 execution done
Analysis-1 execution done
Analysis-2 execution done
Analysis-3 execution done
Analysis-4 execution done
Analysis-5 execution done
Analysis-6 execution done
Analysis-7 execution done
ALL DONE!!!

Process 0 Terminating

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSees'
tapisInput['Main Script'] = OpenSeesPy_script_PY_TACC

tapisInput['GET_TACC_OPENSEESPY'] = 'True'; # You can set this to False to test whether the pip-installed version of OpenSeesPy works on Tacc.

tapisInput['UseMPI'] = 'False'
tapisInput['MODULE_LOADS_LIST'] = 'python'

tapisInput['PIP_INSTALLS_LIST'] = 'numpy,matplotlib'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSees',
 'Main Script': 'Ex1a.Canti2D.Push.tacc.py',
 'GET_TACC_OPENSEESPY': 'True',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'python',
 'PIP_INSTALLS_LIST': 'numpy,matplotlib',
 'name': 'designsafe-agnostic-app_OpenSees_Ex1a.Canti2D.Push.tacc.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

2B. OpenSeesPy: Multi-Process using concurrent.futures#

keep UseMPI False

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [OpenSeesPy_script_PY_futures_TACC])

Test Locally#

# test here first --
os.system(f'python {OpsScriptsPath_Local}/{OpenSeesPy_script_PY_futures_TACC}')

['/home/jupyter/CommunityData/Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic/Ex1a.Canti2D.Push.futures.tacc.py']
Analysis-2 execution done with ok=0
Analysis-1 execution done with ok=0
Analysis-4 execution done with ok=0
Analysis-7 execution done with ok=0
Analysis-3 execution done with ok=0
Analysis-5 execution done with ok=0
Analysis-6 execution done with ok=0
Analysis-8 execution done with ok=0
[case 4], status=0
[case 6], status=0
[case 2], status=0
[case 1], status=0
[case 3], status=0
[case 5], status=0
[case 7], status=0
[case 8], status=0

All results:
{'case_id': 1, 'status': 0}
{'case_id': 2, 'status': 0}
{'case_id': 3, 'status': 0}
{'case_id': 4, 'status': 0}
{'case_id': 5, 'status': 0}
{'case_id': 6, 'status': 0}
{'case_id': 7, 'status': 0}
{'case_id': 8, 'status': 0}
ALL DONE!!!

Process 0 Terminating

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSees'
tapisInput['Main Script'] = OpenSeesPy_script_PY_futures_TACC

tapisInput['GET_TACC_OPENSEESPY'] = 'True'; # You can set this to False to test whether the pip-installed version of OpenSeesPy works on Tacc.

tapisInput['UseMPI'] = 'False'
tapisInput['MODULE_LOADS_LIST'] = 'python'

tapisInput['PIP_INSTALLS_LIST'] = 'numpy,futures'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSees',
 'Main Script': 'Ex1a.Canti2D.Push.futures.tacc.py',
 'GET_TACC_OPENSEESPY': 'True',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'python',
 'PIP_INSTALLS_LIST': 'numpy,futures',
 'name': 'designsafe-agnostic-app_OpenSees_Ex1a.Canti2D.Push.futures.tacc.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

2C. OpenSeesPy: Multi-Process using mpi4py#

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [OpenSeesPy_script_PY_TACC])

Test Locally#

# test here first --
os.system(f'python {OpsScriptsPath_Local}/{OpenSeesPy_script_PY_TACC}')

using OpenSeesPy wheel
['/home/jupyter/CommunityData/Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic/Ex1a.Canti2D.Push.tacc.py']
Analysis-0 execution done
Analysis-1 execution done
Analysis-2 execution done
Analysis-3 execution done
Analysis-4 execution done
Analysis-5 execution done
Analysis-6 execution done
Analysis-7 execution done
ALL DONE!!!

Process 0 Terminating

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSees'
tapisInput['Main Script'] = OpenSeesPy_script_PY_TACC

tapisInput['GET_TACC_OPENSEESPY'] = 'True'; # You can set this to False to test whether the pip-installed version of OpenSeesPy works on Tacc.

tapisInput['UseMPI'] = 'False'
tapisInput['MODULE_LOADS_LIST'] = 'python'

tapisInput['PIP_INSTALLS_LIST'] = 'numpy,matplotlib'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSees',
 'Main Script': 'Ex1a.Canti2D.Push.tacc.py',
 'GET_TACC_OPENSEESPY': 'True',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'python',
 'PIP_INSTALLS_LIST': 'numpy,matplotlib',
 'name': 'designsafe-agnostic-app_OpenSees_Ex1a.Canti2D.Push.tacc.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

2D. OpenSeesPy: Multi-Process using mpi4py#

The scripts used in this notebook, with the .tacc.py in the filename have a logical process to check whether the TACC-compiles OpenSeesPy is available – recommended for TACC.

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [OpenSeesPy_script_PY_mpi4py_TACC])

Test Locally#

# test here first --
os.system(f'mpiexec -np 3 python  {OpsScriptsPath_Local}/{OpenSeesPy_script_PY_mpi4py_TACC}')

mpi4py -- python pid 1 of 3 started
mpi4py -- python pid 0 of 3 started
mpi4py -- python pid 2 of 3 started
pid 1 of np=3 Analysis-1 execution done
pid 0 of np=3 Analysis-0 execution done
pid 2 of np=3 Analysis-2 execution done
pid 1 of np=3 Analysis-4 execution done
pid 0 of np=3 Analysis-3 execution done
pid 2 of np=3 Analysis-5 execution done
pid 2 of np=3 ALL DONE!!!
pid 0 of np=3 Analysis-6 execution done
pid 0 of np=3 ALL DONE!!!
pid 1 of np=3 Analysis-7 execution done
pid 1 of np=3 ALL DONE!!!

Process 0 Terminating
Process 0 Terminating
Process 0 Terminating

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'OpenSees'
tapisInput['Main Script'] = OpenSeesPy_script_PY_mpi4py_TACC

tapisInput['GET_TACC_OPENSEESPY'] = 'True'; # You can set this to False to test whether the pip-installed version of OpenSeesPy works on Tacc.

tapisInput['UseMPI'] = 'True'
tapisInput['MODULE_LOADS_LIST'] = 'python'

tapisInput['PIP_INSTALLS_LIST'] = 'numpy,mpi4py'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'OpenSees',
 'Main Script': 'Ex1a.Canti2D.Push.mpi4py.tacc.py',
 'GET_TACC_OPENSEESPY': 'True',
 'UseMPI': 'True',
 'MODULE_LOADS_LIST': 'python',
 'PIP_INSTALLS_LIST': 'numpy,mpi4py',
 'name': 'designsafe-agnostic-app_OpenSees_Ex1a.Canti2D.Push.mpi4py.tacc.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

2E. OpenSeesPy: PyLauncher#

you can let pylauncher manage your analysis tasks – you can do this for both the TCL and Python versions of OpenSees.

OpsUtils.show_text_file_in_accordion(OpsScriptsPath_Local, [OpenSeesPy_Script_wArgv,PyLauncher_Input_Script,
PyLauncher_FileList_File])

Test Locally#

we can only test the analysis script

# test the script first
os.system(f'python {OpsScriptsPath_Local}/{OpenSeesPy_Script_wArgv} --NodalMass 11.59 --outDir outCase38') # adding the -h to show help

using OpenSeesPy wheel
['/home/jupyter/CommunityData/Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic/Ex1a.Canti2D.Push.argv.tacc.py', '--NodalMass', '11.59', '--outDir', 'outCase38']
we have input arguments
NodalMass 11.59
LColList [101, 121, 200, 240, 301, 360, 401, 481]
outDir outCase38
Lcol 101
2 0 101
<class 'int'>
Analysis-0 execution done
Lcol 121
2 0 121
<class 'int'>
Analysis-1 execution done
Lcol 200
2 0 200
<class 'int'>
Analysis-2 execution done
Lcol 240
2 0 240
<class 'int'>
Analysis-3 execution done
Lcol 301
2 0 301
<class 'int'>
Analysis-4 execution done
Lcol 360
2 0 360
<class 'int'>
Analysis-5 execution done
Lcol 401
2 0 401
<class 'int'>
Analysis-6 execution done
Lcol 481
2 0 481
<class 'int'>
Analysis-7 execution done
ALL DONE!!!

Process 0 Terminating

PyLauncher Input#

You need 3 files:

Your OpenSees-Analysis script needs to accept and process command-line arguments – these are your analysis variables. Example:

print(sys.argv)
if len(sys.argv)>1:
    print('we have input arguments')
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--NodalMass", type=float, default=5.18)
parser.add_argument("--LCol", type=float, default=None)  # NEW: single run value
parser.add_argument("--LColList", type=csv_floats, default=[101,121,200,240,301,360,401,481])
parser.add_argument("--outDir", type=str, default='outData_PY_tacc')
args = parser.parse_args()
NodalMass = args.NodalMass
LColList = args.LColList
outDir = args.outDir

if args.LCol is not None:
    LColList = [args.LCol]          # launcher mode: one case per process
else:
    LColList = args.LColList        # interactive mode: multiple cases

print('NodalMass',NodalMass)
print('LColList',LColList)
print('outDir',outDir)

your Main Script calls the PyLauncher:

Example:

import pylauncher
pylauncher.ClassicLauncher("Ex1a.Canti2D.Push.argv.tacc.runsList.txt",debug="host+job")

The Task File containing the commands. Example:

python Ex1a.Canti2D.Push.argv.tacc.py --NodalMass 11.59 --outDir outCase38
python Ex1a.Canti2D.Push.argv.tacc.py --NodalMass 11.79 --outDir outCase39
python Ex1a.Canti2D.Push.argv.tacc.py --NodalMass 11.99 --outDir outCase40
python Ex1a.Canti2D.Push.argv.tacc.py --LcolList 100,200,300 --outDir outCase41
python Ex1a.Canti2D.Push.argv.tacc.py --LcolList 105,205,305 --outDir outCase42
python Ex1a.Canti2D.Push.argv.tacc.py --LcolList 110,210,310 --outDir outCase43
....

app input for this run#

tapisInput = tapisInputAll.copy(); # grab the common input

tapisInput['Main Program'] = 'python' # we are calling pylauncher
tapisInput['Main Script'] = PyLauncher_Input_Script

tapisInput['GET_TACC_OPENSEESPY'] = 'True'; # You can set this to False to test whether the pip-installed version of OpenSeesPy works on Tacc.

tapisInput['UseMPI'] = 'False'
tapisInput['MODULE_LOADS_LIST'] = 'python,pylauncher'

tapisInput['PIP_INSTALLS_LIST'] = 'numpy,matplotlib'

tapisInput['name'] = tapisInput['appId'] + '_' + tapisInput['Main Program'] + '_' + tapisInput['Main Script']

display(tapisInput)

{'maxMinutes': 6,
 'execSystemId': 'stampede3',
 'execSystemLogicalQueue': 'skx-dev',
 'nodeCount': 1,
 'coresPerNode': 48,
 'allocation': 'DS-HPC1',
 'archive_system': 'MyData',
 'storage_system': 'CommunityData',
 'input_folder': 'Training/Computational-Workflows-on-DesignSafe/Examples/OpenSees_Basic',
 'appId': 'designsafe-agnostic-app',
 'appVersion': 'latest',
 'ZIP_OUTPUT_SWITCH': 'True',
 'PATH_MOVE_OUTPUT': '$WORK/tmp_TestTapisApps',
 'Main Program': 'python',
 'Main Script': 'Ex1a.Canti2D.Push.argv.tacc.py.callPylauncher.py',
 'GET_TACC_OPENSEESPY': 'True',
 'UseMPI': 'False',
 'MODULE_LOADS_LIST': 'python,pylauncher',
 'PIP_INSTALLS_LIST': 'numpy,matplotlib',
 'name': 'designsafe-agnostic-app_python_Ex1a.Canti2D.Push.argv.tacc.py.callPylauncher.py'}

Submit#

# -----------------------------------------------------
jobReturns = OpsUtils.run_tapis_job(t,tapisInput,get_job_metadata=True,get_job_history=True,get_job_filedata=True,askConfirmJob = askConfirmJob,askConfirmMonitorRT = askConfirmMonitorRT)
# -----------------------------------------------------

print('Done!!!')

Done!!!

Run Any OpenSees using DS Agnostic App

Contents

Run Any OpenSees using DS Agnostic App#

Workflows Demonstrated#

Tcl Interpreter#

OpenSeesPy#

Unique to OpenSeesPy on TACC#

Redirecting OUTPUT#

Initialize#

Initialize Python#

Load Specialized Utilities Library#

Examples-Files Location#

to run scripts locally on jupyter hub:#

Connect to Tapis#

Initialize ALL-Job Input#

SLURM-Specific Input#

Common Input for All Jobs#

App-Specific Input#

Define App-Info Input#

Check if App Exists#

Access App Schema on Tapis#

Agnostic-App Input#

Display All Output thus far#

Job-Execution Controls#

1. OpenSees – TCL Interpreter#

1A. OpenSees: Single Thread#

Test File Locally#

app input for this run#

Submit#

1B. OpenSeesMP via Agnostic App#

app input for this run#

Submit#

1C. OpenSeesSP via Agnostic App#

app input for this run#

Submit#

1D. PyLauncher With OpenSees-Tcl via Agnostic App#

PyLauncher Input#

app input for this run#

Submit#

2. OpenSeesPy Interpreter#

2A. OpenSeesPy: Single Thread#

Test Locally#

app input for this run#

Submit#

2B. OpenSeesPy: Multi-Process using concurrent.futures#

Test Locally#

app input for this run#

Submit#

2C. OpenSeesPy: Multi-Process using mpi4py#

Test Locally#

app input for this run#

Submit#

2D. OpenSeesPy: Multi-Process using mpi4py#

Test Locally#

app input for this run#

Submit#

2E. OpenSeesPy: PyLauncher#

Test Locally#

PyLauncher Input#

app input for this run#

Submit#