Build A Custom Tapis App

Build A Custom Tapis App#

Silvia Mazzoni, DesignSafe, 2026

Build a General “Agnostic” App and a Reduced-Scope OpenSeesPy App#

You can access these apps via the web portal at

https://designsafe-ci.org/workspace/designsafe-agnostic-app
https://designsafe-ci.org/workspace/designsafe-openseespy-s3 Even though submitting the job via the portal is not efficient when you have to submit it more than once, looking at the app input via the portal is very helpful in gaining insight into the app itself since the inputs are presented with options and documentation.

Purpose of this Notebook#

This notebook is a practical, end-to-end guide for designing, generating, packaging, and registering custom Tapis v3 ZIP-runtime apps on DesignSafe HPC systems (e.g., Stampede3).

It focuses on two closely related outcomes:

A general “agnostic” Tapis app: designsafe-agnostic-app
A flexible, reusable wrapper capable of running:
- OpenSees (serial)
- OpenSeesMP (MPI)
- OpenSeesPy
- Python-based workflows
- Other command-line solvers or scripts
A reduced-scope, opinionated OpenSeesPy app: designsafe-openseespy-s3
A simplified derivative designed for:
- portal-friendly usage
- fewer user-facing parameters
- constrained, safer execution patterns

The notebook parameterizes shared logic first, then shows how to specialize it—demonstrating how one robust app design can support many execution styles without duplicating code.

What This Notebook Produces#

Running this notebook generates a complete, versionable ZIP app bundle, including:

app.json
The Tapis App definition: inputs, parameters, resource requests, queue mapping, and runtime settings.
profile.json
Execution-system–specific environment configuration (e.g., module loads, defaults), allowing the same app logic to remain portable.
tapisjob_app.sh
The core wrapper script that runs on the allocated compute node(s).
This script is treated as a first-class software artifact, with:
- explicit setup / run / post-process phases
- structured logging
- MPI and non-MPI launch control
- controlled staging and cleanup
A packaged .zip archive
Ready to be registered as a new Tapis app version or used to update an existing one.

This notebook is designed to be re-run repeatedly as you refine the app, ensuring that artifacts remain consistent, reproducible, and traceable.

Side-By-Side Comparison: Agnostic App vs. OpenSeesPy-Only App#

Feature	designsafe-agnostic-app	designsafe-openseespy-s3
Scope	Run any executable: OpenSees, OpenSeesMP, OpenSeesPy, python3, custom tools	Run OpenSeesPy only
Main Program options	Multiple (OpenSees, OpenSeesMP, python3)	Hidden → always python3
UseMPI	Exposed to user	Exposed but optional
PIP installation	List + file	Hidden default list
Module loading	User controls via list/file	Hidden defaults (python, opensees, hdf5)
ZIP/Move output	Supported	Removed (not supported)
Complexity	Full-featured	Simplest possible
Intended for	Advanced users, HPC workflows, general apps	Web portal users running OpenSeesPy scripts

MPI vs. Non-MPI Apps: Interpreting isMpi#

Tapis apps include an isMpi flag that controls how Tapis launches the job, not what your wrapper is allowed to do.

Key clarifications:

An app with isMpi: false can still run MPI internally
Requesting multiple nodes does not automatically invoke MPI
The wrapper (tapisjob_app.sh) ultimately decides:
- whether MPI is used
- how many ranks
- which launcher (mpirun, srun, etc.)

This notebook demonstrates:

pure serial execution
explicit MPI launch inside the wrapper
hybrid workflows (serial preprocessing → MPI solve → serial postprocessing)

This design gives you maximum control while remaining compatible with Tapis orchestration.

Jupyter as an IDE#

The Jupyter Notebook turns app development into a repeatable, one-click pipeline instead of a fragile set of manual steps. It becomes the living, executable “source of truth” for both the app logic and its documentation.

The notebook parameterizes the common logic and then specializes it for each app, so you can see:

how to design a flexible, general app; and
how to derive a simplified, opinionated variant (here, OpenSeesPy-only) for portal-friendly use.

Creating a robust app requires many iterations.
When using a Jupyter Notebook you just hit the “Restart the Kernel and Run All Cells” button to run the entire app-building workflow, from creating the app to submitting a job and visualizing the output.

Execution-System Constraints#

When using OpenSees, these apps rely on its availability in the execution system. Stampede3 meets this requirement.

If you do not need OpenSees, or are using your own version of it, you may install the Agnostic app in any system.

What follows is a practical walkthrough for defining, packaging, and registering these two apps using the Tapis v3 API.

About this Notebook#

This notebook builds the designsafe-agnostic-app: a Tapis app wrapper that can run OpenSees / OpenSeesMP / OpenSeesSP, OpenSeesPy, and general Python tasks, as well as other command-line programs.

The goal is not just to produce one app, but to give you a template and guide for writing your own apps:

The notebook demonstrates how to work from a single source and generate multiple app variants (e.g., the full agnostic app vs. a more focused OpenSeesPy app).
The Tapis-app’s shell script is broken down into many features that are designed to be modular, optional, and controlled by environment variables or app inputs.
A core design goal is to minimize expensive file movement through Tapis. Moving large or numerous files via Tapis (into or out of the execution directory) can dominate runtime and congest shared channels. Many of the features in the app are explicitly designed to move work onto the shared filesystem (via ‘rsync’, ‘unzip’, and ‘mv’ on the cluster), so jobs run faster and the Tapis system scales better for everyone.

Design Philosophy: Modular, Optional, Single-Source#

The notebook and the ‘tapisjob_app.sh’ script are designed so that:

Each feature is self-contained (usually guarded by an environment variable: ‘GET_TACC_OPENSEESPY’, ‘PATH_COPY_IN_LIST’, ‘ZIP_OUTPUT_SWITCH’, etc.).
Features complement each other, some may be interdependent, but none is mandatory for the app to function.
The same core shell script supports:
- A fully-featured, “agnostic” app (OpenSees + Python + generic features).
- A more targeted OpenSeesPy-focused app, which only enables a subset of those features.

This notebook shows how to keep one single source for the logic and selectively turn features on/off when building different apps. In practice, apps are never “done” after a single iteration; this single-source model makes it realistic to maintain and evolve a family of related apps over time.

Why a Jupyter Notebook as the Single Source?#

A Jupyter Notebook is an ideal single-source, click-button environment for building and maintaining Tapis apps. Instead of juggling separate shell scripts, JSON files, and command-line calls, the entire app lifecycle lives in one place: the notebook.

With this approach:

All steps are captured in one workflow Writing the app files (‘tapisjob_app.sh’, ‘app.json’, optional helper scripts), registering/updating the app, and submitting a test job can all be driven from a single “Run All” action. Anything else would require you to perform those steps manually—editing files by hand, copying them to the system, running ‘tapis’ CLI commands in the right order, and hoping you didn’t miss a step.
Reproducible “click-to-rebuild” apps The notebook acts as an executable recipe: whenever you want to change the app (new feature, new version, new defaults), you edit cells, re-run them, and the notebook regenerates the app definition and script in a consistent way. This reduces “drift” between your code and your registration on Tapis.
Single source for logic and documentation Markdown cells describe the intent and usage; code cells implement it. The same notebook that writes ‘app.json’ also explains every input, parameter, and feature. When you change the app, you update the notebook in one place, instead of hunting through external docs.
Interactive inspection and debugging The notebook can show you each generated file with and without line numbers:
- Line numbers help you quickly locate errors reported by JSON validators or Tapis (e.g., “line 127” in ‘app.json’).
- The plain (no line numbers) view is perfect for copying file contents into other tools when necessary.
Flexible for multiple app variants Because the notebook contains all the branching logic, you can generate the full agnostic app and more specialized apps (e.g., OpenSeesPy-only) from the same code. Parameters, feature flags, and small configuration changes are handled programmatically, rather than by hand-editing multiple divergent copies.

In short, the Jupyter Notebook turns app development into a **repeatable, one-click pipeline** instead of a fragile set of manual steps. It becomes the living, executable “source of truth” for both the app logic and its documentation.

How to Use This Notebook as Your Template#

When you adapt this notebook for your own app:

Start from the generic features:
- Keep the logging, timers, directory management, and output movement.
- Decide which file-movement helpers (‘UNZIP_FILES_LIST’, ‘PATH_COPY_IN_LIST’, ‘ZIP_OUTPUT_SWITCH’, ‘PATH_MOVE_OUTPUT’) make sense for your workflow.
Add your domain-specific blocks:
- For OpenSees, keep or extend the existing module loads and OpenSeesPy handling.
- For other solvers, use these as patterns to load different modules or shared libraries.
Adapt the Python features as needed:
- Decide whether you want a strict requirements file, a simple list-of-packages switch, or both.
Document everything in ‘app.json’:
- For each input and parameter, provide a meaningful description.
- Use the notebook’s line-numbered view and validation helpers as you iterate.
Reuse and extend the logging and timers:
- They are extremely useful for profiling jobs and should be considered foundational infrastructure in every new app you build.

The result is a single-source, modular, and reusable pattern that you can carry forward into future apps—whether they’re for structural analysis, data post-processing, or entirely different scientific workflows.

Why Well-Documented Apps Matter#

Keeping rich documentation inside ‘app.json’ is important:

You only need to update one source as the app evolves.
Users see clear descriptions directly in the Tapis UI (or in CLI help), rather than searching for an external PDF or web page.
It helps future you (and collaborators) understand:
- What each input does,
- Which options are safe to change,
- How environment variables map to script behavior.

Because the notebook builds and validates the JSON, it becomes the natural place to update both logic and documentation together.

Anatomy of a Tapis App#

A Tapis v3 App is composed of a small set of files and configuration artifacts that work together to define:

how a job looks to the user
how it runs on the HPC system
how files are staged, executed, and archived
how the scientific code or workflow is launched

Although a Tapis app can be extremely flexible—supporting OpenSees, OpenSeesMP, OpenSeesPy, Python, or arbitrary executables—the underlying structure is always the same.

This section breaks down each component, explains what it does, where it lives, why it is needed, and how Tapis interacts with it during a job. (Click on each header to expand)

1. app.json — The App Definition (Required)
Defines the app’s identity, inputs, parameters, and execution system

app.json is the **formal definition** of the application. This is the file that is *registered* into Tapis, and therefore must follow the Tapis App schema.

It defines:

App identity

App ID
Version
Description
Category
Ownership / permissions

Execution system

Stampede3 (or another execution system)
Queue, node count, cores, memory
Scheduler profile
Archiving rules

Runtime configuration

runtime: “ZIP” tells Tapis to fetch + unpack a ZIP file at job start
Path to the ZIP package
Whether MPI is enabled at the Tapis level (isMpi: false for our apps)

App parameters (visible to the user in the portal)

Examples:

Main Program (OpenSees, python3, etc.)
Main Script name
UseMPI toggle
Optional command-line arguments

These appear in the portal UI or Tapis CLI and are forwarded directly into the wrapper script.

Environment variables

These control application-specific behavior such as:

Which modules to load
Which pip packages to install
Whether to copy TACC-compiled OpenSeesPy
Optional ZIP/unzip behavior

They are automatically exported into the job’s environment.

Input/Output behavior

Required “Input Directory”
Archive inclusion/exclusion patterns

In short: app.json defines what the app is and how users interact with it.

2. Scheduler Profile — System-level environment initialization
Initializes the compute-node environment before the wrapper script runs

A scheduler profile defines how the compute node environment is set up **before** your wrapper script runs. It dictates availability of the module command, default environment variables, and job-launch behavior.

Your apps use:

--tapis-profile tacc-no-modules

Because it ensures:

A clean environment
No automatically loaded modules
Full control inside tapisjob_app.sh

Scheduler profile = system setup. envVariables = app configuration.

3. tapisjob_app.sh — Wrapper script (the executable logic) (Required)
Performs all runtime logic; loads modules, installs pip, launches user script

This is the **heart of the app at runtime**.

Tapis does not execute your OpenSees or Python files directly. Instead, it runs this script, which performs all operational steps:

Logs & timers#

Creates:

SLURM-job-summary.log
SLURM-full-environment.log and timestamps total runtime + main program runtime.

Validates arguments#

Ensures the app received:

Main Program
Main Script
UseMPI
Additional CLI arguments

Normalizes the environment#

Ensures python3 is used instead of python
Verifies input directory exists
Shows Job UUID and system paths

Loads modules#

Using:

MODULE_LOADS_LIST
or MODULE_LOADS_FILE

This is necessary because the scheduler profile loads nothing.

Installs pip packages#

Supports:

PIP_INSTALLS_LIST
PIP_INSTALLS_FILE

Executed directly on the compute node.

Optional: Copies TACC-compiled OpenSeesPy#

If GET_TACC_OPENSEESPY=True, it copies:

OpenSeesPy.so → ./opensees.so

This enables users to import opensees reliably.

Chooses launcher#

Decides whether to prepend:

ibrun

for MPI jobs.

Runs the user script#

Executes:

[ibrun] <BINARYNAME> <INPUTSCRIPT> [args]

This is the actual scientific computation.

Post-processing#

Removes temporary files
Produces timing logs
Returns to parent directory

This script is what makes the app function.

4. App ZIP Package — Runtime bundle delivered to the compute node (Required)
Bundles the wrapper script and optional documentation into a portable runtime image

Tapis apps using "runtime": "ZIP" require a single ZIP file containing:

tapisjob_app.sh
ReadMe.md (optional)
profile.json (optional)
Any helper files

This ZIP is stored in a Tapis-accessible storage location:

tapis://designsafe.storage.default/silvia/apps/<appname>/<version>/<zipfile>

When a job runs, Tapis:

Copies the ZIP into the job directory
Unpacks it
Executes tapisjob_app.sh

This ZIP therefore functions like a lightweight container.

5. Input Directory — User-provided model files (User-Provided)
User-provided scripts and data used in the computation

Each app declares one required:

Input Directory

This directory must contain:

The user’s main script (model.tcl, runOSPy.py, …)
Any supporting input files
Any ZIPs to be expanded
Any requirements files (for Python or modules)

At runtime, Tapis stages this directory into:

$JOB_WORKING_DIR/inputDirectory/

Your wrapper script then cds into it before running calculations.

6. README.md — Human-Facing Documentation (Optional but Highly Recommended)
Documentation for human users (optional but recommended)

Although not required by Tapis, providing a ReadMe.md improves:

Portal usability
Training workflows
Reproducibility
Collaboration

It typically includes:

App description
Usage instructions
How to import OpenSeesPy
MPI guidance
Example jobs
Known limitations

This file is packaged inside the app ZIP but is not executed.

Together, these pieces create a robust, portable, and reproducible HPC application.

How Tapis Apps Run#

When a job is launched, Tapis performs a simple but powerful sequence:

1. User submits job

* inputs parameters + input directory
* via WebPortal, Jupyter Notebook, or CLI

2. Tapis validates the job request

Tapis validates the job request against app.json (arguments, file inputs, environment variables)

3. Tapis stages the inputs

Create the job working directory
Copy the user’s Input Directory
Copy the app’s ZIP bundle

4. Unpack the ZIP Runtime on the Compute Node

Extract tapisjob_app.sh
Make the wrapper script executable

5. Submit SLURM Job

Submit to SLURM: Tapis sends the job request to Stampede3’s SLURM scheduler.
Using the queue, time limit, and resources defined in app.json

6. SLURM Job starts

SLURM allocates one or more compute nodes.

On the first compute node, SLURM runs:

./tapisjob.sh <args>

The compute node has a clean environment — it’s not the login-node environment. That is why you must load modules inside the script, because no modules are pre-loaded.

7. Wrapper Script Runs

tapisjob.sh executes the wrapper script tapisjob_app.sh. It Calls:

./tapisjob_app.sh <MainProgram> <MainScript> <UseMPI> [args...]

The wrapper script ‘tapisjob_app.sh’ is executed inside the compute-node environment, not on the login node.
This script is unique to the app. The Agnostic-App Script does the following:
- Loads modules
- Installs pip packages
- Copies TACC OpenSeesPy if requested
- Chooses MPI or serial launcher
- Calls the main binary file.
- Logs everything

8. Output Archiving

Tapis copies job output into:

$WORK/tapis-jobs-archive/<date>/<jobname>-<UUID>/

Excludes ZIP files if specified
Preserves logs for debugging

9. Job complete

* logs & results are now available in the portal

Summary Tapis lifecycle: stage → unpack → run → archive.

Tapis as a Platform: Task-Specific and General Apps#

By combining:

Domain-specific blocks (like OpenSees/OpenSeesPy),
Python environment management,
And generic HPC/Tapis helpers,

this notebook demonstrates the power and versatility of Tapis as a platform:

You can build highly task-specific apps (e.g., a particular OpenSeesPy workflow for a research project).
You can also build general-purpose apps (e.g., a generic Python post-processing wrapper) that others in the TACC/DesignSafe community can reuse.

Because the logic, documentation, and configuration live together in a single notebook-driven source, it’s easier to:

Share apps,
Iterate on them,
And improve our collective institutional knowledge about running complex workflows on shared HPC infrastructure.

Tapis Documentation Resources#

https://tapis.readthedocs.io/en/latest/technical/apps.html

https://tapis-project.github.io/live-docs/?service=Apps

https://github.com/tapis-project/tapipy/blob/main/tapipy/resources/openapi_v3-apps.yml

Agnostic-App Features#

The Agnostic app is designed to be reusable, adaptable, and extensible. This is achieved by including packaged features that can be included or excluded in future apps.

The app-specific execution logic is defined in tapisjob_app.sh
The user-facing inputs and defaults are defined in app.json

Each feature in the wrapper is modular, optional, and controlled by environment variables or app inputs. Features are organized by why they exist (design intent), not just by what they do.

0. “Feature Families” (Design Intent)

Observability & Debugging

- make jobs explainable without reruns - make support/debugging possible from logs alone

Safe File Staging

- support copy-in and bundles without creating archive mess - avoid accidental deletion or path hazards

Environment Construction

- deterministic module loading - robust Python behavior (*python* vs *python3*) - reproducible pip installs

Execution Semantics

- explicit MPI vs non-MPI launch choice - binary-aware defaults (OpenSees vs Python)

Extensibility Hooks

- user-controlled pre/post steps without modifying the wrapper

Output Strategy

- optional zip repacking - optional in-system moves to WORK/SCRATCH for performance

1. Observability & Debugging

1.1 Summary Logging (*SUMMARY_SHORT*) The app writes a compact human-focused summary log (default: *SLURM-job-summary.log*) pinned to the original SLURM script root directory.

It records:

App id/version/description
System paths (HOME, WORK, SCRATCH)
User configuration:
- JobUUID, inputDirectory, INPUTSCRIPT, UseMPI, BINARYNAME, argument list
Feature flags and env variables:
- module/pip inputs, copy/unzip inputs, hooks, output controls
launcher selection (MPI vs direct)
runtime timers (run-only + total)

This is the first file to read when anything goes wrong.

1.2 Full Environment Logging (*FULL_ENV_LOG*) The wrapper also writes a verbose environment dump (*SLURM-full-environment.log*) containing *env | sort*.

Use cases:

module conflicts
path ordering surprises
MPI/runtime environment differences between jobs

2. Safe File Staging

2.1 Directory discipline: Script root vs Input Directory

The wrapper separates: - ***SCRIPT_ROOT_DIR***: where the SLURM job starts - ***inputDirectory***: where user files are staged (and where execution happens)

It cds into inputDirectory for execution, then returns to script root for packaging/moves. This prevents a common class of mistakes where “global” actions run in the wrong directory.

2.2 Copy-in staging (*PATH_COPY_IN_LIST*)

Optional staging of external paths (WORK/SCRATCH/HOME) into the working directory using *rsync -av*.

Why this exists:

keep the Input Directory small
reuse shared datasets
create a runtime layout without hard-coding absolute paths inside scripts

2.3 Copy-in cleanup (*DELETE_COPIED_IN_ON_EXIT*) — manifest-driven + safe

When enabled, the wrapper: - records what was copied in a manifest - uses a Bash *EXIT* trap to cleanup (success or failure) - deletes **only** the manifest-listed items - rejects unsafe paths (absolute paths, *..* traversal) - logs each deletion

This allows “temporary convenience inputs” without polluting the final archive.

2.4 ZIP expansion (*UNZIP_FILES_LIST*) Optional unzip of one or more ZIP bundles staged in the Input Directory.

- supports names with or without *.zip* - quiet unzip (*unzip -o -q*) - logs missing zips as warnings

Use this when you bundle many small files into a single upload artifact.

3. Environment Construction

3.1 Defensive *module* initialization

If *module* is not on PATH, the wrapper sources */etc/profile.d/modules.sh* (when present). This prevents jobs from failing on minimal profiles like *tacc-no-modules*.

3.2 Module loading: file + list (both supported)

Two mechanisms (can be used together):

MODULE_LOADS_FILE:
- supports purge, use , load , ?optional, and bare module names
- best for version-controlled, documented module stacks
MODULE_LOADS_LIST:
- comma-separated list
- best for quick one-offs

3.3 Python normalization (newer behavior) Even on HPC systems, *python* and *python3* can resolve to different interpreters.

To remove ambiguity, the wrapper:

normalizes any python-ish BINARYNAME to python3
injects a PATH shim so python executes python3
(optionally) shims pip → pip3
logs command -v python/python3 and versions

This prevents “works on my node” failures caused by hidden interpreter drift.

3.4 Python package installs: file + list

Two mechanisms (can be used together):

PIP_INSTALLS_FILE: pip3 install -r
PIP_INSTALLS_LIST: per-package pip3 install

Both are fail-fast: pip errors stop the job with clear logging.

4. OpenSees-specific features

4.1 Default OpenSees Tcl module loads

If *BINARYNAME* is *OpenSees*, *OpenSeesMP*, or *OpenSeesSP*, the wrapper loads: - *hdf5/1.14.4* - *opensees*

This is a “binary-aware default” so users don’t have to remember module boilerplate.

4.2 OpenSeesPy injection (*GET_TACC_OPENSEESPY*)

If enabled, the wrapper: - loads *python/3.12.11*, *hdf5/1.14.4*, *opensees* - copies *${TACC_OPENSEES_BIN}/OpenSeesPy.so* to *./opensees.so* - removes *./opensees.so* after the run

This is the recommended OpenSeesPy path on Stampede3 (more robust than PyPI wheels).

5. Extensibility hooks

5.1 Pre-job hook (*PRE_JOB_SCRIPT*) Runs *after* environment construction but *before* the main executable.

- relative paths resolved as './< script >' inside the Input Directory - executable runs directly; otherwise runs via *bash*

Default policy: warnings on failure, continue job (policy is intentionally permissive for experimentation).

5.2 Post-job hook (*POST_JOB_SCRIPT*) Runs after the main executable with the same resolution and execution rules.

Typical uses:

post-processing
summarizing results
moving/organizing additional artifacts
light cleanup

6. Execution semantics: launcher choice

6.1 Sequential vs MPI (*UseMPI*)

The wrapper selects: - direct run if *UseMPI* is false-like - *ibrun* if *UseMPI* is true-like

It logs the decision and the final command line.

This keeps MPI explicit and prevents “accidental MPI” jobs.

7. Output strategy

7.1 Optional repack to ZIP (*ZIP_OUTPUT_SWITCH*)

If enabled, the wrapper: - zips the entire Input Directory into *inputDirectory.zip* - deletes the original directory tree

This reduces file counts and makes transfers more efficient.

7.2 Optional in-system output move (*PATH_MOVE_OUTPUT*)

If set, the wrapper: - creates */_/* - moves the primary archive there - copies top-level job logs there too

This is a performance feature:

move to WORK for interactive inspection in JupyterHub
move to SCRATCH for chained HPC workflows

8. Practical guidance: mapping features to inputs

If you want a simple “mental model”:

Make the run work:
- set Main Program, Main Script, UseMPI
Make the environment correct:
- use MODULE_LOADS_FILE / MODULE_LOADS_LIST
- use PIP_INSTALLS_FILE / PIP_INSTALLS_LIST
- enable GET_TACC_OPENSEESPY when using OpenSeesPy
Make inputs available:
- use UNZIP_FILES_LIST for bundles
- use PATH_COPY_IN_LIST for external datasets
- enable DELETE_COPIED_IN_ON_EXIT if copy-ins are temporary
Make results usable:
- use ZIP_OUTPUT_SWITCH for large file trees
- use PATH_MOVE_OUTPUT to land outputs in WORK/SCRATCH quickly

A Note on Python Environments: Limitations and Practical Tradeoffs#

One important limitation of the agnostic app’s Python support is that it relies on an existing system-level Python environment, rather than creating or managing a fully user-defined virtual environment by default. While the app can be extended to support user-provided virtual environments, this is intentionally not part of the core workflow, because Python environments are something that should ideally be set up once, tested thoroughly, and reused, not rebuilt on every job.

Why Not Build a Virtual Environment Inside the App?#

Creating or activating a custom virtual environment at runtime adds complexity and cost:

Environment creation is slow and would happen for every job, which is wasteful.
Environment reproducibility becomes harder unless carefully pinned.
Python’s open-source ecosystem has many interdependencies, which means configuration issues can arise frequently, especially in HPC environments where multiple compiler and MPI stacks coexist.

For these reasons, the recommended practice is:

If you need a custom, long-lived Python environment, create it once in ‘$WORK’ or a similar persistent location and build a separate, dedicated Tapis app that activates that environment. This isolates environment management from job execution and avoids rebuilding environments every time.

Why Provide ‘PIP_INSTALLS_FILE’ and ‘PIP_INSTALLS_LIST’?#

Even though installing packages at runtime is not ideal performance-wise, we included these options because:

They allow lightweight customization without requiring a separate environment-management app.
They help maintain portability and reduce user burden—no need to manage a Python venv manually.
The small extra setup time is usually negligible compared to TACC job runtimes.
The convenience and reliability of installing a few packages at submission time outweigh the risks of depending on stale or incompatible environments.

But Beware: Python Is Flexible and Fragile#

Python’s strength—its huge open-source ecosystem—is also the source of occasional instability:

Packages may depend on different compiler toolchains, MPI bindings, or C libraries.
Minor version changes can break expected behaviors.
System environments may differ subtly between compute nodes and login nodes.

Because of this, our philosophy in the agnostic app is:

Keep the Python layer as simple, minimal, and controllable as possible. Add only the packages you need, and prefer stable, TACC-supported modules whenever available.

You can add a user-defined virtual environment to your workflow later, but it should be handled intentionally—preferably in a separate, specialized app designed just for environment creation and maintenance.

This approach keeps the agnostic app robust, portable, and predictable while still giving users enough flexibility to extend Python functionality when needed.

Tapis-App-Development Workflow#

This notebook automates the full lifecycle of creating, deploying, and testing two Tapis v3 Apps:

designsafe-agnostic-app (general-purpose)
designsafe-openseespy-s3 (OpenSeesPy-only)

Below is a detailed breakdown of each step in the workflow. (Click on each header to expand)

0. Connect to Tapis
You need to authenticate (TACC/DesignSafe Username and password) and get a token

Before anything else, you must authenticate with the Tapis v3 API so you can: - upload files, - register apps, - modify permissions, - and submit jobs.

In this step, we:

load API credentials (client ID/secret)
request an OAuth2 token
create a Python tapis client object
confirm access to the execution system (stampede3) and the storage system (designsafe.storage.default)

This step must succeed before any of the subsequent steps are attempted.

1. Create app.json
Describes the app, its inputs, execution system, and wrapper script

The **app definition file** is the heart of every Tapis app.

It contains:

app name, version, and description
which execution system to use
what queue, how many nodes/cores, time limits
what runtime type to use (ZIP, Docker, etc.)
what inputs the user must supply
environment variables that control runtime behavior
how to archive results after job completion

In this step, the notebook programmatically builds two JSON objects:

Agnostic app: full-featured, configurable
OpenSeesPy app: reduced-scope, simplified interface

These JSON objects are saved as:

app.json for the agnostic app
openseespy-app.json for the reduced-scope one

2. Create tapisjob_app.sh
Runs your analysis (e.g., ibrun OpenSees main.tcl)

This is the **wrapper script** that runs *inside the HPC job*.

It performs all runtime logic:

prints app info and job UUID
changes into the input directory
loads required TACC modules
(optionally) copies TACC-compiled OpenSeesPy (opensees.so)
installs Python packages
chooses an MPI launcher (ibrun) when appropriate
executes the user’s script
logs timing, environment summaries, and success/failure
optionally zips the output and/or moves results to WORK/HOME/SCRATCH
cleans up temporary files

The agnostic app uses a longer, more capable script.
The OpenSeesPy app uses a simplified variant with reduced features.

The notebook writes both scripts to disk for packaging.

2a. Zip the App
This app is a ZIP-runtime app (see app.json)

Since this is a **ZIP runtime** app, Tapis requires a .zip archive containing: - the tapisjob_app.sh file (required) - optional documentation files (ReadMe.md) - optional supporting scripts

The notebook automatically:

zips the bundle
saves it with a versioned name, e.g.
- designsafe-agnostic-app.zip
- designsafe-openseespy-s3.zip

This ZIP file becomes the app’s containerImage in app.json.

3. Create profile.json (Optional)
(Optional) Loads modules/environment -- use an existing one or define a new profile

Most apps rely on a scheduler profile to control: - module behavior - environment setup - SLURM initialization

In this case, both apps use:

--tapis-profile tacc-no-modules

This means:

TACC loads no modules by default
The wrapper script is responsible for loading Python, OpenSees, HDF5, etc.

If a custom profile were needed, the notebook would generate it here.

4. Create ReadMe.md
Instructions for the app user

A user-facing markdown file is automatically generated for each app.

It includes:

what the app does
what parameters it expects
how to use it on DesignSafe
how to import OpenSeesPy correctly
examples for both serial and MPI usage

These files are included in the app ZIP bundle.

The notebook validates this JSON, ensuring it’s syntactically correct and consistent with the script.

5. Upload Files to Storage
To the deployment path in your storage system

Next, you upload:

the ZIP archive
the app JSON
the ReadMe

to a designated deployment folder such as:

designsafe.storage.default:/silvia/apps/designsafe-agnostic-app/<version>/

Tapis retrieves these files at runtime, so they must be readable.

The notebook performs this upload automatically using the Python client.

6. Register the App with Tapis
With Tapis via CLI or Python

You call:

client.apps.createAppVersion(...)

or, if updating:

client.apps.patchAppVersion(...)

At this step, the app becomes visible in:

the DesignSafe “My Apps” list,
the Tapis apps registry.

The notebook registers:

the agnostic app version
the OpenSeesPy-only app version

Both apps are immediately usable after registration.

7. Make/Unmake App Public (Optional)
Make or unmake the app usable to others on TACC (optional)

If desired, we can make the apps publicly usable on Stampede3:

client.apps.grantRole(...)

This step is optional. Personal apps only need permissions for your user account.

8. Set File Permissions
If the app is public, make the app files accessible to others

If the app is made public, the underlying ZIP files must also be made world-readable so other users can launch them.

The notebook includes helper commands to apply the correct:

chmod
Tapis files.setPermissions

This notebook teaches both:

how to build general-purpose HPC apps, and
how to package simplified “easy button” apps for portal users.

Making a Tapis App Public#

You make a Tapis app public when you want other users—any DesignSafe or TACC user—to run your app directly, without needing you to share it privately. A public app:

Appears in other users’ ‘apps’ listings.
Can be executed by anyone with access to the execution system.
Can be launched through the DesignSafe Web Portal by navigating to: ‘https://www.designsafe-ci.org/workspace/’ (once your app is public and indexed).

To make an app truly public, you must complete both:

(1) mark the app public in Tapis using Tapipy, and

(2) ensure your app files are readable on the filesystem.

1. Choose a Public-Friendly Location for 'appPath_Tapis'

If the app is intended to be public:

Place ‘appPath_Tapis’ in an HPC directory readable by all TACC users, typically ‘$WORK’ on Stampede3.
This ensures Tapis—and any user who wants to download or reference your ‘.zip’—can access it.

Even for private apps, ‘$WORK’ is recommended because copying from ‘$WORK’ to the execution directory is fast and reliable.

2. Mark the App Public Using **Tapipy**

With Tapipy, “public” is controlled by **sharing the app with the special “public” role**.

Share the App Publicly

from tapipy.tapis import Tapis

t = Tapis(base_url=<BASE_URL>, username=<USER>, password=<PASSWORD>)
t.apps.share_app_public(appId="APP_ID", appVersion="VERSION")

Unshare (Make Private Again)

t.apps.unshare_app_public(appId="APP_ID", appVersion="VERSION")

3. Verify That the App Is Public

Use Tapipy to retrieve the latest version and inspect the 'isPublic' field:

app = t.apps.getAppLatestVersion(appId="APP_ID")
print(app.isPublic)

A correct public app will show:

True

If ‘False’, your app is not public yet—even if permissions are correct.

4. Ensure Filesystem Permissions Allow Public Access

Making an app “public” in Tapis does **not** bypass Unix file permissions. Other users—and Tapis itself when executing the app—must have:

Read access to the ‘.zip’ file
Execute (traverse) permission on every directory in the path

4.1. Make the App Bundle Readable

bash:

chmod go+r yourfile.zip

Python:

import os, stat

path = "yourfile.zip"
st = os.stat(path)
file_perms = stat.S_IRGRP | stat.S_IROTH  # group + others read
os.chmod(path, st.st_mode | file_perms)

4.2. Ensure All Directories Are Traversable

Directories must have ‘+x’ for group and others:

bash:

chmod go+x /work2/groupID/username
chmod go+x /work2/groupID/username/system
chmod go+x /work2/groupID/username/system/apps
chmod go+x /work2/groupID/username/system/apps/app_name
chmod go+x /work2/groupID/username/system/apps/app_name/app_version

Python:

import os, stat

dir_perms = stat.S_IXGRP | stat.S_IXOTH  # group + others execute

dirs = [
    "/work2/groupID/username",
    "/work2/groupID/username/system",
    "/work2/groupID/username/system/apps",
    "/work2/groupID/username/system/apps/app_name",
    "/work2/groupID/username/system/apps/app_name/app_version",
]

for d in dirs:
    st = os.stat(d)
    os.chmod(d, st.st_mode | dir_perms)

Note: ‘/work2’ on Stampede3 is not world-readable by default, so setting traversal permissions is required.

Because ‘os.chmod()’ uses ‘st.st_mode | perms’:

Your own permissions remain unchanged.
We only add missing group/other bits.
Apply permissions after copying all app files.

Summary Checklist for Public Tapis Apps

Requirement	Completed By
App bundle in a readable shared location (‘$WORK’)	You
App marked public	‘t.apps.share_app_public(…)’
Verified ‘“isPublic”: true’	‘t.apps.getAppLatestVersion()’
File readable by group/others	‘chmod go+r’
Directories traversable by group/others	‘chmod go+x’
App available in DesignSafe Web Portal	Automatic once public

Troubleshooting: “My Public App Still Doesn’t Work”

Even after setting 'isPublic = true' and fixing permissions, you may find that:

Other users can’t see or run your app.
The DesignSafe Web Portal can’t load it at ‘https://www.designsafe-ci.org/workspace/’.
Jobs fail because the app bundle can’t be read.

Below are the most common causes and quick checks.

1. The App Isn’t Actually Public

Symptom: Other users can’t see the app in their listings or launch it from the portal.

Check with Tapipy:

app = t.apps.getAppLatestVersion(appId="APP_ID")
print(app.isPublic)

If this prints ‘False’, you haven’t successfully shared it.

Fix:

t.apps.share_app_public(appId="APP_ID", appVersion="VERSION")

2. Wrong ‘appId’ or ‘appVersion’

Symptom: You shared one version, but users (or the portal) are trying to use another.

Check all versions:

apps = t.apps.getApps(appId="APP_ID")
for a in apps:
    print(a.id, a.version, a.isPublic)

Fix:

Make sure the version you intend to expose is the one with ‘isPublic = True’.

Share that specific version:

t.apps.share_app_public(appId="APP_ID", appVersion="INTENDED_VERSION")

3. File Permissions Still Too Tight

Symptom: Users see the app, but jobs fail with errors about missing or unreadable files, or they can’t copy the ‘.zip’.

Checklist:

App bundle is readable by group + others:
```
chmod go+r yourfile.zip
```

All directories in the path are traversable (‘+x’) by group + others:

chmod go+x /work2/groupID/username
chmod go+x /work2/groupID/username/system
chmod go+x /work2/groupID/username/system/apps
chmod go+x /work2/groupID/username/system/apps/app_name
chmod go+x /work2/groupID/username/system/apps/app_name/app_version

If in doubt: Have another user run ‘ls yourfile.zip’ and ‘ls’ each directory in the path. If they can’t traverse or see it, permissions are still too restricted.

4. ‘appPath_Tapis’ or Archive Path Doesn’t Match Reality

Symptom: The app is public and permissions look correct, but Tapis can’t find or unpack the ‘.zip’ when a job starts.

Things to check in your app definition (JSON):

‘archiveSystem’ / ‘execSystemId’: point to the correct system (e.g., Stampede3).
‘appPath_Tapis’ (or whatever you call the directory): matches the actual directory on the system.
‘appArchivePath’ (or similar field): matches the actual filename (including subdirectories if used).

If you moved the ‘.zip’ after creating the app, or changed directory names, you must update the app definition and re-register it.

5. DesignSafe Web Portal Not Showing Updates Yet

Symptom: ‘isPublic = true’ and the app works via Tapipy/CLI, but ‘https://www.designsafe-ci.org/workspace/’ isn’t showing or launching the latest version.

Quick checks:

Confirm ‘isPublic’ on the intended version:

app = t.apps.getAppLatestVersion(appId="APP_ID")
print(app.id, app.version, app.isPublic)

Make sure ‘appId’ matches the name the portal expects (case, hyphens, etc.).
If you recently changed ‘isPublic’ or app metadata, give the portal a little time to refresh its index, or log out / back in and try again.

If it still doesn’t appear after a reasonable delay, verify everything else in this checklist, then contact DesignSafe support with the ‘appId’ and ‘version’.

6. Users Don’t Have Access to the Execution System

Symptom: App appears in listings, but when users try to run it, they get authorization or system-access errors.

Check:

Which ‘execSystemId’ the app uses (e.g., Stampede3).
Whether the other user has:
- An active allocation / account on that system.
- Proper onboarding (e.g., they can log in or submit other jobs there).

If they don’t have access to the execution system, they won’t be able to run your app, even if it’s public.

Resource on Making App Public

https://tapis-project.github.io/live-docs/?service=Apps#tag/Sharing/operation/shareAppPublic

https://github.com/tapis-project/tapipy/blob/main/tapipy/resources/openapi_v3-apps.yml

Notebook Workflow#

Outline

NOTE: The following outline may be missing a few steps that may have been added later in development

```
# Initialize Process
## Initialize Python Environment
### Load Specialized Utilities Library
## Set Process-Control Switches
### Make-App Switches
### Test-App Switches
### Make-Public Switches
## Set App-Author Info
## Set Execution-System
## Set App Path for Development
## Set App Path for Deployment
## Get Today's Date
# Connect to Tapis
## Get username
## Get User- and Execution-System-Specific Work Paths
### Work Path in the HPC System
### Work Path in the JupyterHub System
# Configure App
## Set App ID Data
### Set Update Type
## Set App Version
## Set File Locations
### Set appPath_Local
### Set appPath_Tapis
# Create the App Files
## A. Create **Readme.MD** – App User Documentation
## B. Create/Select **profile.json** – Environment Setup
## C. Create **app.json** – App Definition
## D. Create **tapisjob_app.sh** – Wrapper Script
### 0. Script initialization: safety flags, required args, and global context
### 1a. Summary log setup
### 1b. Environment Log Setup
### 2. Argument and environment-variable summary helpers
### 2a. Log App (Arguments bash_script_echoSummary_ARGS)
### 2b. Log Environment Variable 
### 3. Log MPI/SLURM diagnostics
### 4a. Error-path timing summary (run vs. total)
### 4b. Success-path timing summary (binary run only)
### 5. Final total-runtime footer (successful script completion)
### 6. Optional pre-run copy of input files/directories
### 7. Optional ZIP expansion of input bundles
### 8. Defensive setup of the module command (before user-defined module loads)
### 9. Loading modules from a user-provided file
### 10. Loading modules from a comma-separated list (MODULE_LOADS_LIST)
### 11. Load OpenSees Modules (If Running OpenSees)
### 12. Installing Python packages from a requirements file (PIP_INSTALLS_FILE)
### 13. Installing Python packages from a comma-separated list (PIP_INSTALLS_LIST)
### 14. Choosing how to launch the app (sequential vs MPI)
### 15. Running the job binary (with timers and error handling)
### 16. OpenSeesPy: copy TACC-compiled OpenSeesPy.so into the run directory
### 17. Optional Cleanup: remove temporary TACC OpenSeesPy library after the run
### 18. Optional: repack the output directory into a single ZIP (ZIP_OUTPUT_SWITCH)
### 19. Optional: move main output to a faster storage destination (PATH_MOVE_OUTPUT)
### 20. Optional: Pre-Job Hook -- User-Defined script run BEFORE main binary (PRE_JOB_SCRIPT)
### 21. Optional: Post-Job Hook -- User-Defined script run AFTER main binary (POST_JOB_SCRIPT)
### 22. Change Directory (cd) INTO Input Directory
### 23. Change Directory (cd) OUT OF Input Directory
### 24. Assemble Main Wrapper File: **tapisjob_app.sh**
### 25. Replace batch_script patches into the Main Wrapper File
#### 25a. Replace batch_script patches -- All Apps
#### 25b. Replace batch_script patches -- Agnostic App
#### 25c. Replace batch_script patches -- OpenSeesPy App
## E. Create **tapisjob_app.zip** – App Zip File
## F. File Check -- Visualize File Contents in Local (Development) Path
### Show Files for Content -- No Line Numbers
### Show Files for Debugging -- SHOW Line Numbers
# Validate App Files Locally
# Deploy the App
## Upload Files to appPath_Tapis
### Make the App Directory
### Upload/Copy Files to Deployment System
### Check Files on Deployment System To Verify Upload
# Register The App
## List All Tapis Apps to Verify Registration
## Access App Schema on Tapis to Validate Registration
# Manage Public App
## Manage App isPublic Status
### Make The App Public (optional)
### or Remove The App From Public Access (optional)
### Verify isPublic Status### Set Permissions for Public App
## Set Permissions for Public App 
### File Permissions
### Path/Directory Permissions
# Test App (done in a separate notebook)
```

</div>

Initialize#

Configure Python#

import shutil
import ipywidgets as widgets
from IPython.display import display, clear_output, Markdown
import textwrap, time
import stat
from pathlib import Path
import json

Load Specialized Utilities Library#

I have developed a collection of python defs that are intended to simplify coding.

# Import Utilities Library
import os,sys
PathOpsUtils = os.path.expanduser('~/CommunityData/Training/Computational-Workflows-on-DesignSafe/OpsUtils')
if not PathOpsUtils in sys.path: sys.path.append(PathOpsUtils)
from OpsUtils import OpsUtils

# We will use this utility often to view the utility functions:
OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'show_text_file_in_accordion.py')

Set Process-Control Switches#

Make-App Switches#

do_makeApp = True
# do_makeApp = False # remove

do_makeApp_OpsPy = True
# do_makeApp_OpsPy = False # remove

Note that the OpsPy app relies on the main app

if do_makeApp == False:
    do_makeApp_OpsPy = False

Test_OpenSees_TCL = True
# Test_OpenSees_TCL = False # REMOVE

Test_OpenSees_PY = True
# Test_OpenSees_PY = False # REMOVE

Test_OpenSees_PY_OpsPy = True
# Test_OpenSees_PY_OpsPy = False # REMOVE

Make-Public Switches#

makePublic = True
# makePublic = False # remove

makeUnPublic = False

makePublic_OpsPy = True
# makePublic_OpsPy = False # remove

makeUnPublic_OpsPy = False

Set App-Author Info#

you can paste this in your files, where needed

app_Author_Info = 'Silvia Mazzoni, DesignSafe (silviamazzoni@yahoo.com)'

Set Execution System#

TACC system where the app will be submit the SLURM job. This app can be run on any system in TACC. You can let the user make this selection.

However, this app was developed and tested for stampede3.

Options:

stamped3: https://docs.tacc.utexas.edu/hpc/stampede3/
vista: https://docs.tacc.utexas.edu/hpc/vista/
frontera: https://docs.tacc.utexas.edu/hpc/frontera/

exec_system_id_app = "stampede3"; # options: 'stamped3','frontera','vista'

Set App Path for Development#

appPath_Local_base = f'~/MyData/myAuthoredTapisApps'; # your choice

Set App Path for Deployment#

Define Work Path in Tapis format

app_system_id = 'cloud.data'

app_system_id_OpsPy = 'cloud.data'

Get Today’s date#

you can paste this in your files, where needed

from datetime import date

today = date.today()

today_formatted = today.strftime("%B %d, %Y")   # December 04, 2025
print(today_formatted)

February 14, 2026

Connect to Tapis#

OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'connect_tapis.py')

force_connect = False
# force_connect = True; # REMOVE do this only if you want to restart the clock on the token.
t=OpsUtils.connect_tapis(force_connect=force_connect)

 -- Checking Tapis token --
 Token loaded from file. Token is still valid!
 Token expires at: 2026-02-14T20:12:52+00:00
 Token expires in: 3:31:40.598734
-- AUTHENTICATED VIA SAVED TOKEN --

Get username#

you may need it for some paths

OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'get_tapis_username.py')

username = OpsUtils.get_tapis_username(t)
print('username',username)

username silvia

Get User- and Execution-System-Specific Work Paths#

While JupyterHub treats Work as part of its environment, Tapis needs its full path, which is user-specific

Work Path in the HPC system#

Use tapis to obtain this user & system-dependent base path

exec_system_envVar_List = ['WORK','HOME','SCRATCH']; # collect these, just in case

exec_system_path_dict = {}
for thisKey in exec_system_envVar_List:
    exec_system_path_dict[thisKey] = t.systems.hostEval(systemId=exec_system_id_app,envVarName=thisKey).name
display(exec_system_path_dict)

user_WorkPath_base = exec_system_path_dict["WORK"]

{'WORK': '/work2/05072/silvia/stampede3',
 'HOME': '/home1/05072/silvia',
 'SCRATCH': '/scratch/05072/silvia'}

Work Path in the JupyterHub System#

Check this path within your JupyterHub System.

user_WorkPath_base_local = f'~/Work/{exec_system_id_app}'

print('user_WorkPath_base:',user_WorkPath_base)
print('user_WorkPath_base_local:',user_WorkPath_base_local)

user_WorkPath_base: /work2/05072/silvia/stampede3
user_WorkPath_base_local: ~/Work/stampede3

Configure App#

Set App ID Data#

app_id = 'designsafe-agnostic-app'
app_description = 'Agnostic Tapis App for General Python Execution as well as OpenSees, OpenSeesMP, OpenSeesSP, OpenSeesPy'
app_helpUrl = ''

app_id_OpsPy = 'designsafe-openseespy-s3'
app_description_OpsPy = f'Basic App to run OpenSeesPy on {exec_system_id_app}.'
app_helpUrl_OpsPy = ''

Check if App Exists#

If it exists you will see a version number.

current_app_version = OpsUtils.get_latest_app_version(t,app_id)
print('current_app_version',current_app_version)

current_app_version 1.3.10

current_app_version_OpsPy = OpsUtils.get_latest_app_version(t,app_id_OpsPy)
print('current_app_version_OpsPy',current_app_version_OpsPy)

current_app_version_OpsPy 1.2.14

Set App Version#

we have an utility for that will autoincrement an existing app’s version

OpsUtils.show_text_file_in_accordion(PathOpsUtils, ['increment_tapis_app_version.py','get_latest_app_version.py','bump_app_version.py'])

Set Update Type#

updateType = 'patch'; # options: major, minor, patch

Determine new version number for this update#

if do_makeApp:
    app_version = OpsUtils.increment_tapis_app_version(t,app_id,updateType)

app exists, now latest_app_version 1.3.10
Update type: patch
now app_version 1.3.11

if do_makeApp_OpsPy:
    app_version_OpsPy = OpsUtils.increment_tapis_app_version(t,app_id_OpsPy,updateType)    

app exists, now latest_app_version 1.2.14
Update type: patch
now app_version 1.2.15

Set File Locations#

appPath_Local Your local development directory. This is where you create, modify, and organize all application files before packaging or uploading.

appPath_Tapis The destination directory on the HPC system where your application files will be uploaded and stored and will ultimately reside (including the packaged .zip). These files do not run from this location, Tapis copies it over to the execution directory.

Special Case: Public App#

If the app is intended to be public, ensure that appPath_Tapis is located in a directory accessible to all TACC users (for example, your WORK directory on Stampede3) so others can download or reference the files as needed.
Even if the app is not public, placing appPath_Tapis in your WORK directory on Stampede3 is still recommended, as copying files from WORK to the execution-system directory (also on Stampede3) is typically faster and more reliable.

Set appPath_Local#

if do_makeApp:
    appPath_Local = f'{appPath_Local_base}/{app_id}/{app_version}'; # your choice
    appPath_Local = os.path.abspath(os.path.expanduser(appPath_Local))
    os.makedirs(appPath_Local, exist_ok=True)
    print(f'appPath_Local: {appPath_Local}\n exists:',os.path.exists(appPath_Local))

appPath_Local: /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11
 exists: True

if do_makeApp_OpsPy:
    appPath_Local_OpsPy = f'{appPath_Local_base}/{app_id_OpsPy}/{app_version_OpsPy}'; # your choice
    appPath_Local_OpsPy = os.path.abspath(os.path.expanduser(appPath_Local_OpsPy))
    os.makedirs(appPath_Local_OpsPy, exist_ok=True)
    print(f'appPath_Local_OpsPy: {appPath_Local_OpsPy}\n exists:',os.path.exists(appPath_Local_OpsPy))

appPath_Local_OpsPy: /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15
 exists: True

Set appPath_Tapis#

appPath_Tapis0 = user_WorkPath_base; # we will use this path to create folders and copy app files using TAPIS if we cannot access the system from here
appPath_Tapis0_local = f'{user_WorkPath_base_local}'; # we will use this path to create folders and copy app files using python/os commands if TAPIS is slow

appPath_Tapis_local_anchor = os.path.abspath(os.path.expanduser(appPath_Tapis0_local))
print('appPath_Tapis_local_anchor',appPath_Tapis_local_anchor)

appPath_Tapis0 += '/apps'
appPath_Tapis0_local  += '/apps'

appPath_Tapis0 = os.path.expanduser(appPath_Tapis0)
appPath_Tapis0_local = os.path.expanduser(appPath_Tapis0_local)

print('appPath_Tapis0',appPath_Tapis0)
print('appPath_Tapis0_local',appPath_Tapis0_local)

appPath_Tapis_local_anchor /home/jupyter/Work/stampede3
appPath_Tapis0 /work2/05072/silvia/stampede3/apps
appPath_Tapis0_local /home/jupyter/Work/stampede3/apps

if do_makeApp:
    appPath_Tapis = f"{appPath_Tapis0}/{app_id}/{app_version}"
    appPath_Tapis_local =f"{appPath_Tapis0_local}/{app_id}/{app_version}"
    container_filename = f'{app_id}.zip'
    
    print('appPath_Tapis',appPath_Tapis)
    print('appPath_Tapis_local',appPath_Tapis_local)
    print('container_filename',container_filename)

appPath_Tapis /work2/05072/silvia/stampede3/apps/designsafe-agnostic-app/1.3.11
appPath_Tapis_local /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11
container_filename designsafe-agnostic-app.zip

if do_makeApp_OpsPy:
    appPath_Tapis_OpsPy = f"{appPath_Tapis0}/{app_id_OpsPy}/{app_version_OpsPy}"   
    appPath_Tapis_local_OpsPy =  f"{appPath_Tapis0_local}/{app_id_OpsPy}/{app_version_OpsPy}"
    container_filename_OpsPy = f'{app_id_OpsPy}.zip'
    
    print('appPath_Tapis_OpsPy',appPath_Tapis_OpsPy)
    print('appPath_Tapis_local_OpsPy',appPath_Tapis_local_OpsPy)
    print('container_filename_OpsPy',container_filename_OpsPy)    

appPath_Tapis_OpsPy /work2/05072/silvia/stampede3/apps/designsafe-openseespy-s3/1.2.15
appPath_Tapis_local_OpsPy /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15
container_filename_OpsPy designsafe-openseespy-s3.zip

Create the App Files#

A Tapis app requires a small set of core files that define what the app is, how it runs, and what users see when launching it. These files together form the runtime “package” that Tapis deploys onto the HPC system.

1. Readme.md — App Documentation (optional but recommended)#

A human-readable guide for users of your app. Include:

What the app does
Expected inputs and outputs
Example usage
Notes on OpenSees/OpenSeesPy behavior (if relevant)

This file does not affect execution, but it is extremely helpful for portal users.

You should generate this file LAST, and update it every time you modify the app. (ChatGPT is great for this task).

2. Scheduler Profile — Environment Setup (optional but common)#

A profile file (e.g., tacc-no-modules, or a custom profile) that defines:

Which modules are available to load
What environment variables are pre-set
Whether the system provides module support automatically

This profile is executed on the compute node before your wrapper script runs by tapisJob.sh. You may use an existing TACC profile or define your own.

In this case, I have opted to use a blank profile and load the modules manually in the tapisJob_app.sh.
However, creating a profile once and using it during job submittal can save a lot of slurm-job time. This route can reduce some user errors, but can also add new ones.

3. app.json — Tapis App Definition#

This JSON file is the heart of the app. It tells Tapis:

The app name, version, and description
What inputs the user must provide
Parameters and flags (e.g., MPI usage, script names)
Execution system (Stampede3) and queue
How files should be staged and archived
What runtime image to unpack (the ZIP file)

Tapis reads this file to create, validate, and register your app.

4. tapisjob_app.sh — Wrapper Script (the executable logic)#

This is the script Tapis actually runs on the HPC compute node. It performs:

Environment and module setup
Optional pip installations
Optional OpenSeesPy .so copy
Logging, timers, and job summaries
Launching the main executable (OpenSeesMP, python3, etc.)
Cleanup and end-of-job reporting

This file is packaged into the ZIP runtime image and becomes the entry point for the app.

A. Create Readme.MD – App User Documentation#

This file is helpful in communicating content to the app user.

if do_makeApp:
    thisFilename = 'ReadMe.MD'
    thisText_ReadMeMd = textwrap.dedent("""

# __app_id__
***__app_description__***

- **Version:** __app_version__  
- **Author:** *__app_Author_Info__*  
- **Date:** __today_formatted__  
- **Platform:** DesignSafe / TACC Stampede3  
- **Runtime:** ZIP (HPC Batch)  
- **Default queue:** *skx-dev* (can be overridden at submission)

---

## 1. Purpose and Design Philosophy

*__app_id__* is a **general-purpose, HPC-oriented Tapis application** designed to support *many* computational workflows without baking assumptions into the app itself.

Instead of creating separate apps for:

* OpenSees vs OpenSeesMP
* Tcl vs Python
* Serial vs MPI
* Small vs large output jobs

this app acts as a **configurable execution driver**.

All behavior is controlled by:

* **app inputs**
* **environment variables**
* **wrapper logic**

This makes the app:

* reusable
* transparent
* automatable
* easy to fork and extend

Tapis orchestrates the job.
SLURM executes it.
**The wrapper enforces semantics and safety.**

---

## 2. High-Level Execution Model

At runtime, the following happens:

1. Tapis stages *inputDirectory*
2. A SLURM batch job is submitted
3. *tapisjob_app.sh* executes on the first node
4. The wrapper:

   * prepares the environment
   * stages inputs
   * selects MPI vs non-MPI execution
   * runs the main executable
   * manages outputs
   * produces structured logs

The **minimum mental model** for the app is:

**[UseMPI?]  BINARYNAME  INPUTSCRIPT  ARGUMENTS**

<details><summary><b>Detailed Execution Mode</b></summary>
1. Tapis stages your **Input Directory** to the job working directory.
2. SLURM starts the batch job on Stampede3.
3. *tapisjob_app.sh* runs on the first allocated node and:
   - sets up summary and full environment logs
   - *cd*s into the Input Directory
   - prepares inputs (optional copy-in, optional unzip)
   - loads modules (optional file + optional list)
   - normalizes Python (*python* → *python3*)
   - installs Python packages (optional file + optional list)
   - optionally injects TACC-compiled OpenSeesPy (*opensees.so*)
   - optionally runs pre/post hooks
   - chooses MPI launcher (*ibrun*) or direct run
   - runs your executable + script + args
   - optionally zips output and/or moves results inside the exec system
   - records timers and exits with clear error handling

    
</details>



## 3. Input parameters (what each one means)

This section is the “user manual” for every input you see in the portal.

### 3.1 File input

#### **Input Directory** (required)
**What it is:** A *single directory* staged by Tapis into the job.  
**What should be inside:**
- your main script (Tcl or Python)
- any supporting files your script needs (models, data, configs)
- optional helper files:
  - *modules.txt* (for *MODULE_LOADS_FILE*)
  - *requirements.txt* (for *PIP_INSTALLS_FILE*)
  - *prehook.sh* / *posthook.sh* (for hook variables)
  - zipped bundles referenced by *UNZIP_FILES_LIST*

**Runtime behavior:** The wrapper *cd*s into this directory before running the main command.  
**Implication:** relative paths in your script should assume this directory is the working directory.

---

### 3.2 Required app arguments

#### **Main Program** (required)
**What it is:** The executable to run (binary name).  
**Common values:**
- *OpenSees* (serial Tcl)
- *OpenSeesMP* / *OpenSeesSP* (MPI Tcl)
- *python3* (Python workflows, including OpenSeesPy)

**Where it must come from:**
- available via modules (recommended), or
- present in the working directory / PATH

**Wrapper notes:**
- If *Main Program* is *python* or *python3*, the wrapper normalizes to *python3*.

---

#### **Main Script** (required)
**What it is:** The filename of the input script passed to the executable.  
**Rules:**
- filename only (no path)
- must exist inside the **Input Directory**

Examples:
- *model.tcl*
- *run_analysis.py*
- *Ex1a.Canti2D.Push.argv.tacc.py*

---

#### **UseMPI** (required)
Controls whether the wrapper launches the executable through *ibrun*.

| UseMPI value | What runs |
|---|---|
| *False* | *<Main Program> <Main Script> [args...]* |
| *True*  | *ibrun <Main Program> <Main Script> [args...]* |

**Use *True* when:**
- OpenSeesMP / OpenSeesSP
- Python + *mpi4py*

**Use *False* when:**
- serial OpenSees (Tcl)
- serial Python / OpenSeesPy
- Python using threading / *concurrent.futures* within a node

> Note: the wrapper treats many “true-like” values as True (*True*, *1*, *Yes*, case-insensitive).

---

#### **CommandLine Arguments** (optional)
Free-form arguments appended after the Main Script.

Example:
```text
--NodalMass 4.19 --outDir outCase1
```

Final command structure:
```bash
[ibrun] <MainProgram> <MainScript> <Arguments...>
```

---

### 3.3 Scheduler inputs

#### **TACC Scheduler Profile** (defaulted)
The app uses the *tacc-no-modules* profile by default so **no modules are implicitly loaded**.
This is intentional: module state is controlled explicitly by the wrapper to improve reproducibility.

#### **TACC Reservation** (optional)
Provide a reservation string if you have one.

---

## 4. Environment variables (advanced configuration)

These values are presented as app inputs in the portal. Most are optional. If you never set them, the wrapper runs with conservative defaults.

### 4.1 OpenSeesPy injection

#### **GET_TACC_OPENSEESPY** (default: *True*)
If True-like, the wrapper attempts to use the **TACC-compiled OpenSeesPy** by:
- loading *python/3.12.11*, *hdf5/1.14.4*, *opensees*
- copying *${TACC_OPENSEES_BIN}/OpenSeesPy.so* into the working directory as *./opensees.so*

**Use this when:**
- you want reliable OpenSeesPy on Stampede3 (recommended)

**In your Python script:**
```python
import opensees as ops
```

**Notes / failure modes:**
- if *TACC_OPENSEES_BIN* is unset or *OpenSeesPy.so* is missing, the wrapper logs a warning and skips the copy.

---

### 4.2 Module loading (two mechanisms)
The two mechanisms are complementary -- you can use both.

#### A. **MODULE_LOADS_FILE** (optional)
A filename (in the Input Directory) containing module commands, one per line.

Supported line formats:
- *purge*
- *use <path>*
- *load <module>*
- *?module* (optional *try-load*)
- bare module names

This is best for **version-controlled, documented module stacks**. It also makes submittal via the web-portal interface easier.

#### B. **MODULE_LOADS_LIST** (optional)
Comma-separated list of modules to load, e.g.:
```text
python/3.12.11,opensees,hdf5/1.14.4,pylauncher
```

**Tip:** use *MODULE_LOADS_FILE* when the setup is more than a few modules or needs comments.

---

### 4.3 Python package installs (two mechanisms)
The two mechanisms are complementary -- you can use both.

#### A. **PIP_INSTALLS_FILE** (optional)
A requirements-style file (in the Input Directory), e.g. *requirements.txt*.

Wrapper behavior:
- runs *pip3 install -r <file>*
- fails the job if pip fails (with a clear error)

It makes submittal via the web-portal interface easier.

#### B. **PIP_INSTALLS_LIST** (optional)
Comma-separated list of packages, e.g.:
```text
mpi4py,pandas,numpy,matplotlib
```

Wrapper behavior:
- installs each package with *pip3 install <pkg>*
- fails the job if any install fails

---

### 4.4 Input preparation

#### A. **UNZIP_FILES_LIST** (optional)
Comma-separated list of ZIP files *in the Input Directory* to expand before execution.
Entries may omit the *.zip* suffix.

Use this when:
- you staged one bundled zip instead of many small files

#### B. **PATH_COPY_IN_LIST** (optional)
Comma-separated list of **absolute paths** (within the execution system) to copy into the working directory before execution.

Example:
```text
$WORK/FileSet2,$SCRATCH/FileSet3/thisFile.at2
```

Use this when:
- you need large/shared datasets without duplicating them into the Input Directory
- you want a specific runtime layout inside the working directory

#### C. **DELETE_COPIED_IN_ON_EXIT** (default: *0*)
If set to *1* / True-like, the wrapper deletes only the copied-in items listed in its manifest on exit.

Safety rules:
- refuses absolute paths
- refuses *..* traversal
- deletes only what landed in the working directory

Use this when:
- copy-in files are “temporary conveniences” and should not be archived

---

### 4.5 Pre/Post hooks

#### A. **PRE_JOB_SCRIPT** (optional)
Script to run after environment setup but before the main executable.
- if relative, interpreted as *./script* inside the Input Directory
- if executable, run directly; otherwise run via *bash*

#### B. **POST_JOB_SCRIPT** (optional)
Script to run after the main executable (same resolution rules as pre-hook).

**Default policy:** hook failures are logged as warnings and the job continues (you can change this policy in the wrapper if desired).

---

### 4.6 Output management

#### A. **ZIP_OUTPUT_SWITCH** (default: *False*)
If True-like:
- zips the entire Input Directory after execution into *inputDirectory.zip*
- removes the original directory

Use this when:
- output is large and contains many small files
- you want a single artifact to move / download

#### B. **PATH_MOVE_OUTPUT** (optional)
If set, the wrapper moves the main output artifact into:
```text
<PATH_MOVE_OUTPUT>/_<JobUUID>/
```
and copies top-level logs into that same folder.

Recommended:
- move to *$WORK/...* for interactive inspection in JupyterHub
- move to *$SCRATCH/...* for chained HPC workflows

---

## 5. Logs you should look at first

Every job produces:
- ***SLURM-job-summary.log*** (compact “what happened”)
- ***SLURM-full-environment.log*** (full *env | sort* dump)

The summary log also records:
- launcher decision
- module/pip actions
- timers (run-only and total)

---

## 6. Typical patterns

### Serial OpenSees (Tcl)
- Main Program: *OpenSees*
- UseMPI: *False*

### OpenSeesMP / OpenSeesSP (MPI)
- Main Program: *OpenSeesMP* (or *OpenSeesSP*)
- UseMPI: *True*

### OpenSeesPy (serial)
- Main Program: *python3*
- UseMPI: *False*
- *GET_TACC_OPENSEESPY=True*

### Python + mpi4py
- Main Program: *python3*
- UseMPI: *True*
- *PIP_INSTALLS_LIST=mpi4py* (or requirements file)

---

## 7. Summary

*__app_id__* provides a single, well-instrumented execution interface for:
- OpenSees (Tcl), OpenSeesMP/SP (MPI), OpenSeesPy
- general Python workflows
- reusable HPC job patterns (copy-in, unzip, hooks, packaging, output movement)

It is designed to be **debuggable, reproducible, and extensible**, and to serve as a template for future apps.

    
    """)
    thisText_ReadMeMd = thisText_ReadMeMd.replace("__app_id__", app_id)
    thisText_ReadMeMd = thisText_ReadMeMd.replace("__app_Author_Info__", app_Author_Info)
    thisText_ReadMeMd = thisText_ReadMeMd.replace("__app_version__", app_version)
    thisText_ReadMeMd = thisText_ReadMeMd.replace("__app_description__", app_description)
    thisText_ReadMeMd = thisText_ReadMeMd.replace("__today_formatted__", today_formatted)
    with open(f"{appPath_Local}/{thisFilename}", "w") as f:
        f.write(thisText_ReadMeMd)
    # write it here
    with open(f"./{thisFilename}_{app_id}", "w") as f:
        f.write(thisText_ReadMeMd)

if do_makeApp:
    OpsUtils.show_text_file_in_accordion(appPath_Local, thisFilename, showLineNumbers=False)

if do_makeApp_OpsPy:
    thisFilename = 'ReadMe.MD'
    thisText_ReadMeMd_OpsPy = textwrap.dedent("""

# __app_id_OpsPy__
***__app_description_OpsPy__***


* **Version:** __app_version_OpsPy__
* **Author:** *__app_Author_Info__*
* **Date:** __today_formatted__
* **Runtime:** ZIP
* **Execution system:** Stampede3 (SLURM)

---

## 1. Overview

**__app_id_OpsPy__** is a lightweight, ZIP-runtime **Tapis batch app** designed to run **OpenSeesPy** workflows on **Stampede3** through the **DesignSafe** platform.

The app intentionally keeps configuration minimal while still supporting:

* Serial Python runs
* MPI-based Python runs using `mpi4py`
* Automatic staging of inputs and outputs
* Optional module loading
* Optional pip installs
* Detailed job summary logging

This app is ideal for:

* Teaching and tutorials
* Small–to–moderate OpenSeesPy models
* Parameter studies
* MPI-enabled OpenSeesPy workflows
* Users who want **zero manual SLURM scripting**

---

## 2. What the App Does (Execution Flow)

When a job starts, the app performs the following steps:

1. **Stages the Input Directory** into the job working directory
2. **Loads user-specified TACC modules** (via `MODULE_LOADS_LIST`)
3. **Optionally installs Python packages** using pip
4. **Copies the TACC-compiled OpenSeesPy library** (`OpenSeesPy.so`) into the working directory as:

   ```
   ./opensees.so
   ```
5. **Selects the launcher**

   * Serial execution → direct `python3`
   * MPI execution → `ibrun python3`
6. **Runs your Python script**
7. **Writes a compact, human-readable job summary log**
8. **Cleans up temporary OpenSeesPy artifacts**
9. **Archives outputs back to DesignSafe storage**

No Docker image is used.
All execution occurs directly on Stampede3 compute nodes under SLURM.

---

## 3. Required Input

### **Input Directory (REQUIRED)**

Upload a single directory containing:

* Your **main OpenSeesPy script** (`.py`)
* Any data files the script reads
* Optional:

  * `requirements.txt`
  * auxiliary Python modules
  * model input files

The directory is staged and exposed as:

```
$PWD/inputDirectory/
```

Your script is executed **from inside this directory**.

---

## 4. App Arguments (Portal Inputs)

### **1. Main Program**

*Fixed and hidden*

```
python3
```

---

### **2. Main Script (REQUIRED)**

The **filename only** of your Python script
(must exist inside the Input Directory)

Example:

```
run_model.py
```

---

### **3. UseMPI (True / False)**

Controls how the script is launched.

| UseMPI | Behavior                  |
| ------ | ------------------------- |
| False  | `python3 script.py`       |
| True   | `ibrun python3 script.py` |

**Guidance**

* Use **True** when using:

  * `mpi4py`
  * OpenSeesPy MPI domain decomposition
* Use **False** for:

  * serial scripts
  * `concurrent.futures` on a single node

---

## 5. Environment Variables (Pre-Configured)

These variables are defined in the app and usually **do not need to be changed**.

| Variable              | Default                                  | Purpose                                                |
| --------------------- | ---------------------------------------- | ------------------------------------------------------ |
| `GET_TACC_OPENSEESPY` | True                                     | Copies TACC-compiled OpenSeesPy into the job directory |
| `MODULE_LOADS_LIST`   | `python/3.12.11,opensees,hdf5/1.14.4`    | Modules loaded before execution                        |
| `PIP_INSTALLS_LIST`   | `mpi4py,pandas,numpy,matplotlib,futures` | Python packages installed via pip                      |

You may override these values if needed.

---

## 6. Importing OpenSeesPy Correctly

Because the app **injects a TACC-compiled shared library** into the working directory, your script must import OpenSeesPy as:

```python
import opensees as ops
```

or:

```python
import opensees
```

❌ **Do not** use:

```python
import openseespy.opensees
```

unless you intentionally install the PyPI wheel and disable `GET_TACC_OPENSEESPY`.

---

## 7. MPI Usage with OpenSeesPy

For MPI workflows, your script should explicitly use `mpi4py`:

```python
from mpi4py import MPI
import opensees as ops
```

Portal settings:

* **UseMPI:** True
* **nodeCount / coresPerNode:** set appropriately

The app will automatically launch with:

```
ibrun python3 your_script.py
```

---

## 8. Job Logging & Diagnostics

Each job produces a **compact summary log** named:

```
SLURM-job-summary.log
```

This file includes:

* App metadata
* Loaded modules
* Installed pip packages
* Launch mode (MPI vs serial)
* Runtime durations
* Error diagnostics (if the job fails)

This log is intended for **human-readable debugging** and complements SLURM output files.

---

## 9. Output & Archiving

All files produced during execution remain inside the Input Directory and are archived to:

```
$WORK/tapis-jobs-archive/<date>/<jobname>-<jobuuid>/
```

Archived content includes:

* Job summary log
* Environment logs
* Script outputs
* Any files created by your Python workflow

---

## 10. Common Failure Modes

| Symptom                 | Likely Cause                          |
| ----------------------- | ------------------------------------- |
| `ImportError: opensees` | Incorrect import statement            |
| MPI job hangs           | `UseMPI=True` but script not MPI-safe |
| pip install failure     | Incompatible package version          |
| Job exits immediately   | Script filename mismatch              |

Check **SLURM-job-summary.log** first.

---

## 11. Intended Scope

This app is designed for:

* OpenSeesPy-based research workflows
* Education and training
* Lightweight automation through the DesignSafe portal

It is **not** intended to replace:

* Custom SLURM scripts
* Large-scale production pipelines
* Long-running, multi-stage workflows

For those use cases, consider developing a custom Tapis app or using OpenSeesMP-specific apps.

---

## 12. License & Reuse

Developed by DesignSafe.
This app may be reused, forked, and extended for broader OpenSeesPy workflows.


    """)
    thisText_ReadMeMd_OpsPy = thisText_ReadMeMd_OpsPy.replace("__app_Author_Info__", app_Author_Info)
    thisText_ReadMeMd_OpsPy = thisText_ReadMeMd_OpsPy.replace("__app_id_OpsPy__", app_id_OpsPy)
    thisText_ReadMeMd_OpsPy = thisText_ReadMeMd_OpsPy.replace("__app_version_OpsPy__", app_version_OpsPy)
    thisText_ReadMeMd_OpsPy = thisText_ReadMeMd_OpsPy.replace("__app_description_OpsPy__", app_description_OpsPy)
    thisText_ReadMeMd_OpsPy = thisText_ReadMeMd_OpsPy.replace("__today_formatted__", today_formatted)
    with open(f"{appPath_Local_OpsPy}/{thisFilename}", "w") as f:
        f.write(thisText_ReadMeMd_OpsPy)

    # write it here
    with open(f"./{thisFilename}_{app_id_OpsPy}", "w") as f:
        f.write(thisText_ReadMeMd_OpsPy)

if do_makeApp_OpsPy:
    OpsUtils.show_text_file_in_accordion(appPath_Local, [thisFilename], showLineNumbers=False)

B. Create/Select profile.json – Environment Setup#

This file defines the modules that will be loaded before your script runs. It is executed on the compute node.

You can define this environement once, or you can use available environments, such as opensees.

list of existing profiles, see if any are useful to you#

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'Existing Profiles')
display(here_accordion)

with here_out:
    systemProfiles = t.systems.getSchedulerProfiles(orderBy='name')
    for thisProfile in systemProfiles:
        this_out = widgets.Output()
        this_accordion = widgets.Accordion(children=[this_out])
        # here_accordion.selected_index = 0
        this_accordion.set_title(0, thisProfile.name)
        display(this_accordion)
        with this_out:
            print(thisProfile)

C. Create app.json – App Definition#

Defines the app’s metadata, inputs, parameters, and execution configuration.

thisText_options_COMMANDLINE_ARGS = textwrap.dedent(""",
        {
          "name": "CommandLine Arguments",
          "description": "Optional command-line arguments appended after Main Script (e.g., '--npts 2000 --dir X' or any format consistent with how your input script parses them).",
          "arg": null,
          "inputMode": "INCLUDE_ON_DEMAND",
          "notes": {"isHidden": __isHidden__}
        }""")

thisText_options_ENV_VARS = textwrap.dedent(""",
        {
          "key": "UNZIP_FILES_LIST",
          "value": "",
          "inputMode": "INCLUDE_ON_DEMAND",
          "description": "Comma-separated list of ZIP files in the Input Directory to unzip before the run. Example: 'inputs.zip,gm_files.zip'.",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "PATH_COPY_IN_LIST",
          "value": "",
          "inputMode": "INCLUDE_ON_DEMAND",
          "description": "Absolute Path (within the Execution System) of folder that will be copied into the job working directory **before** execution.  (Example: '$HOME/FileSet1,$WORK/FileSet2,$SCRATCH/FileSet3/thisFile.at2')",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "DELETE_COPIED_IN_ON_EXIT",
          "value": "0",
          "inputMode": "INCLUDE_ON_DEMAND",
          "description": "If set to a true-like value, removes files or directories that were copied into the job working directory via PATH_COPY_IN_LIST after the job completes, preventing temporary inputs from being included in the final archive.",
          "notes": { "isHidden": __isHidden__ }
        },
        {
          "key": "MODULE_LOADS_FILE",
          "value": "",
          "inputMode": "INCLUDE_ON_DEMAND",
          "description": "Name of a file in the Input Directory containing a list of modules to load (newline- or comma-separated). Example: 'modules.txt'.",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "PIP_INSTALLS_FILE",
          "value": "",
          "inputMode": "INCLUDE_ON_DEMAND",
          "description": "Name of a file in the Input Directory containing a list of Python packages to pip install (newline- or comma-separated). Example: 'requirements.txt'.",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "ZIP_OUTPUT_SWITCH",
          "value": "False",
          "inputMode": "INCLUDE_BY_DEFAULT",
          "description": "If 'True', zip the job output directory into a single archive before Tapis archiving. NOTE: the value must be defined as a string.",
          "notes": {"isHidden": __isHidden__,
                      "enum_values": [{"True": "True: Zip All Output into a file"},{"False": "False: No Zipping"}]}
        },
        {
          "key": "PATH_MOVE_OUTPUT",
          "value": "",
          "inputMode": "INCLUDE_BY_DEFAULT",
          "description": "Destination path (Absolute and within the Execution System) where outputs will be moved **after** execution. (E.g., '$HOME/OutSet1', '$WORK/OutSet2', '$SCRATCH/OutSet3')",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "PRE_JOB_SCRIPT",
          "value": "",
          "inputMode": "INCLUDE_BY_DEFAULT",
          "description": "Filename of user-defined PRE-JOB script (or absolute path). This file must reside in the Input Directory. It is run after the system has been configured, but before the main binary. (e.g. prehook.sh,$WORK/.../pre-hook.sh)",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "POST_JOB_SCRIPT",
          "value": "",
          "inputMode": "INCLUDE_BY_DEFAULT",
          "description": "Filename of user-defined POST-JOB script (or absolute path). This file must reside in the Input Directory. It is run after the the main binary. (e.g. prehook.sh,$WORK/.../pre-hook.sh)",
          "notes": {"isHidden": __isHidden__}
        }

        
""")

thisFilename = 'app.json'

# ------------------------------------------------------------------
# Configurable knobs for different systems / queues / resources
# ------------------------------------------------------------------

print('exec_system_id_app',exec_system_id_app)

# Default resources (you can override per app / per system)
node_count = 1
if exec_system_id_app == "frontera":
    exec_system_queue = "development" ; # frontera
    cores_per_node = 56
else:
    exec_system_queue = "skx-dev"; # stampede3
    cores_per_node = 48
memory_mb = 192000   # 192 GB
max_minutes = 120    # 2 hours

# Typically archive to the same system, but you can decouple this
# archive_system_id = exec_system_id_app
# archive_system_dir = "HOST_EVAL($WORK)/tapis-jobs-archive/${JobCreateDate}/${JobName}-${JobUUID}"
# use MyData:
archive_system_id = 'designsafe.storage.default'
archive_system_dir = username + "/tapis-jobs-archive/${JobCreateDate}/${JobName}-${JobUUID}"

# Tapis MPI flags (wrapper also has a user-level UseMPI flag)
# the following defines whether the tapis_app.sh is run using mpi, not your script
isMpi = 'false'
if isMpi == 'true':
    mpiCmd = '"ibrun"'
else:
    mpiCmd = 'null'

# Scheduler profile (matches TACC config)
#   Examples:
#   - 'tacc-no-modules'         (no preloaded modules; user must load via MODULE_LOADS_LIST)
#   - 'python312_stampede3'     (if you later create one)
thisSchedulerProfile = 'tacc-no-modules'

isHidden = 'false'

thisText_appJson_Raw = textwrap.dedent("""
{
  "id": "__app_id__",
  "version": "__app_version__",
  "description": "__app_description__",
  "owner": "${apiUserId}",
  "enabled": true,
  "runtime": "ZIP",
  "runtimeVersion": null,
  "runtimeOptions": null,
  "containerImage": "__container_filename_path__",
  "jobType": "BATCH",
  "maxJobs": -1,
  "maxJobsPerUser": -1,
  "strictFileInputs": true,
  "jobAttributes": {
    "execSystemConstraints": null,
    "execSystemId": "__execSystemId__",
    "execSystemExecDir": "${JobWorkingDir}",
    "execSystemInputDir": "${JobWorkingDir}",
    "execSystemOutputDir": "${JobWorkingDir}",
    "execSystemLogicalQueue": "__execSystemLogicalQueue__",
    "archiveSystemId": "__archiveSystemId__",
    "archiveSystemDir": "__archiveSystemDir__",
    "archiveOnAppError": true,
    "isMpi": __isMpi__,
    "mpiCmd": __mpiCmd__,
    "parameterSet": {
      "appArgs": [
        {
          "name": "Main Program",
          "description": "Binary executable to run. (e.g., OpenSees, OpenSeesMP, OpenSeesSP, python3 -- OpenSeesPy: use python3).    The executable must be available in the job's execution system. Some executables require you to load specific modules.",
          "arg": "python3",
          "inputMode": "REQUIRED",
          "notes": {
                      "isHidden": __isHidden__,
                      "enum_values": [{"OpenSees": "OpenSees"},{"OpenSeesMP": "OpenSeesMP"},{"OpenSeesSP": "OpenSeesSP"},{"python3": "Python"}]
                  }
        },
        {
          "name": "Main Script",
          "description": "Filename (no path) of the input script passed to the executable (Example: Ex1a.Canti2D.Push.mpi4py.tacc.py). This file must reside in the Input Directory.  Note: This App uses TACC-Compiled OpenSeesPy: use 'import opensees' or 'import opensees as ops' in your script.",
          "arg": null,
          "inputMode": "REQUIRED",
          "notes": {
                        "inputType": "fileInput",
                        "isHidden": false
                  }
        },
        {
          "name": "UseMPI",
          "description": "Flag indicating whether the application should launch the main program with an MPI parallel-execution command (ibrun). **True**: enable distributed-memory parallelism, allowing multi-core or multi-node execution. (Suitable for OpenSeesMP / OpenSeesSP / Python with mpi4py (OpenSeesPy)). **False**: execution stays on one node. (Suitable for OpenSees, Python, or Python with concurrent.futures for one-node parallelism.)",
          "arg": "False",
          "inputMode": "REQUIRED",
          "notes": {
            "isHidden": false,
            "enum_values": [
              {"True": "True — Enable MPI mode -- Use multi-node or multi-core parallelism."},
              {"False": "False — No MPI -- Use single-node process." }
            ]
          }
        }__COMMANDLINE_ARGS__
      ],
      "containerArgs": [],
      "schedulerOptions": [
        {
          "name": "TACC Scheduler Profile",
          "description": "Scheduler profile (e.g., tacc-no-modules) -- the app loads the modules you specify.",
          "inputMode": "__SchedulerProfile_FIXITY__",
          "arg": "--tapis-profile __schedulerProfile__",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "name": "TACC Reservation",
          "description": "If you have a TACC reservation, enter the reservation string here.",
          "inputMode": "INCLUDE_ON_DEMAND",
          "arg": null,
          "notes": {
              "isHidden": false
          }
        }        
      ],
      "envVariables": [
        {
          "key": "GET_TACC_OPENSEESPY",
          "value": "__GET_TACC_OPENSEESPY_DEFAULT__",
          "inputMode": "INCLUDE_BY_DEFAULT",
          "description": "If 'True', use the TACC-compiled OpenSeesPy (not the PyPI wheel). In your script, import OpenSeesPy using 'import opensees' or 'import opensees as ops'.",
          "notes": {
                      "isHidden": __isHidden__,
                      "enum_values": [{"True": "True: Copy TACC-Compiled OpenSeesPy"},{"False": "False: no TACC-Compiled OpenSeesPy"}]
                    }
        },
        {
          "key": "PIP_INSTALLS_LIST",
          "value": "mpi4py,pandas,numpy,matplotlib,futures",
          "inputMode": "__PIP_INSTALLS_LIST_inputMode__",
          "description": "Comma-separated list of Python packages to pip install before the run. Example: 'numpy,scipy,mpi4py' Defaults:'mpi4py,pandas,numpy,scipy'.",
          "notes": {"isHidden": __isHidden__}
        },
        {
          "key": "MODULE_LOADS_LIST",
          "value": "python/3.12.11,opensees,hdf5/1.14.4,pylauncher",
          "inputMode": "__MODULE_LOADS_LIST_inputMode__",
          "description": "Comma-separated list of TACC modules to load before the run. Defaults: 'opensees,hdf5/1.14.4' 'python/3.12.11' and 'pylauncher' are included if  GET_TACC_OPENSEESPY=True.",
          "notes": {"isHidden": __isHidden__}
        }__ENV_VARS__
      ],
      "archiveFilter": {
        "includes": [],
        "excludes": ["__container_filename__"],
        "includeLaunchFiles": true
      }
    },
    "fileInputs": [
      {
        "name": "Input Directory",
        "inputMode": "__fileInputs_InputDirectory_INPUTMODE__",
        "sourceUrl": null,
        "targetPath": "inputDirectory",
        "envKey": "inputDirectory",
        "description": "Directory containing the main script and any supporting files (models, data, etc.). (Example: tapis://designsafe.storage.community/app_examples/opensees/OpenSeesPy)",
        "notes": {
          "selectionMode": "directory",
          "isHidden": false
        }
      }
    ],
    "fileInputArrays": [],
    "nodeCount": __nodeCount__,
    "coresPerNode": __coresPerNode__,
    "memoryMB": __memoryMB__,
    "maxMinutes": __maxMinutes__,
    "subscriptions": [],
    "tags": []
  },
  "tags": [
    "portalName: DesignSafe",
    "portalName: CEP"
  ],
  "notes": {
    "label": "__app_id__",
    "helpUrl": "__app_helpUrl__",
    "hideNodeCountAndCoresPerNode": false,
    "isInteractive": __isInteractive__,
    "icon": "__icon__",
    "category": "__category__"
  }
}
""")

exec_system_id_app stampede3

# common content
app_icon = 'OpenSees'
app_category = 'Simulation'
app_isInteractive = 'false'
thisText_appJson_Raw = thisText_appJson_Raw.replace("__icon__", app_icon)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__category__", app_category)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__isInteractive__", app_isInteractive)

# System / queue / archive placeholders
thisText_appJson_Raw = thisText_appJson_Raw.replace("__execSystemId__", exec_system_id_app)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__execSystemLogicalQueue__", exec_system_queue)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__archiveSystemId__", archive_system_id)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__archiveSystemDir__", archive_system_dir)

# Resource placeholders
thisText_appJson_Raw = thisText_appJson_Raw.replace("__nodeCount__", str(node_count))
thisText_appJson_Raw = thisText_appJson_Raw.replace("__coresPerNode__", str(cores_per_node))
thisText_appJson_Raw = thisText_appJson_Raw.replace("__memoryMB__", str(memory_mb))
thisText_appJson_Raw = thisText_appJson_Raw.replace("__maxMinutes__", str(max_minutes))

# MPI + scheduler profile
thisText_appJson_Raw = thisText_appJson_Raw.replace("__isMpi__", isMpi)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__mpiCmd__", mpiCmd)
thisText_appJson_Raw = thisText_appJson_Raw.replace("__schedulerProfile__", thisSchedulerProfile)

thisText_appJson_Raw = thisText_appJson_Raw.replace("__fileInputs_InputDirectory_INPUTMODE__", 'REQUIRED')

thisText_Raw = thisText_appJson_Raw

# different settings for the two apps:
MODULE_LOADS_LIST_inputMode_app = 'INCLUDE_ON_DEMAND'
MODULE_LOADS_LIST_inputMode_appOpsPy = 'INCLUDE_BY_DEFAULT'
PIP_INSTALLS_LIST_inputMode_app = 'INCLUDE_ON_DEMAND'
PIP_INSTALLS_LIST_inputMode_appOpsPy = 'INCLUDE_BY_DEFAULT'


__GET_TACC_OPENSEESPY_DEFAULT___app = "False"
__GET_TACC_OPENSEESPY_DEFAULT___appOpsPy = "True"

if do_makeApp:
    thisText_appJson = thisText_Raw   
    # App-Specific Input ----------------
    # Basic placeholder replacements
    thisText_appJson = thisText_appJson.replace("__app_id__", app_id)
    thisText_appJson = thisText_appJson.replace("__app_version__", app_version)
    thisText_appJson = thisText_appJson.replace("__app_description__", app_description)
    thisText_appJson = thisText_appJson.replace(
        "__container_filename_path__",
        f"/{appPath_Tapis}/{container_filename}"
    )
    thisText_appJson = thisText_appJson.replace("__container_filename__", container_filename)

    thisText_appJson = thisText_appJson.replace("__app_helpUrl__", app_helpUrl)
    thisText_appJson = thisText_appJson.replace("__SchedulerProfile_FIXITY__", 'INCLUDE_BY_DEFAULT')

    thisText_appJson = thisText_appJson.replace("__COMMANDLINE_ARGS__", thisText_options_COMMANDLINE_ARGS)
    thisText_appJson = thisText_appJson.replace("__ENV_VARS__", thisText_options_ENV_VARS)    

    thisText_appJson = thisText_appJson.replace("__isHidden__", isHidden)

    thisText_appJson = thisText_appJson.replace("__MODULE_LOADS_LIST_inputMode__", MODULE_LOADS_LIST_inputMode_app)
    thisText_appJson = thisText_appJson.replace("__PIP_INSTALLS_LIST_inputMode__", PIP_INSTALLS_LIST_inputMode_app)
    
    thisText_appJson = thisText_appJson.replace("__GET_TACC_OPENSEESPY_DEFAULT__", __GET_TACC_OPENSEESPY_DEFAULT___app)

    
    
    with open(f"{appPath_Local}/{thisFilename}", "w") as f:
        f.write(thisText_appJson)

    # print('thisText_appJson',thisText_appJson)

if do_makeApp:
    OpsUtils.show_text_file_in_accordion(appPath_Local, [thisFilename], background='#d4fbff', showLineNumbers=False)

if do_makeApp_OpsPy:
    isHidden_OpsPy = 'true'
    
    thisFilename = 'app.json'
    thisText_appJson_OpsPy = thisText_Raw
    # App-Specific Input ----------------
    # Basic placeholder replacements
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__app_id__", app_id_OpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__app_version__", app_version_OpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__app_description__", app_description_OpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace(
        "__container_filename_path__",
        f"/{appPath_Tapis_OpsPy}/{container_filename_OpsPy}"
    )
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__container_filename__", container_filename_OpsPy)

    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__app_helpUrl__", app_helpUrl_OpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__isHidden__", isHidden_OpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__SchedulerProfile_FIXITY__", 'FIXED')
    
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__COMMANDLINE_ARGS__", '')
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__ENV_VARS__", '')

    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__MODULE_LOADS_LIST_inputMode__", MODULE_LOADS_LIST_inputMode_appOpsPy)
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__PIP_INSTALLS_LIST_inputMode__", PIP_INSTALLS_LIST_inputMode_appOpsPy)    
    
    thisText_appJson_OpsPy = thisText_appJson_OpsPy.replace("__GET_TACC_OPENSEESPY_DEFAULT__", __GET_TACC_OPENSEESPY_DEFAULT___appOpsPy)    
    
    with open(f"{appPath_Local_OpsPy}/{thisFilename}", "w") as f:
        f.write(thisText_appJson_OpsPy)

if do_makeApp_OpsPy:
    OpsUtils.show_text_file_in_accordion(appPath_Local, [thisFilename], background='#d4fbff', showLineNumbers=False)

D. Create tapisjob_app.sh – Wrapper Script#

Wrapper script executed by the job; this is the command that launches your code (e.g., runs OpenSeesMP, Python, or a script)

We are braking up this file into individual chunks, each with its own task. We can then choose whether to include each task.

1. Script initialization: safety flags, required args, and global context#

This first block sets up the execution contract for the SLURM wrapper: how it’s called, which environment variables must exist, and some global metadata/timers.

What this block does

A. Safe shell behavior + debug trace#

#!/bin/bash
set -euo pipefail
set -x

#!/bin/bash – run with Bash explicitly.
set -e – exit immediately if any command returns a non-zero status.
set -u – treat use of unset variables as an error.
set -o pipefail – if any command in a pipeline fails, the whole pipeline fails.
set -x – print each command before executing it (very helpful for debugging SLURM jobs).

Together, these make the script fail fast and visibly instead of silently limping along with partial state.

B. App metadata (filled in by the template)#

echo "  App_Id            : __app_id__"
echo "  App_Version       : __app_version__"
echo "  App_Description   : __app_description__"

These placeholders are filled by the app definition (app.json / template). Printing them at the top:

Confirms which app/version is actually running.
Helps when scanning raw job output or debugging “which app did I launch?”

C. Required positional arguments#

BINARYNAME="${1:?missing binary name}"
INPUTSCRIPT0="${2:?missing input script}"
UseMPI="${3:?missing mpi-call switch}"
shift 3

The wrapper requires three positional arguments:

BINARYNAME – the executable to run e.g., OpenSees, OpenSeesMP, or python3.
INPUTSCRIPT0 – the path to the input script e.g., models/bridge.tcl or analysis.py.
UseMPI – a flag indicating whether to use MPI (later interpreted as true/false-like in the launcher logic).

The :? syntax enforces these as mandatory: if any is missing, the script aborts with a clear error like missing binary name. shift 3 then removes these from $@, leaving only user/script arguments in $*.

You also log:

echo "ARGS: $*"

so you can see the remaining command-line arguments passed through to the binary.

D. Environment-derived parameters#

INPUTSCRIPT="${INPUTSCRIPT0##*/}"
echo "INPUTSCRIPT: $INPUTSCRIPT"

inputDirectory="${inputDirectory:?inputDirectory not set}"
echo "inputDirectory: $inputDirectory"

INPUTSCRIPT is normalized to just the basename of the input file (foo.tcl instead of path/to/foo.tcl). This is the name you actually run inside the working directory.
inputDirectory is required to be set in the environment (from the Tapis app / job definition). If it’s missing, the script fails early with inputDirectory not set.

This clearly separates:

Where the input bundle lives (inputDirectory), from
Which file inside that directory is the main driver (INPUTSCRIPT).

E. Job metadata from Tapis#

JobUUID="${_tapisJobUUID:-}"
echo "JobUUID: ${JobUUID}"

Pulls the Tapis job UUID from _tapisJobUUID if present.
Logs it so:
- You can correlate this run with Tapis records,
- Later blocks (like PATH_MOVE_OUTPUT) can use JobUUID to create per-job output directories.

F. Remember the script’s starting directory#

SCRIPT_ROOT_DIR="$(pwd)"

This captures the directory where the wrapper started, which you later use to:

Anchor the summary log (SUMMARY_SHORT),
Normalize relative paths provided by the user,
Reason about where the job “began” vs. where it might cd during execution.

G. Normalize Python binary name#

if [[ "$BINARYNAME" == "python3" || "$BINARYNAME" == "python" ]]; then
    echo " -- overwrite python with python3, if needed --"
    BINARYNAME="python3"
    python -V || true
    python3 -V || true
fi

If the caller passed either python or python3:

You force BINARYNAME=”python3” to avoid ambiguity. This makes the environment consistent and avoids issues with different python symlinks.
You print both python -V and python3 -V (without failing if they’re missing) so the logs show exactly which Python interpreters are visible on the path.

This is especially important for OpenSeesPy and other Python-based workflows, where the exact Python version matters.

H. Start the “total script” timer#

TOTAL_START_EPOCH=$(date +%s)
TOTAL_START_HUMAN="$(date)"

These mark the beginning of the entire job wrapper’s lifetime:

TOTAL_START_EPOCH – numeric timestamp for precise duration calculations.
TOTAL_START_HUMAN – human-readable timestamp for the summary log.

Later echoTimers blocks use this to report:

How long the full script ran (setup + run + post-processing),
Both in h/m/s and in raw seconds, for both normal completion and error exits.

bash_script_run_INITIALIZE = textwrap.dedent("""
    #!/bin/bash
    set -euo pipefail
    set -x

    # ---- app written by __app_Author_Info__ ----
    echo "  App_Id            : __app_id__"
    echo "  App_Version       : __app_version__"
    echo "  App_Description   : __app_description__"
    
    echo " ---- required args ---- "
    echo
    BINARYNAME="${1:?missing binary name}"
    INPUTSCRIPT0="${2:?missing input script}"
    UseMPI="${3:?missing mpi-call switch}"
    shift 3

    echo "ARGS: $*"

    echo " ---- env params ---- "
    INPUTSCRIPT="${INPUTSCRIPT0##*/}"
    echo "INPUTSCRIPT: $INPUTSCRIPT"
    inputDirectory="${inputDirectory:?inputDirectory not set}"
    echo "inputDirectory: $inputDirectory"
    
    # -- Job info
    JobUUID="${_tapisJobUUID:-}"
    echo "JobUUID: ${JobUUID}"

    SCRIPT_ROOT_DIR="$(pwd)"

    # Normalize python binary name
    if [[ "$BINARYNAME" == "python3" || "$BINARYNAME" == "python" ]]; then
        echo " -- overwrite python with python3, if needed --"
        BINARYNAME="python3"
        python -V || true
        python3 -V || true
    fi

    # ---- TIMERS: total script ----
    TOTAL_START_EPOCH=$(date +%s)
    TOTAL_START_HUMAN="$(date)"
""")

2. Summary Log Setup#

This block sets up a compact log files for each job run.

What this block does

SUMMARY_SHORT (default: SLURM-job-summary.log in SCRIPT_ROOT_DIR): A compact, human-focused summary pinned to the directory where the SLURM script starts (SCRIPT_ROOT_DIR).
- If SUMMARY_SHORT is not set, it is initialized to ${SCRIPT_ROOT_DIR}/SLURM-job-summary.log.
- If the user provides a relative path, it is converted into an absolute path under SCRIPT_ROOT_DIR. This guarantees that the summary log always lives in a predictable location associated with the job’s starting directory.

After resolving these paths, the script:

Echoes the chosen log file locations to stdout so the user immediately sees where logs will be written.
Initializes the compact summary log with a banner and basic app metadata:
- App ID, version, description, and help URL (filled in by template placeholders such as app_id, app_version, etc.).
Records key system paths ($HOME, $WORK, $SCRATCH) and the SLURM-SCRIPT_ROOT_DIR, providing a quick reference for where job data may live.
Prints a user-configuration summary, including:
- JobUUID (the Tapis/Job identifier),
- inputDirectory,
- INPUTSCRIPT,
- UseMPI,
- any additional argument/parameter details (echoSummary_ARGS),
- any relevant environment variable summaries (echoSummary_ENV_VARS).
Logs timing and environment info, including:
- A pointer to the full environment log,
- The overall job start time in both human-readable and epoch formats.

The complete list of the environment variables is, optionally, logged in the verbose script, shown next.

bash_script_echoSummary_START = textwrap.dedent("""

    echo "=start=============================================================="
    echo "===================== SUMMARY-LOG SETUP ==========================="
    echo "==================================================================="
    # Compact, human-focused summary (default name below)
    # SUMMARY_SHORT="${SUMMARY_SHORT:-./SLURM-job-summary.log}"
   
    # Compact, human-focused summary (default name below), pinned to start dir
    if [[ -z "${SUMMARY_SHORT:-}" ]]; then
      SUMMARY_SHORT="${SCRIPT_ROOT_DIR}/SLURM-job-summary.log"
    elif [[ "${SUMMARY_SHORT}" != /* ]]; then
      # If user gave a relative path, make it absolute from the start dir
      SUMMARY_SHORT="${SCRIPT_ROOT_DIR}/${SUMMARY_SHORT}"
    fi

    echo "Compact summary log: ${SUMMARY_SHORT}"  
    
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "================== JOB SUMMARY ====================================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "  App Id            : __app_id__" >> "$SUMMARY_SHORT"
    echo "  App Version       : __app_version__" >> "$SUMMARY_SHORT"
    echo "  App Description   : __app_description__" >> "$SUMMARY_SHORT"
    echo "  App helpURL       : __app_helpUrl__" >> "$SUMMARY_SHORT"
    echo "================== SYSTEM-PATH DEFINITIONS ========================" >> "$SUMMARY_SHORT"
    printf '  $HOME    : %s\n'   "${HOME}" >> "$SUMMARY_SHORT"
    printf '  $WORK    : %s\n'   "${WORK:-<unset>}" >> "$SUMMARY_SHORT"
    printf '  $SCRATCH : %s\n'   "${SCRATCH:-<unset>}" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "  SLURM-SCRIPT_ROOT_DIR   : ${SCRIPT_ROOT_DIR}" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "================= USER-CONFIGURATION SUMMARY ======================" >> "$SUMMARY_SHORT"
    echo "JobUUID        : ${JobUUID}" >> "$SUMMARY_SHORT"
    echo "inputDirectory : ${inputDirectory}" >> "$SUMMARY_SHORT"
    echo "INPUTSCRIPT    : ${INPUTSCRIPT}" >> "$SUMMARY_SHORT"
    echo "UseMPI     : ${UseMPI}" >> "$SUMMARY_SHORT"
    __echoSummary_ARGS__
    echo "============== APP-DEFINED ENVIRONMENT VALUES ==+==================" >> "$SUMMARY_SHORT"
    echo "  MODULE_LOADS_LIST   : ${MODULE_LOADS_LIST:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  PIP_INSTALLS_LIST   : ${PIP_INSTALLS_LIST:-<unset>}" >> "$SUMMARY_SHORT"
    __echoSummary_ENV_VARS__
    __echoSummary_MPI__
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "Environment: see full-env log for full env dump" >> "$SUMMARY_SHORT"
    echo "Total start time: ${TOTAL_START_HUMAN} (epoch ${TOTAL_START_EPOCH})" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"

    echo "=end===============================================================" >> "$SUMMARY_SHORT"

""")

3. Environment Log Setup#

This block configures how the job’s runtime environment is captured for debugging and reproducibility, while avoiding excessive clutter in the main SLURM output file.

What this block does

The script separates environment logging into two optional files:

REDACTED_ENV_LOG (default: `./SLURM-environment.redacted.log`)#

This is the default and recommended environment snapshot.

It creates a sorted dump of the full environment (env | sort) but:

Redacts sensitive variables (e.g., variables containing TOKEN, SECRET, PASSWORD, KEY, AWS, etc.).
Masks credentials embedded in URLs (e.g., scheme://user:pass@host → scheme://<REDACTED>@host).
Writes the output to a dedicated file.
Does not print the environment to stdout.

This keeps your SLURM job output clean while preserving a reproducible and security-conscious runtime snapshot.

You can disable it by setting:

REDACTED_ENV_LOG=""

FULL_ENV_LOG (default: disabled)#

If explicitly enabled, this writes the complete, unredacted environment to a separate file:

FULL_ENV_LOG=./my-full-env.log

Because this may contain credentials or tokens, it is disabled by default and should only be used for deep debugging.

Additional Behavior#

The environment is written only to log files — it is no longer printed to the main job stdout, preventing excessive noise in .out files.
Log files are created with restricted permissions (umask 077) to reduce accidental exposure.
The summary log (SUMMARY_SHORT) records which environment logs were created.

Why Environment Logging Matters in HPC#

On HPC systems (such as Stampede3 at TACC), your runtime environment is dynamically constructed at job launch. It may include:

Loaded modules and toolchains
MPI and compiler versions
SLURM-provided variables
Scratch paths and allocation settings
Software stack adjustments made by Tapis

Small changes in modules, paths, or compiler/MPI versions can alter numerical results, performance, or even job behavior. Capturing the environment ensures that:

Runs are reproducible
Differences between jobs can be diagnosed
Toolchain or module changes can be traced
Support teams can debug issues efficiently

Together, the summary log and the redacted environment log provide:

A lightweight, human-readable job summary
A reproducible runtime snapshot
Improved security
Cleaner SLURM output

bash_script_echoSummary_VERBOSE = textwrap.dedent(r"""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "===================== ENVIRONMENT-LOG SETUP =======================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"

    # --- files (set to "" to disable) ---
    # Redacted env log is the safe default
    REDACTED_ENV_LOG="${REDACTED_ENV_LOG:-./SLURM-environment.redacted.log}"

    # Full env log is OFF by default (enable only if you really want it)
    FULL_ENV_LOG="${FULL_ENV_LOG:-}"

    # Create files as private as possible (env can contain tokens)
    umask 077

    # Regex (case-insensitive) for variable names that should be redacted
    # Tune as needed for your environment.
    REDACT_ENV_NAME_REGEX='(TOKEN|SECRET|PASSWORD|PASS|PWD|KEY|API|AUTH|BEARER|COOKIE|CREDENTIAL|PRIVATE|SSH|AWS|AZURE|GCP|GOOGLE|SLACK|GITHUB|GITLAB|JWT|SAS|SIGNATURE|SESSION|SENTRY|MONGO|DBPASS|DB_PASSWORD|DATABASE_URL|CONNECTION_STRING)'

    redact_env_stream () {
      # Reads KEY=VALUE lines, outputs redacted KEY=... when KEY matches regex,
      # and also masks credentials embedded in URLs like scheme://user:pass@host
      awk -v re="$REDACT_ENV_NAME_REGEX" '
        BEGIN { IGNORECASE=1 }
        {
          line=$0
          split(line, a, "=")
          key=a[1]
          val=substr(line, length(key)+2)

          # redact by variable name
          if (key ~ re) {
            print key "=<REDACTED>"
            next
          }

          # redact creds embedded in URLs: scheme://user:pass@host -> scheme://<REDACTED>@host
          gsub(/:\/\/[^\/:@]+:[^\/@]+@/, "://<REDACTED>@", line)

          print line
        }
      '
    }

    if [[ -n "${REDACTED_ENV_LOG}" ]]; then
      echo "Redacted environment will be written to: ${REDACTED_ENV_LOG}" >> "$SUMMARY_SHORT"
      {
        echo "==================================================================="
        echo "REDACTED ENVIRONMENT DUMP (env | sort)"
        echo "Generated: $(date -Is)"
        echo "Redaction rule: names matching /${REDACT_ENV_NAME_REGEX}/ -> <REDACTED>"
        echo "Also masks URL credentials like scheme://user:pass@host"
        echo "==================================================================="
        env | sort | redact_env_stream
        echo "==================================================================="
      } > "${REDACTED_ENV_LOG}"
    else
      echo "Redacted environment dump disabled (REDACTED_ENV_LOG is empty)." >> "$SUMMARY_SHORT"
    fi

    if [[ -n "${FULL_ENV_LOG}" ]]; then
      echo "WARNING: full environment dump ENABLED: ${FULL_ENV_LOG}" >> "$SUMMARY_SHORT"
      {
        echo "==================================================================="
        echo "FULL ENVIRONMENT DUMP (env | sort)"
        echo "Generated: $(date -Is)"
        echo "==================================================================="
        env | sort
        echo "==================================================================="
      } > "${FULL_ENV_LOG}"
    else
      echo "Full environment dump not enabled (FULL_ENV_LOG is empty)." >> "$SUMMARY_SHORT"
    fi

    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

4. Argument and environment-variable summary helpers#

These two small snippets are plugged into the main summary block via the ‘echoSummary_ARGS’ and ‘echoSummary_ENV_VARS’ placeholders. They keep the Jupyter-generated script readable while still providing a rich job summary.

4a. Log App (Arguments bash_script_echoSummary_ARGS)#

This helper records what is actually being run and with which arguments:

What this block does

* BINARYNAME The executable or driver being launched (e.g., OpenSees, OpenSeesMP, python, etc.). Capturing this is useful when you have multiple entry points or versions and want to verify which one this job used.

ARGS (‘$*’) The full, space-separated list of command-line arguments passed into the app’s main binary. This gives a single, human-readable line summarizing the effective runtime configuration (input file, flags, options) as seen by the executable.

Together these two lines give you a quick “command line snapshot” for reproducing the run.

bash_script_echoSummary_ARGS = textwrap.dedent("""
    echo "BINARYNAME     : ${BINARYNAME}" >> "$SUMMARY_SHORT"
    echo "ARGS           : $*" >> "$SUMMARY_SHORT"
""")

4b. Log Environment Variable#

This helper captures higher-level environment knobs that control how the job environment is prepared, plus some optional MPI/SLURM diagnostics.

What this block does

Environment “knobs” (each shows **** if not defined):

MODULE_LOADS_LIST / MODULE_LOADS_FILE Describe which environment modules (e.g., hdf5, opensees) should be loaded. One is for inline lists; the other can point to a file listing modules.
PIP_INSTALLS_LIST / PIP_INSTALLS_FILE Optional Python package installs to perform at runtime (inline list vs. file-driven). This is useful for lightweight, job-specific Python environments.
GET_TACC_OPENSEESPY Switch/flag indicating whether to fetch a TACC-provided OpenSeesPy setup.
UNZIP_FILES_LIST Files or archives to unzip before execution (e.g., input bundles).
PATH_COPY_IN_LIST Paths to copy into the job’s working directory prior to running the app.
DELETE_COPIED_IN_ON_EXIT If set to a true-like value, removes files or directories that were copied into the job working directory via PATH_COPY_IN_LIST after the job completes, preventing temporary inputs from being included in the final archive..
ZIP_OUTPUT_SWITCH Controls whether output should be zipped at the end of the job.
PATH_MOVE_OUTPUT Destination path to move packaged output to (e.g., a work or archive directory).

These lines make it very easy to see, after the fact, how the environment and I/O preparation were configured for a given run.

bash_script_echoSummary_ENV_VARS = textwrap.dedent("""
    echo "  MODULE_LOADS_FILE        : ${MODULE_LOADS_FILE:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  PIP_INSTALLS_FILE        : ${PIP_INSTALLS_FILE:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  GET_TACC_OPENSEESPY      : ${GET_TACC_OPENSEESPY:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  UNZIP_FILES_LIST         : ${UNZIP_FILES_LIST:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  PATH_COPY_IN_LIST        : ${PATH_COPY_IN_LIST:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  DELETE_COPIED_IN_ON_EXIT : ${DELETE_COPIED_IN_ON_EXIT:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  ZIP_OUTPUT_SWITCH        : ${ZIP_OUTPUT_SWITCH:-<unset>}" >> "$SUMMARY_SHORT"
    echo "  PATH_MOVE_OUTPUT         : ${PATH_MOVE_OUTPUT:-<unset>}" >> "$SUMMARY_SHORT"
""")

5. Log MPI/SLURM diagnostics#

The block then prints some optional MPI info, purely for logging.

What this block does

It computes a rank (RANK) and world size (SIZE) in a robust way, checking common environment variables from:
- PMI (PMI_RANK, PMI_SIZE),
- OpenMPI (OMPI_COMM_WORLD_RANK, OMPI_COMM_WORLD_SIZE),
- SLURM (SLURM_PROCID, SLURM_NTASKS).
It also records the short hostname (HOST).

These values are then written to SUMMARY_SHORT with explanatory messages. As the inline comment notes, these MPI-derived values are not used to control the app—they’re just there to document how the job “looks” from an MPI/SLURM perspective, which can be very helpful when debugging parallel vs. non-parallel runs or Slurm task layouts.

bash_script_echoSummary_MPI = textwrap.dedent("""
    echo "================= MPI INFO (optional info) =======================" >> "$SUMMARY_SHORT"
    echo "# -- mpi info: pick up rank and world size across IMPI/OpenMPI/SLURM. none of these are used by the app/SLURM job!" >> "$SUMMARY_SHORT"
    RANK=${PMI_RANK:-${OMPI_COMM_WORLD_RANK:-${SLURM_PROCID:-0}}}
    SIZE=${PMI_SIZE:-${OMPI_COMM_WORLD_SIZE:-${SLURM_NTASKS:-1}}}
    HOST=$(hostname -s)
    echo "RANK: $RANK -- RANK should be zero since it belongs to the SLURM-JOB-APP, which is not run via an MPI!" >> "$SUMMARY_SHORT"
    echo "of SIZE: $SIZE" >> "$SUMMARY_SHORT"
    echo "on Host: $HOST" >> "$SUMMARY_SHORT"
    echo "MPI rank/size: $RANK / $SIZE" >> "$SUMMARY_SHORT"
""")

6. Error-path timing summary (run vs. total)#

This block computes and records timing information when the job exits via an error path. It assumes that RUN_START_EPOCH and TOTAL_START_EPOCH were captured earlier in the script.

What this block does

A. **Capture end times**

RUN_END_EPOCH=$(date +%s)
RUN_END_HUMAN="$(date)"
...
TOTAL_END_EPOCH=$(date +%s)
TOTAL_END_HUMAN="$(date)"

RUN_END_EPOCH / RUN_END_HUMAN The end time (in seconds since epoch and human-readable form) for just the binary run portion of the job (e.g., OpenSees / OpenSeesMP execution window).
TOTAL_END_EPOCH / TOTAL_END_HUMAN The end time for the entire script, including setup, pre/post-processing, and the binary run.

B. Compute durations and split into h/m/s

RUN_DURATION=$(( RUN_END_EPOCH - RUN_START_EPOCH ))
RH=$(( RUN_DURATION / 3600 ))
RM=$(( (RUN_DURATION % 3600) / 60 ))
RS=$(( RUN_DURATION % 60 ))

TOTAL_DURATION=$(( TOTAL_END_EPOCH - TOTAL_START_EPOCH ))
TH=$(( TOTAL_DURATION / 3600 ))
TM=$(( (TOTAL_DURATION % 3600) / 60 ))
TS=$(( TOTAL_DURATION % 60 ))

RUN_DURATION is the elapsed time (in seconds) for the binary run only. It’s then decomposed into RH (hours), RM (minutes), and RS (seconds).
TOTAL_DURATION is the elapsed time for the full script lifetime, likewise decomposed into TH, TM, and TS.

This split makes it easy to glance at both the total runtime and the heavy compute portion.

C. Log timing details to the summary (on error)

echo "Run end time (on error): ${RUN_END_HUMAN}" >> "$SUMMARY_SHORT"
echo "Run end epoch (on error): ${RUN_END_EPOCH}" >> "$SUMMARY_SHORT"
echo "Binary run runtime (on error): ${RH}h ${RM}m ${RS}s (${RUN_DURATION} seconds)" >> "$SUMMARY_SHORT"
...
echo "Total script runtime (on error): ${TH}h ${TM}m ${TS}s (${TOTAL_DURATION} seconds)" >> "$SUMMARY_SHORT"

All values are appended to SUMMARY_SHORT with explicit “(on error)” annotations, so the user knows this timing snapshot comes from an abnormal termination path rather than the normal job footer.

The binary runtime lines answer: “How long did the core application actually run before failing?”
The total script runtime lines answer: “How long was this SLURM job alive in total, including setup and teardown?”

This error-timer block gives a clear post-mortem view of when the failure occurred and how much wall-clock time was spent in the main compute segment versus the overall job.

bash_script_echoTimers_START = textwrap.dedent("""

        echo "=start==============================================================" >> "$SUMMARY_SHORT"
        echo "===================== TIMERS AFTER RUN ============================" >> "$SUMMARY_SHORT"
        echo "===================================================================" >> "$SUMMARY_SHORT"
        # ---- TIMERS: run + total on error ----
        RUN_END_EPOCH=$(date +%s)
        RUN_END_HUMAN="$(date)"
        RUN_DURATION=$(( RUN_END_EPOCH - RUN_START_EPOCH ))
        RH=$(( RUN_DURATION / 3600 ))
        RM=$(( (RUN_DURATION % 3600) / 60 ))
        RS=$(( RUN_DURATION % 60 ))
    
        TOTAL_END_EPOCH=$(date +%s)
        TOTAL_END_HUMAN="$(date)"
        TOTAL_DURATION=$(( TOTAL_END_EPOCH - TOTAL_START_EPOCH ))
        TH=$(( TOTAL_DURATION / 3600 ))
        TM=$(( (TOTAL_DURATION % 3600) / 60 ))
        TS=$(( TOTAL_DURATION % 60 ))
    
    
        echo "==============" >> "$SUMMARY_SHORT"
        echo "Run end time (on error): ${RUN_END_HUMAN}" >> "$SUMMARY_SHORT"
        echo "Run end epoch (on error): ${RUN_END_EPOCH}" >> "$SUMMARY_SHORT"
        echo "Binary run runtime (on error): ${RH}h ${RM}m ${RS}s (${RUN_DURATION} seconds)" >> "$SUMMARY_SHORT"
        echo "" >> "$SUMMARY_SHORT"
        echo "Total end time (on error): ${TOTAL_END_HUMAN}" >> "$SUMMARY_SHORT"
        echo "Total end epoch (on error): ${TOTAL_END_EPOCH}" >> "$SUMMARY_SHORT"
        echo "Total script runtime (on error): ${TH}h ${TM}m ${TS}s (${TOTAL_DURATION} seconds)" >> "$SUMMARY_SHORT"
        echo "==============" >> "$SUMMARY_SHORT"
        echo "=end===============================================================" >> "$SUMMARY_SHORT"

""")

7. Success-path timing summary (binary run only)#

This block records how long the main binary ran when it completes successfully. It assumes RUN_START_EPOCH was set earlier, right before launching the main executable.

What this block does

1. **Capture end time of the binary run**

RUN_END_EPOCH=$(date +%s)
RUN_END_HUMAN="$(date)"

RUN_END_EPOCH is the end time in seconds since the Unix epoch.
RUN_END_HUMAN is the same time in a human-readable string (from date).

Compute elapsed runtime and split into h/m/s

RUN_DURATION=$(( RUN_END_EPOCH - RUN_START_EPOCH ))
RH=$(( RUN_DURATION / 3600 ))
RM=$(( (RUN_DURATION % 3600) / 60 ))
RS=$(( RUN_DURATION % 60 ))

RUN_DURATION is the total elapsed time (in seconds) for the main binary run.
RH, RM, RS break that into hours, minutes, and seconds for easier reading.

Append a concise timing block to the summary log

echo "==============" >> "$SUMMARY_SHORT"
echo "Run end time: ${RUN_END_HUMAN}" >> "$SUMMARY_SHORT"
echo "Run end epoch: ${RUN_END_EPOCH}" >> "$SUMMARY_SHORT"
echo "Binary run runtime: ${RH}h ${RM}m ${RS}s (${RUN_DURATION} seconds)" >> "$SUMMARY_SHORT"
echo "==============" >> "$SUMMARY_SHORT"

These lines write a small, clearly delimited footer to SUMMARY_SHORT that tells you:

When the binary finished (Run end time, both human and epoch),
How long it ran (Binary run runtime in h:m:s and raw seconds).

Unlike the error-path timer block, this snippet is used for the normal, successful completion of the binary run and focuses only on the binary’s runtime, not the total script lifetime.

bash_script_echoTimers_AFTER = textwrap.dedent("""

    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "===================== TIMERS AT END RUN ===========================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    # ---- TIMER: binary run success ----
    RUN_END_EPOCH=$(date +%s)
    RUN_END_HUMAN="$(date)"
    RUN_DURATION=$(( RUN_END_EPOCH - RUN_START_EPOCH ))
    RH=$(( RUN_DURATION / 3600 ))
    RM=$(( (RUN_DURATION % 3600) / 60 ))
    RS=$(( RUN_DURATION % 60 ))


    echo "==============" >> "$SUMMARY_SHORT"
    echo "Run end time: ${RUN_END_HUMAN}" >> "$SUMMARY_SHORT"
    echo "Run end epoch: ${RUN_END_EPOCH}" >> "$SUMMARY_SHORT"
    echo "Binary run runtime: ${RH}h ${RM}m ${RS}s (${RUN_DURATION} seconds)" >> "$SUMMARY_SHORT"
    echo "==============" >> "$SUMMARY_SHORT"

    echo "=end===============================================================" >> "$SUMMARY_SHORT"

""")

8. Final total-runtime footer (successful script completion)#

This block adds a “job is fully done” footer to the summary log, capturing how long the entire SLURM script ran when it exits cleanly.

What this block does

Visual separator

echo "===============================================================" >> "$SUMMARY_SHORT"

Adds a strong horizontal divider so it’s obvious where the final timing section begins.

Capture total script end time

TOTAL_END_EPOCH=$(date +%s)
TOTAL_END_HUMAN="$(date)"
TOTAL_DURATION=$(( TOTAL_END_EPOCH - TOTAL_START_EPOCH ))
TH=$(( TOTAL_DURATION / 3600 ))
TM=$(( (TOTAL_DURATION % 3600) / 60 ))
TS=$(( TOTAL_DURATION % 60 ))

TOTAL_END_EPOCH / TOTAL_END_HUMAN: When the script fully finishes (in epoch seconds and human-readable form).
TOTAL_DURATION: Wall-clock time from TOTAL_START_EPOCH (set near the top of the script) to final exit — this includes:
- Environment/module setup
- Any pre-processing
- The main binary run
- Any post-processing and packaging
TH, TM, TS: The total duration broken into hours, minutes, and seconds.

Write a clear “script is done” block

echo "==============" >> "$SUMMARY_SHORT"
echo "Total end time: ${TOTAL_END_HUMAN}" >> "$SUMMARY_SHORT"
echo "Total end epoch: ${TOTAL_END_EPOCH}" >> "$SUMMARY_SHORT"
echo "Total script runtime (including setup + post): ${TH}h ${TM}m ${TS}s (${TOTAL_DURATION} seconds)" >> "$SUMMARY_SHORT"
echo "DONE!!!" >> "$SUMMARY_SHORT"
echo "==============" >> "$SUMMARY_SHORT"

This footer:

Marks the final end time of the job,
Reports the full script runtime, explicitly noting that it includes setup and post-processing,
Finishes with a loud, human-readable “DONE!!!” so that a quick scan of the summary log immediately confirms a successful, end-to-end completion.

bash_script_echoTimers_END = textwrap.dedent("""

    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "===================== TIMERS AT END TOTAL =========================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"

    # ---- TIMER: total script success ----
    TOTAL_END_EPOCH=$(date +%s)
    TOTAL_END_HUMAN="$(date)"
    TOTAL_DURATION=$(( TOTAL_END_EPOCH - TOTAL_START_EPOCH ))
    TH=$(( TOTAL_DURATION / 3600 ))
    TM=$(( (TOTAL_DURATION % 3600) / 60 ))
    TS=$(( TOTAL_DURATION % 60 ))

    echo "==============" >> "$SUMMARY_SHORT"
    echo "Total end time: ${TOTAL_END_HUMAN}" >> "$SUMMARY_SHORT"
    echo "Total end epoch: ${TOTAL_END_EPOCH}" >> "$SUMMARY_SHORT"
    echo "Total script runtime (including setup + post): ${TH}h ${TM}m ${TS}s (${TOTAL_DURATION} seconds)" >> "$SUMMARY_SHORT"
    echo "DONE!!!" >> "$SUMMARY_SHORT"
    echo "==============" >> "$SUMMARY_SHORT"
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
    
""")

9. Optional pre-run copy of input files/directories (with optional cleanup)#

This block implements an optional “copy-in” step that pulls files or directories into the current working directory before the main run, and (optionally) removes those copied items after the job completes.

The behavior is entirely driven by environment variables and requires no changes to the application code.

PATH_COPY_IN_LIST — what to copy in
DELETE_COPIED_IN_ON_EXIT — whether copied items should be deleted at job end

What this block does

A. Section header in the summary log

echo "===================================================================" >> "$SUMMARY_SHORT"
echo " ---- copy files or directories over, optional ---- " >> "$SUMMARY_SHORT"

This simply marks, in SUMMARY_SHORT, that a copy-in phase may occur.

B. Initialize copy-in tracking and cleanup controls

COPY_IN_MANIFEST="${COPY_IN_MANIFEST:-.tapis_copy_in_manifest.txt}"
DELETE_COPIED_IN_ON_EXIT="${DELETE_COPIED_IN_ON_EXIT:-0}"
: > "${COPY_IN_MANIFEST}"

COPY_IN_MANIFEST records exactly which paths were copied into the working directory during this job.
The manifest is created (or cleared) at the start of the run.
Cleanup is opt-in and only occurs if:
```
DELETE_COPIED_IN_ON_EXIT=1
```

This ensures deletion is explicit, traceable, and reproducible.

C. Register a cleanup handler (runs on success or failure)

trap cleanup_copied_in EXIT

A cleanup function is registered using a Bash EXIT trap, meaning it runs:

after a successful job
after an application error
after an unexpected script failure

This guarantees consistent cleanup behavior whenever it is enabled.

D. Check whether any paths were requested

if [[ -n "${PATH_COPY_IN_LIST:-}" ]]; then

If PATH_COPY_IN_LIST is unset or empty, the entire block is skipped.
If set, it must be a comma-separated list of paths to files or directories to copy into the working directory.

Example:

PATH_COPY_IN_LIST="/work2/data/mesh,/scratch/configs,input.dat"

E. Parse the comma-separated list

IFS=',' read -ra _copy_items <<< "${PATH_COPY_IN_LIST}"

This splits the list into an array (_copy_items) so each entry can be handled independently.

F. Iterate over requested paths (trimming, validation, copy, and tracking)

for _src in "${_copy_items[@]}"; do
  # Trim leading/trailing whitespace
  _src="${_src#"${_src%%[![:space:]]*}"}"
  _src="${_src%"${_src##*[![:space:]]}"}"

  [[ -z "${_src}" ]] && continue

  if [[ -e "${_src}" ]]; then
    rsync -av -- "${_src}" .
    _base="$(basename -- "${_src}")"
    printf '%s\n' "${_base}" >> "${COPY_IN_MANIFEST}"
    echo "Copied in from: ${_src} -> $(pwd)" >> "$SUMMARY_SHORT"
  else
    echo "WARNING: path to copy does not exist: ${_src}"
    echo "WARNING: path to copy does not exist: ${_src}" >> "$SUMMARY_SHORT"
  fi
done

For each candidate path:

• Whitespace trimming

Leading and trailing spaces are removed, allowing clean usage like:

```bash
PATH_COPY_IN_LIST="input1, input2, /some/other/dir"
```

• Skip empty entries

Trailing commas or consecutive commas produce empty entries, which are safely ignored.

• Existence check

* If the source exists:

  * `rsync -av` copies it into the **current working directory (`.`)**.
  * Both files and directories are supported.
  * The **destination name** (basename) is written to the copy-in manifest for potential cleanup.
  * The action is logged to `SUMMARY_SHORT`.

* If the source does not exist:

  * A warning is printed to stdout and recorded in the summary log for diagnostics.

G. Optional cleanup at job completion

If the user enables cleanup:

DELETE_COPIED_IN_ON_EXIT=1

the cleanup handler:

reads the copy-in manifest
deletes only the paths that were copied in
refuses to delete:
- absolute paths
- paths containing ..
- anything outside the working directory

Each deletion is logged to SUMMARY_SHORT, ensuring transparency and auditability.

Why this pattern is used

This approach provides:

Flexible input staging without hard-coding paths into the app
Fast local access to large or frequently accessed files
Reproducibility via explicit environment variables
Safety via manifest-based deletion
Clean job directories when temporary inputs are no longer needed

It is especially useful for large meshes, auxiliary scripts, configuration directories, or scratch-only inputs that should not persist beyond the job lifecycle.

bash_script_option_COPY_FILES = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=========== PRE-RUN COPY OF INPUT FILES/DIRECTORIES ===============" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"


    COPY_IN_MANIFEST="${COPY_IN_MANIFEST:-.copy_in_manifest.txt}"
    DELETE_COPIED_IN_ON_EXIT="${DELETE_COPIED_IN_ON_EXIT:-0}"
    
    # Create/clear manifest for this run
    : > "${COPY_IN_MANIFEST}"
    
    cleanup_copied_in() {
      # Only delete if explicitly enabled
      [[ "${DELETE_COPIED_IN_ON_EXIT}" == "1" ]] || return 0
    
      # Only act if manifest exists and is non-empty
      [[ -s "${COPY_IN_MANIFEST}" ]] || return 0
    
      echo "Cleanup enabled: deleting copied-in items listed in ${COPY_IN_MANIFEST}" | tee -a "$SUMMARY_SHORT"
    
      # Delete only paths inside the current working directory
      while IFS= read -r _rel; do
        [[ -z "${_rel}" ]] && continue
    
        # Safety: disallow absolute paths and parent traversal
        if [[ "${_rel}" = /* ]] || [[ "${_rel}" == *".."* ]]; then
          echo "WARNING: refusing to delete suspicious path from manifest: ${_rel}" | tee -a "$SUMMARY_SHORT"
          continue
        fi
    
        # Safety: ensure it actually exists in $PWD
        if [[ -e "${_rel}" ]]; then
          rm -rf -- "${_rel}"
          echo "Deleted copied-in item: ${_rel}" | tee -a "$SUMMARY_SHORT"
        fi
      done < "${COPY_IN_MANIFEST}"
    }
    
    # Ensure cleanup runs on exit (success or failure)
    trap cleanup_copied_in EXIT




    echo " ---- copy files or directories over, optional ---- " >> "$SUMMARY_SHORT"
    if [[ -n "${PATH_COPY_IN_LIST:-}" ]]; then
      IFS=',' read -ra _copy_items <<< "${PATH_COPY_IN_LIST}"

      for _src in "${_copy_items[@]}"; do
        # Trim leading/trailing whitespace
        _src="${_src#"${_src%%[![:space:]]*}"}"
        _src="${_src%"${_src##*[![:space:]]}"}"

        # Skip empty entries (e.g., trailing comma)
        [[ -z "${_src}" ]] && continue

        if [[ -e "${_src}" ]]; then
          # Copy into working directory (preserve name; rsync will create dir if source is a dir)
          rsync -av -- "${_src}" .
    
          # Track what landed in the working dir so we can delete it later if enabled
          _base="$(basename -- "${_src}")"
          printf '%s\n' "${_base}" >> "${COPY_IN_MANIFEST}"
          
          echo "Copied in from: ${_src} -> $(pwd)" >> "$SUMMARY_SHORT"
        else
          echo "WARNING: path to copy does not exist: ${_src}"
          echo "WARNING: path to copy does not exist: ${_src}" >> "$SUMMARY_SHORT"
        fi
      done
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

10. Optional ZIP expansion of input bundles#

This block implements an optional “unzip inputs” step that expands one or more ZIP archives into the current working directory before the main run. It is driven by the UNZIP_FILES_LIST environment variable.

NOTE: This script is executed after the file-copy script, so that you may unzip files that have been copied into the working diretory!!

What this block does

1. **Section header + echo requested list**

echo "===================================================================" >> "$SUMMARY_SHORT"
echo " ---- expand input ZIP, optional ---- "      >> "$SUMMARY_SHORT"
UNZIP_FILES_LIST="${UNZIP_FILES_LIST:-}"
echo "UNZIP_FILES_LIST list: ${UNZIP_FILES_LIST}" >> "$SUMMARY_SHORT"

Writes a header into SUMMARY_SHORT so the log clearly shows when ZIP expansion is being handled.
Normalizes UNZIP_FILES_LIST to an empty string if unset.
Logs the raw UNZIP_FILES_LIST value for debugging (what the job thought it should unzip).

Parse the comma-separated list (first pass)

if [[ -n "$UNZIP_FILES_LIST" ]]; then
  IFS=',' read -ra ZIP_LIST <<< "$UNZIP_FILES_LIST"
  for f in "${ZIP_LIST[@]}"; do
    # trim whitespace
    f="$(echo "$f" | xargs)"
  done
fi

If UNZIP_FILES_LIST is non-empty, it is split on commas into ZIP_LIST.
Each entry is passed through xargs to trim whitespace (e.g., to tolerate file1, file2 , data/job.zip).
This first loop is effectively a “sanity pass” for the raw list.

Parse again and actually unzip (second pass)

UNZIP_FILES_LIST="${UNZIP_FILES_LIST:-}"
echo "UNZIP_FILES_LIST list: ${UNZIP_FILES_LIST}" >> "$SUMMARY_SHORT"
if [[ -n "$UNZIP_FILES_LIST" ]]; then
  IFS=',' read -ra ZIP_LIST <<< "$UNZIP_FILES_LIST"
  for f in "${ZIP_LIST[@]}"; do
    # trim whitespace
    f="$(echo "$f" | xargs)"
    [[ -z "$f" ]] && continue

The list is echoed again (still helpful for debugging when reading the summary).
UNZIP_FILES_LIST is split again and each item is:
- Whitespace-trimmed,
- Skipped if empty ([[ -z “$f” ]] && continue), so stray commas do not cause errors.

Normalize filenames and unzip

    # add .zip if missing
    case "$f" in
      *.zip) zipfile="$f" ;;
      *)     zipfile="${f}.zip" ;;
    esac

    if [[ -f "$zipfile" ]]; then
      echo "Unzipping $zipfile ..."
      unzip -o -q "$zipfile"
      echo "Unzipped: $zipfile into $(pwd)" >> "$SUMMARY_SHORT"
    else
      echo "Warning: $zipfile not found, skipping."
      echo "WARNING: $zipfile not found, skipping unzip" >> "$SUMMARY_SHORT"
    fi
  done
fi

For each cleaned entry:

If the user did not include .zip, the script automatically appends it, so both input and input.zip are accepted.
If the resolved zipfile exists in the current directory:
- It is unzipped in-place with unzip -o -q:
  - -o overwrites existing files without prompting,
  - -q keeps output quiet in the terminal.
- A one-line summary (“Unzipped: … into $(pwd)”) is written to SUMMARY_SHORT.
If the file does not exist:
- A warning is printed to stdout and also logged to SUMMARY_SHORT, so missing ZIPs are obvious when reviewing job output.

In short, this block provides a flexible, environment-driven way to expand one or more input ZIP bundles (meshes, scripts, parameter sets, etc.) into the job’s working directory without hardcoding filenames in the app logic.

bash_script_option_UNZIP = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=========== ZIP EXPANSION OF INPUT BUNDLES ========================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"

    echo " ---- expand input ZIP, optional ---- "      >> "$SUMMARY_SHORT"
    UNZIP_FILES_LIST="${UNZIP_FILES_LIST:-}"
    echo "UNZIP_FILES_LIST list: ${UNZIP_FILES_LIST}" >> "$SUMMARY_SHORT"
    if [[ -n "$UNZIP_FILES_LIST" ]]; then
      IFS=',' read -ra ZIP_LIST <<< "$UNZIP_FILES_LIST"
      for f in "${ZIP_LIST[@]}"; do
        # trim whitespace
        f="$(echo "$f" | xargs)"
      done
    fi

    UNZIP_FILES_LIST="${UNZIP_FILES_LIST:-}"
    echo "UNZIP_FILES_LIST list: ${UNZIP_FILES_LIST}" >> "$SUMMARY_SHORT"
    if [[ -n "$UNZIP_FILES_LIST" ]]; then
      IFS=',' read -ra ZIP_LIST <<< "$UNZIP_FILES_LIST"
      for f in "${ZIP_LIST[@]}"; do
        # trim whitespace
        f="$(echo "$f" | xargs)"
        [[ -z "$f" ]] && continue
    
        # add .zip if missing
        case "$f" in
          *.zip) zipfile="$f" ;;
          *)     zipfile="${f}.zip" ;;
        esac
    
        if [[ -f "$zipfile" ]]; then
          echo "Unzipping $zipfile ..."
          unzip -o -q "$zipfile"
          echo "Unzipped: $zipfile into $(pwd)" >> "$SUMMARY_SHORT"
        else
          echo "Warning: $zipfile not found, skipping."
          echo "WARNING: $zipfile not found, skipping unzip" >> "$SUMMARY_SHORT"
        fi
      done
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"

""")

11. Defensive setup of the module command (before user-defined module loads)#

This block does not actually load any modules. Instead, it makes sure the module command itself is defined before later logic tries to use it with a user-provided module list or file.

What this block does

Why this is necessary, even though module names come from the user later:

On many HPC systems, module is not a standalone executable; it’s a shell function or alias injected by login/profile scripts (e.g., /etc/profile.d/modules.sh).
Batch jobs (like Tapis/SLURM jobs) often run in a non-interactive, non-login shell, which may not source those profile scripts automatically.
If that happens and we immediately try to do something like module load hdf5 opensees, the job will fail with:
```
module: command not found
```
even though the system does support modules.

This snippet therefore:

Writes a small header to SUMMARY_SHORT noting that we’re setting up the module environment.
Checks if module is available:
```
if ! command -v module >/dev/null 2>&1; then
```
This catches cases where the shell hasn’t been initialized with the Modules environment.
If module is missing, it manually sources the standard Modules init script:
```
if [[ -f /etc/profile.d/modules.sh ]]; then
    source /etc/profile.d/modules.sh
fi
```
This is a defensive “bootstrap” step: it recreates what a login shell would normally do, so that module load … will work.

After this block has run, we can safely process user-provided module lists/files (via MODULE_LOADS_LIST, MODULE_LOADS_FILE, etc.) knowing that the module command exists. It’s basically insurance against subtle “works in interactive shell, fails in batch job” problems.

bash_script_option_MODULE_ENV_SETUP = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "============== DEFENSIVE SETUP OF MODULE COMMAND ==================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " ---- set up module environment ---- " >> "$SUMMARY_SHORT"
    echo "Ensure 'module' command is available (defensive; usually provided by profile)" >> "$SUMMARY_SHORT"
    if ! command -v module >/dev/null 2>&1; then
      if [[ -f /etc/profile.d/modules.sh ]]; then
        # shellcheck source=/etc/profile.d/modules.sh
        source /etc/profile.d/modules.sh
      fi
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

12. Loading modules from a user-provided file#

This block defines a helper function to read a module recipe file and then uses it (if MODULE_LOADS_FILE is set) to configure the environment in a controlled, reproducible way.

What this block does

A. What the helper does

load_modules_from_file is a small parser for a module config file. It’s designed so users (or the app) can ship a simple text file that describes all module operations, instead of hard-coding them in the script.

Key behaviors:

a. Graceful skip if file doesn’t exist

[[ -f "$reqfile" ]] || { echo "No module file: $reqfile (skipping)"; return 0; }

If the file isn’t there, it just prints a note and returns successfully (no hard failure).

b. Line-by-line parsing with comments and whitespace

while IFS= read -r raw || [[ -n "$raw" ]]; do
  line="${raw%%#*}"                         # strip inline comments
  line="$(printf '%s' "$line" | awk '{$1=$1}1')"  # trim whitespace
  [[ -z "$line" ]] && continue              # skip empty/comment-only lines

Supports full-line comments and inline comments (# …).
Trims whitespace so users can format the file nicely.
Skips blank/comment-only lines.

c. Supported commands / syntaxes

Each non-empty line is interpreted with a case:

purge
```
module purge
```
Clears the environment module stack. Logged to SUMMARY_SHORT. Useful to reset toolchains.
“use ”
```
module use /some/modulefiles/path
```
Adds a directory to the MODULEPATH, allowing access to additional modulefiles.
“load ”
```
module load gcc/13.2
```
Explicit load directive; user writes exactly what they’d type in a shell.
?something (line begins with literal ?)
```
module try-load something
```
Optional modules: attempt to load but don’t treat failure as fatal. This is handy for “use if available, otherwise ignore” cases.
Any other non-empty token
```
module load $line
```
For simple lines like hdf5 or opensees, it assumes module load .

Every action is mirrored into SUMMARY_SHORT for later auditing.

B. How it’s used with MODULE_LOADS_FILE

After defining the helper, the script:

a. Logs the configuration:

echo "MODULE_LOADS_FILE: ${MODULE_LOADS_FILE:-}" >> "$SUMMARY_SHORT"

b. If MODULE_LOADS_FILE is set and points to a real file, it:

Prints a message to stdout (Loading modules from file: …),
Does a defensive module purge before applying the file:
```
module purge || true
echo "module purge (before MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
```
This helps avoid weird toolchain conflicts from whatever modules might already be loaded (from system defaults, profiles, etc.). The file itself can still contain its own purge if needed, but this gives you a clean starting point.
Calls load_modules_from_file “$MODULE_LOADS_FILE” to apply the recipe.
Runs module list || true so the final module state is visible in the job output.
Logs that user-defined modules were loaded.

C. Why do this if we also support a “list” variable?

You’re giving users two ways to specify modules:

A file-based recipe (MODULE_LOADS_FILE), which:
- Supports comments,
- Supports ordering, purge, use, load, and optional ?module,
- Can be version-controlled alongside the app.
A simple list variable (MODULE_LOADS_LIST, handled elsewhere), which is great for quick overrides or programmatic injection.

This block specifically handles the file-based, richer syntax:

It centralizes all the module operations in one place (the file), instead of scattering module load … calls throughout the script.
It gives power users a way to express more complex sequences (e.g., purge, use /path, load toolchain, ?debug-tools) while still keeping the main wrapper script generic.
It makes the environment reproducible and inspectable: you can archive the module file with the job, and the summary log clearly shows every module action that was executed.

bash_script_option_MODULE_LOAD_FILE = textwrap.dedent(r"""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "========== LOADING MODULES FROM A USER-PROVIDED FILE ==============" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " ---- helper function: load modules from a file (supports comments, purge, use, ?optional, load) ----" >> "$SUMMARY_SHORT"
    load_modules_from_file() {
      local reqfile="$1"
      [[ -f "$reqfile" ]] || { echo "No module file: $reqfile (skipping)"; return 0; }

      while IFS= read -r raw || [[ -n "$raw" ]]; do
        # strip inline comments and trim
        line="${raw%%#*}"
        line="$(printf '%s' "$line" | awk '{$1=$1}1')"
        [[ -z "$line" ]] && continue

        case "$line" in
          purge)
            module purge
            echo "module purge (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            ;;
          "use "*)
            # NOTE: do NOT lowercase paths (could be case-sensitive)
            usepath="${line#use }"
            usepath="$(printf '%s' "$usepath" | awk '{$1=$1}1')"
            module use "$usepath"
            echo "module use $usepath (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            ;;
          "load "*)
            orig="${line#load }"
            orig="$(printf '%s' "$orig" | awk '{$1=$1}1')"
            mod="${orig,,}"   # lowercase module name

            if [[ "$mod" != "$orig" ]]; then
              echo "NOTE: lowercased module name: '$orig' -> '$mod' (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi

            if module load "$mod"; then
              echo "module load $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            else
              echo "WARNING: module load failed: $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi
            ;;
          \?*)
            # optional module lines that start with literal '?'
            orig="${line#\?}"
            orig="$(printf '%s' "$orig" | awk '{$1=$1}1')"
            mod="${orig,,}"   # lowercase module name

            if [[ "$mod" != "$orig" ]]; then
              echo "NOTE: lowercased optional module name: '$orig' -> '$mod' (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi

            if module try-load "$mod"; then
              echo "module try-load $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            else
              echo "WARNING: module try-load failed: $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi
            ;;
          *)
            orig="$line"
            mod="${orig,,}"   # lowercase module name

            if [[ "$mod" != "$orig" ]]; then
              echo "NOTE: lowercased module name: '$orig' -> '$mod' (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi

            if module load "$mod"; then
              echo "module load $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            else
              echo "WARNING: module load failed: $mod (from MODULE_LOADS_FILE)" >> "$SUMMARY_SHORT"
            fi
            ;;
        esac

      done < "$reqfile"
    }

    echo "===========================================================" >> "$SUMMARY_SHORT"
    echo " ---- modules to load from file or list (overrides/augments defaults) ----" >> "$SUMMARY_SHORT"
    echo "MODULE_LOADS_FILE: ${MODULE_LOADS_FILE:-}" >> "$SUMMARY_SHORT"
    if [[ -n "${MODULE_LOADS_FILE:-}" && -f "$MODULE_LOADS_FILE" ]]; then
      echo "Loading modules from file: $MODULE_LOADS_FILE"
      load_modules_from_file "$MODULE_LOADS_FILE"
      module list || true
      echo "Loaded user-defined modules from file: $MODULE_LOADS_FILE" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

13. Loading modules from a comma-separated list (MODULE_LOADS_LIST)#

This block provides a lightweight, environment-variable–driven way to load modules, complementary to the file-based approach handled by MODULE_LOADS_FILE.

What this block does

It assumes that the module command has already been made available (by the earlier module-environment setup step) and that any default or file-based module configuration has already been applied. This list-based mechanism is ideal for **quick overrides or additions** without editing a module file.

What it does:

Log the incoming configuration

The script writes the current value of MODULE_LOADS_LIST into SUMMARY_SHORT, so you can see exactly which modules were requested for this job:
- If the variable is unset or empty, it logs an empty value and does nothing else.
- If it’s set, the value (e.g., gcc/13.2,hdf5, opensees) is recorded as-is for later debugging.
Parse the comma-separated list

If MODULE_LOADS_LIST is non-empty:
- It is split on commas into an array of module names.
- Each entry is passed through xargs to trim leading/trailing whitespace, so both hdf5 and ” hdf5 ” work the same way.
- Empty entries (e.g., from trailing commas or accidental ,,) are skipped safely.
Load each requested module

For each non-empty module token:
- A message like loading module … is printed to stdout so the job output shows what’s happening in real time.
- module load is invoked to actually bring the module into the environment.
- The action is mirrored into SUMMARY_SHORT as module load (from MODULE_LOADS_LIST) so you have a persistent record in the summary log.

Why this exists in addition to the module file:

The file-based approach (MODULE_LOADS_FILE) is best for structured, version-controlled environment recipes (with purge, use, optional ?module, etc.).
The list-based approach (MODULE_LOADS_LIST) is best for:
- Quick tweaks in a job submission,
- Adding one or two extra modules on top of existing defaults,
- Programmatic injection of modules from the Tapis job JSON or another wrapper.

Together, they let you keep a stable, shared “baseline” module file while still allowing per-job or per-user customization via a simple environment variable.

bash_script_option_MODULE_LOAD_LIST = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "==== LOADING MODULES FROM A USER-PROVIDED COMMA-SEPARATED LIST ====" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "MODULE_LOADS_LIST: ${MODULE_LOADS_LIST:-}" >> "$SUMMARY_SHORT"

    if [[ -n "${MODULE_LOADS_LIST:-}" ]]; then
      echo "Loading modules from: $MODULE_LOADS_LIST"
      IFS=',' read -ra MOD_LIST <<< "$MODULE_LOADS_LIST"

      for mod in "${MOD_LIST[@]}"; do
        mod="$(echo "$mod" | xargs)"          # trim whitespace
        [[ -z "$mod" ]] && continue
        mod="${mod,,}"                       # <-- convert to lowercase

        echo "loading module $mod ..."
        module load "$mod"
        echo "module load $mod (from MODULE_LOADS_LIST)" >> "$SUMMARY_SHORT"
      done
    fi

    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

14. Load OpenSees Modules (If Running OpenSees)#

Conceptually, this snippet says:

What this block does

> “If the main binary is an OpenSees **Tcl** executable, make sure the standard OpenSees modules are loaded.”

Concretely:

It writes a header to SUMMARY_SHORT for bookkeeping.
It checks BINARYNAME:
- If it’s OpenSees, OpenSeesMP, or OpenSeesSP, it:
  - module load hdf5/1.14.4 || true
  - module load opensees || true
  - logs Loaded default OpenSees-Tcl modules: hdf5/1.14.4, opensees.

So this is your baseline environment for Tcl-based OpenSees runs, independent of any user-supplied module file or list.

bash_script_option_OPS_MODULES_LOAD = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "===================== LOAD OPENSEES MODULES =======================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    # ---- OpenSees Tcl: load default modules if using OpenSees binaries ----
    if [[ "$BINARYNAME" == "OpenSees" || "$BINARYNAME" == "OpenSeesMP" || "$BINARYNAME" == "OpenSeesSP" ]]; then
        module load hdf5/1.14.4 || true
        module load opensees || true
        echo "Loaded default OpenSees-Tcl modules: hdf5/1.14.4, opensees" >> "$SUMMARY_SHORT"
    fi
    
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

15. Add the pylauncher module, just in case.#

Just making sure that it is available if needed.

bash_script_option_PYLAUNCHER_MODULES_LOAD = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "===================== LOAD PyLauncher MODULES =======================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    if [[ "$BINARYNAME" == "python3" || "$BINARYNAME" == "python" || "$BINARYNAME" == "Python3" || "$BINARYNAME" == "Python" ]]; then
        module load pylauncher || true
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

16. Forcing `python` to Use `python3` in a Batch App#

This block ensures that any command that calls python actually runs python3, even if the system has multiple Python installations and python would normally resolve to a different interpreter.

This matters because:

Your wrapper script may launch the main program with python3, but user code or dependencies may still invoke python (for example via subprocess.run(["python", ...]), os.system("python ..."), Makefiles, or CLI tools installed with pip).
On HPC systems, python and python3 often point to different versions/environments, which can lead to confusing runtime errors (missing packages, wrong ABI, wrong OpenSeesPy build, etc.).

What this block does

A. Creates a small “shim” directory inside the job’s start directory:

SCRIPT_ROOT_DIR/.pyshim/bin

B. Writes a lightweight wrapper script named python into that directory:

When anything runs python ..., the wrapper executes python3 ... with the same arguments.

C. (Optional) Writes a wrapper script named pip:

When anything runs pip ..., it executes pip3 ....
This helps because many workflows assume pip exists (not just pip3).

D. Prepends the shim directory to PATH:

This is the key step: placing the shim directory first in PATH guarantees that python resolves to the shim (and thus python3) before any other python executable found elsewhere.

E. Logs resolution and versions to the summary log:

Records what python and python3 resolve to (command -v ...)
Records the reported versions (python -V, python3 -V)
This makes debugging easy if something still behaves unexpectedly.

Why This Works in Batch Jobs

alias python=python3 is not reliable in non-interactive batch shells.
Changing system-wide alternatives is not appropriate (and often not possible) on shared HPC systems.
A PATH-prepended shim is simple, local to the job, and reliably affects:
- the wrapper script itself
- user scripts
- subprocess calls
- pip-installed console entry points that internally call python

Notes and Best Practices

Place this block after module loads (so you wrap the final Python environment you intend to use).
If a later module load changes PATH, it could override ordering; in that case, place the shim block after those loads.
This block does not change python3; it only ensures that python (and optionally pip) consistently follow python3 (and pip3).

bash_script_option_PYTHON_ALIAS = textwrap.dedent("""
    # -------------------------------------------------------------------
    # Force `python` to mean `python3` (and optionally `pip` -> `pip3`)
    # Put this AFTER your module loads (so it wraps the final python).
    # -------------------------------------------------------------------
    PY_SHIM_DIR="${SCRIPT_ROOT_DIR}/.pyshim/bin"
    mkdir -p "$PY_SHIM_DIR"
    
    cat > "${PY_SHIM_DIR}/python" <<'EOF'
    #!/usr/bin/env bash
    exec python3 "$@"
    EOF
    chmod +x "${PY_SHIM_DIR}/python"
    
    # Optional but usually helpful (many tools call `pip`)
    cat > "${PY_SHIM_DIR}/pip" <<'EOF'
    #!/usr/bin/env bash
    exec pip3 "$@"
    EOF
    chmod +x "${PY_SHIM_DIR}/pip"
    
    # Prepend shim dir so it wins over any other `python` in PATH
    export PATH="${PY_SHIM_DIR}:$PATH"
    
    # Log what will be used
    echo "python resolves to: $(command -v python)"   >> "$SUMMARY_SHORT"
    echo "python3 resolves to: $(command -v python3)" >> "$SUMMARY_SHORT"
    python -V  >> "$SUMMARY_SHORT" 2>&1 || true
    python3 -V >> "$SUMMARY_SHORT" 2>&1 || true
""")

17. Installing Python packages from a requirements file (PIP_INSTALLS_FILE)#

This block provides a file-based mechanism for installing Python dependencies at runtime—similar in spirit to MODULE_LOADS_FILE, but specifically for pip packages.

What this block does

1. Logs the section and prints the Python version

The script writes a header into SUMMARY_SHORT and prints the active Python version:

echo "---- Python / pip setup / from FILE ----" >> "$SUMMARY_SHORT"
python3 -V || true

This helps diagnose environment or module-loading problems.
|| true ensures the job does not fail simply because python3 -V is missing—actual failure only occurs during the pip install.

2. Checks for a valid requirements file

if [[ -n "${PIP_INSTALLS_FILE:-}" && -f "$PIP_INSTALLS_FILE" ]]; then

This ensures:

PIP_INSTALLS_FILE is non-empty
The referenced file exists on the execution system

If either condition fails, the block logs a message and safely skips installation. No error is thrown just because the user omitted this option.

3. Installs packages with pip3 install -r (with full error handling)

If the requirements file exists:

echo "Installing Python packages from file: $PIP_INSTALLS_FILE" >> "$SUMMARY_SHORT"
echo "pip3 install -r $PIP_INSTALLS_FILE" >> "$SUMMARY_SHORT"

if ! pip3 install -r "$PIP_INSTALLS_FILE"; then
    rc=$?
    echo "ERROR: pip3 install failed for requirements file '$PIP_INSTALLS_FILE' (exit code $rc)" >> "$SUMMARY_SHORT"
    echo "ERROR: pip3 install failed for requirements file '$PIP_INSTALLS_FILE' (exit code $rc)" >&2
    exit "$rc"
fi

Key behaviors:

Logs both the file path and the exact pip command for reproducibility.
If pip install fails:
- Writes a clear ERROR line into SUMMARY_SHORT
- Writes the same message to stderr (so it appears cleanly in Tapis logs)
- Exits with a non-zero return code so the job is correctly marked FAILED.

This mirrors the same error-handling model used in the list-based installer.

Why this pattern is useful

Keeps Python dependency definitions out of the wrapper script and in a normal requirements.txt file.
Allows job- or version-specific overrides without modifying the underlying app.
Fully Tapis-aware: failures propagate correctly and are easy for users to understand.
Logs exactly what was attempted, simplifying debugging (“what environment did this job actually run in?”).

How it complements PIP_INSTALLS_LIST

Feature	`PIP_INSTALLS_FILE`	`PIP_INSTALLS_LIST`
Best for	Stable, version-controlled environments	Quick per-job tweaks
Input type	File (requirements.txt style)	Comma-separated string
Error handling	Fail-fast with clear message	Fail-fast with clear message
Ideal use case	Reproducible runtime environments	Lightweight additions or overrides

Together, they provide a robust, flexible environment management pattern for Python inside Tapis/DesignSafe apps.

bash_script_option_PIP_FILE = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "= INSTALLING PYTHON PACKAGES FROM A USER-PROVIDED REQUIREMENTS FILE =" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "---- Python / pip setup / from FILE ----" >> "$SUMMARY_SHORT"
    python3 -V || true

    if [[ -n "${PIP_INSTALLS_FILE:-}" && -f "$PIP_INSTALLS_FILE" ]]; then
      echo "Installing Python packages from file: $PIP_INSTALLS_FILE" >> "$SUMMARY_SHORT"
      echo "pip3 install -r $PIP_INSTALLS_FILE" >> "$SUMMARY_SHORT"

      if ! pip3 install -r "$PIP_INSTALLS_FILE"; then
        rc=$?
        echo "ERROR: pip3 install failed for requirements file '$PIP_INSTALLS_FILE' (exit code $rc)" >> "$SUMMARY_SHORT"
        echo "ERROR: pip3 install failed for requirements file '$PIP_INSTALLS_FILE' (exit code $rc)" >&2
        exit "$rc"
      fi

    else
      echo "PIP_INSTALLS_FILE not provided or file does not exist; skipping pip installs." >> "$SUMMARY_SHORT"
    fi

    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

# import os

# absPath = os.path.abspath('../../../shared/Examples/OpenSees')
# MyDataPath = os.path.expanduser('~/MyData/')
# MyPath = absPath.replace(MyDataPath,'')

# print('absPath',absPath)
# print('MyDataPath',MyDataPath)
# print('MyPath',MyPath)


# # print(os.path.abspath('../../../shared/Examples/OpenSees'))

18. Installing Python packages from a comma-separated list (PIP_INSTALLS_LIST)#

This block is the list-based counterpart to PIP_INSTALLS_FILE. Instead of pointing to a requirements file, you can provide Python packages directly through the PIP_INSTALLS_LIST environment variable.

What this block does

1. Log the section and report the active Python version

The script writes a header into SUMMARY_SHORT and prints the Python version for debugging:

echo "---- Python / pip setup / from LIST ----" >> "$SUMMARY_SHORT"
python3 -V || true

python3 -V || true guarantees that missing Python does not break the job at this stage — the block only fails during an actual pip install error.

2. Check whether a package list was provided

if [[ -n "${PIP_INSTALLS_LIST:-}" ]]; then

If PIP_INSTALLS_LIST is empty or unset, the block logs that it is skipping installation.
If set (e.g., numpy, scipy==1.13, pandas), the block continues.

This allows convenient per-job overrides without touching the app definition.

3. Split the comma-separated package list

IFS=',' read -ra PKG_LIST <<< "$PIP_INSTALLS_LIST"
for pkg in "${PKG_LIST[@]}"; do
    pkg="$(echo "$pkg" | xargs)"
    [[ -z "$pkg" ]] && continue

Splits the list into an array PKG_LIST.
Uses xargs to trim whitespace, so both numpy and " numpy " work.
Ignores empty entries so "numpy,,scipy" does not cause errors.

This makes the interface resilient to user formatting.

4. Install each package and fail cleanly on errors

For every valid package:

echo "pip3 install $pkg (from PIP_INSTALLS_LIST)" >> "$SUMMARY_SHORT"
if ! pip3 install "$pkg"; then
    rc=$?
    echo "ERROR: pip3 install failed for package '$pkg' (exit code $rc)" >> "$SUMMARY_SHORT"
    echo "ERROR: pip3 install failed for package '$pkg' (exit code $rc)" >&2
    exit "$rc"
fi

Key behaviors:

Logs the pip command for reproducibility.
If pip3 install fails:
- Writes a clear error message to SUMMARY_SHORT
- Writes the same message to stderr (visible in portal / tapis jobs logs)
- Exits with a non-zero return code so Tapis correctly marks the job as FAILED.

This is the correct, Tapis-friendly pattern for surfacing human-readable errors.

Why this matters

This block now behaves like a “proper installation step” in HPC workflows:

Transparent: every attempted install shows up in the log.
Fail-fast: jobs don’t continue in a broken environment.
Tapis-aware: the failure is clearly communicated to both humans and the Tapis job system.

How this complements PIP_INSTALLS_FILE

Feature	`PIP_INSTALLS_FILE`	`PIP_INSTALLS_LIST`
Best for	Stable, version-controlled environments	Quick per-job tweaks and overrides
Input type	Requirements-style text file	Comma-separated string
Good for	Reproducibility	Experimentation, dynamic injection from job JSON
Error handling	Fail-fast on requirements file errors	Fail-fast on individual package errors

Using both options gives you a flexible but robust installation strategy:

A file-based baseline,
Plus a list-based runtime knob for additional packages.

bash_script_option_PIP_LIST = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "= INSTALLING PYTHON PACKAGES FROM A USER-PROVIDED COMMA-SEPARATED LIST =" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "---- Python / pip setup / from LIST ----" >> "$SUMMARY_SHORT"
    python3 -V || true

    if [[ -n "${PIP_INSTALLS_LIST:-}" ]]; then
      echo "Installing Python packages from PIP_INSTALLS_LIST list: $PIP_INSTALLS_LIST" >> "$SUMMARY_SHORT"
      IFS=',' read -ra PKG_LIST <<< "$PIP_INSTALLS_LIST"
      for pkg in "${PKG_LIST[@]}"; do
        pkg="$(echo "$pkg" | xargs)"   # trim whitespace
        [[ -z "$pkg" ]] && continue
        echo "pip3 install $pkg (from PIP_INSTALLS_LIST)" >> "$SUMMARY_SHORT"
        if ! pip3 install "$pkg"; then
          rc=$?
          echo "ERROR: pip3 install failed for package '$pkg' (exit code $rc)" >> "$SUMMARY_SHORT"
          echo "ERROR: pip3 install failed for package '$pkg' (exit code $rc)" >&2
          exit "$rc"
        fi
      done
    else
      echo "PIP_INSTALLS_LIST is empty; skipping pip installs." >> "$SUMMARY_SHORT"
    fi

    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

19. Choosing how to launch the app (sequential vs MPI)#

This block decides whether to wrap the binary in an MPI launcher or run it directly, and records that choice in the summary log.

What this block does

**Key inputs:**

BINARYNAME – which executable we’re running (OpenSees, OpenSeesMP, etc.).
UseMPI – a user-facing flag (string) that says whether MPI should be used. It’s interpreted in a flexible, “human-ish” way.

What it does:

Log the section and the requested MPI flag

It writes a header and the current value of UseMPI to SUMMARY_SHORT, so later you can see what the job thought it should do regarding MPI.
Decide the launcher and populate LAUNCH

The logic fills an array LAUNCH, which will be prepended when actually running the job:
- Case 1: BINARYNAME == “OpenSees”
```
LAUNCH=()
```
  - No launcher is used: this is a purely sequential run.
  - Logged as Launcher: none (OpenSees sequential).
- Case 2: UseMPI is “false-like”
```
elif [[ ! "${UseMPI:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
  LAUNCH=()
```
  - If UseMPI is anything other than true/True/TRUE/yes/Yes/1 (or similar), then we again run without an MPI launcher.
  - Logged as Launcher: none (UseMPI false-like).
- Case 3: MPI is requested
```
else
  LAUNCH=(ibrun)
```
  - If UseMPI matches a “true-like” string, the script sets LAUNCH=(ibrun).
  - This means the final command will be ibrun on Stampede3.
  - Logged as Launcher: ibrun (UseMPI true-like).

Why this pattern:

Keeps the launcher choice centralized and explicit, instead of scattering ibrun versus direct runs in multiple places.
Makes UseMPI flexible and user-friendly (accepting true, True, YES, 1, etc.).
Ensures that plain OpenSees (Tcl, single-process) defaults to sequential, while MPI-capable binaries can opt into parallelism cleanly via UseMPI.

bash_script_run_CHOOSE_LAUNCHER = textwrap.dedent("""
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "=========== CHOOSING THE LAUNCHER (SEQUENTIAL VS MPI) =============" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " ---- choose launcher ---- " >> "$SUMMARY_SHORT"
    echo "UseMPI ${UseMPI}" >> "$SUMMARY_SHORT"

    LAUNCH=()
    if [[ "$BINARYNAME" == "OpenSees" ]]; then
      LAUNCH=()        # direct run for sequential
      echo "Launcher: none (OpenSees sequential)" >> "$SUMMARY_SHORT"
    elif [[ ! "${UseMPI:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
      LAUNCH=()
      echo "Launcher: none (UseMPI false-like)" >> "$SUMMARY_SHORT"
    else
      LAUNCH=(ibrun)
      echo "Launcher: ibrun (UseMPI true-like)" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

20. Running the job binary (with timers and error handling)#

This block is the core execution step: it runs the main binary (with or without an MPI launcher), records timing information, and handles errors in a consistent way.

What this block does

This block: A. Start the “binary run” timer

Captures:
- RUN_START_EPOCH – epoch time when the binary starts.
- RUN_START_HUMAN – human-readable start time.
Logs the start time to SUMMARY_SHORT.

This pairs with the echoTimers_START / bash_script_echoTimers_AFTER blocks so you can see how long the binary itself ran.

B. Optional Python version check

If BINARYNAME == “python3”, it runs python3 -V || true:
- Prints the Python version to stdout (helpful when debugging OpenSeesPy or other Python workflows).
- || true avoids killing the job if that command fails.

C. Actually run the application

The script then:

Prints a little visual marker to stdout (************************* run!!!) and a separator in SUMMARY_SHORT.
Chooses how to run based on LAUNCH (which was set earlier by bash_script_run_CHOOSE_LAUNCHER):

Case I – With launcher (MPI / ibrun, etc.)

If LAUNCH has elements:
- Logs a line like Running: ibrun OpenSeesMP input.tcl to SUMMARY_SHORT.
- Executes: “${LAUNCH[@]}” “$BINARYNAME” “$INPUTSCRIPT” “$@”
- If that command fails (nonzero status):
  - Captures the return code in rc.
  - Logs Program exited with error status: $rc to the summary.
  - Calls echoTimers_START to record on-error timing (run + total).
  - Exits the wrapper with the same rc.

Case II – No launcher (sequential)

If LAUNCH is empty:
- Logs Running: $BINARYNAME $INPUTSCRIPT $* to SUMMARY_SHORT.
- Runs the binary directly: “$BINARYNAME” “$INPUTSCRIPT” “$@”
- On failure:
  - Same behavior as above: capture rc, log it, call echoTimers_START, and exit with rc.

This gives you the same error-handling path whether you’re running sequentially or under ibrun.

D. Mark successful completion

If the binary exits with status 0:

Writes a big banner to SUMMARY_SHORT:
- Separator line
- Run completed with NO ERROR!!!!
- Separator line

At this point, the end-of-run timer block (bash_script_echoTimers_AFTER + bash_script_echoTimers_END) will typically run to record the normal, successful runtime for both the binary and the whole script.

bash_script_run_RUN_JOB = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "===================== RUNNING THE JOB BINARY ======================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    # ---- TIMER: binary run only ----
    RUN_START_EPOCH=$(date +%s)
    RUN_START_HUMAN="$(date)"
    echo "Binary run start time: ${RUN_START_HUMAN} (epoch ${RUN_START_EPOCH})" >> "$SUMMARY_SHORT"

    if [[ "$BINARYNAME" == "python3" ]]; then
        python3 -V || true
    fi

    if [[ "$BINARYNAME" == "python" ]]; then
        python -V || true
    fi    


    echo '************************* run!!!'
    echo "==============" >> "$SUMMARY_SHORT"

    
   
    if [[ ${#LAUNCH[@]} -gt 0 ]]; then
      echo "Running: ${LAUNCH[*]} $BINARYNAME $INPUTSCRIPT $*" >> "$SUMMARY_SHORT"
      "${LAUNCH[@]}" "$BINARYNAME" "$INPUTSCRIPT" "$@"
      rc=$?
    else
      echo "Running: $BINARYNAME $INPUTSCRIPT $*" >> "$SUMMARY_SHORT"
      "$BINARYNAME" "$INPUTSCRIPT" "$@"
      rc=$?
    fi
    
    if [[ $rc -ne 0 ]]; then
      echo "Program exited with error status: $rc" >> "$SUMMARY_SHORT"
    
      __echoTimers_START__

      echo "ERROR: Application run failed with status $rc" >&2
      echo "HINT: Possible cause: Failed to import openseespy or missing modules." >&2
      exit "$rc"
    fi

    echo "###############################################################################" >> "$SUMMARY_SHORT"
    echo "Run completed with NO ERROR!!!!" >> "$SUMMARY_SHORT"
    echo "###############################################################################" >> "$SUMMARY_SHORT"
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

21. OpenSeesPy: copy TACC-compiled OpenSeesPy.so into the run directory#

Because the PyPI wheel for OpenSeesPy is not guaranteed to match the exact Python build and system libraries on TACC, this block provides a safer alternative:

If requested, use the TACC-compiled OpenSeesPy shared library instead of relying on whatever pip install openseespy would pull in.

What this block does

Concretely, this block:

Checks whether the user actually requested the TACC OpenSeesPy

It looks at GET_TACC_OPENSEESPY and treats “true-like” strings as on:
- Accepted values: true, True, TRUE, yes, Yes, 1, etc.
- If not true-like, the entire block is skipped.
This makes it an opt-in switch from the Tapis job JSON or environment.
Loads the Python + OpenSees environment expected by the TACC build

Inside the GET_TACC_OPENSEESPY true branch, it defensively does:
- module load python/3.12.11 || true
- module load hdf5/1.14.4 || true
- module load opensees || true
These are the modules that match the environment used to compile the TACC OpenSeesPy library. Loading them ensures:
- ABI and library compatibility for OpenSeesPy.so,
- Consistency with the TACC-supported toolchain.
The || true prevents a hard failure if one of these loads doesn’t succeed, but the summary log still notes that this path was taken.
Validate and use TACC_OPENSEES_BIN as the source location

The script then checks:
- If TACC_OPENSEES_BIN is unset or empty:
  - It logs a warning to SUMMARY_SHORT that the path is missing and skips the copy.
- Else if ${TACC_OPENSEES_BIN}/OpenSeesPy.so doesn’t exist:
  - It logs a warning that the file wasn’t found and skips the copy.
This protects against misconfigured environments or typos in the path.
Copy the TACC OpenSeesPy into the execution directory as opensees.so

If everything is valid:
- It echoes to stdout: Copying TACC OpenSeesPy -> ./opensees.so
- Performs:
```
cp "${TACC_OPENSEES_BIN}/OpenSeesPy.so" ./opensees.so
```
- Logs to SUMMARY_SHORT exactly what was copied and where:
  - Source: ${TACC_OPENSEES_BIN}/OpenSeesPy.so
  - Destination: $(pwd)/opensees.so
This puts a known-good, TACC-compiled OpenSeesPy shared library directly in the working directory, where Python will pick it up (e.g., via local opensees.so import behavior) without relying on an external wheel.
Document the modules that were loaded for this path

Finally, it appends a note to SUMMARY_SHORT:
- Loaded default TACC OpenSeesPy modules: python/3.12.11, hdf5/1.14.4, opensees
So when you inspect the job summary, you can see that:
- The TACC OpenSeesPy path was used,
- Which modules/environment were assumed for it.

When paired with a later “cleanup” block that removes ./opensees.so after the run, this pattern lets you:

Inject a cluster-native OpenSeesPy build for the duration of the job,
Avoid subtle wheel/ABI mismatches from pip install,
Keep the execution directory clean once the run is done.

bash_script_option_COPY_OPENSEESPY = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=============== COPY TACC-COMPILED OPENSEESPY =====================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " ---- OpenSeesPy: copy TACC-compiled opensees.so if requested ----" >> "$SUMMARY_SHORT"
    if [[ "${GET_TACC_OPENSEESPY:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
        module load python/3.12.11 || true
        module load hdf5/1.14.4 || true
        module load opensees || true
        if [[ -z "${TACC_OPENSEES_BIN:-}" ]]; then
          echo "WARNING: GET_TACC_OPENSEESPY=True but TACC_OPENSEES_BIN is not set; skipping copy of OpenSeesPy." >> "$SUMMARY_SHORT"
        elif [[ ! -f "${TACC_OPENSEES_BIN}/OpenSeesPy.so" ]]; then
          echo "WARNING: ${TACC_OPENSEES_BIN}/OpenSeesPy.so not found; skipping copy of OpenSeesPy." >> "$SUMMARY_SHORT"
        else
          echo "Copying TACC OpenSeesPy -> ./opensees.so"
          cp "${TACC_OPENSEES_BIN}/OpenSeesPy.so" ./opensees.so
          echo "Copied TACC OpenSeesPy: " >> "$SUMMARY_SHORT"
          echo "    ${TACC_OPENSEES_BIN}/OpenSeesPy.so -> $(pwd)/opensees.so" >> "$SUMMARY_SHORT"
        fi
        echo "Loaded default TACC OpenSeesPy modules: python/3.12.11, hdf5/1.14.4, opensees" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

22. Optional Cleanup: remove temporary TACC OpenSeesPy library after the run#

This block cleans up the OpenSeesPy shared library that may have been copied into the execution directory by the GET_TACC_OPENSEESPY option.

What this block does

**What it does:**

Log that a cleanup phase is running

It writes a header and a short label (“remove OpenSeesPy file (Optional)”) into SUMMARY_SHORT so it’s clear that a post-run OpenSeesPy cleanup step was attempted.
Check whether TACC OpenSeesPy was requested

Just like the copy block, it uses:
```
if [[ "${GET_TACC_OPENSEESPY:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
```
This ensures that cleanup only runs if the job opted into using the TACC OpenSeesPy build (i.e., the same flag that gated the copy step). If the user never requested GET_TACC_OPENSEESPY, this block quietly does nothing.
Remove the local opensees.so file

Inside the true branch:
- It calls:
```
rm -f ./opensees.so || true
```
  - rm -f removes the file if it exists and does nothing (no error) if it doesn’t.
  - || true prevents any unexpected rm issue from killing the job.
- It logs to SUMMARY_SHORT that the file was removed.

Why this cleanup matters:

The copy step deliberately places a cluster-specific OpenSeesPy.so into the current directory as opensees.so so the job can use a known-compatible library.
Leaving that file behind could:
- Confuse future runs (if they expect a different version),
- Be mistakenly packaged or moved as user output,
- Make it ambiguous whether the job used a TACC-native build or a different OpenSeesPy installation.

By removing ./opensees.so at the end only when GET_TACC_OPENSEESPY was enabled, you:

Keep the execution directory clean,
Make the TACC-compiled library clearly ephemeral and job-scoped,
Avoid interfering with any other environment that might exist outside this job.

bash_script_option_DELETE_OPENSEESPY = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=============== REMOVE TACC-COMPILED OPENSEESPY ===================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "remove OpenSeesPy file (Optional)" >> "$SUMMARY_SHORT"
    if [[ "${GET_TACC_OPENSEESPY:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
        rm -f ./opensees.so || true
        echo "Removed OpenSeesPy file ./opensees.so" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

23. Optional: repack the output directory into a single ZIP (ZIP_OUTPUT_SWITCH)#

This block optionally converts the job’s output directory into one ZIP archive and updates ArchiveName so that the later “move output” phase will move either:

What this block does

* the original directory (no zipping), or * the ZIP file (if zipping is enabled).

How it works#

Before this block runs, you set:

ArchiveName="${inputDirectory}"

So by default, ArchiveName points to the output folder itself.

This block then:

Logs configuration

It writes a header and echoes ZIP_OUTPUT_SWITCH into SUMMARY_SHORT, so you can see whether this option was turned on for the job.
Checks whether zipping is requested

If ZIP_OUTPUT_SWITCH matches a true-like value (true, yes, 1, etc.), the script:
- Sets
```
ArchiveName="inputDirectory.zip"
```
  overriding the earlier value. From this point on, any later “move output” logic should operate on inputDirectory.zip instead of the directory.
Creates the ZIP archive
- Runs:
```
zip -r -q "${ArchiveName}" "./${inputDirectory}"
```
  which recursively zips ./${inputDirectory} into inputDirectory.zip in the current working directory.
- Logs to SUMMARY_SHORT that the archive was created and from which path.
This reduces the job’s output to one big file, which is often much more robust with Tapis archive/transfer limits and typically faster to move.
Deletes the original directory
- Removes the original ${inputDirectory} tree with rm -rf.
- Logs the removal in the summary.
After this, the execution directory contains the ZIP instead of the exploded folder, and ArchiveName now correctly points to the ZIP. The archive/move phase that follows doesn’t need to care which path was taken — it simply moves ArchiveName, whether that’s a folder (no zip) or a single ZIP file (zip enabled).

Why this is helpful#

Avoids hitting limits on number of files in Tapis archiving.
Makes the archive/move phase shorter and simpler (one artifact).
Keeps the interface to later steps clean: they just look at ArchiveName and don’t need branching logic for “folder vs zip.”

bash_script_option_ZIP_OUTPUT = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=============== REPACK THE OUTPUT DIRECTORY TO ZIP ================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " ---- optional re-pack an output folder ----" >> "$SUMMARY_SHORT"
    echo "ZIP_OUTPUT_SWITCH: ${ZIP_OUTPUT_SWITCH:-}" >> "$SUMMARY_SHORT"

    if [[ "${ZIP_OUTPUT_SWITCH:-}" =~ ^([Tt][Rr][Uu][Ee]|1|[Yy][Ee]?[Ss]?)$ ]]; then
      ArchiveName="inputDirectory.zip"
      
      zip -r -q "${ArchiveName}" "./${inputDirectory}"
      echo "Zipped output: ${ArchiveName} from $(pwd)" >> "$SUMMARY_SHORT"

      rm -rf "${inputDirectory}"
      echo "removed"
      echo "Removed original inputDirectory: ${inputDirectory}" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

24. Optional: move main output to a faster storage destination (PATH_MOVE_OUTPUT)#

What this block does

This block is the **final handoff step** for job results. Instead of leaving large output directly in the execution directory for Tapis to archive (which can be slow), it optionally **moves the main output to a user-chosen location** on the system (e.g., $SCRATCH or $WORK) and copies over top-level helper files.

Because earlier logic sets:

ArchiveName=”${inputDirectory}” by default, and
ArchiveName=”inputDirectory.zip” if ZIP_OUTPUT_SWITCH is enabled,

this block will move either the output folder or the ZIP file, depending on how the job was configured.

1. Check whether a destination was requested#

The block reads PATH_MOVE_OUTPUT:

If PATH_MOVE_OUTPUT is unset or empty → no move is performed, everything stays in the execution directory.
If it is set → we treat it as the base directory where outputs should be moved.

Typical choices (user guidance):

$SCRATCH – Best for short-term, high-volume data. Large, fast, but not backed up; good for heavy, transient outputs.
$WORK – Best for large project storage that needs to persist across jobs and sessions.
$HOME – Not recommended for big outputs:
- Intended for permanent small files: configs, scripts, dotfiles, etc.
- Typically has limited capacity and is not meant for bulk simulation results.

You expose this as an option in the app so users can choose the storage tier that matches their use case.

2. Construct a job-specific destination path#

If PATH_MOVE_OUTPUT is set:

dest=”${PATH_MOVE_OUTPUT}” Start from the user-provided base path.
Append the job identifier:
```
dest="${dest}/_${JobUUID}"
```
- This creates a unique subdirectory per job, named with the JobUUID (prefixed by _), which:
  - Prevents collisions between runs,
  - Makes it easy to find outputs for a specific Tapis job later.
Create the directory:
```
mkdir -p -- "$dest"
```
- Ensures the full directory path exists before moving/copying.

3. Move the main output artifact (ArchiveName)#

mv -v -- "$ArchiveName" "$dest/"
echo "Moved main output: ${ArchiveName} -> ${dest}/" >> "$SUMMARY_SHORT"

Moves the primary result (either the folder or the ZIP, depending on earlier steps) into the job-specific directory under PATH_MOVE_OUTPUT.
Uses -v so the move is visible in stdout, and logs the move to SUMMARY_SHORT.

This is the key step that makes Tapis archiving fast: by moving the heavy data to a different filesystem (e.g., $SCRATCH), the execution directory stays small and light, so Tapis has much less to copy.

4. Copy additional top-level files for convenience#

find . -maxdepth 1 -type f -exec cp -t "$dest/" {} +
echo "Copied additional top-level files from $(pwd) -> ${dest}/" >> "$SUMMARY_SHORT"

Finds all top-level regular files in the current directory (e.g., logs, small config files, summary logs).
Copies them into the same destination directory, without removing them from the execution directory.
This gives you a consolidated result folder containing:
- The main output (folder or ZIP),
- Top-level logs and other important small files.

Meanwhile, the execution directory retains minimal, lightweight content so that:

Tapis’s default archive remains small and fast,
The “real” payload is safely stored in your chosen system path ($SCRATCH, $WORK, etc.).

This pattern lets the app treat PATH_MOVE_OUTPUT + ArchiveName as the main hook for high-volume outputs, while still giving Tapis a quick, small archive to handle.

bash_script_option_MOVE_OUTPUT = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "===================== MOVE MAIN OUTPUT TO FASTER STORAGE DESTINATION ===========" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo " --- move the result to destinations (if requested) ---" >> "$SUMMARY_SHORT"
    dest=""
    if [[ -n "${PATH_MOVE_OUTPUT:-}" ]]; then
      dest="${PATH_MOVE_OUTPUT}"
      
      echo " ---- move $ArchiveName ---- "
      echo "add JobUUID to destination path"
      dest="${dest}/_${JobUUID}"
      mkdir -p -- "$dest"
      mv -v -- "$ArchiveName" "$dest/"
      echo "Moved main output: ${ArchiveName} -> ${dest}/" >> "$SUMMARY_SHORT"

      find . -maxdepth 1 -type f -exec cp -t "$dest/" {} +
      echo "Copied additional top-level files from $(pwd) -> ${dest}/" >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
    
""")

25. Optional: Pre-Job Hook – User-Defined script run BEFORE main binary (PRE_JOB_SCRIPT)#

This block implements an optional user-defined “pre-job” hook that runs before the main OpenSees/OpenSeesPy execution begins. It allows users to insert their own setup logic — such as preparing input files, generating parameters, running a Python pre-processor, or checking environment conditions — directly inside the job’s execution directory.

What this block does

This block:

Announces the hook in the job summary log A header entry is written to SLURM-job-summary.log to indicate that the pre-job hook is being evaluated.
Checks whether the user provided the environment variable
```
PRE_JOB_SCRIPT
```
If the variable is empty or unset, the app simply logs:
```
No PRE_JOB_SCRIPT provided; skipping pre-job hook.
```
Resolves the script’s location
- If the user provides:
  - a full absolute path → used as-is
  - a filename only → the wrapper assumes the file is inside the job’s input directory (./)
Executes the script appropriately The logic distinguishes between:
- Executable scripts (chmod +x ) → run directly
- Non-executable files → run using bash
- Missing or invalid paths → log a warning
Error handling If the script fails, the wrapper:
- Logs a warning with the exit code
- Does not stop the job by default (the policy is intentionally lenient so users can choose whether a hook failure should stop the entire job)
The wrapper includes a commented line showing where a stricter “fail-fast” policy could be activated.

Why this feature matters#

This hook gives users considerable flexibility without modifying the core app, enabling tasks such as:

Creating randomized parameter sets
Unzipping or reorganizing input files
Generating model files on the fly
Preparing database connections
Logging metadata to custom files
Running small diagnostic checks before the HPC job starts

The hook is safe, optional, and entirely user-controlled.

How to use it#

Users simply include in their Tapis job submission:

"envVariables": {
    "PRE_JOB_SCRIPT": "prepare_inputs.sh"
}

…and place prepare_inputs.sh inside the Input Directory, or supply a full absolute path.

bash_script_option_PRE_JOB_SCRIPT = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "======================== PRE-JOB HOOK =============================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "OPTIONAL: pre-job hook" >> "$SUMMARY_SHORT"
    if [[ -n "${PRE_JOB_SCRIPT:-}" ]]; then
      echo "PRE_JOB_SCRIPT specified: ${PRE_JOB_SCRIPT}" >> "$SUMMARY_SHORT"
    
      # If user passed just a filename, assume it is in the current directory (inputDirectory)
      _pre="${PRE_JOB_SCRIPT}"
      if [[ ! "$_pre" = /* ]]; then
        _pre="./${_pre}"
      fi
    
      if [[ -x "$_pre" ]]; then
        echo "Running pre-job script (executable): $_pre" >> "$SUMMARY_SHORT"
        if ! "$_pre"; then
          rc=$?
          echo "WARNING: pre-job script exited with status $rc" >> "$SUMMARY_SHORT"
          # Decide policy: fail hard or continue
          # exit "$rc"
        fi
      elif [[ -f "$_pre" ]]; then
        echo "Running pre-job script via bash: $_pre" >> "$SUMMARY_SHORT"
        if ! bash "$_pre"; then
          rc=$?
          echo "WARNING: pre-job script (bash) exited with status $rc" >> "$SUMMARY_SHORT"
          # exit "$rc"
        fi
      else
        echo "WARNING: PRE_JOB_SCRIPT not found: $_pre" >> "$SUMMARY_SHORT"
      fi
    else
      echo "No PRE_JOB_SCRIPT provided; skipping pre-job hook." >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

26. Optional: Post-Job Hook – User-Defined script run AFTER main binary (POST_JOB_SCRIPT)#

This block implements an optional user-defined “post-job” hook that runs after the main executable finishes, but before the script leaves the job directory and before final timers/output handling are logged. It allows users to attach custom post-processing steps directly to the job workflow.

What this block does

This block:

Announces the hook in the job summary log It writes a section header to SLURM-job-summary.log:
```
OPTIONAL: post-job hook
```
so users can clearly see whether a post-job script was requested and how it behaved.
Checks whether the user provided POST_JOB_SCRIPT If the environment variable is unset or empty, the app logs:
```
No POST_JOB_SCRIPT provided; skipping post-job hook.
```
and proceeds without running anything extra.
Resolves the script path Similar to the pre-hook:
- If POST_JOB_SCRIPT is an absolute path, it is used directly.
- If it is just a filename, the wrapper assumes it lives in the current working directory (usually the inputDirectory):
```
_post="./${POST_JOB_SCRIPT}"
```
Executes the script in a flexible way The handler distinguishes between:
- Executable files (chmod +x post_hook.sh) → run directly:
```
"$_post"
```
- Non-executable files → run via:
```
bash "$_post"
```
- Missing/invalid paths → a warning is written to the summary log.
Error handling If the post-job script fails (non-zero exit code), the wrapper:
- Logs a warning containing the exit status
- By default, does not abort the job at this late stage
The code includes a commented exit “$rc” line to show where a stricter “fail on post-hook error” policy could be enabled if desired.

Why this feature matters#

The post-job hook provides a convenient place to run custom post-processing inside the same job, without editing the main Tapis app or wrapper script. Typical use cases include:

Aggregating or compressing output files
Creating summary figures or CSV tables
Running validation checks on the results
Writing additional custom logs or metadata
Pushing results into user-specific directory structures (within the execution system)
Cleaning up intermediate scratch data while keeping key outputs

Because the hook runs after the main program finishes, it’s a natural place to attach “last step” logic.

How to use it#

Users can specify the hook via an environment variable in their Tapis job:

"envVariables": {
  "POST_JOB_SCRIPT": "postprocess_results.sh"
}

and place postprocess_results.sh in the Input Directory (or specify an absolute path).

The wrapper will then:

resolve the path,
execute it either as an executable or via bash,
and log any warnings if the script exits with a non-zero status.

bash_script_option_POST_JOB_SCRIPT = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "======================== POST-JOB HOOK ============================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "OPTIONAL: post-job hook" >> "$SUMMARY_SHORT"
    if [[ -n "${POST_JOB_SCRIPT:-}" ]]; then
      echo "POST_JOB_SCRIPT specified: ${POST_JOB_SCRIPT}" >> "$SUMMARY_SHORT"
    
      _post="${POST_JOB_SCRIPT}"
      if [[ ! "$_post" = /* ]]; then
        _post="./${_post}"
      fi
    
      if [[ -x "$_post" ]]; then
        echo "Running post-job script (executable): $_post" >> "$SUMMARY_SHORT"
        if ! "$_post"; then
          rc=$?
          echo "WARNING: post-job script exited with status $rc" >> "$SUMMARY_SHORT"
          # Decide policy: fail or continue; usually continue:
          # exit "$rc"
        fi
      elif [[ -f "$_post" ]]; then
        echo "Running post-job script via bash: $_post" >> "$SUMMARY_SHORT"
        if ! bash "$_post"; then
          rc=$?
          echo "WARNING: post-job script (bash) exited with status $rc" >> "$SUMMARY_SHORT"
          # exit "$rc"
        fi
      else
        echo "WARNING: POST_JOB_SCRIPT not found: $_post" >> "$SUMMARY_SHORT"
      fi
    else
      echo "No POST_JOB_SCRIPT provided; skipping post-job hook." >> "$SUMMARY_SHORT"
    fi
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

27. Change Directory (cd) INTO Input Directory#

Change to Input Directory (with Logging)

What this block does

This script block switches the working directory to the job’s designated input directory and records that action in the short summary log.

It appends a clearly marked header and footer to SUMMARY_SHORT, making it easy to see when the script attempts to cd into $inputDirectory.

The ‘cd – “$inputDirectory”’ command safely handles paths that may contain spaces or begin with a dash, and the subsequent pwd call confirms the new working directory, writing the resolved path back to SUMMARY_SHORT for traceability and debugging.

bash_script_cd_InputDirectory_IN = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "==================== CD INTO INPUT DIRECTORY ======================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    cd -- "$inputDirectory" 
    echo "Changed directory to: $(pwd)" >> "$SUMMARY_SHORT"
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

28. Change Directory (cd) OUT OF Input Directory#

Change Back to Parent Directory (with Logging)

What this block does

This script block moves the working directory up one level in the directory hierarchy and logs the change to SUMMARY_SHORT.

It first appends a visual separator line to make the action easy to spot in the summary log, then runs ‘cd ..’ to go to the parent directory.

Finally, it records the new working directory using pwd, writing the resolved path to SUMMARY_SHORT so it’s clear where subsequent commands will execute.

bash_script_cd_InputDirectory_OUT = textwrap.dedent("""
    echo "=start==============================================================" >> "$SUMMARY_SHORT"
    echo "=================== CD BACK FROM INPUT DIRECTORY ==================" >> "$SUMMARY_SHORT"
    echo "===================================================================" >> "$SUMMARY_SHORT"
    echo "cd back one folder"
    cd ..
    echo "changed directory back to: $(pwd)" >> "$SUMMARY_SHORT"
    echo "=end===============================================================" >> "$SUMMARY_SHORT"
""")

Assemble Main Wrapper File: tapisjob_app.sh#

This tapisjob_app.sh skeleton is the master wrapper script that the notebook assembles and then packs into the app ZIP. All the placeholders in it get replaced with the app-specific blocks you defined earlier (initialize, logging, module/pip setup, OpenSeesPy handling, timers, etc.).

tapisjob_app.sh is the orchestrator. The notebook builds it from your modular chunks, then zips it before uploading, together with the rest of the app files, to the TACC system. Every job launched by this app runs through this script, which standardizes logging, environment setup, execution, and output handling for OpenSees/OpenSeesPy workflows on DesignSafe/TACC.

Details of tapisjob_app.sh (generated job wrapper)

This file is the main SLURM/Tapis job driver that the app runs for each submission. It is auto-generated in the notebook by stitching together modular code blocks (the __run_*__, __option_*__, and __echoSummary_*__ placeholders). Once assembled, this script is included in the app’s runtime ZIP and uploaded to the TACC system.

At runtime, tapisjob_app.sh is responsible for:

1. Initialization and argument validation#

Enforces required arguments: binary name, input script, and the UseMPI flag.
Captures inputDirectory, JobUUID, and the starting directory.
Starts global timers for the entire script and enables safe shell behavior:
```
set -euo pipefail
set -x
```

2. Job summary + environment logging#

Initializes the compact summary log (SUMMARY_SHORT) and the full environment log.
Records app metadata, key paths ($HOME, $WORK, $SCRATCH), user configuration, and environment variables that control:
- module loading,
- pip installs,
- file copy-in,
- unzipping,
- output movement and zipping.

3. Move into the input directory & optional pre-run preparation#

cd -- "$inputDirectory"

From inside the job’s working folder, the script may:

Copy in extra files or directories specified via PATH_COPY_IN_LIST.
Unzip any input archives (UNZIP_FILES_LIST) so the run sees expanded input data.

If copy-in is enabled, the script also initializes a copy-in manifest, which records exactly which files or directories were staged into the working directory. This manifest is later used for optional cleanup.

4. Environment setup (modules + pip)#

The script performs layered environment configuration:

Ensures the module command is available.
Optionally loads modules from:
- a module file (MODULE_LOADS_FILE), and/or
- a comma-separated list (MODULE_LOADS_LIST).
Optionally stages the TACC-compiled OpenSeesPy shared library (GET_TACC_OPENSEESPY).
Optionally installs Python packages from:
- a requirements-style file (PIP_INSTALLS_FILE), and/or
- a comma-separated list (PIP_INSTALLS_LIST).

After this phase, the job has a reproducible, fully documented runtime environment.

5. Launcher selection and main run#

Chooses how to launch the binary:
- direct execution (sequential), or
- ibrun (MPI) if UseMPI is true-like.
Starts a binary-run timer, executes the application, and:
- On error: records run/total timings, logs the error code, and exits with that code.
- On success: logs a “NO ERROR” message and continues.

6. Post-run cleanup inside the input directory (runtime artifacts)#

If TACC OpenSeesPy was staged in (GET_TACC_OPENSEESPY), the temporary opensees.so is removed after the run to avoid polluting the directory.
Other short-lived runtime artifacts created solely for execution (temporary launch helpers, scratch symlinks, etc.) are cleaned up here if applicable.

7. Optional cleanup of copy-in files (end-of-job hygiene)#

If the user enables:

DELETE_COPIED_IN_ON_EXIT=1

the script performs a controlled cleanup of files and directories that were copied in before the run:

A cleanup function is registered via a Bash EXIT trap, ensuring it runs:
- on normal completion,
- on application failure,
- or on unexpected script termination.
The cleanup logic:
- reads the copy-in manifest created during the pre-run copy phase,
- deletes only the paths that were explicitly copied into the working directory,
- refuses to delete absolute paths, parent traversals (..), or anything outside the job directory.
Each deletion is logged to SUMMARY_SHORT for transparency and traceability.

This keeps job directories clean while ensuring deletion is explicit, auditable, and opt-in.

8. Return to the parent directory and archive preparation#

cd ..
ArchiveName="${inputDirectory}"

Back in the parent directory, the script treats ArchiveName as the main output artifact:

By default, this is the original output folder.
If ZIP_OUTPUT_SWITCH is enabled:
- the folder is repacked into inputDirectory.zip,
- the original directory is removed,
- ArchiveName is updated to reference the ZIP instead.

9. Optional output move (fast archive strategy)#

If PATH_MOVE_OUTPUT is set:
- a job-specific subdirectory (using JobUUID) is created under the chosen base path (e.g., $SCRATCH or $WORK),
- the main output artifact (ArchiveName) is moved there,
- top-level logs and summaries are copied alongside it.

This minimizes the size of the execution directory and makes Tapis archiving significantly faster.

10. Final timing footer and completion#

Writes the total script end time and full runtime (setup + run + post-processing) into SUMMARY_SHORT.
Prints a final DONE! marker to clearly indicate wrapper completion.

bash_script_tapisJob_app_base = textwrap.dedent("""\

    __run_INITIALIZE__
    
    __echoSummary_START__
    
    __echoSummary_VERBOSE__

    __cd_InputDirectory_IN__

    __option_COPY_FILES__

    __option_UNZIP__

    __option_MODULE_ENV_SETUP__

    __option_MODULE_LOAD_FILE__

    __option_MODULE_LOAD_LIST__

    __option_COPY_OPENSEESPY__

    __option_OPS_MODULES_LOAD__

    __option_PyLauncher_MODULES_LOAD__

    __option_PYTHON_ALIAS__
    
    __option_PIP_FILE__

    __option_PIP_LIST__
    
    echo "###############################################################################" >> "$SUMMARY_SHORT"
    echo "DONE with installations" >> "$SUMMARY_SHORT"
    echo "" >> "$SUMMARY_SHORT"

    __option_PRE_JOB_SCRIPT__

    __run_CHOOSE_LAUNCHER__

    __run_RUN_JOB__

    __echoTimers_AFTER__

    __option_DELETE_OPENSEESPY__

    __option_POST_JOB_SCRIPT__

    __cd_InputDirectory_OUT__

    ArchiveName="${inputDirectory}"

    __option_ZIP_OUTPUT__
    
    __option_MOVE_OUTPUT__

    __echoTimers_END__

    echo "DONE!"
""")

Replace batch_script patches into the Main Wrapper File#

A. Replace batch_script patches – All Apps#

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__run_INITIALIZE__", bash_script_run_INITIALIZE)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__app_Author_Info__", app_Author_Info)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__cd_InputDirectory_IN__", bash_script_cd_InputDirectory_IN)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__cd_InputDirectory_OUT__", bash_script_cd_InputDirectory_OUT)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__run_CHOOSE_LAUNCHER__", bash_script_run_CHOOSE_LAUNCHER)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__run_RUN_JOB__", bash_script_run_RUN_JOB)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__echoSummary_START__", bash_script_echoSummary_START)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__echoTimers_START__", bash_script_echoTimers_START)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__echoTimers_AFTER__", bash_script_echoTimers_AFTER)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__echoTimers_END__", bash_script_echoTimers_END)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_MODULE_ENV_SETUP__", bash_script_option_MODULE_ENV_SETUP)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_MODULE_LOAD_LIST__", bash_script_option_MODULE_LOAD_LIST)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_OPS_MODULES_LOAD__", bash_script_option_OPS_MODULES_LOAD)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_PyLauncher_MODULES_LOAD__", bash_script_option_PYLAUNCHER_MODULES_LOAD)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_PIP_LIST__", bash_script_option_PIP_LIST)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_COPY_OPENSEESPY__", bash_script_option_COPY_OPENSEESPY)
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_DELETE_OPENSEESPY__", bash_script_option_DELETE_OPENSEESPY)

bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_PYTHON_ALIAS__", bash_script_option_PYTHON_ALIAS)


    
bash_script_tapisJob_app_base = bash_script_tapisJob_app_base.replace("__option_PIP_FILE__", bash_script_option_PIP_FILE)

B. Replace batch_script patches – Agnostic App#

if do_makeApp:
    bash_script_tapisJob_app = bash_script_tapisJob_app_base
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__echoSummary_ARGS__", bash_script_echoSummary_ARGS)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__echoSummary_ENV_VARS__", bash_script_echoSummary_ENV_VARS)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__echoSummary_MPI__", bash_script_echoSummary_MPI)
    
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__echoSummary_VERBOSE__", bash_script_echoSummary_VERBOSE)
    
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_COPY_FILES__", bash_script_option_COPY_FILES)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_UNZIP__", bash_script_option_UNZIP)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_MODULE_LOAD_FILE__", bash_script_option_MODULE_LOAD_FILE)
    

    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_PRE_JOB_SCRIPT__", bash_script_option_PRE_JOB_SCRIPT)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_POST_JOB_SCRIPT__", bash_script_option_POST_JOB_SCRIPT)
    
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_ZIP_OUTPUT__", bash_script_option_ZIP_OUTPUT)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__option_MOVE_OUTPUT__", bash_script_option_MOVE_OUTPUT)

    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__app_id__", app_id)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__app_version__", app_version)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__app_description__", app_description)
    bash_script_tapisJob_app = bash_script_tapisJob_app.replace("__app_helpUrl__", app_helpUrl)

    
    thisFilename_sh = "tapisjob_app.sh"

    with open(f"{appPath_Local}/{thisFilename_sh}", "w") as f:
        f.write(bash_script_tapisJob_app)

if do_makeApp:
    OpsUtils.show_text_file_in_accordion(appPath_Local, [thisFilename_sh], background='#d4fbff', showLineNumbers=False)

C. Replace batch_script patches – OpenSeesPy App#

if do_makeApp_OpsPy:
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_base
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__echoSummary_ARGS__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__echoSummary_ENV_VARS__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__echoSummary_MPI__", '')
    
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__echoSummary_VERBOSE__", '')
    
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_COPY_FILES__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_UNZIP__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_MODULE_LOAD_FILE__", '')

    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_PRE_JOB_SCRIPT__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_POST_JOB_SCRIPT__", '')    
    
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_ZIP_OUTPUT__", '')
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__option_MOVE_OUTPUT__", '')

    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__app_id__", app_id_OpsPy)
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__app_version__", app_version_OpsPy)
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__app_description__", app_description_OpsPy)
    bash_script_tapisJob_app_OpsPy = bash_script_tapisJob_app_OpsPy.replace("__app_helpUrl__", app_helpUrl_OpsPy)
    
    thisFilename_sh_OpsPy = "tapisjob_app.sh"

    with open(f"{appPath_Local_OpsPy}/{thisFilename_sh_OpsPy}", "w") as f:
        f.write(bash_script_tapisJob_app_OpsPy)
    

if do_makeApp_OpsPy:
    OpsUtils.show_text_file_in_accordion(appPath_Local, [thisFilename_sh_OpsPy], background='#d4fbff', showLineNumbers=False)

E. Create tapisjob_app.zip – App Zip File#

OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'zip_file.py')

if do_makeApp:
    zip_path = os.path.join(appPath_Local, container_filename)
    OpsUtils.zip_file(zip_path,thisFilename_sh,bash_script_tapisJob_app)

zip_path /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11/designsafe-agnostic-app.zip

if do_makeApp_OpsPy:
    zip_path_OpsPy = os.path.join(appPath_Local_OpsPy, container_filename_OpsPy)
    OpsUtils.zip_file(zip_path_OpsPy,thisFilename_sh_OpsPy,bash_script_tapisJob_app_OpsPy)

zip_path /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15/designsafe-openseespy-s3.zip

F. File Check – Visualize File Contents in Local (Development) Path#

Look at the files we have written and check for typos or formatting errors.

Validation and Line-Numbered Views

The notebook shows the app-definition files with and without line numbers:

* **Without line numbers**: ideal when you need to copy the JSON content into another tool or system without extra markup.
* **With line numbers**: ideal for debugging JSON validation errors (e.g., “invalid JSON at line 127, column 10”). You can quickly navigate to the offending line in the notebook view.

This small UX choice dramatically simplifies the **app-validation cycle**: inspect, fix, re-validate, repeat.

Show Files for Content – No Line Numbers#

Good for copying content

showLineNumbers = False
accordionTitle = f'Local Files NO LINE NUMBERS'
if do_makeApp or do_makeApp_OpsPy:
    here_out = widgets.Output()
    here_accordion = widgets.Accordion(children=[here_out])
    # here_accordion.selected_index = 0
    here_accordion.set_title(0, accordionTitle)
    display(here_accordion)
    
    with here_out:
        if do_makeApp:
            appfiles = os.listdir(appPath_Local); # same as before
            print('app_id:',app_id)
            print('appPath_Local:',appPath_Local)
            print('appfiles:',appfiles)
            OpsUtils.show_text_file_in_accordion(appPath_Local, appfiles, background='lightyellow', showLineNumbers=showLineNumbers)
            if len(appfiles)==0:
                here_accordion.set_title(0, 'ERROR!!!!! THERE ARE NO FILES!!!')
        if do_makeApp_OpsPy:
            appfiles_OpsPy = os.listdir(appPath_Local_OpsPy); # same as before
            print('\napp_id_opsPy:',app_id_OpsPy)
            print('appPath_Local_OpsPy:',appPath_Local_OpsPy)
            print('appfiles_OpsPy:',appfiles_OpsPy)
            OpsUtils.show_text_file_in_accordion(appPath_Local_OpsPy, appfiles_OpsPy, background='lightyellow', showLineNumbers=showLineNumbers)
            if len(appfiles_OpsPy)==0:
                here_accordion.set_title(0,'ERROR!!!!!! THERE ARE NO FILES!!!')

Show Files for Debugging – SHOW Line Numbers#

Useful for Error-Source Identification in Debugging

showLineNumbers = True
accordionTitle = f'Local Files SHOW LINE NUMBERS'
if do_makeApp or do_makeApp_OpsPy:
    here_out = widgets.Output()
    here_accordion = widgets.Accordion(children=[here_out])
    # here_accordion.selected_index = 0
    here_accordion.set_title(0, accordionTitle)
    display(here_accordion)
    
    with here_out:
        if do_makeApp:
            appfiles = os.listdir(appPath_Local); # same as before
            print('app_id:',app_id)
            print('appPath_Local:',appPath_Local)
            print('appfiles:',appfiles)
            OpsUtils.show_text_file_in_accordion(appPath_Local, appfiles, background='lightyellow', showLineNumbers=showLineNumbers)
            if len(appfiles)==0:
                here_accordion.set_title(0,'ERROR!!!!!! THERE ARE NO FILES!!!')
        if do_makeApp_OpsPy:
            appfiles_OpsPy = os.listdir(appPath_Local_OpsPy); # same as before
            print('\napp_id_opsPy:',app_id_OpsPy)
            print('appPath_Local_OpsPy:',appPath_Local_OpsPy)
            print('appfiles_OpsPy:',appfiles_OpsPy)
            OpsUtils.show_text_file_in_accordion(appPath_Local_OpsPy, appfiles_OpsPy, background='lightyellow', showLineNumbers=showLineNumbers)
            if len(appfiles_OpsPy)==0:
                here_accordion.set_title(0,'ERROR!!!!!! THERE ARE NO FILES!!!')

Validate App Files Locally#

OpsUtils.show_text_file_in_accordion(PathOpsUtils, 'validate_app_folder.py')

if do_makeApp:
    appfiles = os.listdir(appPath_Local)
    if len(appfiles_OpsPy)==0:
        print('ERROR!!!!! THERE ARE NO FILES!!!')
    validation = OpsUtils.validate_app_folder(appPath_Local,appfiles)
    if not validation:
        print('Validation Failed: stopping here!!!!')
        a = 3/0

🔍 Validating app folder: /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11

✅ All required files are present.

📄 App ID: designsafe-agnostic-app
📄 App Name: (missing)
📄 Version: 1.3.11
🔧 Parameters: []
📦 Inputs: []
📤 Outputs: []

App Keys: ['id', 'version', 'description', 'owner', 'enabled', 'runtime', 'runtimeVersion', 'runtimeOptions', 'containerImage', 'jobType', 'maxJobs', 'maxJobsPerUser', 'strictFileInputs', 'jobAttributes', 'tags', 'notes']

✅ Basic validation complete. App folder looks good!

if do_makeApp_OpsPy:
    appfiles_OpsPy = os.listdir(appPath_Local_OpsPy)
    if len(appfiles_OpsPy)==0:
        print('ERROR!!!!! THERE ARE NO FILES!!!')
    validation = OpsUtils.validate_app_folder(appPath_Local_OpsPy,appfiles_OpsPy)
    if not validation:
        print('Validation Failed: stopping here!!!!')
        a = 3/0

🔍 Validating app folder: /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15

✅ All required files are present.

📄 App ID: designsafe-openseespy-s3
📄 App Name: (missing)
📄 Version: 1.2.15
🔧 Parameters: []
📦 Inputs: []
📤 Outputs: []

App Keys: ['id', 'version', 'description', 'owner', 'enabled', 'runtime', 'runtimeVersion', 'runtimeOptions', 'containerImage', 'jobType', 'maxJobs', 'maxJobsPerUser', 'strictFileInputs', 'jobAttributes', 'tags', 'notes']

✅ Basic validation complete. App folder looks good!

Deploy the App#

Upload Files to appPath_Tapis#

Although Tapis can create directories and upload files, filesystem operations through the API tend to be slow.

If you have direct access to the target directory (e.g., via JupyterHub or SSH), it is usually faster to create folders and copy files using Python’s os and shutil utilities—though note that the destination filesystem itself may still be slow, regardless of the method used.

File-Transfer Options:
- Using Tapis: When using Tapis to transfer files, you must specify URI-style paths (e.g., tacc.work2://…).
  - This method does not work from within DesignSafe’s JupyterHub because there is not SSH protocol there. here is the error:
    ForbiddenError: message: FILES_CLIENT_SSH_PERM_DENIED OboTenant: designsafe OboUser: silvia Operation: mkdir System: cloud.data EffectiveUser: silvia Host: cloud.data.tacc.utexas.edu Path: /work2 Error: SFTP error (SSH_FX_PERMISSION_DENIED): Permission denied
- Using Python/Shell: When using Python or shell commands, you instead provide local filesystem paths (e.g., ../Work/…).
  - The method is fast and reliable because Work is mounted on JupyterHub in DesignSafe.

The process described below defines arguments for both methods (appPath_Tapis and appPath_Tapis_local) so you can choose whether to perform uploads via Tapis or directly through Python, depending on what is most convenient for your workflow.

FileUpload_method = 'Python'; # options: 'Python' or 'Tapis'

Make the App Directory#

The apps in this folder are the ones that area actually uploaded.

if do_makeApp:
    print('FileUpload_method',FileUpload_method)
    if FileUpload_method == 'Tapis':
        print('app_system_id',app_system_id)
        print('appPath_Tapis',appPath_Tapis)
        t.files.mkdir(systemId=app_system_id, path=appPath_Tapis)
        print('\nCreated appPath_Tapis',appPath_Tapis)
    else:
        print('appPath_Tapis_local',appPath_Tapis_local)
        os.makedirs(appPath_Tapis_local, exist_ok=True)
        print('\nCreated: appPath_Tapis',appPath_Tapis)    

FileUpload_method Python
appPath_Tapis_local /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11

Created: appPath_Tapis /work2/05072/silvia/stampede3/apps/designsafe-agnostic-app/1.3.11

if do_makeApp_OpsPy:
    print('FileUpload_method',FileUpload_method)
    if FileUpload_method == 'Tapis':
        print('app_system_id_OpsPy',app_system_id_OpsPy)
        print('appPath_Tapis_OpsPy',appPath_Tapis_OpsPy)
        t.files.mkdir(systemId=app_system_id_OpsPy, path=appPath_Tapis_OpsPy)
        print('\nCreated: appPath_Tapis_OpsPy',appPath_Tapis_OpsPy)
    else:
        print('appPath_Tapis_local_OpsPy',appPath_Tapis_local_OpsPy)
        os.makedirs(appPath_Tapis_local_OpsPy, exist_ok=True)
        print('\nCreated: appPath_Tapis_local_OpsPy',appPath_Tapis_local_OpsPy)
        

FileUpload_method Python
appPath_Tapis_local_OpsPy /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

Created: appPath_Tapis_local_OpsPy /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

Upload/Copy Files to Deployment System#

if do_makeApp:
    appfiles = os.listdir(appPath_Local)
    if len(appfiles)==0:
        print('ERROR!!!!! THERE ARE NO FILES!!!')
    for fname in appfiles:
        fpath = f'{appPath_Local}/{fname}'
        if FileUpload_method == 'Tapis':
            dest_file_path=f'{appPath_Tapis}/{fname}'
            t.upload(source_file_path=fpath,
                     system_id=app_system_id,
                     dest_file_path=dest_file_path)
            print(f'\nTapis-uploaded {fpath} to {dest_file_path} on {app_system_id}')
        else:
            shutil.copy(fpath, appPath_Tapis_local)
            print(f'\nOS-copied {fpath} to {appPath_Tapis_local}')

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11/ReadMe.MD to /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11/app.json to /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11/tapisjob_app.sh to /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-agnostic-app/1.3.11/designsafe-agnostic-app.zip to /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11

if do_makeApp_OpsPy:
    appfiles_OpsPy = os.listdir(appPath_Local_OpsPy)
    if len(appfiles_OpsPy)==0:
        print('ERROR!!!!! THERE ARE NO FILES!!!')
    for fname in appfiles_OpsPy:
        fpath = f'{appPath_Local_OpsPy}/{fname}'
        if FileUpload_method == 'Tapis':
            dest_file_path=f'{appPath_Tapis_OpsPy}/{fname}'
            t.upload(source_file_path=fpath,
                     system_id=app_system_id_OpsPy,
                     dest_file_path=f'{appPath_Tapis_OpsPy}/{fname}')
            print(f'\nTapis-uploaded {fpath} to {dest_file_path} on {app_system_id}')
        else:
            shutil.copy(fpath, appPath_Tapis_local_OpsPy)
            print(f'\nOS-copied {fpath} to {appPath_Tapis_local_OpsPy}')

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15/ReadMe.MD to /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15/app.json to /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15/tapisjob_app.sh to /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

OS-copied /home/jupyter/MyData/myAuthoredTapisApps/designsafe-openseespy-s3/1.2.15/designsafe-openseespy-s3.zip to /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15

Check Files on Deployment System To Verify Upload#

import glob
if do_makeApp:
    here_out = widgets.Output()
    here_accordion = widgets.Accordion(children=[here_out])
    here_accordion.selected_index = 0
    here_accordion.set_title(0, f'Verify Upload -- app-id={app_id}')
    display(here_accordion)
    
    with here_out:
        print('app_system_id:',app_system_id)
        print('appPath_Tapis:',appPath_Tapis)
        print('')
        print('appPath_Tapis_local:',appPath_Tapis_local)
        if FileUpload_method == 'Tapis':
            appfiles = t.files.listFiles(systemId=app_system_id, path=appPath_Tapis)
            for thisF in appfiles:
                print(thisF)
                print('')            
        else:
            appfiles = os.listdir(appPath_Tapis_local)
            OpsUtils.show_text_file_in_accordion(appPath_Tapis_local, appfiles, background='cream', showLineNumbers=False)
        if len(appfiles)==0:
            here_accordion.set_title(0,'ERROR!!!!!! THERE ARE NO FILES!!!')

if do_makeApp_OpsPy:
    here_out = widgets.Output()
    here_accordion = widgets.Accordion(children=[here_out])
    here_accordion.selected_index = 0
    here_accordion.set_title(0, f'Verify Upload -- app-id={app_id_OpsPy}')
    display(here_accordion)
    
    with here_out:
        print('app_system_id:',app_system_id_OpsPy)
        print('appPath_Tapis:',appPath_Tapis_OpsPy)
        print('')
        print('appPath_Tapis_local:',appPath_Tapis_local_OpsPy)
        if FileUpload_method == 'Tapis':
            appfiles_OpsPy = t.files.listFiles(systemId=app_system_id_OpsPy, path=appPath_Tapis_OpsPy)
            for thisF in appfile_OpsPys:
                print(thisF)
                print('')            
        else:
            appfiles_OpsPy = os.listdir(appPath_Tapis_local_OpsPy)
            OpsUtils.show_text_file_in_accordion(appPath_Tapis_local_OpsPy, appfiles_OpsPy, background='cream', showLineNumbers=False)
        if len(appfiles_OpsPy)==0:
            here_accordion.set_title(0,'ERROR!!!!!! THERE ARE NO FILES!!!')

Register The App#

This creates the actual App record that Jobs can run.

This is when we send the json content to Tapis, where it it “memorizes” it.

if do_makeApp:
    # Create (or create a new version) of the app
    with open(f'{appPath_Local}/app.json') as f:
        app_def = json.load(f)
    t.apps.createAppVersion(**app_def)

if do_makeApp_OpsPy:
    # Create (or create a new version) of the app
    with open(f'{appPath_Local_OpsPy}/app.json') as f:
        app_def = json.load(f)
    t.apps.createAppVersion(**app_def)

List All Tapis Apps to Verify Registration#

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List all apps')
display(here_accordion)

with here_out:
    listType = 'ALL' # Include all items requester is authorized to view. Includes check for READ or MODIFY permission.
    select = 'id,created,description,version,owner' # Attributes to return in each result.
    orderBy = 'created(asc)'
    results = t.apps.getApps( orderBy=orderBy,
                             select=select)  
    for thisRes in results:
        print('--')
        print(thisRes)

Access App Schema on Tapis to Validate Registration#

OpsUtils.show_text_file_in_accordion(PathOpsUtils, ['getAppLatestVersion.py','display_tapis_app_schema.py'])

appMetaData = t.apps.getAppLatestVersion(appId=app_id)

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List the new app')
display(here_accordion)

with here_out:
    OpsUtils.display_tapis_app_schema(appMetaData)
thisAppVersion = appMetaData.version
isPublic = appMetaData.isPublic
here_accordion.set_title(0, f'app schema: {app_id}  -- version = {thisAppVersion}    --  isPublic = {isPublic}')

appMetaData_OpsPy = t.apps.getAppLatestVersion(appId=app_id_OpsPy)

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List the new app')
display(here_accordion)

with here_out:
    OpsUtils.display_tapis_app_schema(appMetaData_OpsPy)
thisAppVersion_OpsPy = appMetaData_OpsPy.version
isPublic_OpsPy = appMetaData_OpsPy.isPublic
here_accordion.set_title(0, f'app schema: {app_id_OpsPy}  -- version = {thisAppVersion_OpsPy}    --  isPublic = {isPublic_OpsPy}')

Manage Public App#

Manage App isPublic Status#

Make The App Public (optional)#

if makePublic:
    print('makePublic')
    t.apps.shareAppPublic(appId=app_id)

makePublic

if makePublic_OpsPy:
    print('makePublic_OpsPy')
    t.apps.shareAppPublic(appId=app_id_OpsPy)

makePublic_OpsPy

or Remove The App From Public Access (optional)#

if makeUnPublic:
    print('makeUnPublic')
    t.apps.unShareAppPublic(appId=app_id)

if makeUnPublic_OpsPy:
    print('makeUnPublic_OpsPy')
    t.apps.unShareAppPublic(appId=app_id_OpsPy)

Verify isPublic Status#

appMetaData = t.apps.getAppLatestVersion(appId=app_id)

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List the new app')
display(here_accordion)

with here_out:
    OpsUtils.display_tapis_app_schema(appMetaData)
thisAppVersion = appMetaData.version
isPublic = appMetaData.isPublic
here_accordion.set_title(0, f'app schema: {app_id}  -- version = {thisAppVersion}    --  isPublic = {isPublic}')

appMetaData_OpsPy = t.apps.getAppLatestVersion(appId=app_id_OpsPy)

here_out = widgets.Output()
here_accordion = widgets.Accordion(children=[here_out])
# here_accordion.selected_index = 0
here_accordion.set_title(0, f'List the new app')
display(here_accordion)

with here_out:
    OpsUtils.display_tapis_app_schema(appMetaData_OpsPy)
thisAppVersion_OpsPy = appMetaData_OpsPy.version
isPublic_OpsPy = appMetaData_OpsPy.isPublic
here_accordion.set_title(0, f'app schema: {app_id_OpsPy}  -- version = {thisAppVersion_OpsPy}    --  isPublic = {isPublic_OpsPy}')

Set Permissions for Public App#

If you want to make your app public, You must make files readable and the folders executable so that the app can copy the app-definition file (.zip) to the user’s execution directory.

To allow anyone to copy a file from your folder, you must ensure:

The file is readable (+r) by your group (g) and others (o):

bash:
```
chmod go+r yourfile.zip
```

python:

file_perms = stat.S_IRGRP | stat.S_IROTH

Every directory in the path is executable (traversable) (+x) by your group (g) and others (o):

bash:

chmod go+x /workId/groupID/username
chmod go+x /workId/groupID/username/system
chmod go+x /workId/groupID/username/system/apps
chmod go+x /workId/groupID/username/system/apps/app_name
chmod go+x /workId/groupID/username/system/apps/app_name/app_version

python:

dir_perms = stat.S_IXGRP |  stat.S_IXOTH

NOTE: On Stampede3 /work2 is group-write by default, but NOT others-readable.

In Python, Permissions are changed with st.st_mode | perms, so:

Your existing user perms are preserved.
Any existing group/other bits are preserved; we’re only adding what’s missing.
You will assign permissions once the process of creating folders and copying app files is complete.

def add_perms(path: Path, perms: int):
    """OR in the given permission bits without removing existing ones."""
    st = os.stat(path)
    old_mode = st.st_mode & 0o777
    new_mode = old_mode | perms
    print(f"try: {path} : {oct(old_mode)} -> {oct(new_mode)}")
    os.chmod(path, new_mode)
    print(f"{path} : {oct(old_mode)} -> {oct(new_mode)}")
    

File Permissions#

For the files: group&others get read (can copy)

We want to make all the files accesible to the group+others: Tapis will want the .zip file. Users will want the others.

file_perms = stat.S_IRGRP | stat.S_IROTH

if makePublic:
    if do_makeApp:
        appfiles = os.listdir(appPath_Tapis_local)
        for fname in appfiles:
            file_path = os.path.join(appPath_Tapis_local, fname)
            add_perms(file_path, file_perms)
        if len(appfiles)==0:
            print('ERROR!!!!! THERE ARE NO FILES!!!')
    if do_makeApp_OpsPy:
        appfiles_OpsPy = os.listdir(appPath_Tapis_local_OpsPy)
        for fname in appfiles_OpsPy:
            file_path_OpsPy = os.path.join(appPath_Tapis_local_OpsPy, fname)
            add_perms(file_path_OpsPy, file_perms)
        if len(appfiles_OpsPy)==0:
            print('ERROR!!!!! THERE ARE NO FILES!!!')

try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/ReadMe.MD : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/ReadMe.MD : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/designsafe-agnostic-app.zip : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/designsafe-agnostic-app.zip : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/tapisjob_app.sh : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/tapisjob_app.sh : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/app.json : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11/app.json : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/ReadMe.MD : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/ReadMe.MD : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/tapisjob_app.sh : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/tapisjob_app.sh : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/app.json : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/app.json : 0o660 -> 0o664
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/designsafe-openseespy-s3.zip : 0o660 -> 0o664
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15/designsafe-openseespy-s3.zip : 0o660 -> 0o664

Path/Directory Permissions#

For directories: group&others get execute (can traverse but not list contents)

We need to set these permission for every level of the path.

dir_perms = stat.S_IXGRP |  stat.S_IXOTH

# define a python function that builds directories.
def dirs_between(anchor: Path, descendant: Path):
    """
    Yield directories from anchor down to descendant (inclusive),
    assuming descendant is inside anchor.
    """
    anchor = anchor.resolve()
    descendant = descendant.resolve()

    # safety check: make sure descendant is under anchor
    try:
        descendant.relative_to(anchor)
    except ValueError:
        raise ValueError(f"{descendant} is not inside {anchor}")

    dirs = []
    current = descendant
    while True:
        dirs.append(current)
        if current == anchor:
            break
        current = current.parent

    return list(reversed(dirs))

if makePublic:
    if do_makeApp:
        start_dir = Path(appPath_Tapis_local_anchor)  # don't go above this
        end_dir = Path(os.path.abspath(os.path.expanduser(appPath_Tapis_local)))
        print('start_dir',start_dir)
        print('end_dir',end_dir)
        permission_dirs = dirs_between(start_dir,end_dir)
        # permission_dirs = dirs_below(end_dir)
        
        # set permissions
        for thisDir in permission_dirs:
            add_perms(thisDir, dir_perms)
        

start_dir /home/jupyter/Work/stampede3
end_dir /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11
try: /home/jupyter/Work/stampede3 : 0o711 -> 0o711
/home/jupyter/Work/stampede3 : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps : 0o711 -> 0o711
/home/jupyter/Work/stampede3/apps : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app : 0o711 -> 0o711
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11 : 0o755 -> 0o755
/home/jupyter/Work/stampede3/apps/designsafe-agnostic-app/1.3.11 : 0o755 -> 0o755

if makePublic_OpsPy:
    if do_makeApp_OpsPy:
        start_dir_OpsPy = Path(os.path.abspath(os.path.expanduser(user_WorkPath_base_local)))  # don't go above this
        end_dir_OpsPy = Path(os.path.abspath(os.path.expanduser(appPath_Tapis_local_OpsPy)))
        print('start_dir_OpsPy',start_dir_OpsPy)
        print('end_dir_OpsPy',end_dir_OpsPy)
        permission_dirs_OpsPy = dirs_between(start_dir_OpsPy,end_dir_OpsPy)
        
        # set permissions
        for thisDir_OpsPy in permission_dirs_OpsPy:
            add_perms(thisDir_OpsPy, dir_perms)

start_dir_OpsPy /home/jupyter/Work/stampede3
end_dir_OpsPy /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15
try: /home/jupyter/Work/stampede3 : 0o711 -> 0o711
/home/jupyter/Work/stampede3 : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps : 0o711 -> 0o711
/home/jupyter/Work/stampede3/apps : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3 : 0o711 -> 0o711
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3 : 0o711 -> 0o711
try: /home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15 : 0o755 -> 0o755
/home/jupyter/Work/stampede3/apps/designsafe-openseespy-s3/1.2.15 : 0o755 -> 0o755

print('Done!!!')

Done!!!