Download All Job Output

Try on DesignSafe

Download All Job Output#

Get list of ALL job-output files recursively with OPTIONAL download

by Silvia Mazzoni, DesignSafe, 2025

I have created a single python function that will drill into the job output files recursively and returns both relative and full paths (not absolute, these are paths within tapis).
This function gives the user the option to download the files locally.

In this notebook we will copy the files over to our local project directory within DesignSafe.

Using local utilities library

Connect to Tapis#

Yes, you need to first connect to Tapis, this authenticates you

t=OpsUtils.connect_tapis()

 -- Checking Tapis token --
 Token loaded from file. Token is still valid!
 Token expires at: 2025-12-07T00:02:57+00:00
 Token expires in: 3:03:22.438761
-- AUTHENTICATED VIA SAVED TOKEN --

User Input – job id#

jobUuid = '4dfa35e1-15cd-48fd-a090-f348544dee1f-007'

Download Files#

projectDir = os.path.expanduser('~/MyData/tmp_removeME')
if os.path.exists(projectDir):
    print(f"The path exists: {projectDir}")
else:
    print(f"Path does not exist: {projectDir}")
    os.makedirs(projectDir)
    print(f"Create directory: {projectDir}")

Path does not exist: /home/jupyter/MyData/tmp_removeME
Create directory: /home/jupyter/MyData/tmp_removeME

TapisJobFilesDict = OpsUtils.get_tapis_job_all_files(t, jobUuid, displayIt=10, target_dir=projectDir, overwrite=True)

List files in new location#

AllContents = os.listdir(projectDir)
print(f"\n Get contents using: os.listdir('{projectDir}'): {AllContents}")

 Get contents using: os.listdir('/home/jupyter/MyData/tmp_removeME'): ['opensees.zip', 'tapisjob.env', 'tapisjob.out', 'tapisjob.sh', 'tapisjob_app.sh', '.ipynb_checkpoints', 'inputDirectory']

List files recursively#

AllFiles = OpsUtils.get_files_recursive(path=projectDir)

----------------------------

DIRECTORY: /home/jupyter/MyData/tmp_removeME
  5 files & 2 directories:
    FILE: /home/jupyter/MyData/tmp_removeME/opensees.zip
    FILE: /home/jupyter/MyData/tmp_removeME/tapisjob.env
    FILE: /home/jupyter/MyData/tmp_removeME/tapisjob.out
    FILE: /home/jupyter/MyData/tmp_removeME/tapisjob.sh
    FILE: /home/jupyter/MyData/tmp_removeME/tapisjob_app.sh
----------------------------

DIRECTORY: /home/jupyter/MyData/tmp_removeME/inputDirectory

Delete directory#

Clean up: this was just a temporary directory

# let's use shutil utility to delete non-empty directory. BE CAREFUL!
import shutil

if os.path.exists(projectDir):
    shutil.rmtree(projectDir)
    print(f"Deleted directory and all contents: {projectDir}")
else:
    print(f"Directory not found: {projectDir}")

Deleted directory and all contents: /home/jupyter/MyData/tmp_removeME
Contents