Explore All Jobs#
by Silvia Mazzoni, DesignSafe, 2025
You can use getJobList() to retrieve high-level metadata for all your jobs, such as job names, UUIDs, submission times, statuses, and applications used.
This is perfect for building a summary DataFrame to explore your jobs and decide which one to examine in detail.
You can access a Tapis Job directly – Whether you created a job via the web-portal app, or directly via Tapis in a Jupyter Notebook.
Here we will connect to Tapis, get the metadata and save it to a pandas dataframe.
The data returned by the tapis command ‘t.jobs.getJobList’ returns a nested json dictionary. We will use a function to flatten it into a dataframe of relevant metrics.
We will be using a python function that we will be able to reuse.
Using local utilities library
Connect to Tapis#
t=OpsUtils.connect_tapis()
-- Checking Tapis token --
Token loaded from file. Token is still valid!
Token expires at: 2025-08-21T02:49:32+00:00
Token expires in: 3:41:23.801206
-- LOG IN SUCCESSFUL! --
tapis getJobList() method#
The getJobList() method retrieves a list of jobs, which can be filtered using a variety of fields. This is typically your first step when exploring job data. We are not using search criteria in the getJobList command because they are unreliable. You can filter the data once received.
For better filtering, we will take the results from the getJobList and convert them into a pandas dataframe.
I have put it directly into a python function so we can use it in future modules.
Convert TapisResult object to dataframe#
getJobList() returns a list of TapisResult objects, not plain Python dictionaries.
These TapisResult objects behave kind of like dictionaries (you can do job.id, job.status, etc), but they are actually custom Python objects.
If you want to easily convert them into something like a pandas DataFrame, or save as JSON, you need plain Python dictionaries.
The . dict attribute of a Python object gives you its internal dictionary — essentially all its fields as key/value pairs.
job.dict for job in jobslist turns your list of TapisResult objects into a list of plain dictionaries that pandas can easily consume.
The Process:#
pull jobs
convert to dicts
build the dataframe
reorder columns
optionally display it
get_tapis_jobs_df.py
# ../OpsUtils/OpsUtils/Tapis/get_tapis_jobs_df.py
def get_tapis_jobs_df(t, displayIt=False, NmaxJobs=500):
"""
Retrieve a list of jobs from Tapis and organize them into a Pandas DataFrame.
This function fetches up to NmaxJobs from the user's Tapis account, converts the
results into a structured DataFrame, adds a convenient index column, and moves key
metadata columns (like name, uuid, status) to the front for easier exploration.
It can also optionally display the DataFrame (entire or just the head) right in
the notebook for quick inspection.
Parameters
----------
t : Tapis
An authenticated Tapis client (from connect_tapis()).
displayIt : bool or str, default=False
If 'head' or 'displayHead', displays only the first few rows.
If True or 'displayAll', displays the entire DataFrame.
If False, no display output (just returns the DataFrame).
NmaxJobs : int, default=500
Maximum number of jobs to retrieve from Tapis.
Returns
-------
pandas.DataFrame
DataFrame containing metadata for the fetched jobs.
Example
-------
df = get_tapis_jobs_df(t, displayIt='head', NmaxJobs=1000)
"""
# Silvia Mazzoni, 2025
from datetime import datetime, timezone
import pandas as pd
# Get jobs from Tapis
jobslist = t.jobs.getJobList(limit=NmaxJobs)
# Convert TapisResult objects to dictionaries
jobsdicts = [job.__dict__ for job in jobslist]
# Build DataFrame
df = pd.DataFrame(jobsdicts)
# Add index column for convenience
df["index_column"] = df.index
# add formatted data
for thisK in ['created','remoteStarted', 'ended','lastUpdated']:
df[f'{thisK}_dt'] = pd.to_datetime(df[thisK], utc=True)
df[f'{thisK}_unix'] = df[f'{thisK}_dt'].astype('int64') // 10**9
df[f'{thisK}_date'] = df[f'{thisK}_unix'].apply(
lambda x: datetime.fromtimestamp(x, tz=timezone.utc).date()
)
# Reorder columns: put key ones first if they exist
startCols = ['index_column', 'name', 'uuid', 'status', 'appId', 'appVersion']
existingStartCols = [col for col in startCols if col in df.columns]
remainingCols = [col for col in df.columns if col not in existingStartCols]
columns = existingStartCols + remainingCols
df = df[columns]
# Optional display logic
if displayIt != False:
print(f'Found {len(df)} jobs')
if displayIt in [True] or displayIt.lower() in ['display','displayall','all']:
display(df)
elif displayIt.lower() in ['head', 'displayHead']:
display(df.head())
return df
Get the last 3 Jobs#
jobs_all_df = OpsUtils.get_tapis_jobs_df(t,NmaxJobs = 3,displayIt=True)
Found 3 jobs
| index_column | name | uuid | status | appId | appVersion | owner | created | condition | remoteStarted | ... | created_date | remoteStarted_dt | remoteStarted_unix | remoteStarted_date | ended_dt | ended_unix | ended_date | lastUpdated_dt | lastUpdated_unix | lastUpdated_date | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | opensees-mp-s3-3.6.0_2024-06-13T18:18:01 | 0d3c401b-1807-45a7-904c-ea50d381d2ca-007 | FAILED | opensees-mp-s3 | 3.6.0 | silvia | 2024-06-20T21:29:15.384516Z | None | None | ... | 2024-06-20 | NaT | -9223372037 | 1677-09-21 | 2024-06-20 21:40:18.629466+00:00 | 1718919618 | 2024-06-20 | 2024-06-20 21:40:18.629466+00:00 | 1718919618 | 2024-06-20 |
| 1 | 1 | opensees-mp-s3-3.6.0_2024-06-13T18:18:01 | a983892a-f8a7-45cd-91a9-fe87747bb49c-007 | FAILED | opensees-mp-s3 | 3.6.0 | silvia | 2024-06-13T18:18:10.809303Z | None | None | ... | 2024-06-13 | NaT | -9223372037 | 1677-09-21 | 2024-06-13 18:19:12.957455+00:00 | 1718302752 | 2024-06-13 | 2024-06-13 18:19:12.957455+00:00 | 1718302752 | 2024-06-13 |
| 2 | 2 | opensees-interactive-3.7.0_2024-10-08T20:57:15 | 65e05dee-13d9-4b56-8d83-f9957991f28a-007 | FAILED | opensees-interactive | 3.7.0 | silvia | 2024-10-08T20:57:24.066497Z | None | 2024-10-08T20:57:46.786802Z | ... | 2024-10-08 | 2024-10-08 20:57:46.786802+00:00 | 1728421066 | 2024-10-08 | 2024-10-08 21:36:31.742985+00:00 | 1728423391 | 2024-10-08 | 2024-10-08 21:36:31.742985+00:00 | 1728423391 | 2024-10-08 |
3 rows × 27 columns
Get all your Jobs#
jobs_all_df = OpsUtils.get_tapis_jobs_df(t,displayIt=True)
Found 500 jobs
| index_column | name | uuid | status | appId | appVersion | owner | created | condition | remoteStarted | ... | created_date | remoteStarted_dt | remoteStarted_unix | remoteStarted_date | ended_dt | ended_unix | ended_date | lastUpdated_dt | lastUpdated_unix | lastUpdated_date | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | opensees-mp-s3-3.6.0_2024-06-13T18:18:01 | 0d3c401b-1807-45a7-904c-ea50d381d2ca-007 | FAILED | opensees-mp-s3 | 3.6.0 | silvia | 2024-06-20T21:29:15.384516Z | None | None | ... | 2024-06-20 | NaT | -9223372037 | 1677-09-21 | 2024-06-20 21:40:18.629466+00:00 | 1718919618 | 2024-06-20 | 2024-06-20 21:40:18.629466+00:00 | 1718919618 | 2024-06-20 |
| 1 | 1 | opensees-mp-s3-3.6.0_2024-06-13T18:18:01 | a983892a-f8a7-45cd-91a9-fe87747bb49c-007 | FAILED | opensees-mp-s3 | 3.6.0 | silvia | 2024-06-13T18:18:10.809303Z | None | None | ... | 2024-06-13 | NaT | -9223372037 | 1677-09-21 | 2024-06-13 18:19:12.957455+00:00 | 1718302752 | 2024-06-13 | 2024-06-13 18:19:12.957455+00:00 | 1718302752 | 2024-06-13 |
| 2 | 2 | opensees-interactive-3.7.0_2024-10-08T20:57:15 | 65e05dee-13d9-4b56-8d83-f9957991f28a-007 | FAILED | opensees-interactive | 3.7.0 | silvia | 2024-10-08T20:57:24.066497Z | None | 2024-10-08T20:57:46.786802Z | ... | 2024-10-08 | 2024-10-08 20:57:46.786802+00:00 | 1728421066 | 2024-10-08 | 2024-10-08 21:36:31.742985+00:00 | 1728423391 | 2024-10-08 | 2024-10-08 21:36:31.742985+00:00 | 1728423391 | 2024-10-08 |
| 3 | 3 | opensees-interactive-3.7.0_2024-08-11T02:15:19 | d5ca8c92-6858-4c2d-84c8-de3a6891fab6-007 | FAILED | opensees-interactive | 3.7.0 | silvia | 2024-08-11T02:15:36.208122Z | None | 2024-08-11T02:15:58.677513Z | ... | 2024-08-11 | 2024-08-11 02:15:58.677513+00:00 | 1723342558 | 2024-08-11 | 2024-08-12 02:18:52.616413+00:00 | 1723429132 | 2024-08-12 | 2024-08-12 02:18:52.627855+00:00 | 1723429132 | 2024-08-12 |
| 4 | 4 | opensees-interactive-3.7.0_2024-08-14T01:48:24 | aba14450-27bc-42ae-93f7-2330afdd6462-007 | FAILED | opensees-interactive | 3.7.0 | silvia | 2024-08-14T01:48:32.046028Z | None | 2024-08-14T01:48:54.607485Z | ... | 2024-08-14 | 2024-08-14 01:48:54.607485+00:00 | 1723600134 | 2024-08-14 | 2024-08-14 03:29:59.323250+00:00 | 1723606199 | 2024-08-14 | 2024-08-14 03:29:59.333848+00:00 | 1723606199 | 2024-08-14 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 495 | 495 | opensees-interactive-latest_2025-06-19T21:12:20 | d04b3140-acde-4d64-a8c1-7d8c46f5b25c-007 | FAILED | opensees-interactive | latest | silvia | 2025-06-19T21:27:49.686635Z | None | 2025-06-19T21:28:12.183247Z | ... | 2025-06-19 | 2025-06-19 21:28:12.183247+00:00 | 1750368492 | 2025-06-19 | 2025-06-19 21:29:01.579244+00:00 | 1750368541 | 2025-06-19 | 2025-06-19 21:29:01.579244+00:00 | 1750368541 | 2025-06-19 |
| 496 | 496 | opensees-interactive-latest_2025-06-19T21:12:20 | fb630e5c-69f1-4370-b49e-b01e9d3da673-007 | FAILED | opensees-interactive | latest | silvia | 2025-06-19T22:38:07.428425Z | None | 2025-06-19T22:38:40.331526Z | ... | 2025-06-19 | 2025-06-19 22:38:40.331526+00:00 | 1750372720 | 2025-06-19 | 2025-06-19 22:39:08.866665+00:00 | 1750372748 | 2025-06-19 | 2025-06-19 22:39:08.866665+00:00 | 1750372748 | 2025-06-19 |
| 497 | 497 | OpenSees_agnostic-app-test_zipFolderOut | 967f77be-8108-4750-bafa-f1bf9ddc34d8-007 | FINISHED | agnostic-app-test | 0.0.21 | silvia | 2025-08-20T01:40:15.974840Z | None | 2025-08-20T01:41:34.213701Z | ... | 2025-08-20 | 2025-08-20 01:41:34.213701+00:00 | 1755654094 | 2025-08-20 | 2025-08-20 01:42:28.713746+00:00 | 1755654148 | 2025-08-20 | 2025-08-20 01:42:28.713746+00:00 | 1755654148 | 2025-08-20 |
| 498 | 498 | opensees-MP-multiMotion-dapi | 8d9405ff-ffc0-46f3-b20a-930596ff4481-007 | FAILED | opensees-mp-s3 | latest | silvia | 2025-06-09T23:56:14.759209Z | None | None | ... | 2025-06-09 | NaT | -9223372037 | 1677-09-21 | 2025-06-09 23:57:25.251713+00:00 | 1749513445 | 2025-06-09 | 2025-06-09 23:57:25.251713+00:00 | 1749513445 | 2025-06-09 |
| 499 | 499 | OpsTrain_JobSubmit_WebPortal | f3633ed7-116a-45ec-bb8f-100978f6e5b7-007 | FINISHED | opensees-mp-s3 | latest | silvia | 2025-06-03T05:57:51.692995Z | None | 2025-06-03T05:59:15.487231Z | ... | 2025-06-03 | 2025-06-03 05:59:15.487231+00:00 | 1748930355 | 2025-06-03 | 2025-06-03 06:02:53.709377+00:00 | 1748930573 | 2025-06-03 | 2025-06-03 06:02:53.709377+00:00 | 1748930573 | 2025-06-03 |
500 rows × 27 columns