Compute Environments

DesignSafe provides three places where computation happens. Each serves a different purpose, and most researchers move between them as a project evolves: develop interactively in JupyterHub, then submit production runs to HPC.

The environments form a tiered approach that lowers the barrier to HPC, AI, and ML one step at a time.

1. Portal apps run simulation codes (including SimCenter tools) on HPC without writing any orchestration code.

2. JupyterHub adds interactive work for small problems plus programmatic HPC job submission with dapi.

3. JupyterLab HPC brings interactive sessions onto dedicated CPU and GPU nodes, with AI/ML stacks such as CUDA, PyTorch, and TensorFlow.

4. Command-line access to Stampede3, Frontera, and Lonestar6 remains available for researchers who want to manage SLURM directly.

Tools and data stay co-located at TACC through every tier, so moving up a tier never means moving your files somewhere else.

JupyterHub¶

JupyterHub is where most day-to-day work happens. Each session gets a dedicated Kubernetes container at TACC with up to 8 CPU cores and 20 GB RAM. Sessions start immediately with no queue wait. The browser-based environment includes notebooks, a terminal, a file manager, and a text editor — all sharing the same filesystem.

Move to HPC when the workload needs more memory, more cores, multi-node parallelism (MPI), or longer runtimes than an interactive session allows.

JupyterLab HPC: interactive sessions on compute nodes¶

For heavier interactive work, JupyterLab HPC (CPU or GPU) and Jupyter HPC Native sessions run on a dedicated compute node — not a container. The session is scheduled by SLURM like any job, and you get the full node exclusively; GPU sessions run on Vista (NVIDIA H200).

Use JupyterHub for development and as the launchpad for batch jobs. Use JupyterLab HPC to debug at full-node scale, inspect job outputs directly on $SCRATCH (which JupyterHub cannot see), or run interactive workloads a container can’t hold.

Which Jupyter should I pick?¶

Why move up, measured¶

The payoff of picking the right tier is not marginal. The same 10,000-simulation fragility study runs in about 2 hours in a JupyterHub session and about 3 minutes on a dedicated Stampede3 node. Drag the study size to see when the queue wait stops mattering.

CPU vs GPU¶

A CPU spends most of its silicon on control logic and cache serving a few powerful cores, which makes it good at general-purpose, branch-heavy work. A GPU spends nearly all of its silicon on thousands of small cores doing the same arithmetic in lockstep, which makes it dramatically faster for dense linear algebra, and that is exactly what ML training and many physics kernels are.

On DesignSafe, GPU work runs through JupyterLab HPC GPU sessions (dedicated GPU node, CUDA, PyTorch, TensorFlow preloaded) or through GPU queues in batch jobs. If the workload is dominated by matrix multiplication, request a GPU; if it is dominated by conditional logic, file handling, or a legacy solver, more CPU cores serve better.

Virtual machines¶

Virtual machines (VMs) run applications without a queue wait. MATLAB, ADCIRC Interactive, STKO, and QGIS run on shared VMs at TACC; STKO and QGIS provide a full graphical desktop through NICE DCV, which streams a remote desktop to the browser.

OpenSees Express is a submit-only VM: jobs are sent to it through the portal (Tcl scripts, serial only) or via Tapis — there is no SSH and no interactive use. Its appeal is zero queue wait; the trade-off is that the VM is shared without per-user isolation, so performance varies with other users’ load. Use it for short serial Tcl runs; anything parallel or long belongs on HPC. The legacy OpenSees Interactive VM is being retired in favor of JupyterHub.

VMs share hardware across users, so they work best for lightweight tasks and quick tests.

HPC systems¶

HPC (High-Performance Computing) systems handle production-scale computation. These are clusters of interconnected machines (nodes), each with dozens of CPU cores and hundreds of gigabytes of memory. They are shared systems — thousands of researchers submit jobs to the same hardware, so SLURM manages access through job queues. Researchers using DesignSafe never interact with SLURM directly; Tapis generates SLURM scripts automatically. Long-running simulations, multi-core parallel analyses, and parametric sweeps with hundreds of runs all belong on HPC.

DesignSafe researchers have access to three TACC systems:

Nodes, cores, and memory¶

A laptop is a shared-memory machine. Every core sees the same RAM, so parallel programs can simply share data. A supercomputer is a distributed-memory machine, thousands of separate computers connected by a fast network. Cores on the same node still share memory, but cores on different nodes exchange data by passing messages, which is exactly what MPI does. This distinction drives how jobs are sized.

A node is a complete physical computer. Each node has multiple cores (CPUs) that execute work in parallel, sharing the same pool of RAM.

When submitting a job, specify node_count, cores_per_node, and max_minutes. Total cores = node_count x cores_per_node. For MPI jobs, each core runs one parallel process (rank). For PyLauncher sweeps, each core runs one independent task.

All cores on a node share memory. If each process needs more memory, request fewer cores per node:

SLURM and queues¶

SLURM is the job scheduler on all TACC systems. When a job is submitted, SLURM places it in a queue (also called a partition). Each queue groups nodes with similar hardware and enforces limits on node count and runtime. Researchers using DesignSafe never write SLURM scripts directly — Tapis generates them automatically from the job parameters.

Stampede3 queues (full policy in the Stampede3 User Guide):

SKX nodes are the most numerous (1,060) and a good default. The skx-dev queue is designed for short test runs with low wait times — but it allows only one running and one queued job per user at a time, so plan test submissions sequentially. Always test there before submitting production jobs.

Queue limits change over time; the tables above are a snapshot, and TACC’s own queue tables (linked per system) are authoritative. The live limits are also one call away in dapi, where ds.tapis.systems.getSystem(systemId="stampede3").batchLogicalQueues lists every queue with its node, core, and runtime caps.

Frontera queues (56 cores, 192 GB per node; Frontera User Guide):

Lonestar6 queues (128 cores, 256 GB per node; Lonestar6 User Guide):

Choosing a queue¶

1. Estimate memory per process. If each MPI rank needs 8 GB and the node has 192 GB, use at most 24 cores per node.

2. Determine total cores needed. A model decomposed into 96 subdomains needs 96 cores (e.g., 2 nodes x 48 cores).

3. Pick the queue that fits. Use skx-dev or development for testing. Use production queues for real runs.

4. Check system load. Live queue status: Stampede3, Frontera, Lonestar6.

Allocations and Service Units¶

A DesignSafe account is a TACC account, and computing time comes as a TACC allocation, a grant of Service Units tied to a research project. New users can start with a 500 SU startup allocation, renewable up to 10,000 SUs per year, and larger allocations are available by request; TACC manages both the allocations and the access. The allocation name signals its scope. DS-Corral covers storage and VM resources, DS-Portal-* allocations back the portal apps, and PI-specific DS-* allocations fund individual research groups.

Running jobs charges Service Units (SUs):

SUs = nodes x hours x charge_rate

A job on 4 SKX nodes for 2 hours at 1 SU/node-hour costs 8 SUs. The same job on SPR nodes at 2 SUs/node-hour costs 16 SUs. Nodes are billed entirely regardless of how many cores are used. Every job incurs a minimum charge of 15 minutes.

HPC-enabled tools (OpenSeesMP, OpenFOAM, ADCIRC) require an allocation. If you don’t have one, submit a ticket through the DesignSafe help desk. Remaining balance and allocation codes are on the TACC Dashboard.