Paths in Python

Try on DesignSafe

Paths in Python#

Working with Paths in Python: os module vs Shell Commands

by Silvia Mazzoni, DesignSafe, 2025

When writing scripts that manipulate files or navigate directories, it’s critical to use tools that are portable, robust, and easy to maintain across systems.

On DesignSafe (and generally on HPC systems), your scripts might run on JupyterHub, a virtual machine, or a batch environment—so writing path operations in pure Python ensures they behave consistently everywhere.

import os

os Python Module#

The os.path module is part of Python’s built-in os library and helps you work with file system paths in a clean, portable way. Instead of relying on system-specific shell commands like cd, pwd, or ls, os.path and related os functions let you write code that works consistently across Linux, macOS, and Windows.

Unlike shell commands, which only display output to the screen, Python’s os and os.path functions return actual values (like strings, lists, or booleans) that you can use directly in your code. For example, os.listdir() gives you a list of filenames you can iterate over—no parsing, no copy-pasting from the terminal.

When comparing options:

  • os and os.path are Python-native — safe, readable, and portable.

  • Shell commands like ls or cd are fragile and don’t return data you can use in Python.

  • subprocess is powerful, but better suited for running full programs or system-level tools, not basic path operations.

If you’re building a workflow that needs to run reliably in different environments—like JupyterHub, VMs, or HPC systems—use os.path.

Why use os.path?#

  • Portable: handles differences between Linux, macOS, and Windows automatically (slashes, etc.).

  • Works with Python variables: no need to parse command output.

  • Safe and predictable: your scripts don’t depend on external shell programs.

By contrast:

  • Shell commands via os.system() are less predictable and can’t return values to Python variables.

  • The subprocess module is very powerful for calling external programs (like running OpenSees or SLURM scripts), but it’s overkill for simple file operations.

Examples with os and os.path#

Task

Recommended Python Code

Shell Equivalent

Get current directory

os.getcwd()

pwd

List files in a directory

os.listdir(path)

ls path

Change directory

os.chdir(path)

cd path

Expand ~ to home directory

os.path.expanduser('~')

echo ~

Build safe path

os.path.join(dir, file)

cd dir/subdir (fragile)

Get absolute path from relative

os.path.abspath('myfile.txt')

realpath myfile.txt

Check if a path exists

os.path.exists(path)

test -e path or ls

Check if path is file or directory

os.path.isfile(path), os.path.isdir()

file path, ls -ld path

Create a directory

os.mkdir(path)

mkdir path

Create all parent directories

os.makedirs(path, exist_ok=True)

mkdir -p path

Rename/move a file

os.rename('old', 'new')

mv old new

Remove a file

os.remove('file.txt')

rm file.txt

Remove an empty directory

os.rmdir('folder')

rmdir folder

Remove a directory (recursively)

shutil.rmtree('folder')

rm -r folder

Copy a file

shutil.copy('src', 'dst')

cp src dst

Copy a directory

shutil.copytree('src', 'dst')

cp -r src dst

✅ Tip: Use the os and os.path modules for general-purpose, portable path and file manipulation. Use shutil when you need higher-level operations like copying or deleting directories – see below for more on shutil.

Check if a path exists#

Checking if a path exists with os.path.exists

Before working with any file or directory — such as listing its contents, reading a file, or writing data — it’s good practice to first check whether the path actually exists on the filesystem.

Python’s os.path module provides a simple way to do this:

Why this matters#

  • Running commands on paths that don’t exist often causes errors like FileNotFoundError.

  • This is especially important on DesignSafe, where storage may be mounted differently on JupyterHub, the OpenSees VM, or Stampede3. A path valid on one system might not exist on another.

  • Checking existence first makes your scripts more robust and user-friendly — you can print helpful messages or even create missing directories.

Adding os.path.exists checks into your workflow helps you avoid many common bugs and ensures your code handles different environments gracefully.

# initialize:
os.chdir(os.path.expanduser('~')) # start at the home directory, we will discuss these commands

# Check if path exists:
thisPath = 'MyData'

if os.path.exists(thisPath):
    print(f"The path exists: {thisPath}")
else:
    print(f"Path does not exist: {thisPath}")

The path exists: MyData
Contents