ISPyPSA Command Line Interface

The ispypsa command provides a user-friendly interface for running ISPyPSA workflows. It's the simplest and quickest way to run the predefined workflows. However, if more flexibility is required the API might be a better option.

Overview

The ispypsa command and sub-tasks allow you to:

• Download data
• Extract data from ISP workbooks
• Generate model input files
• Run capacity expansion and operational models
• Manage workflow outputs

Most tasks require a configuration file that specifies paths and model parameters. See the Workflow overview section for a high-level overview of the default ISPyPSA workflow.

Installation

See the Installation section in the Getting Started guide.

Basic Usage

The basic command structure is:

uvplain python

uv run ispypsa config=<config_file> [task]

ispypsa config=<config_file> [task]

The config=<config_file> argument is required for task execution and must point to a valid ISPyPSA configuration YAML file.

Important

For the sake of brevity the commands on the rest of this page are given in plain Python, but can be run with uv run as well.

Examples

# List available tasks (no config required)
ispypsa list

# Run a specific task with config
ispypsa config=my_config.yaml create_and_run_capacity_expansion_model

# Run all tasks with config
ispypsa config=my_config.yaml

Tasks

The ISPyPSA CLI workflow consists of a series of tasks, each dependent on the outputs of previous tasks. The following diagram shows the task dependency graph:

save_config
│
└──> cache_required_iasr_workbook_tables
     │
     └──> create_ispypsa_inputs
          │
          ├──> create_pypsa_friendly_inputs
          │    │
          │    ├──> create_and_run_capacity_expansion_model
          │    │    │
          └────┼────┼──> create_operational_timeseries
               │    │    │
               └────┴────┴──> create_and_run_operational_model

Important

Each task depends on the outputs of previous tasks. If a particular task is run, but the previous tasks' runs on which it depends isn't up to date, then the CLI will detect this and also run the previous tasks. The detection of a previous task's being 'up to date' is based on two checks 1) its input files haven't been modified since it last ran and 2) its output files exist. If either 1) or 2) aren't true then a task is not up to date and will be rerun. This applies to both the primary target task and all of its dependencies.

Note

ISPyPSA provides independent download and plotting tasks. These tasks are not part of the main workflow task dependency and need be explicity run when needed.

download_workbook

Downloads the ISP workbook Excel file from the data repository.

Usage with config file:

ispypsa config=config.yaml download_workbook

Usage with direct parameters (no config file needed):

ispypsa workbook_version=6.0 workbook_path=data/workbooks/6.0.xlsx download_workbook

Parameters (if not using config):

• workbook_version (required): Version of workbook to download (e.g., "6.0")
• workbook_path (required): Full path where workbook should be saved (must end with .xlsx)

Outputs:

• Excel workbook file at the specified path

Notes:

• This task will overwrite any existing workbook file at the target location
• If download fails partway through, partial files will remain
• Task always runs when invoked (never considers itself up-to-date)

download_trace_data

Downloads trace data (demand, wind, solar) from the data repository.

Usage with config file:

# Download example dataset (default - smaller, for testing)
ispypsa config=config.yaml download_trace_data

# Download full dataset (override config default)
ispypsa config=config.yaml trace_dataset_type=full download_trace_data

Usage with direct parameters (no config file needed):

# Download example dataset to specified directory
ispypsa save_directory=data/traces download_trace_data

# Download full dataset to specified directory
ispypsa save_directory=data/traces trace_dataset_type=full download_trace_data

# Specify all parameters
ispypsa save_directory=data/traces trace_dataset_type=example trace_dataset_year=2024 download_trace_data

Parameters (if not using config):

• save_directory: Directory where trace data should be saved
• trace_dataset_type (optional): Either "example" or "full"
  • Defaults to "example"
  • "example": Smaller dataset suitable for testing and development
  • "full": Complete dataset for production runs
• trace_dataset_year (optional): Year of dataset
  • Defaults to 2024
  • Currently only 2024 is supported

Outputs:

• Trace data files in the specified directory
• Files organized in subdirectories: isp_{year}/project/, isp_{year}/zone/, isp_{year}/demand/

Notes:

• This task will re-download all files, overwriting any existing trace data
• If download fails partway through, partial files will remain
• Task always runs when invoked (never considers itself up-to-date)
• The "full" dataset manifest is currently empty and will be populated in future releases
• Direct mode is useful for downloading data before creating a config file

cache_required_iasr_workbook_tables

Extracts data from the ISP Excel workbook and caches it as CSV files.

ispypsa config=config.yaml cache_required_iasr_workbook_tables

Inputs:

• IASR Excel workbook (specified in config paths.workbook_path)

Outputs:

• CSV files in the workbook cache directory (location specified in config paths. parsed_workbook_cache)

create_ispypsa_inputs

Generates ISPyPSA format input tables from the cached workbook data.

ispypsa config=config.yaml create_ispypsa_inputs

Inputs:

• Cached workbook CSV files (location specified in config paths. parsed_workbook_cache)

Outputs:

• ISPyPSA input tables in {run_directory}/{ispypsa_run_name}/ispypsa_inputs/ (run_directory and ispypsa_run_name specified in config)

create_pypsa_friendly_inputs

Converts ISPyPSA format tables to PyPSA-compatible format and generates time series data for capacity expansion.

ispypsa config=config.yaml create_pypsa_friendly_inputs

Inputs:

• ISPyPSA input tables in {run_directory}/{ispypsa_run_name}/ispypsa_inputs/ (run_directory and ispypsa_run_name specified in config)
• Trace data (demand, wind, solar) (location specified in config paths. parsed_traces_directory)

Outputs:

• PyPSA-friendly tables in {run_directory}/{ispypsa_run_name}/pypsa_friendly/ (run_directory and ispypsa_run_name specified in config)
• Time series data in {run_directory}/{ispypsa_run_name}/pypsa_friendly/capacity_expansion_timeseries/ (run_directory and ispypsa_run_name specified in config)

create_and_run_capacity_expansion_model

Creates the PyPSA network object and runs the capacity expansion optimization.

ispypsa config=config.yaml create_and_run_capacity_expansion_model

Inputs:

• PyPSA-friendly tables in {run_directory}/{ispypsa_run_name}/pypsa_friendly/ (run_directory and ispypsa_run_name specified in config)
• Time series data in {run_directory}/{ispypsa_run_name}/pypsa_friendly/capacity_expansion_timeseries/ (run_directory and ispypsa_run_name specified in config)

Outputs:

• Optimized capacity expansion results in {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion.nc

Plotting Option:

By default, a suite of plots is automatically generated after the capacity expansion model runs. This behaviour is controlled by the create_plots setting in config.yaml, which is set to True in the default workflow.

When enabled, the task will create and save a suite of plots to the {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion_plots/ directory.

This provides a convenient way to visualise model outputs without needing to run the create_capacity_expansion_plots task separately.

You can disable plot creation by setting create_plots: False in your config or by passing it as a command-line argument:

# Disable plot generation for this run
ispypsa config=config.yaml create_plots=False create_and_run_capacity_expansion_model

Skip Optimization Option:

You can skip the optimization step and only build the network using the run_optimisation flag:

ispypsa config=config.yaml run_optimisation=False create_and_run_capacity_expansion_model

This is particularly useful for:

• Testing that your model configuration is valid
• Verifying network construction without waiting for optimization
• Debugging model setup issues
• Creating a network file for manual inspection or custom optimization

When this flag is set to False, the task will:

1. Build the complete PyPSA network with all components and constraints
2. Save the unoptimized network to the output file
3. Skip the potentially time-consuming optimization step

create_capacity_expansion_plots

Generates plots from the capacity expansion model results.

ispypsa config=config.yaml create_capacity_expansion_plots

Inputs:

• Tabular results from capacity expansion model in {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion_tables/ (generated by create_and_run_capacity_expansion_model)

Outputs:

• Plot files (interactive HTML format) in {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion_plots/
• Results viewer website: {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion_results_viewer.html

Notes:

• This task is not part of the main workflow dependency chain and must be run explicitly.
• It will fail if the capacity expansion results are not present.
• Generated plots include:
• Regional and sub-regional dispatch plots
• Transmission flow plots for all links
• Transmission capacity expansion plots
• Regional transmission expansion plots

create_operational_timeseries

Creates time series data for operational modeling.

ispypsa config=config.yaml create_operational_timeseries

Inputs:

• ISPyPSA input tables in {run_directory}/{ispypsa_run_name}/ispypsa_inputs/ (run_directory and ispypsa_run_name specified in config)
• Trace data (demand, wind, solar) (location specified in config paths. parsed_traces_directory)

Outputs:

• Operational time series data in {run_directory}/{ispypsa_run_name}/pypsa_friendly/operational_timeseries/

create_and_run_operational_model

Prepares the PyPSA network object for operational modeling using fixed capacities from capacity expansion and runs the operational optimization.

ispypsa config=config.yaml create_and_run_operational_model

Inputs:

• Capacity expansion results in {run_directory}/{ispypsa_run_name}/outputs/capacity_expansion.nc
• PyPSA-friendly tables in {run_directory}/{ispypsa_run_name}/pypsa_friendly/
• Operational time series data in {run_directory}/{ispypsa_run_name}/pypsa_friendly/operational_timeseries/

Outputs:

• Operational optimization results in {run_directory}/{ispypsa_run_name}/outputs/operational.nc

Running Without Capacity Expansion

The operational model can be built and run even if the capacity expansion optimization was skipped (using run_optimisation=False). However, this will significantly affect the operational model outputs because:

• The network will use initial capacities rather than optimized capacities
• No new generation or transmission capacity will have been added
• The operational model may be infeasible if initial capacities are insufficient
• Results will not reflect least-cost capacity investment decisions

Skip Optimization Option:

You can skip the optimization step and only prepare the network using the run_optimisation flag:

ispypsa config=config.yaml run_optimisation=False create_and_run_operational_model

This is particularly useful for:

• Testing operational model setup without running the full optimization
• Verifying that capacity expansion results load correctly
• Debugging time series data integration
• Creating a prepared network for custom operational analysis

When this flag is set to False, the task will:

1. Load the capacity expansion results
2. Update the network with operational time series data
3. Fix the optimal capacities from capacity expansion
4. Save the prepared network without running the rolling horizon optimization
5. Skip the potentially very long operational optimization process

Plotting Option:

Similar to capacity expansion, plots can be automatically generated after the operational model runs. This behaviour is controlled by the create_plots setting in config.yaml, which defaults to True.

When enabled, the task will create and save a suite of plots to the {run_directory}/{ispypsa_run_name}/outputs/operational_plots/ directory.

You can disable plot creation by setting create_plots: False in your config or by passing it as a command-line argument:

# Disable plot generation for this run
ispypsa config=config.yaml create_plots=False create_and_run_operational_model

create_operational_plots

Generates plots from the operational model results.

ispypsa config=config.yaml create_operational_plots

Inputs:

• Tabular results from operational model in {run_directory}/{ispypsa_run_name}/outputs/operational_tables/ (generated by create_and_run_operational_model)

Outputs:

• Plot files (interactive HTML format) in {run_directory}/{ispypsa_run_name}/outputs/operational_plots/
• Results viewer website: {run_directory}/{ispypsa_run_name}/outputs/operational_results_viewer.html

Notes:

• This task is not part of the main workflow dependency chain and must be run explicitly.
• It will fail if the operational results are not present.
• Generated plots include:
• Regional and sub-regional dispatch plots
• Transmission flow plots for all links

list

Shows all available tasks and their status. No config file required.

ispypsa list

save_config

Saves a copy of the configuration file to the run directory. This task runs automatically as a dependency of other tasks to preserve the exact configuration used for reproducibility.

ispypsa config=config.yaml save_config

Inputs:

• Configuration YAML file (specified with config=)

Outputs:

• Copy of the configuration file in {run_directory}/{ispypsa_run_name}/ (preserves the exact config used for the run)

Note: This task always runs (never considers itself up-to-date) to ensure the config file is always current.

Configuration

The config= argument is required for most tasks. It accepts either absolute or relative paths:

# Absolute path
ispypsa config=/home/user/projects/my_config.yaml task_name

# Relative path (from current directory)
ispypsa config=../configs/my_config.yaml task_name

# File in current directory
ispypsa config=my_config.yaml task_name

See the example configuration file for the required format.

Examples

Complete Workflow

Run all tasks one by one to generate model results:

# Extract workbook data
ispypsa config=config.yaml cache_required_iasr_workbook_tables

# Generate ISPyPSA inputs
ispypsa config=config.yaml create_ispypsa_inputs

# At this stage the ISPyPSA inputs could be edited to adjust build costs or any other
# inputs set out in {run_directory}/{ispypsa_run_name}/ispypsa_inputs/

# Convert to PyPSA format and run capacity expansion
ispypsa config=config.yaml create_and_run_capacity_expansion_model

# Create operational time series
ispypsa config=config.yaml create_operational_timeseries

# Run operational optimization
ispypsa config=config.yaml create_and_run_operational_model

# Generate plots (optional - also runs automatically if create_plots=True)
ispypsa config=config.yaml create_capacity_expansion_plots
ispypsa config=config.yaml create_operational_plots

Simplified Complete Workflow

The complete workflow can be run in fewer steps, with intermediate tasks run automatically when their outputs are required.

# Generate ISPyPSA inputs
ispypsa config=config.yaml create_ispypsa_inputs

# At this stage the ISPyPSA inputs could be edited to adjust build cost or any other
# inputs set out in {run_directory}/{ispypsa_run_name}/ispypsa_inputs/

# Run complete workflow (all remaining tasks)
ispypsa config=config.yaml create_and_run_operational_model

Running from Different Directories

The CLI works correctly from any directory and handles relative paths appropriately:

# From project root - no config needed for list
ispypsa list

# From project root - config for task execution
ispypsa config=ispypsa_config.yaml create_ispypsa_inputs

# From a subdirectory
cd analysis
ispypsa config=../ispypsa_config.yaml create_ispypsa_inputs

# Using an absolute path (works from anywhere)
ispypsa config=/home/user/ispypsa/config.yaml create_ispypsa_inputs

Using Different Configurations

You can easily switch between different model configurations:

# Development configuration
ispypsa config=configs/dev_config.yaml create_and_run_capacity_expansion_model

# Production configuration with different parameters
ispypsa config=configs/prod_config.yaml create_and_run_capacity_expansion_model

# Test configuration with smaller dataset
ispypsa config=configs/test_config.yaml create_and_run_capacity_expansion_model

Skip Optimization Flags

You can skip the optimization step in modeling tasks using these flags:

# Skip capacity expansion optimization (only prepare the network)
ispypsa config=config.yaml run_optimisation=False create_and_run_capacity_expansion_model

# Skip operational optimization (only prepare the network, capacity expansion will also be skipped)
ispypsa config=config.yaml run_optimisation=False create_and_run_operational_model

# Run both optimizations normally (default behavior)
ispypsa config=config.yaml run_optimisation=True create_and_run_operational_model

These flags are useful for:

• Debugging network construction without waiting for optimization
• Testing model setup and configuration
• Preparing networks for manual optimization or analysis

Path Issues

If you encounter path-related errors:

1. Use debug=True to see how paths are being resolved
2. Try using absolute paths in your config file
3. Ensure all directories exist before running tasks
4. Check file permissions for input/output directories

Advanced Usage

How ispypsa Works

The ispypsa command is built on the doit task automation tool with ISPyPSA-specific enhancements:

1. run command: Uses doit's native run command with config=value parameter support
2. Other commands: Uses doit's built-in commands (list, help, etc.) directly
3. Lazy loading: Configuration is only loaded when tasks actually execute
4. Path resolution: All paths work relative to where you run the command

Passing Additional doit Options

You can pass any doit option to the run command:

# Run with verbose output
ispypsa config=config.yaml -v create_and_run_capacity_expansion_model

# Always execute task ignoring up-to-date checks
ispypsa config=config.yaml -a create_ispypsa_inputs

# Continue executing even after task failure
ispypsa config=config.yaml --continue create_and_run_operational_model

Common doit options for the run command:

• -v / --verbosity: Set verbosity level (0-2)
• -a / --always-execute: Always execute tasks even if up-to-date
• --continue: Continue executing tasks even after failure
• -s / --single: Execute only specified tasks ignoring their dependencies

Commands Without Config

Some commands work without requiring a config file:

# List all available tasks
ispypsa list

# Show help
ispypsa help

# Show help for specific command
ispypsa help run

Working Directory Behavior

The CLI maintains your current working directory:

• All paths in config files work relative to where you run the command
• No directory changes occur during execution
• Output files are created according to paths in your config file