What is qq?

qq is a wrapper around batch scheduling systems designed to simplify job submission and management. It is inspired by NCBR's Infinity ABS but aims to be more decentralized and easier to extend. In the long term, it will support both PBS Pro and Slurm, while making it straightforward to add compatibility with other batch systems as needed.

Although qq and Infinity ABS share the same philosophy and use very similar commands, they share no code.

Disclaimer: qq is developed for the internal use of the Robert Vácha Lab and may not work on your clusters or for your applications.

Installation

This section explains how to easily install qq on different clusters.

All installation scripts assume that you are using bash as your shell. If you use a different shell, follow this section of the manual.

Installing on robox

To install qq on the robox cluster (computers of the RoVa Lab), log in to your desktop and run:

curl -fsSL https://github.com/Ladme/qq/releases/latest/download/qq-robox-install.sh | bash

This command downloads the latest version of qq, installs it to your home directory on your desktop, and then installs it to the home directory of the computing nodes.

To finish the installation, either open a new terminal or source your .bashrc file.

Note: This does not install qq on other desktops with separate local home directories.
If you want to use qq on other desktops, you'll need to install it there separately.
(Not installing it there however helps prevent accidentally running jobs on someone else's desktop.)

Installing on sokar

To install qq on the sokar cluster (managed by NCBR), log in to sokar.ncbr.muni.cz and run:

curl -fsSL https://github.com/Ladme/qq/releases/latest/download/qq-sokar-install.sh | bash

This downloads the latest version of qq and installs it to the shared home directory of the cluster nodes.

To complete the installation, open a new terminal or source your .bashrc file.

Installing on metacentrum

To install qq on Metacentrum, log in to any Metacentrum frontend and run:

curl -fsSL https://github.com/Ladme/qq/releases/latest/download/qq-metacentrum-install.sh | bash

This command downloads the latest version of qq and installs it into your home directory on brno12-cerit. The script then adds qq's location on this storage to your PATH across all Metacentrum machines.

Because qq runs significantly slower when stored on non-local storage, the installation script also configures the .bashrc files of all Metacentrum machines to automatically copy qq from brno12-cerit to their local scratch space on login. This improves the responsiveness of qq operations.

To complete the installation, either open a new terminal or source your .bashrc file.

Installing manually

Installing a pre-built version

To install a pre-built version of qq on a single computer or on several computers sharing the same home directory, run:

curl -fsSL https://github.com/Ladme/qq/releases/latest/download/qq-install.sh | \
bash -s -- $HOME https://github.com/Ladme/qq/releases/latest/download/qq-release.tar.gz

To finish the installation, either open a new terminal or source your .bashrc file.

Installing a pre-built version for other shells

If you're not using bash, you'll need to modify the qq-install.sh script.

First, download it:

curl -OL https://github.com/Ladme/qq/releases/latest/download/qq-install.sh

Then edit this line to match your shell's RC file:

BASHRC="${TARGET_HOME}/.bashrc"
# For example, if you use zsh:
BASHRC="${TARGET_HOME}/.zshrc"

Next, make the script executable and run it:

chmod u+x qq-install.sh
./qq-install.sh $HOME https://github.com/Ladme/qq/releases/latest/download/qq-release.tar.gz

Building qq from source

To build and install qq yourself, you'll need git and uv installed.

First, clone the qq repository:

git clone git@github.com:Ladme/qq.git

Then navigate to the project directory and install the dependencies:

cd qq
uv sync --all-groups

Build the package using PyInstaller:

uv run pyinstaller qq.spec

PyInstaller will create a directory named qq inside dist. Copy that directory wherever you want and add it to your PATH.

If you want the qq cd command to work, add the following shell function to your shell's RC file:

qq() {
    if [[ "$1" == "cd" ]]; then
        for arg in "$@"; do
            if [[ "$arg" == "--help" ]]; then
                command qq "$@"
                return
            fi
        done
        target_dir="$(command qq cd "${@:2}")"
        cd "$target_dir" || return
    else
        command qq "$@"
    fi
}

Running a job

This section demonstrates how to run a basic qq job by performing a simple Gromacs simulation on the robox cluster. This assumes that you have already successfully installed qq on the cluster.

1. Preparing an input directory

Start by creating a directory for the job on a shared storage (you can also submit from a local storage on your computer but that is not recommended). This directory should contain all necessary simulation input files — in this example, an mdp, gro, cpt, ndx,top, and itp files.

2. Preparing a run script

Next, prepare a run script that creates a tpr file from the input files using gmx grompp, and then runs the simulation with gmx mdrun. We’ll configure the simulation to use 8 OpenMP threads.

Note that all qq run scripts must start with the correct shebang line:

#!/usr/bin/env -S qq run

A complete example of a run script:

#!/usr/bin/env -S qq run

# activate the Gromacs module
metamodule add gromacs/2024.3-cuda

# prepare a TPR file
gmx_mpi grompp -f md.mdp -c eq.gro -t eq.cpt -n index.ndx -p system.top -o md.tpr

# run the simulation using 8 OpenMP threads
gmx_mpi mdrun -deffnm md -ntomp 8 -v

Save this file as run_job.sh and make it executable:

chmod u+x run_job.sh

3. Submitting the job

Submit the job using qq submit:

qq submit run_job.sh -q cpu --ncpus 8 --walltime 1d

This submits run_job.sh to the cpu queue, requesting 8 CPU cores and a walltime of one day. All other parameters are determined by the queue or qq’s default settings.

The batch system then schedules the job for execution. Once a suitable compute node is available, the job runs through qq run, a wrapper around bash that prepares the working directory, copies files, executes the script, and performs cleanup. You can read more about how exactly this works in this section of the manual.

4. Inspecting the job

After submission, you can inspect the job using qq info, access its working directory on the compute node with qq go, or terminate it using qq kill. For an overview of all qq commands, see this section of the manual.

5. Getting the results

Once the job finishes, the resulting Gromacs output files will be transferred from the working directory back to the original input directory. You can verify that everything completed successfully using qq info.

If your job failed (crashed) or was killed, the output files will not be transferred to ensure your input directory remains in a consistent state. In these cases, the working directory on the compute node is preserved, allowing you to inspect the job files directly there using qq go, or to explicitly copy them back to the input directory using qq sync.

Job types

qq currently supports two job types: standard and loop.

standard jobs are the default type. Any job for which you don't specify a job-type when submitting is considered standard. Read more about standard jobs here.

loop jobs automatically submit their continuation before finishing. Read more about them here.

Standard jobs

A standard job is the default qq job type. This section describes the full lifecycle of a standard qq job.

1. Submitting the job

Submitting a qq job is done using qq submit.

qq submit submits the job to the batch system and generates a qq info file containing metadata and details about the job. This info file is named after the submitted script, has the .qqinfo extension, and is located in the input directory (often also called the submission or, somewhat confusingly, job directory).

Once submitted, the batch system takes over, finding a suitable place and time to execute your job. As a user, you don't need to do anything else except wait for the job to run.

2. Preparing the working directory

When the batch system allocates a machine for your job, the qq run environment takes over. It first prepares a working directory for the job on the execution node.

If you requested the job to run in the input directory (by submitting with --workdir=input_dir or the equivalent --workdir=job_dir), the input directory is used directly as the working directory, and no additional setup is required.

If you requested the job to run on scratch (the default for PBS), a working directory is created inside your allocated scratch space, and all files and directories in the input directory are copied there — except for the qq info file and the "archive" directory (discussed later).
During submission, you can also specify files you explicitly do not want to copy to the working directory.

Once the working directory is ready, qq updates the info file to mark the job state as running. Only then is your submitted script executed.

3. Executing the script

Your submitted script is then executed using bash.

The script should exit with code 0 if everything ran successfully, or a non-zero code to indicate an error. The exit code is passed back to qq, which sets the appropriate job state (finished for 0, failed for anything else).

Standard output from your script is saved to a file named after your script with the .out extension. Standard e rror output is stored in asimilar file with the .err extension.

4. Finalizing execution

After the script finishes, qq performs cleanup.

If your job ran in the input directory, cleanup is simple: qq updates the job's state (finished or failed) in the qq info file, and execution ends.

If your job ran on scratch, cleanup depends on the script's exit status.

If it finished successfully, all files from the working directory are copied back to the input directory, and the working directory (or its contents) is deleted. Finally, qq sets the job state to finished.

If the job failed, the working directory is left intact on the execution machine for inspection (you can open it using qq go). The job state is set to failed.

Regardless of the result, qq creates an output file (named after your script with the .qqout extension) in the input directory. This file contains basic information about what qq did and when the job finished.

It may take a few seconds for the batch system to write this file, so if it doesn't appear right away, be patient.

Killing a qq job

If your job is killed (either manually via qq kill or automatically by the batch system, for example if it exceeds walltime), all files remain in the working directory on the execution machine. qq stops the running script and marks the job state as killed.

Additional notes

  • Most operations during working directory setup and cleanup are automatically retried in case of errors. This helps prevent job crashes caused by temporary storage or network issues. If an operation fails, qq waits a few minutes and retries — up to three attempts. After three failures, qq stops and reports an error. Note that qq does not retry execution of your script itself.
  • If your job fails with an exit code between 90–99, this usually means a qq operation failed. Check the qq output file (.qqout) for more details. An exit code of 99 indicates a critical or unexpected error, which usually means a bug in qq. Please report such cases.

Loop jobs

Loop jobs are jobs that automatically submit their continuation at the end of execution. This section describes how they differ from standard jobs. Please read the section about standard jobs first — otherwise, this may be difficult to follow.

To turn a job into a loop job, you must set two qq submit options:

  • job-type to loop, and
  • loop-end to specify the last cycle of the loop job.

Loop job cycles

Each loop job consists of multiple cycles. Every cycle is a separate job from the batch system's perspective. Before a cycle finishes, it submits the next one and then ends. The next cycle continues where the previous one left off.

You can control the starting cycle using the loop-start submission option (defaults to 1). To set the final cycle, use the loop-end option. The cycle specified as loop-end will be the last one executed.

Archive directory

Each loop job creates an archive directory inside the input directory. This directory is not copied to the job's working directory, so it can safely hold large amounts of data. In loop jobs, the archive serves two main purposes:

  • to identify and initialize the current cycle of the loop job,
  • to store data from previous cycles without copying them to the working directory.

You can control the archive directory's name using the archive submission option (default: storage).

Archived files should follow a specific filename format that includes the job cycle number they belong to. You can define this format using the archive-format submission option (default: job%04d). In this format, %04d is replaced by the cycle number — for example, job0001 for cycle 1, job0002 for cycle 2, job0143 for cycle 143 and so on.

When a new cycle is submitted (either manually or automatically by the previous one), qq sets the current cycle number based on the highest cycle number found in the names of the archived files. In other words: in each cycle of a loop job, at least one file must be added to the archive whose name includes the number of the next cycle. Otherwise, the job submission will fail with an error.

(If no archive directory or archived files exist, the cycle number defaults to loop-start.)

Working with the archive

You typically should not transfer files from and to the archive directly inside your submitted script. If you follow the proper naming etiquette, the qq run environment will handle all archiving operations for you.

At the start of each cycle, after copying files from the input directory to the working directory, qq run checks the archive and automatically copies all files associated with the current cycle into the working directory. For example, if the current cycle number is 8 and archive-format is job%04d, any file in the archive containing job0008 in its name will be automatically copied to the working directory. These files can then be used to initialize the next cycle of the job.

After the submitted script finishes successfully, qq moves all files matching the archive-format (for any cycle) to the archive directory. For example, if the 8th cycle produces the files job0008.txt, job0008.dat, and job0009.init, and the archive format is job%04d, all three files will be moved to the archive. Only after these files are archived are the remaining files in the working directory moved to the input directory. This ensures that archived files don't clutter the input directory or get copied to the next cycle's working directory.

In summary, unlike with Infinity, you do not need to explicitly fetch files from and to the archive, you just need to name them accordingly and qq will archive them automatically.

If the script fails or the job is killed, no archival is performed. All files remain in the working directory, as with standard jobs.

Resubmiting

After the current cycle finishes the execution of the submitted script, archives the relevant files, and copies the other files to the input directory, qq resubmits the job. This means that the next cycle is submitted from the original input directory. The resubmission may occur from either the original input machine or the current main execution node, depending on the batch system.

The new job (the next cycle) waits for the previous one to finish completely before starting. When it begins, even before creating its working directory, qq archives runtime files from the previous cycle, renaming them according to the specified archive-format.

If the current cycle of the loop job corresponds to loop-end, no resubmission is performed.

Extending a loop job

Sometimes, after a job completes N cycles, you may realize you need M more. To extend the job, simply submit it again from the same input directory with loop-end set to N + M, either on the command line or in the submission script.

Importantly: you do not need to delete any runtime files from the previous cycle — and you probably shouldn't. qq submit can detect that you are extending an existing loop job and will handle the continuation correctly. This has the added benefit that the runtime files from the Nth cycle will be properly archived.

Forcing qq to not resubmit

You can manually force qq to not submit the next cycle of a loop job, even if the current cycle number has not yet reached loop-end, by returning the value of the environment variable QQ_NO_RESUBMIT from within the script:

#!/usr/bin/env -S qq run

# qq job-type loop
# qq loop-end 100
# qq archive storage
# qq archive-format md%04d

...

# if a specific condition is met, do not resubmit but finish successfully
if [ -n "${SOME_CONDITION}" ]; then
    exit "${QQ_NO_RESUBMIT}"
fi

exit 0

If qq detects this exit code, it will not submit the next cycle of the loop job. The current cycle will still be marked as successfully finished (exit code 0).

Working directories

A working directory is the directory where a qq job is actually executed. Typically, it resides on a compute node’s local storage, but it can also be on a shared filesystem — or even be the same as the input directory.

How the working directory is created depends on the batch system.

Clusters of the Metacentrum family

On clusters of the Metacentrum family (such as robox, sokar, and of course all Metacentrum clusters), the working directory is, by default, created on the local scratch storage of the main compute node assigned to the job. You can, however, explicitly choose to use SSD scratch, shared scratch, in-memory scratch (if available), or even use the input directory itself as the working directory.

To control where the working directory is created, use the work-dir option (or the equivalent spelling workdir) of the qq submit command:

  • --work-dir=scratch_local – Default option on Metacentrum-family clusters. Creates the working directory on an appropriate local scratch storage. Depending on the setup, it may also be created on SSD scratch.
  • --work-dir=scratch_ssd – Creates the working directory on SSD-based scratch storage.
  • --work-dir=scratch_shared – Creates the working directory on shared scratch storage accessible by multiple nodes.
  • --work-dir=scratch_shm – Creates the working directory in RAM (in-memory scratch). Useful for jobs requiring extremely fast I/O. Note that if your job fails, your data are immediately lost.
  • --work-dir=input_dir – Uses the input directory itself as the working directory. Files are not copied anywhere. Can be slower for I/O-heavy jobs.
  • --work-dir=job_dir – Same as input_dir.

Not all scratch types are available on every compute node. Use qq nodes to see which storage options are supported by each node.

For more details on scratch storage types available on Metacentrum-family clusters, visit the official documentation.

Specifying the working directory size

Local, SSD, and shared scratch

By default, qq allocates 1 GB of storage per CPU core when using a scratch directory. If you need a different amount of storage, you can adjust it using the following qq submit options:

  • --work-size-per-cpu (or --worksize-per-cpu) — specifies the amount of storage per CPU core.
  • --work-size (or --worksize) — specifies the total amount of storage for the entire job.

--work-size overrides --work-size-per-cpu.

Example:

qq submit --work-size=16gb (...)
# or
qq submit --work-size-per-cpu=2gb (...)

In-memory scratch

If you use --work-dir=scratch_shm, you should allocate memory instead of work-size, using the mem or mem-per-cpu options. Make sure the total allocated memory covers both your program’s memory usage and your in-memory storage needs. By default, qq allocates 1 GB of RAM per CPU core for all jobs.

--mem overrides --mem-per-cpu.

Example:

qq submit --mem=32gb (...)
# or
qq submit --mem-per-cpu=4gb (...)

Not requesting scratch

If you use --work-dir=input_dir (or --work-dir=job_dir), the available storage is limited by your shared filesystem quota.

Job states

There are three types of job states that qq uses: batch states, naïve states, and real states.

  • Batch states describe the job's state according to the batch system itself.
  • Naïve states are recorded in qq info files.
  • Real states combine both sources of information to report the most accurate job status.

Batch states are shown in the output of qq jobs and qq stat, while real states are used by all other commands that report a job's status.

Below are the meanings of the most common real states you may encounter:

  • queued – The job has been submitted and is waiting in the queue for execution.
  • held – The job has been submitted but is blocked from execution for some reason.
  • booting – The job has been allocated computing nodes and the working directory is being prepared, but it is not yet ready.
  • running – The job is currently running; its script is being executed or the execution is being finalized.
  • exiting – qq run has finished executing or is submitting the next job cycle (for loop jobs), but the batch system hasn't completed the job yet and the .qqout file isn't available.
  • finished – The job completed successfully (exit code 0), has been synchronized, and the .qqout file is available in the input directory.
  • failed – The job's execution failed (exit code > 0).
  • killed – The job was terminated by the user, an administrator, or the batch system.
  • in an inconsistent state – qq believes the job to be in a specific state which is incompatible with what the batch system reports. This usually indicates either a bug or that the job was manipulated outside qq.
  • unknown – The job is in a state that qq does not recognize.

Runtime files

qq uses four types of runtime files, each with one of the following extensions: .qqinfo, .out, .err, and .qqout.

qqinfo files

A .qqinfo file (also called a "qq info file") is created after submission by qq submit. It stores information used to track the job submitted from that directory. Each qq job requires its own info file for management and control.

Do NOT move, modify, or delete qq info files manually.
Always use qq commands such as qq kill or qq clear to manage them safely.
Moving, editing, or removing a qq info file while a job is running will cause the job to crash, and you may lose its data.

out files

A .out file contains the standard output from the script executed as a qq job. This file is created when the job begins running in the working directory and is copied to the input directory once (or if) the job finishes successfully.

err files

A .err file contains the standard error output from the script executed as a qq job. Like the .out file, it is created when the job starts in the working directory and is copied to the input directory once (or if) the job completes successfully.

qqout files

A .qqout file contains the output from the qq run execution environment. It includes technical information about the job's progress and internal qq operations. This file is placed into the input directory by the batch system only after the job has fully completed.

Environment variables

When a qq job is submitted, several environment variables are automatically set and can be used within the submitted script.

  • QQ_ENV_SET: indicates that the job is running inside the qq environment (always set to true)
  • QQ_INPUT_MACHINE: name of the input machine from which the job was submitted
  • QQ_INPUT_DIR: absolute path to the job's input directory on the input machine
  • QQ_INFO: absolute path to the qq job's info file on the input machine
  • QQ_BATCH_SYSTEM: name of the batch system used to schedule and execute the job

If the QQ_DEBUG environment variable is set when running qq submit, its value is propagated to the job environment as well. This turns on the debug mode, dramatically increasing the verbosity of qq run.

If the job is a loop job, the following additional environment variables are also set:

  • QQ_LOOP_CURRENT: current cycle number of the loop job
  • QQ_LOOP_START: first cycle of the loop job
  • QQ_LOOP_END: last cycle of the loop job
  • QQ_ARCHIVE_FORMAT: filename format used for archived files
  • QQ_NO_RESUBMIT: exit code that can be returned from the body of the script to indicate that the next cycle of the job should not be submitted

Apart from the variables listed here and those provided by the batch system itself, no other environment variables are typically propagated from the submission environment to the job environment.

Additional internal environment variables may be set, but these are not intended for public use and may change or be removed in future versions of qq.

Commands

qq provides a range of commands for submitting, executing, monitoring, and managing your jobs, as well as for displaying information about available compute nodes and submission queues. This section describes how to use each of them.

Each command is run in the terminal using the following syntax:
qq [COMMAND] [ARGS] [OPTIONS]

For example:
qq info 123456 -s
prints a short summary of the job with ID 123456.

To see a list of all available qq commands, simply type:
qq

For detailed information about a specific command, use:
qq [COMMAND] --help

qq cd

The qq cd command is used to navigate to the input directory of a job. It is qq's equivalent of Infinity's pgo when used with a job ID.

Quick comparison with pgo

  • Unlike pgo, qq cd does not have a dual function.
    pgo can either open a new shell on the job's main node or navigate to the job's input directory depending on the arguments provided.
    qq cd, on the other hand, always navigates to the input directory of the specified job in the current shell. It never opens a new shell.

Description

Changes the current working directory to the input directory of the specified job.

qq cd [OPTIONS] JOB_ID

JOB_ID — Identifier of the job whose input directory should be entered.

Examples

qq cd 123456

Changes the current shell's working directory to the input directory of the job with ID 123456. You can use either the short job ID or the full ID including the batch server address.

Notes

  • Works with any job type, including those not submitted using qq submit.

qq clear

The qq clear command is used to remove qq runtime files from the current directory. It is qq's equivalent of Infinity's premovertf.

Quick comparison with premovertf

  • qq clear checks whether the qq runtime files belong to an active or successfully completed qq job.
    If they do not, the files are deleted without asking for confirmation.
    Otherwise, qq prints an error and asks you to rerun the command with --force.
    In contrast, premovertf simply lists the files and always asks for confirmation before deleting them (unless run as premovertf -f).

Description

Deletes qq runtime files from the current directory.

qq clear [OPTIONS]

Options

--force — Force deletion of all qq runtime files, even if they belong to active or successfully completed jobs.

Examples

qq clear

Deletes all qq runtime files (files with extensions .out, .err, .qqinfo, .qqout) from the current directory, provided these files are not associated with any job or belong to a job that has been killed or has failed. If multiple jobs are represented in the directory, only files related to killed or failed jobs are deleted. This helps prevent accidental removal of files from running or successfully finished jobs.

qq clear --force

Deletes all qq runtime files from the current directory, regardless of their job state. In other words, all files with extensions .out, .err, .qqinfo, and .qqout will be removed. This is dangerous — only use the --force flag if you are absolutely sure you know what you are doing!

Notes

  • You should not delete the .qqinfo file of a running job, as this will cause the job to fail!

qq go

The qq go command is used to navigate to the working directory of a job. It is qq's equivalent of Infinity's pgo when used in an input directory.

Quick comparison with pgo

  • Unlike pgo, qq go does not have a dual function.
    pgo can either open a new shell on the job's main node or navigate to the job's input directory depending on the arguments provided. qq go, on the other hand, always opens a new shell on the job's main node in the working directory. If you want to navigate to the input directory instead, use qq cd.
  • qq go always attempts to access the job's working directory if it exists, even if the job has failed or been killed — no --force option is required.

Description

Opens a new shell in the working directory of the specified qq job, or in the working directory of the job submitted from the current directory.

qq go [OPTIONS] JOB_ID

JOB_ID — Identifier of the job whose working directory should be entered. This argument is optional.

If JOB_ID is not provided, qq go searches for qq jobs in the current directory. If multiple matching jobs are found, qq go opens a shell for each one in turn.

Examples

qq go 123456

Opens a new shell in the working directory of the job with ID 123456 on its main working node. You can use either the short job ID or the full ID including the batch server address. If the job does not exist, is not a qq job, its info file is missing, or the working directory no longer exists, the command exits with an error. If the job is not yet running, the command waits until the working directory is ready.

qq go

Opens a new shell in the working directory of the job whose info file is present in the current directory. If multiple suitable jobs are found, qq go opens a shell for each job in turn.

Notes

  • Uses cd for local directories or ssh for remote hosts.
  • Does not change the working directory of the current shell; it always opens a new shell at the destination.

qq info

The qq info command is used to monitor a qq job's state and display information about it. It is qq's equivalent of Infinity's pinfo.

Quick comparison with pinfo

  • You can use qq info with a job ID to obtain information about a qq job without having to navigate to its input directory.
  • Unlike pinfo, qq info focuses only on the most important details about a job.
    The output is intentionally compact and easier to read.

Description

Displays information about the state and properties of the specified qq job, or of qq jobs found in the current directory.

qq info [OPTIONS] JOB_ID

JOB_ID — Identifier of the job to display information for. This argument is optional.

If JOB_ID is not provided, qq info searches for qq jobs in the current directory. If multiple jobs are found, qq info prints information for each job in turn.

Options

-s, --short — Display only the job ID and the current state of the job.

Examples

qq info 123456

Displays the full information panel for the job with ID 123456. You can use either the short ID or the full ID including the batch server address. This only works if the job is a qq job with a valid and accessible info file, and the associated batch server is reachable from the current machine.

qq info

Displays the full information panel for all jobs whose info files are present in the current directory.

qq info -s

Displays short information for all jobs whose info files are present in the current directory. Only the jobs' full IDs and their current states are shown.

qq jobs

The qq jobs command is used to display information about a user's jobs. It is qq's equivalent of Infinity's pjobs.

Quick comparison with pjobs

  • Unlike pjobs, qq jobs always shows the nodes that the job is running on, if any are assigned.
  • Unlike pjobs, qq jobs distinguishes between failed/killed and successfully finished jobs in its output.

Description

Displays a summary of your jobs or the jobs of a specified user. By default, only unfinished jobs are shown.

qq jobs [OPTIONS]

Options

-u, --user TEXT — Username whose jobs should be displayed. Defaults to your own username.

-a, --all — Include both unfinished and finished jobs in the summary.

--yaml — Output job metadata in YAML format.

Examples

qq jobs

Displays a summary of your unfinished jobs (queued, running, or exiting). This includes both qq jobs and any other jobs associated with the current batch server.

qq jobs -u user2

Displays a summary of user2's unfinished jobs.

qq jobs --all

Displays a summary of all your jobs in the batch system, both unfinished and finished. Note that the batch system eventually removes records of finished jobs, so they may disappear from the output over time.

qq jobs --yaml

Prints a summary of your unfinished jobs in YAML format. This output contains all available metadata as provided by the batch system.

Notes

  • This command lists all types of jobs, including those submitted using qq submit and jobs created through other tools.
  • The run times and job states may not exactly match the output of qq info, since qq jobs relies solely on batch system data and does not use qq info files.

qq kill

The qq kill command is used to terminate qq jobs. It is qq's equivalent of Infinity's pkill.

Quick comparison with pkill

  • You can use qq kill with a job ID to terminate a job without having to navigate to its input directory.
  • When prompted to confirm that you want to terminate a job, qq kill only requires pressing a single key (y to confirm or any other key to cancel), instead of typing 'yes' and pressing Enter.
  • qq kill --force will attempt to terminate jobs even if qq considers them finished, failed, or already killed. This is useful for removing stuck or lingering jobs from the batch system.

Description

Terminates the specified qq job, or all qq jobs submitted from the current directory.

qq kill [OPTIONS] JOB_ID

JOB_ID — Identifier of the job to terminate. This argument is optional.

If JOB_ID is not provided, qq kill searches for qq jobs in the current directory. If multiple suitable jobs are found, qq kill terminates each one in turn.

By default, qq kill prompts for confirmation before terminating a job.

Without the --force flag, it will only attempt to terminate jobs that are queued, held, booting, or running — not jobs that are already finished or killed. When the --force flag is used, qq kill attempts to terminate any job regardless of its state, including jobs that qq believes are already finished or killed. This can be used to remove lingering or stuck jobs.

Options

-y, --yes — Terminate the job without asking for confirmation.

--force — Forcefully terminate the job, ignoring its current state and skipping confirmation.

Examples

qq kill 123456

Terminates the job with ID 123456. You can use either the short job ID or the full ID including the batch server address. You will be prompted to confirm the termination by pressing y. This command only works if the specified job is a qq job with a valid and accessible info file, and the batch server must be reachable from the current machine.

qq kill

Terminates all suitable qq jobs whose info files are present in the current directory. You will be asked to confirm each termination individually.

qq kill 123456 -y

Terminates the job with ID 123456 without asking for confirmation (assumes 'yes').

qq kill 123456 --force

Forcefully terminates the job with ID 123456. This kills the job immediately and without confirmation, regardless of qq's recorded job state.

qq killall

The qq killall command is used to terminate all of your qq jobs. It is qq's equivalent of Infinity's pkillall.

Quick comparison with pkillall

  • qq killall can only terminate jobs submitted using qq submit; other jobs are not affected.

Description

Terminates all qq jobs submitted by the current user.

qq killall [OPTIONS]

This command only terminates qq jobs — other jobs in the batch system are not affected.

By default, qq killall prompts for confirmation before terminating the jobs.

Options

-y, --yes — Terminate all jobs without confirmation.

--force — Forcefully terminate all jobs, ignoring their current states and skipping confirmation.

Examples

qq killall

Terminates all your qq jobs with valid and accessible info files. You will be prompted to confirm termination by pressing y.

qq killall -y

Terminates all your qq jobs with valid and accessible info files without asking for confirmation (assumes "yes").

qq killall --force

Forcefully terminates all your qq jobs with valid and accessible info files. No confirmation is requested, and the jobs will be terminated even if qq believes they are already finished, failed, or killed.

qq nodes

The qq nodes command displays the compute nodes available on the current batch server. It is qq's equivalent of Infinity's pnodes.

Quick comparison with pnodes

  • The output of qq nodes is more dynamically formatted than that of pnodes. If an entire group of nodes lacks a specific attribute (e.g., no GPUs, no shared scratch storage), the corresponding column is hidden.
  • Node group assignments are always determined heuristically based on node names. A full match of the alphabetic part of the name is required for nodes to belong to the same group (unlike pnodes, which uses partial matches).

Description

Displays information about the nodes managed by the batch system. By default, only nodes that are available to you are shown.

qq nodes [OPTIONS]

Nodes are grouped heuristically into node groups based on their names.

Options

-a, --all — Display all nodes, including those that are down, inaccessible, or reserved.

--yaml — Output node metadata in YAML format.

Examples

qq nodes

Displays a summary of all nodes in the batch system that are available to you.

qq nodes --all

Displays a summary of all nodes in the batch system, including those that are down, inaccessible, or reserved.

qq nodes --yaml

Prints a summary of all available nodes in YAML format. This output contains the full metadata provided by the batch system.

Notes

  • The availability state of nodes is not always perfectly reliable. Occasionally, nodes that are actually unavailable may still be reported as available.

qq queues

The qq queues command displays the queues available on the current batch server. It is qq's equivalent of Infinity's pqueues.

Quick comparison with pqueues

  • qq queues is generally more accurate at identifying available and unavailable queues than pqueues.
  • The only other notable difference is the output format.

Description

Displays information about the queues available on the current batch server. By default, only queues that are available to you are shown.

qq queues [OPTIONS]

Options

-a, --all — Display all queues, including those that are not available to you.

--yaml — Output queue metadata in YAML format.

Examples

qq queues

Displays a summary of all batch system queues to which you can submit jobs.

qq queues --all

Displays a summary of all queues in the batch system, including those you cannot submit to.

qq queues --yaml

Prints a summary of all available queues in YAML format. This output contains the full metadata provided by the batch system.

qq run

The qq run command represents the execution environment in which a qq job runs. It is qq's equivalent of Infinity's infex script and the infinity-env.

You should not invoke qq run directly. Instead, every script submitted with qq submit must include the following shebang line:

#!/usr/bin/env -S qq run

For more details about what qq run does, see the sections on standard jobs and loop jobs.

Quick comparison with infex and infinity-env

  • Like infinity-env, using the qq run shebang prevents you from accidentally running the script directly.
  • Unlike Infinity, all qq jobs must use this execution environment — no separate helper run script is created when submitting a qq job.
  • qq run also takes over the responsibilities of parchive and presubmit, which have no direct equivalents in qq.

qq stat

The qq stat command displays information about jobs from all users. It is qq's equivalent of Infinity's pqstat.

Quick comparison with pqstat

  • The same differences that apply between qq jobs and pjobs also apply here.

Description

Displays a summary of jobs from all users. By default, only unfinished jobs are shown.

qq stat [OPTIONS]

Options

-a, --all — Include both unfinished and finished jobs in the summary.

--yaml — Output job metadata in YAML format.

Examples

qq stat

Displays a summary of all unfinished (queued, running, or exiting) jobs associated with the current batch server.

qq stat --all

Displays a summary of all jobs in the batch system, both unfinished and finished. Note that the batch system eventually removes information about finished jobs, so they may disappear from the output over time.

qq stat --yaml

Prints a summary of all unfinished jobs in YAML format. This output contains all metadata provided by the batch system.

Notes

  • This command lists all types of jobs, including those submitted using qq submit and jobs created through other tools.
  • The run times and job states may not exactly match the output of qq info, since qq stat relies solely on batch system data and does not use qq info files.

qq submit

The qq submit command is used to submit qq jobs to the batch system. It is qq's equivalent of Infinity's psubmit.

Quick comparison with psubmit

  • qq submit does not ask for confirmation; it behaves like psubmit (...) -y.

  • Options and parameters are specified differently. The only positional argument is the script name — everything else is an option. You can see all supported options using qq submit --help.

    Infinity:

    psubmit cpu run_script ncpus=8,walltime=12h,props=cl_zero -y

    qq:

    qq submit -q cpu run_script --ncpus=8 --walltime=12h --props=cl_zero

  • Options can also be specified directly in the submitted script, or as a mix of in-script and command-line definitions. Command-line options always take precedence.

Description

Submits a qq job to the batch system.

qq submit [OPTIONS] SCRIPT

SCRIPT — Path to the script to submit.

The submitted script must contain the qq run shebang.

When the job is successfully submitted, qq submit creates a .qqinfo file for tracking the job’s state.

Options

There are many available options. For a full list, run qq submit --help.

All options can also be specified directly inside the submitted script (see below).

Specifying options in the script

Instead of specifying submission options on the command line, you can include them directly in the script using qq directives.

qq directives follow this format: # qq <option>=<value> or # qq <option> <value> (both are equivalent).

The word qq is case-insensitive (qq, QQ, Qq, and qQ are all valid), and spacing is flexible. All qq directives must appear at the beginning of the script, before any executable commands.

Example:

#!/usr/bin/env -S qq run

# qq job-type loop
# qq loop-end 10
# qq archive storage
# qq archive-format md%04d

# qq ncpus=8
# qq ngpus=1
# qq walltime=1d

metamodule add ...

In the example above, kebab-case is used for option names, but qq directives also support snake_case, camelCase, and PascalCase.
For example: # qq job-type loop, # qq job_type loop, # qq jobType loop, and # qq JobType loop are all equivalent.

Command-line options always take precedence over options defined in the script body.

Examples

qq submit run_script.sh -q default --ncpus=8 --workdir=scratch_local --worksize-per-cpu=2gb --walltime=2d --props=hyperthreading

Submits the script run_script.sh to the default queue, requesting 8 CPU cores and 16 GB of local scratch space (2 GB per core). The requested walltime is 48 hours, and the job must run on a node with the hyperthreading property. Additional options may come from the script or queue defaults, but command-line options take precedence.

qq submit run_script.sh

Submits the script run_script.sh, taking all submission options from the script itself or from queue/server defaults.

qq sync

The qq sync command fetches files from a job's working directory to its input directory. It is qq's equivalent of Infinity's psync.

Quick comparison with psync

  • Unlike psync, qq sync fetches all files from the working directory by default.
  • If you want to fetch only specific files, you cannot select them interactively — you must provide a list of filenames when running qq sync.

Description

Fetches files from the working directory of the specified qq job, or from the working directory of the job submitted from the current directory.

qq sync [OPTIONS] JOB_ID

JOB_ID — Identifier of the job whose working directory files should be fetched. This argument is optional.

If JOB_ID is not provided, qq sync searches for qq jobs in the current directory. If multiple suitable jobs are found, qq sync fetches files from each one in turn. Files fetched from later jobs may overwrite files from earlier ones in the input directory.

Files are copied from the job's working directory to its input directory, not to the current directory.

Options

-f, --files TEXT — A colon-, comma-, or space-separated list of files to fetch. If not specified, all files are fetched.

Examples

qq sync 123456

Fetches all files from the working directory of the job with ID 123456 to that job's input directory. You can use either the short job ID or the full ID including the batch server address. This only works if the specified job is a qq job with a valid and accessible info file, and if the batch server and main node are reachable from the current machine.

qq sync

Fetches all files from the working directories of all jobs whose info files are present in the current directory.

qq sync 123456 -f file1.txt,file2.txt,file3.txt

Fetches file1.txt, file2.txt, and file3.txt from the working directory of the job with ID 123456 to its input directory. All other files are ignored. Missing files are skipped without error.

Themes

qq is highly customizable, especially when it comes to the appearance of command output. You can customize qq by creating a config.toml file and placing it in ~/.config/qq.

Several ready-to-use themes are available at: https://github.com/Ladme/qq/tree/main/themes

  • light_terminal — By default, qq assumes a dark terminal background. On light backgrounds, the default output may look very ugly. If you use a light terminal background, you should install this qq theme.

  • traffic_lights_theme — Adjusts the colors used for job states: running jobs are green (instead of blue), queued jobs are yellow (instead of purple), and finished jobs are blue (instead of green).

Glossary

  • compute node – Computer where a job can be executed.

  • input directory – Directory from which the job is submitted. Contains the qq info file.

  • job directory – See "input directory".

  • loop job – Job which submits its continuation right before finishing [see more].

  • main working node – Working node (see below) responsible for managing a job.

  • qq info file – YAML-formatted file containing information about a qq job. Necessary for performing operations with the qq job. Located in the input directory.

  • qq job – Job of the batch system submitted using qq submit.

  • standard job – Default qq job [see more].

  • submission directory – See "input directory".

  • work(ing) directory – Directory where the job is being executed [see more].

  • working node – Compute node where the job is being (or has been) executed.